[hyades-dev] Interest in contributing a test execution framework

["Michael Silverstein" <msilverstein@xxxxxxxxxxxxxx>]

Title: Interest in contributing a test execution framework

Hello:

We'd like to gauge interest in a potential contribution to the Hyades project by SilverMark. We've been creating object testing tools since 1996 and have a framework that may be of use to the project. I'll briefly describe it without burdening you all with a lot of details at the outset.

This is a test execution framework implemented in Java, based on concepts described in "Automated testing of Object Oriented-Components Using Intelligent Test Artifacts", http://servlet.java.sun.com/javaone/javaone2000/pdfs/TS-708 and used as a basis for the SilverMark's Test Mentor - Java Edition (www.javatesting.com) commercial product. The essence of it is this:

• The framework provides objects that each possess a type of testing behavior. These objects are called steps.
• Tests are composed of instances of steps, configured to perform specific testing tasks.
• The tests themselves may be considered "executable models". One of our areas of interest is to use the framework as basis for an executable form of the UML Testing Profile.
• There are a variety of types of steps, and it is fairly easy to extend the framework to provide additional behaviors. Some examples of the behavior provided by specific step types are:
• calling methods and constructors
• setting/getting fields
• parsing and iterating over test data
• referencing test assets (asset reuse)
• sequencing other test steps
• running a piece of Java code (script)
• recursively extracting object state
• validation (assertion-like behavior)
• calling out to a JUnit test
• accessing distributed objects such as EJBs or through distribution schemes such as RMI or CORBA
• Steps also share common properties, such as:
• name and description, for documentation
• expected exception (class of exception expected to be thrown by executing the step)
• number of times to execute the step
• performance criterion (maximum time step should take to execute)
• precondition
• postcondition
• failure (step to execute if the owning step fails for any reason)
• invariant (step to execute, which performs an assertion on some state that is assumed to be true under all circumstances)
• maximum execution time
• All of these common properties except for “name” are optional, which reduces the overhead of step configuration. Individual step types have their own configuration properties.
• Steps know how to represent themselves as XML. The objects that represent the results of executing steps also know how to represent themselves as XML.
• The framework includes mechanisms for reusing and passing parameters to test assets, as well as iterating over composite parameters
• Test steps implement the JUnit junit.framework.Test interface, so they look and smell to JUnit like JUnit tests, and can even be run under JUnit

Some benefits of using the framework are (in no particular order):

• You already have a robust, flexible, well tested framework for composing and executing a variety of types of tests
• At execution time test steps use reflection to interact with components under test, so complete tests can be implemented independent of the presence of the system under test
• Steps may be configured incrementally, as more information about the system under test becomes known.
• Test steps can conditionally override Java's private member access checks
• Each test step measures its own execution time, so fine-grained performance metrics are gathered
• Tests and test results are independent of presentation, so a variety of views can be created for them, specific to the target audience
• The framework can be easily extended to support additional testing behaviors
• The framework can be easily extended to gather additional fine-grained metrics during execution. For example, it would not be difficult to extend the steps' behavior to measure memory consumption or code coverage during individual method calls, given the presence of callable API for those metrics.

I've gone on enough. Please let me know if any of this would be of interest to the group.

- Mike

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mike Silverstein                        voice:  (919) 858-8300 x29
SilverMark, Inc.                        fax:    (919) 858-8308
5511 Capital Center Drive               mobile: (919) 669-5906
Suite 510, Raleigh, NC 27606            toll free: (888) 588-0668
http://www.silvermark.com               msilverstein@xxxxxxxxxxxxxx