Junit-based tools integration
Nov 3, 2004
Table of Contents
The Hyades Junit Framework is built on top of Hyades in order to provide common services and model representation for Junit-derived testing tools. It allows external tools to integrate without effort and lets their developer concentrate on their testing domain, using Hyades models and services, rather than on the plumbing that would be required by the integration into Eclipse.
The Hyades Junit Framework extends the Hyades Framework, and provide new extension points for Junit-derived tools. This document is organized in two parts: first, it documents the implementations choices that were made to integrate into Hyades, then it describes the extensibility available to external tools.
The following Test Suite types are introduced:
The following Test Case type is introduced:
Note in this section the distinction made between Hyades notions (using bold notation) and Junit notions (using italic notation). The mapping between Hyades notions (inherited from the UML 2 Testing Profile (aka U2TP) specifications) and Junit notions is following U2TP committee recommendations.
This type of Test Suite represents a TestSuite class. The suite() method (if any) of the TestSuite class is the Behavior of this type of Test Suite. Since the suite() method is unique to the TestSuite class and may not exist, the TestSuite class will be the resource that implements the external behavior of the Test Suite. This type of Test Suite does not own any Test Case.
This type of Test Suite represents a TestCase class. The suite() method (if any) of the TestCase class is the Behavior of this type of Test Suite. Since the suite() method is unique to the TestCase class and may not exist, the TestCase class will be the resource that implements the external behavior of the Test Suite. This type of Test Suite contains Test Cases of type org.eclipse.hyades.test.junit.testMethod.
This type of Test Case represents a test method. Each method with a name starting with “test” will be mapped in the model to a Test Case of this type. The method itself will be the resource that implements the external behavior of this type of Test Case.
In the Junit Hyades Framework, test behaviors are external resources.
External Behavior Resource
The Hyades Junit Framework will provide default content and label providers for displaying Junit-typed Test Suites and Test Cases in the Test Navigator. It will also provide additional actions to their context menu.
The Hyades Junit Framework provides a set of Test Suite Editors for editing the content of the above types of Test Suites. These editors allow to choose the external resource that implements the behavior of the Test Suites and Test Cases.
An automatic synchronization will occur between the external behavior and the test model so that any refactoring or editing of the external behavior is reflected by the test model. For instance, if a new test method is added to a TestCase, the matching “testCase” Test Suite will be appended with a new “testCase” test method.
The Test Suite Editor will also smooth down the vocabulary clash between the Hyades world and the Junit world (Test Case/Test Suite/Test Method). The Test Suite Editor will use the Junit terminology.
The Hyades Junit Framework will use the Hyades basic launch configuration (the “Test” launch configuration) to allow the launch of any Junit test. Junit tests may be launched locally or remotely, depending on the deployment chosen in the launch configuration. If the deployment is local, the framework will be able to work without requiring the RAC to be installed. In that case, the tester process will be launched as a child of the eclipse process, and results will be sent back to Eclipse through a socket (as does the JDT Junit plugin).
One of the major area where Junit derived tools could leverage a common infrastructure is the runtime area.
The Hyades Junit Framework provides a runner implementing a TestListener that produces execution events (serialized as fragments through the RAC and then loaded to an Execution History model). This allows any Junit tool to produce an Execution History model, which is natively viewable in Hyades using the Execution History Viewer.
Since most model implementations are undertaken by Hyades Junit Framework, Junit-derived tools should have very few interactions with the Hyades Test Model. The simplified model API will provide a reduced set of APIs for reading/writing to the underlying test model, and will act as a facade. This API will also allow to go back and forth between the JDT model and the Test Model (since the external behavior of a Test element is a Java element).
Of course, using the Hyades Junit framework does not prevent from using the Hyades framework when necessary. The following Hyades extension points/features should be typically useful:
If any specific handling is required for a test suite (in the areas of Test Suite editing, deployments, execution), the external tool should define its own test type, develop specific extensions for this type when necessary, and re-use Hyades Junit Framework extensions when no specific handling is required.
Depending on the specifities of the external tool test suites, the external tool may re-use a Junit Framework extension, or provide its own extension to the following Hyades extension points:
[I suppose here that the choice is to update the existing URL Testing tools to xxxUnit tool (HTTPUnit or one of his derivates, or competitors)]
The ability to record a sequence of actions from an external environment and generate a Test that allows to automatically replay this sequence. Optionally, the generated test may be edited before it is run.
Today, the notion of recording is not part of the framework. But among the three extensions of this framework provided with Hyades, one of them already has a recorder, which can be considered as a private feature of the extension.
This extension is URL Testing, and the recording feature consists in capturing all http requests performed from the local machine, and generating a test that performs the same sequence of http requests. The user may easily modify the generated test by adding loops in order to stimulate the Web server with many requests at the same time (load testing).
Other testing tools from different fields (UI, components, system) share the same notion of recording, for the same purpose: allowing to quickly generate a test from a real-world usage.
Since recording is a common notion in the test world, it makes sense to introduce this notion into Hyades. The benefits from this introduction are the same as the ones that support Hyades: having a consistent user-experience from one testing tool to another.
The Hyades Test Model allows two levels of specification for test behaviors:
Depending on which level is used, the recording features of Hyades that we may foresee are different.
Today, there is no Hyades extension that records something and directly produces a Test Model with an internal behavior.
In URL testing, the recording involves the production of an intermediate file (with a .rec extension) of a proprietary format. This file is then translated into a Hyades Test model.
This section describes which features the Hyades common infrastructure could provide to:
1) Support and integrate recorders from various test products into a consistent UI (this is the end-user aspect)
2) Facilitate the integration of existing recorders, and provide common components to develop new features of these recorders (this is the tool-vendor aspect).
This section describes the common notions shared by all recorders implementations, and presented to the user. This section does not cover existing notions such as Test or Deployment. The guideline is to add as less as possible of new concepts to the existing framework.
A recording environment is a type of external environment where a recording can be exercised. E.g.: graphical application, http server.
The notion of recording environment is tightly linked, in the test field, to the notion of System Under Test. Usually a given type of recording environment is required to produce a given type of test.
A recording is a recorded sequence of actions performed in a given recording environment. This recording may exists in memory, or in a file of a proprietary format.
Hyades encourages the use of Hyades Trace Format for recordings.
A recording profile is an instance of a Recording Environment that must be setup in order to be able to start a recording.
For instance, a Recording Profile for a graphical application will specify which application to launch, and optionally which steps to perform in order to have the application enter an initial state before the recording can start.
A recording profile for a URL Test could specify which browser to launch, and which initial page to open before the recording can start.
The notion of Recording Profile is a key feature for the users wishing to start a recording, without having to enter the same information every time they do so. For instance, in the case of a UI recording, selecting an existing recording profile will launch the application and make it enter an initial state in a single click, making it ready for recording.
An obvious benefit of integrating recorders into the Hyades framework is a consistent user experience from one tool to another. Being integrated into Eclipse is a first step, but Hyades should bring more commonality. For instance, a user familiar with URL Testing could immediately know where to look to record a User Interface test. Moreover, the interface for launching/stopping the record could be the same for the two products.
This wizard, accessible using File -> New… -> Test -> Recording, aims at creating a recording persisted as a file (the file format depends on the type of recording).
To do so, the user first chooses to use an existing Recording Profile (or to create a new one), then chooses the name and location of the recording file. Once done, the wizard terminates and the Recording View is opened in order to let the user start and stop the recording as required, then save it.
Fig.1 New Recording Wizard State Chart Diagram
This wizard, accessible using File -> New… -> Test -> Test from recording, aims at creating a new test from a recording. The recording may already exists (it is stored in an external resource of a proprietary format), or the user may choose to record the sequence on the fly.
Case 2.1: Creating a test from a an existing recording
The user selects the recording environment, then the recording resource, the test is automatically created from this recording, and the Test Suite Editor is opened with the created test.
Case 2.2: Creating a test from a new recording
The user chooses to make a new recording. He’s prompted to select a Recording Profile or to create a new one, then the wizard is closed, and the Recording view is brought to front in order to allow the recording to start. When the recording is finished, the user clicks on the “Generate Test” in the Recording view: the test is created and the Test Suite Editor is opened with this test.
Note: the existing category for URL Test is “Recording” but the name is not accurate since the wizard doesn’t create a recording but a test suite. “Test From Recording” is suggested instead.
Fig.2 New Test From Recording Wizard State Chart Diagram
This wizard allows to create a new Recording Profile, i.e. choose the Recording Environment (graphical application, http server), and then enter settings specific to the Recording Environment (for instance, the external application path, an optional setup script…).
This wizard is integrated into the New Recording Wizard (Feature 1), New Test From Recording Wizard (case 2.1), or may be invoked standalone from the Recording View (Feature 4).
This is the central view of the recording feature. This view allows to:
- select a recording profile
- create a new recording profile
- start a recording
- pause a recording
- generate a test from the current recording
- save a recording
While recording is in progress, this view shows a tree viewer depicting what is being recorded. This view also provides a status bar showing the current recorder state.
Fig.3 Recording View State Chart Diagram
As explained above, a recording may or may not be persisted to a proprietary format, and is represented in memory using a proprietary form.
To encourage the use of Hyades Trace Model for representing a recording, a set of APIs will be provided to facilitate this use.
These APIs will allow to:
- create an empty in-memory Trace Model and associate it to the current recording in the Recording View
- display the content of this in-memory Trace Model while it is being populated
- generate a persisted Test from this in-memory model by converting the Trace Model to a Test Behavior Model.
If the Hyades Trace Model is chosen as an internal representation for recordings, no work is practically required from the tool vendor on the Recording View extension, nor it has to take care of the proprietary-form-to-Hyades-Test-Model conversion.
This section describes the set of extension points and API provided by Hyades for the purpose of recorders integration. It does not describe internal APIs.
This extension point is used to declare a type of a supported Recording Environment. It also declares what kind of test can be generated from this type of Recording Environment.
This extension is used to declare a Wizard able to create a Recording Profile for a specific Recording Environment.
This extension is used to declare a class able to manage a Recording Session. This extension also specifies which type(s) of Recording Environments are supported by the Recording Session. Managing a Recording Session includes the following tasks:
- setup the recording environment according to a recording profile
- start the recording
- pause the recording
- stop the recording environment
- generate a Test (in memory) from a recording
- save the recording (if the extension declares that this operation is supported)
A recording session may be initialized in two ways:
- from an existing recording resource, if the extension declares what kind of resources are supported
- from a Recording Profile
This extension provides a tree item provider for populating the tree viewer in the Recording Viewer. This provider has to translate items from the proprietary representation of the recording to a hierarchy of graphical tree nodes.
This extension provides a label provider for the status bar displayed in the Recording View.
Note that for contributors wishing to use the Trace Model as an internal representation for their recording, an instance of the “Recording Session” extension point (named Hyades Trace Model Recording Session) will be provided, as well as an instance of the Recording Viewer Provider. This extension will not support recording persistency. This extension delegates some of its tasks to the contributor using the following extension points:
This extension is used to declare a class that supports the feed of an in-memory Trace Model from a recording environment. This extension also declares which type(s) of Recording Environment is (are) supported. This class supports the following features:
- setup the recording environment according to a recording profile
- start the recording
- pause the recording
- stop the recording environment
This class is initialized from a Recording Profile. It is also responsible of feeding the model with new recording items. The notification of new items will naturally be performed using EMF notification system.
This API provides functions for storing and retrieving properties from a Recording Profile. A Recording profile consists of a mandatory property, the type of Recording Environment, and optional properties dependent on the type Recording Environment.
These APIs will be typically used by the New Recording Profile Wizard extensions, and by Recording Session or Test Recording Session Feeder extensions.
The persistence of Recording Profiles is very similar to the persistence of Launch Configuration in Eclipse.