Requirements Definitions
and
Solution Proposals

 

Junit-based tools integration

 

 

 

 

Revision 0.2

 Nov 3, 2004

 

 

 

 

 

 

 

 

 

 

 

 


 

 


Table of Contents

 

1          Introduction.. 4

2          Hyades JUnit Framework Vision.. 5

2.1        Purpose. 5

2.2        Integration into the Hyades Framework.. 5

2.2.1      Test types. 5

2.2.2      Model Representation. 5

2.2.3      Test Navigator integration. 6

2.2.4      Test Suite Editor. 6

2.2.5      Launch Configuration. 6

2.2.6      Runtime. 6

2.3        Extensibility of the Hyades Junit Framework.. 6

2.3.1      Simplified model API 6

2.3.2      Other useful Hyades extensions. 7

3          GUI Testing Tools.. 8

3.1        Comparison of the existing offering.. 8

3.2        Integration in Hyades. 8

3.2.1      Recorder Integration. 8

3.2.2      Test Suite Generation. 8

3.2.3      Execution. 8

3.2.4      Runtime extensions. 8

3.2.5      Execution History Viewer extensions. 8

4          HTTP Testing Tools.. 9

4.1        Comparison of the existing offering.. 9

4.2        Integration in Hyades. 9

4.2.1      Test Script Generation. 9

4.2.2      Execution. 9

4.2.3      Runtime extensions. 9

4.2.4      Execution History Viewer extensions. 9

5          J2EE Testing Tools.. 10

5.1        Comparison of the existing offering.. 10

5.2        Integration in Hyades. 10

5.2.1      Recorder Integration. 10

5.2.2      Test Suite Generation. 10

5.2.3      Execution. 10

5.2.4      Runtime extensions. 10

5.2.5      Execution History Viewer extensions. 10

6          Integration of Recorders.. 11

6.1        Introduction.. 11

6.1.1      What is recording?. 11

6.1.2      What’s available in Hyades today (version 3.0)?. 11

6.1.3      Why adding the notion of Recording to Hyades?. 11

6.2        Constraints. 11

6.2.1      Two levels of test specification. 11

6.2.2      Intermediate recorder files. 11

6.3        Suggested features. 11

6.3.1      Core notions. 12

6.3.2      UI features. 12

6.3.3      Model features. 15

6.4        Extensibility API 15

6.4.1      Extension point: Recording Environment 15

6.4.2      Extension point: New Recording Profile Wizard. 16

6.4.3      Extension point: Recording Session. 16

6.4.4      Extension point: Recording Viewer Provider. 16

6.4.5      Extension point: Recording Status Provider. 16

6.4.6      Extension point: Test Recording Session Feeder. 16

6.4.7      Public API: Recording Profile Persistence. 17


 

1         Introduction

 

2         Hyades JUnit Framework Vision

2.1      Purpose

The Hyades Junit Framework is built on top of Hyades in order to provide common services and model representation for Junit-derived testing tools. It allows external tools to integrate without effort and lets their developer concentrate on their testing domain, using Hyades models and services, rather than on the plumbing that would be required by the integration into Eclipse.

 

The Hyades Junit Framework extends the Hyades Framework, and provide new extension points for Junit-derived tools. This document is organized in two parts: first, it documents the implementations choices that were made to integrate into Hyades, then it describes the extensibility available to external tools.

2.2      Integration into the Hyades Framework

2.2.1      Test types

The following Test Suite types are introduced:

org.eclipse.hyades.test.junit.testSuite

org.eclipse.hyades.test.junit.testCase

 

The following Test Case type is introduced:

org.eclipse.hyades.test.junit.testMethod

2.2.2      Model Representation

Note in this section the distinction made between Hyades notions (using bold notation) and Junit notions (using italic notation). The mapping between Hyades notions (inherited from the UML 2 Testing Profile (aka U2TP) specifications) and Junit notions is following U2TP committee recommendations.

2.2.2.1     org.eclipse.hyades.test.junit.testSuite

This type of Test Suite represents a TestSuite class. The suite() method (if any) of the TestSuite class is the Behavior of this type of Test Suite. Since the suite() method is unique to the TestSuite class and may not exist, the TestSuite class will be the resource that implements the external behavior of the Test Suite. This type of Test Suite does not own any Test Case.

2.2.2.2     org.eclipse.hyades.test.junit.testCase

This type of Test Suite represents a TestCase class. The suite() method (if any) of the TestCase class is the Behavior of this type of Test Suite. Since the suite() method is unique to the TestCase class and may not exist, the TestCase class will be the resource that implements the external behavior of the Test Suite. This type of Test Suite contains Test Cases of type org.eclipse.hyades.test.junit.testMethod.

2.2.2.3     org.eclipse.hyades.test.junit.testMethod

This type of Test Case represents a test method. Each method with a name starting with “test” will be mapped in the model to a Test Case of this type. The method itself will be the resource that implements the external behavior of this type of Test Case.

2.2.2.4     Summary

In the Junit Hyades Framework, test behaviors are external resources.

Hyades Test

Hyades type

External Behavior Resource

Test Suite

org.eclipse.hyades.test.junit.testSuite

TestSuite class

Test Suite

org.eclipse.hyades.test.junit.testCase

TestCase class

Test Case

org.eclipse.hyades.test.junit.testMethod

Test method

 

2.2.3      Test Navigator integration

The Hyades Junit Framework will provide default content and label providers for displaying Junit-typed Test Suites and Test Cases in the Test Navigator. It will also provide additional actions to their context menu.

2.2.4      Test Suite Editor

The Hyades Junit Framework provides a set of Test Suite Editors for editing the content of the above types of Test Suites. These editors allow to choose the external resource that implements the behavior of the Test Suites and Test Cases.

An automatic synchronization will occur between the external behavior and the test model so that any refactoring or editing of the external behavior is reflected by the test model. For instance, if a new test method is added to a TestCase, the matching “testCase” Test Suite will be appended with a new “testCase” test method.

The Test Suite Editor will also smooth down the vocabulary clash between the Hyades world and the Junit world (Test Case/Test Suite/Test Method). The Test Suite Editor will use the Junit terminology.

2.2.5      Launch Configuration

The Hyades Junit Framework will use the Hyades basic launch configuration (the “Test” launch configuration) to allow the launch of any Junit test. Junit tests may be launched locally or remotely, depending on the deployment chosen in the launch configuration. If the deployment is local, the framework will be able to work without requiring the RAC to be installed. In that case, the tester process will be launched as a child of the eclipse process, and results will be sent back to Eclipse through a socket (as does the JDT Junit plugin).

2.2.6      Runtime

One of the major area where Junit derived tools could leverage a common infrastructure is the runtime area.

The Hyades Junit Framework provides a runner implementing a TestListener that produces execution events (serialized as fragments through the RAC and then loaded to an Execution History model). This allows any Junit tool to produce an Execution History model, which is natively viewable in Hyades using the Execution History Viewer.

2.3      Extensibility of the Hyades Junit Framework

2.3.1      Simplified model API

Since most model implementations are undertaken by Hyades Junit Framework, Junit-derived tools should have very few interactions with the Hyades Test Model. The simplified model API will provide a reduced set of APIs for reading/writing to the underlying test model, and will act as a facade. This API will also allow to go back and forth between the JDT model and the Test Model (since the external behavior of a Test element is a Java element).

2.3.2      Other useful Hyades extensions

Of course, using the Hyades Junit framework does not prevent from using the Hyades framework when necessary. The following Hyades extension points/features should be typically useful:

2.3.2.1     Test types

If any specific handling is required for a test suite (in the areas of Test Suite editing, deployments, execution), the external tool should define its own test type, develop specific extensions for this type when necessary, and re-use Hyades Junit Framework extensions when no specific handling is required.

2.3.2.2     Other extension points

Depending on the specifities of the external tool test suites, the external tool may re-use a Junit Framework extension, or provide its own extension to the following Hyades extension points:

3         GUI Testing Tools

 

3.1      Comparison of the existing offering

3.2      Integration in Hyades

3.2.1      Recorder Integration

3.2.2      Test Suite Generation

3.2.3      Execution

3.2.4      Runtime extensions

3.2.5      Execution History Viewer extensions

 

4         HTTP Testing Tools

 

4.1      Comparison of the existing offering

4.2      Integration in Hyades

 

[I suppose here that the choice is to update the existing URL Testing tools to xxxUnit tool (HTTPUnit or one of his derivates, or competitors)]

4.2.1      Test Script Generation

4.2.2      Execution

4.2.3      Runtime extensions

4.2.4      Execution History Viewer extensions

 

5         J2EE Testing Tools

 

5.1      Comparison of the existing offering

5.2      Integration in Hyades

5.2.1      Recorder Integration

5.2.2      Test Suite Generation

5.2.3      Execution

5.2.4      Runtime extensions

5.2.5      Execution History Viewer extensions

 

6         Integration of Recorders

6.1      Introduction

6.1.1      What is recording?

The ability to record a sequence of actions from an external environment and generate a Test that allows to automatically replay this sequence. Optionally, the generated test may be edited before it is run.

6.1.2      What’s available in Hyades today (version 3.0)?

Today, the notion of recording is not part of the framework. But among the three extensions of this framework provided with Hyades, one of them already has a recorder, which can be considered as a private feature of the extension.

This extension is URL Testing, and the recording feature consists in capturing all http requests performed from the local machine, and generating a test that performs the same sequence of http requests. The user may easily modify the generated test by adding loops in order to stimulate the Web server with many requests at the same time (load testing).

6.1.3      Why adding the notion of Recording to Hyades?

Other testing tools from different fields (UI, components, system) share the same notion of recording, for the same purpose: allowing to quickly generate a test from a real-world usage.

Since recording is a common notion in the test world, it makes sense to introduce this notion into Hyades. The benefits from this introduction are the same as the ones that support Hyades: having a consistent user-experience from one testing tool to another.

6.2      Constraints

6.2.1      Two levels of test specification

The Hyades Test Model allows two levels of specification for test behaviors:

Depending on which level is used, the recording features of Hyades that we may foresee are different.

6.2.2      Intermediate recorder files

Today, there is no Hyades extension that records something and directly produces a Test Model with an internal behavior.

In URL testing, the recording involves the production of an intermediate file (with a .rec extension) of a proprietary format. This file is then translated into a Hyades Test model.

6.3      Suggested features

This section describes which features the Hyades common infrastructure could provide to:

1)      Support and integrate recorders from various test products into a consistent UI (this is the end-user aspect)

2)      Facilitate the integration of existing recorders, and provide common components to develop new features of these recorders (this is the tool-vendor aspect).

6.3.1      Core notions

This section describes the common notions shared by all recorders implementations, and presented to the user. This section does not cover existing notions such as Test or Deployment. The guideline is to add as less as possible of new concepts to the existing framework.

6.3.1.1     Notion 1: Recording environment

A recording environment is a type of external environment where a recording can be exercised.  E.g.: graphical application, http server.

The notion of recording environment is tightly linked, in the test field, to the notion of System Under Test. Usually a given type of recording environment is required to produce a given type of test.

6.3.1.2     Notion 2: Recording

A recording is a recorded sequence of actions performed in a given recording environment. This recording may exists in memory, or in a file of a proprietary format.

Hyades encourages the use of Hyades Trace Format for recordings.

6.3.1.3     Notion 3: Recording Profile

A recording profile is an instance of a Recording Environment that must be setup in order to be able to start a recording.

For instance, a Recording Profile for a graphical application will specify which application to launch, and optionally which steps to perform in order to have the application enter an initial state before the recording can start.

A recording profile for a URL Test could specify which browser to launch, and which initial page to open before the recording can start.

The notion of Recording Profile is a key feature for the users wishing to start a recording, without having to enter the same information every time they do so. For instance, in the case of a UI recording, selecting an existing recording profile will launch the application and make it enter an initial state in a single click, making it ready for recording.

6.3.2      UI features

An obvious benefit of integrating recorders into the Hyades framework is a consistent user experience from one tool to another. Being integrated into Eclipse is a first step, but Hyades should bring more commonality. For instance, a user familiar with URL Testing could immediately know where to look to record a User Interface test. Moreover, the interface for launching/stopping the record could be the same for the two products.

6.3.2.1     Feature 1 : New Recording Wizard

This wizard, accessible using File -> New… -> Test -> Recording, aims at creating a recording persisted as a file (the file format depends on the type of recording).

To do so, the user first chooses to use an existing Recording Profile (or to create a new one), then chooses the name and location of the recording file. Once done, the wizard terminates and the Recording View is opened in order to let the user start and stop the recording as required, then save it.

Fig.1 New Recording Wizard State Chart Diagram

 

6.3.2.2     Feature 2: New Test From Recording Wizard

This wizard, accessible using File -> New… -> Test -> Test from recording, aims at creating a new test from a recording. The recording may already exists (it is stored in an external resource of a proprietary format), or the user may choose to record the sequence on the fly.

 

Case 2.1: Creating a test from a an existing recording

The user selects the recording environment, then the recording resource, the test is automatically created from this recording, and the Test Suite Editor is opened with the created test.

 

Case 2.2: Creating a test from a new recording

The user chooses to make a new recording. He’s prompted to select a Recording Profile or to create a new one, then the wizard is closed, and the Recording view is brought to front in order to allow the recording to start. When the recording is finished, the user clicks on the “Generate Test” in the Recording view: the test is created and the Test Suite Editor is opened with this test.

 

Note: the existing category for URL Test is “Recording” but the name is not accurate since the wizard doesn’t create a recording but a test suite. “Test From Recording” is suggested instead.

 

Fig.2 New Test From Recording Wizard State Chart Diagram

 

6.3.2.3     Feature 3: New Recording Profile Wizard

This wizard allows to create a new Recording Profile, i.e. choose the Recording Environment (graphical application, http server), and then enter settings specific to the Recording Environment (for instance, the external application path, an optional setup script…).

This wizard is integrated into the New Recording Wizard (Feature 1), New Test From Recording Wizard (case 2.1), or may be invoked standalone from the Recording View (Feature 4).

6.3.2.4     Feature 4: Recording View

This is the central view of the recording feature. This view allows to:

-         select a recording profile

-         create a new recording profile

-         start a recording

-         pause a recording

-         generate a test from the current recording

-         save a recording

While recording is in progress, this view shows a tree viewer depicting what is being recorded. This view also provides a status bar showing the current recorder state.

Fig.3 Recording View State Chart Diagram

 

6.3.3      Model features

6.3.3.1     Feature 5: Default Hyades Recording Model

As explained above, a recording may or may not be persisted to a proprietary format, and is represented in memory using a proprietary form.

To encourage the use of Hyades Trace Model for representing a recording, a set of APIs will be provided to facilitate this use.

These APIs will allow to:

-         create an empty in-memory Trace Model and associate it to the current recording in the Recording View

-         display the content of this in-memory Trace Model while it is being populated

-         generate a persisted Test from this in-memory model by converting the Trace Model to a Test Behavior Model.

If the Hyades Trace Model is chosen as an internal representation for recordings, no work is practically required from the tool vendor on the Recording View extension, nor it has to take care of the proprietary-form-to-Hyades-Test-Model conversion.

6.4      Extensibility API

This section describes the set of extension points and API provided by Hyades for the purpose of recorders integration. It does not describe internal APIs.

6.4.1      Extension point: Recording Environment

This extension point is used to declare a type of a supported Recording Environment. It also declares what kind of test can be generated from this type of Recording Environment.

6.4.2      Extension point: New Recording Profile Wizard

This extension is used to declare a Wizard able to create a Recording Profile for a specific Recording Environment.

6.4.3      Extension point: Recording Session

This extension is used to declare a class able to manage a Recording Session. This extension also specifies which type(s) of Recording Environments are supported by the Recording Session. Managing a Recording Session includes the following tasks:

-         setup the recording environment according to a recording profile

-         start the recording

-         pause the recording

-         stop the recording environment

-         generate a Test (in memory) from a recording

-         save the recording (if the extension declares that this operation is supported)

A recording session may be initialized in two ways:

-         from an existing recording resource, if the extension declares what kind of resources are supported

-         from a Recording Profile

6.4.4      Extension point: Recording Viewer Provider

This extension provides a tree item provider for populating the tree viewer in the Recording Viewer. This provider has to translate items from the proprietary representation of the recording to a hierarchy of graphical tree nodes.

6.4.5      Extension point: Recording Status Provider

This extension provides a label provider for the status bar displayed in the Recording View.

 

Note that for contributors wishing to use the Trace Model as an internal representation for their recording, an instance of the “Recording Session” extension point (named Hyades Trace Model Recording Session) will be provided, as well as an instance of the Recording Viewer Provider. This extension will not support recording persistency. This extension delegates some of its tasks to the contributor using the following extension points:

6.4.6      Extension point: Test Recording Session Feeder

This extension is used to declare a class that supports the feed of an in-memory Trace Model from a recording environment. This extension also declares which type(s) of Recording Environment is (are) supported. This class supports the following features:

-         setup the recording environment according to a recording profile

-         start the recording

-         pause the recording

-         stop the recording environment

This class is initialized from a Recording Profile. It is also responsible of feeding the model with new recording items. The notification of new items will naturally be performed using EMF notification system.

6.4.7      Public API: Recording Profile Persistence

This API provides functions for storing and retrieving properties from a Recording Profile. A Recording profile consists of a mandatory property, the type of Recording Environment, and optional properties dependent on the type Recording Environment.

These APIs will be typically used by the New Recording Profile Wizard extensions, and by Recording Session or Test Recording Session Feeder extensions.

The persistence of Recording Profiles is very similar to the persistence of Launch Configuration in Eclipse.