Test Automation Platform

This proposal has been withdrawn.

Introduction

Test Automation Platform (TAP) is a proposed open-source project under the Eclipse Technology project. The TAP project will add new extensibility and components to Eclipse for constructing system test automation solutions. This includes the creation of a new Eclipse platform designed for headless test execution. The concept encourages seamless interoperability among many commercial, proprietary, and open-source test-related plug-ins. You are invited to comment on and/or join the project. Please send all feedback to the eclipse.tap newsgroup.

Background

System test automation involves black-box testing of systems that include multiple devices or components that are being tested for interoperability, functionality, performance, stress, etc. System testing is used widely in networking, data centers, etc.

Several components are required for a complete system test automation solution including test authoring, specialized testing components (such as network simulators, traffic generators, etc.), testbed inventory management, testbed reservations, physical testbed topology management, test information management, defect tracking, and many others. Some of these components may be "home-grown" while others may be provided by commercial vendors or available as open-source.

Historically, interoperability between these components has depended on the tools coming from a single provider, has required customized "glue," or has required bilateral cooperation between each pairing of providers. None of these is ideal to foster an environment favoring innovation and "best of breed" components.

There exists a lot of disparate expertise surrounding these various components and interfaces. We do not believe that any one individual, organization, or company has sufficient expertise to unilaterally define the best possible interfaces between these tools. Therefore, we would like to see a community develop where these experts can come together to collaborate on defining excellent interface definitions that collectively allow for complete solutions built from best-of-breed components. Even more than that, we believe that interface definitions are insufficient to foster a healthy ecosystem and therefore would like to see an open-source platform developed that allows each tool to be developed in isolation with confidence that it will interoperate properly with other components in a complete solution. Further, the open source component should include reference implementations of these components so that there is a basic working solution in place, allowing each component being added in to augment the value.

Also, an important aspect of test automation is that it involves two distinct phases. In the first phase, a human is designing, developing, and troubleshooting automated tests. In the second phase, the resulting automated tests must be executed in a "lights-out" regression environment. This execution involves many of the same components and interoperability among them. So test automation tool interoperability involves both a human-user-interface component and "headless" interoperability as well.

Scope

The goal of this proposed project is to use Eclipse as a common foundation on which to build interoperability among otherwise disjoint components of a test automation solution. Using Eclipse's extensibility model and Java APIs, this project will define a variety of open interfaces between blocks of functionality that may need to interoperate. These interfaces are defined using a service model so that innovation is encouraged on both sides of the interface – meaning that once a service is defined, many service providers can be plugged in and, likewise, many different service consumers can take advantage of each service. The project will also create reference implementations of all of these components with basic functionality – while allowing commercial and other implementations to be plugged into these same extensibility points.

Eclipse itself (or other compatible RCPs) provides the framework and user interface for operations that involve a human-user interface. Within the scope of this project is also the creation of a new "headless" runtime executable allowing suitable components to use these same interfaces when executing tests. This is a new Equinox-based runtime with no dependencies on any UI components nor on a workspace (i.e., the org.eclipse.core.resources plug-in). Command-line arguments are used to communicate the reason that the runtime is being invoked and to control the behavior.

More specifically, the objectives for this project include:

  • A new TAP feature that can be added into an Eclipse installation that acts as common middleware for other tools that may also be added into that same installation;
  • Extension point declarations and related Java interface definitions for a variety of interfaces and/or services related to system test automation;
  • Sufficient code to connect service providers to service consumers in an appropriate many-to-many arrangement, insulating service consumers from the complexity of “multiplexing”;
  • Reference implementations with basic functionality of all services and service consumers for each interface so that each tool implementation can be done without requiring any other commercial or proprietary tool; and
  • A new runtime platform based on Eclipse without a UI that is intended for performing test-related operations involving test automation tools built on top of these interfaces and extension points.

Description

While the scope of this project is quite wide – involving interoperability among many classes of different test-related tools, the initial focus will be on one or two of these interfaces. By showing that the model works between two different classes of tools, we hope to encourage more tool providers with expertise on other classes of test-related tools to contribute their expertise in defining additional interfaces.

For the initial phase, we propose to define the interface between test authoring tools (i.e., those tools that are used for creating automated black-box system tests) and the specialized tools used for controlling and monitoring the various components in a physical testbed during testing. These tools used during manual testing (e.g., virtual terminal, web browser, traffic generator, etc.) are also needed during automated testing, so the service provided by the specialized tools are also used by a test execution engine running either during test development (within the GUI environment) or during an execution environment (headless).

We also propose to define the interface between test execution tools (that may be specific to the type of test being executed) and other tools that are interested in post-processing the results of a test execution, such as test reporting tools that will publish the results into a centralized database.

We have a working prototype today of these interfaces including reference service providers and service consumers implementations for each. If and when this project is approved into an incubation phase, we propose to contribute this code as a starting point.

We foresee other interfaces coming along as well as other contributors with relevant expertise joining the project. Of particular interest is an interface between test execution and testbed reservation/topology tools which are essential to a complete solution. We also anticipate, for example, interfaces between test scheduling tools and heterogeneous test execution tools (and so on).

We choose not to define a so-called "reference model" for these various tools and components as we believe that this will tend to limit the innovation that we would like to encourage. We prefer to focus on individual service and interfaces definitions that experts agree will facilitate interoperability and open the door to new ideas that exploit these same interfaces.

We have a commercial product today that is built on top of a runtime version of Eclipse designed to run "headless" for executing tests. We anticipate that some of this expertise can be brought to bear on the goal of creating a suitable runtime platform for tool integration.

Organization

Mentors

To be determined

Committers

The initial set of committers will be from Fanfare:

We welcome participation early in the process from other interested parties with relevant expertise.

Interested Parties

We have discussed this topic with a number of interested parties representing both vendors and customers. The following companies have expressed their intention of being an active part o this community: