Eclipse Test & Performance Tools Platform (TPTP) Project
Approved 4.2 Plan

Last revised 00:35 PDT 28 June 2006 ( marks interesting changes since 4.1 Release)

    Please send comments about this plan to the tptp-pmc@eclipse.org PMC mailing list.

This document lays out the feature and API set for the TPTP 4.2 release.

The first part of this plan deals with the important matters of release deliverables, release milestones, target operating environments, and release-to-release compatibility.  These are all things that need to be clear for any release, even if no features were to change. 

The remainder of the plan consists of plan items for the four projects under the TPTP Top-Level Project.  Each plan item covers a feature or API that is to be added to TPTP, or some aspect of TPTP that is to be improved.  Each plan item has its own entry in the TPTP bugzilla database, with a title and a concise summary (usually a single paragraph) that explains the work item at a suitably high enough level so that everyone can readily understand what the work item is without having to understand the nitty-gritty detail.

Not all plan items represent the same amount of work; some may be quite large, others, quite small. Some plan items may involve work that is localized to a single component; others may involve coordinated changes to several components; other may pervade the entire project.

With the previous release as the starting point, this is the plan for how we will enhance and improve it.  Fixing bugs, improving test coverage, documentation, examples, performance tuning, usability, etc. are considered routine ongoing maintenance activities and are not included in this plan unless they would also involve a significant change to the API or feature set, or involve a significant amount of work. The intent of the plan is to account for all interesting feature work.

Release Deliverables

The following release deliverables are provided:

Release Milestones

The TPTP 4.2 release is targeted for general availability on 30-Jun-2006.  All release deliverables will be available for download as soon as the release has been tested and validated in the target operating configurations.  Interim release milestone are planned at roughly 6 week intervals to facilitate coarse-grained planning and staging. TPTP is participating in Callisto Simultaneous Release of Eclipse projects. The list of milestones below includes all Callisto milestones of TPTP. 

Release Milestones

Milestone Date Description
Iteration -2 (4.2 i-2) Friday, 9-Dec-05 Callisto M3 [Done]
Iteration -1 (4.2 i-1) Friday, 23-Dec-05 Callisto M4 [Done]
Iteration 1 (4.2 i1) Friday, 24-Feb-06 Stable build - API freeze; Callisto M5 [Done 3-Mar-06]
  Friday, 14-Apr-06 Callisto RC0 [Done]
Iteration 2 (4.2 i2) Friday, 14-Apr-06 Stable build - UI freeze [Done 20-Apr-06]
  Friday, 21-Apr-06 Callisto RC1 [Done]
  Friday, 5-May-06 Callisto RC2 [Done]
  Friday, 19-May-06 Callisto RC3 [Done 23-May-06]
Iteration 3 (4.2 i3) Friday, 26-May-06 Stable build [Done 31-May-06]
  Wednesday, 31-May-06 Callisto RC4 [Done]
  Tuesday, 20-Jun-06 Callisto RC5 [Done]
  Wednesday, 28-Jun-06 Callisto RC6 [Done]
Iteration 4 (4.2 i4) Friday, 30-Jun-06 General Availability, English only
Post-iteration Aug-06 (tentative) General Availability, Translation

For a detailed development schedule of TPTP 4.2 release, click here.

Target Operating Environments

In order to remain current, each TPTP release targets reasonably current versions of the underlying operating environments.

Most of the TPTP SDK is "pure" Java™ code and has no direct dependence on the underlying operating system.  The chief dependence is therefore on the Java 2 Platform itself.  The TPTP 4.2 release is written and compiled against version 1.4 of the Java 2 Platform APIs, and targeted to run on version 1.4 of the Java 2 Runtime Environment, Standard Edition.

There are many different implementations of the Java 2 Platform running atop a variety of operating systems. We focus TPTP testing on a handful of popular combinations of operating system and Java 2 Platform; these are our reference platforms.  TPTP undoubtedly runs fine in many operating environments beyond the reference platforms we test.  However, since we do not systematically test them we cannot vouch for them.  Problems encountered when running TPTP on non-reference platforms that cannot be recreated on any reference platform will be given lower priority than problems with running TPTP on a reference platform.

TPTP SDK 4.2 is tested and validated on the following target reference platforms (this list may be updated over the course of the release cycle):

TPTP Agent Controller Reference Platforms
Processor architecture Operating system
Intel IA32 Red Hat Linux v7.1, v7.2, v7.3, v8.0
Intel IA32 Red Hat Linux Advanced Server v2.1
Intel IA32 SuSE Linux v7.2, v7.3
Intel IA32 SuSE Linux Enterprise Server (SLES) v7, v8
Intel IA32 Windows 2000 Advanced Server (service pack 2)
Intel IA32 Windows 2000 Professional (service pack 2)
Intel IA32 Windows 2000 Server (service pack 2)
Intel IA32 Windows NT 4.0 (service pack 6a)
Intel IA32 Windows Server 2003
Intel IA32 Windows XP Professional
iSeries OS/400 V5R1, V5R2
PA-RISC HP-UX v11.0, v11i
RS/6000 AIX v4.4.0, v5.1, v5.2
SPARC Sun Solaris v8, v9
zSeries z/OS V1R7
zSeries SuSE Linux Enterprise Server (SLES) v8
PowerPC/64-bit Red Hat Enterprise Linux AS release 3

Although untested, TPTP should work fine on other OSes that support the same operating system kernel and version.

Internationalization

TPTP is designed as the basis for internationalized products. The user interface elements provided by the TPTP SDK components, including dialogs and error messages, are externalized. The English strings are provided as the default resource bundles.

Latin-1 locales are supported by the TPTP SDK on all of the above operating environments; DBCS locales are supported by the TPTP SDK on the Windows, GTK, and Motif window systems; BIDI locales are supported by the TPTP SDK only on Windows operating environments.

The TPTP SDK supports GB 18030, the new Chinese code page standard, on Windows XP and 2000, and Linux.

TPTP supports ICU4J starting in 4.2 release. This will significantly increase the number of supportable locales. Products needing to localize to newer locales are enabled. German, Traditional Chinese, and Arabic are tested. 

Compatibility with Previous Releases

TPTP 4.2 will be compatible with TPTP 4.1.  The following specifies details of the various aspects of release compatibility.

Themes

The TPTP PMC adopted and specialized the following Eclipse themes which represent the key focus areas for TPTP enhancements in the year ahead.

Projects

The TPTP project is is comprised of four, managed in a coordinated fashion, across which the plans items are allocated.  TPTP subprojects include:

Features

Plan items targeted for this release represent the addition of new features or areas where existing features will be significantly reworked or enhanced.  Plan items are allocated to themes and projects indicated above.

TPTP Platform Project Plan Items
Status Description
Committed / Done Dynamic Probekit and Byte Code Insertion (BCI). Until now probes are created and Java class files are instrumented statically within the Eclipse Workbench. This feature allows for dynamic instrumentation of byte code at the time of class load using a dynamic BCI technology. This will eliminate the need for copying and modifying class files (109684). [Theme: Simple to Use]
Committed / Done Java 2 SE Code Analysis Tool. In an effort to increase end-user tools in TPTP, a Java code review and analysis tool will be implemented using static analysis framework. A set of 70 common code analysis rules for Java 2 SE are provided as a part of the tool. 113791 [Theme: Appealing to Broader Community, Simple to Use]
Committed / Done Improvements to Static Analysis Framework. A number of improvements are planned for static analysis framework - support for user defined configuration parameters through new extension points and associated UI for editing them (113795), display of rule count per category and total selected in analysis dialog (113792), collection and annotation of time spent per rule and per category (113790). [Theme: Design for Extensibility: Be a Better Platform]
Committed / Partially Done Support for New Target Platforms. Adding several new platforms to the list of TPTP supported platforms such as Windows and Linux operating systems on EM64T (64-bit) and Intel Itanium Processor Family hardware. Additionally adding support for latest version of current supported platforms: 108582, 121656, 108577,108578, and 108579. [Themes: Enterprise Ready, Appealing to Broader Community]

Status: Completed support for Windows on EM64T and IPF added and upgraded to newer versions for various currently supported platforms.

Committed / Done Data Aggregation in Java Trace Collector. Full execution trace is not suited for profiling larger applications over a prolonged time period. Aggregation of data is necessary to keep the size of collected data manageable. This feature will implement data aggregation algorithms in JVMPI monitor and exercise already existing model capabilities for storing such data (108646). [Theme: Enterprise Ready, Simple to Use]
Committed / Done Improvements to UI features. Sort by time in symptom analysis results view (102390), log table view (108363), filter log events on complex types (108371). [Theme: Simple to Use]
Committed / Done Performance Improvements. Several performance improvements are planned - trace model (108938), logging (112371 and 112878). [Theme: Scaling Up]
Committed / Done Profiler support for JVMTI. JVMTI is the new standard and replaces JMVPI which will not available starting in Java 1.6. A technology preview of JVMTI-based Java 2 SE profiler will be released. It is a brand new implementation and and represents future direction of TPTP Java profiling and tracing tools (86225). [Theme: Design for Extensibility: Be a Better Platform]
Committed / Done Launch UI Enhancements to support multiple agents. This is a required feature for taking advantage of the flexibility and power of JVMTI standard. A launch dialog should support ability to specify multiple agents and their configuration (93212). [Theme: Design for Extensibility: Be a Better Platform]
Investigating / Deferred Security and Dynamic Discovery API in new agent controller technology is missing implementation (95546,74579). We are investigating opportunity to reuse implementation from backward compatibility layer of the new agent controller. [Theme: Enterprise Ready]
Helpwanted / Deferred Port of TPTP Target Environment to Mac OS X. This calls for porting TPTP C/C++ implemented agent controller and data collection agents, namely JVMPI monitor and native logging to Mac OS X (68111). [Theme: Appealing to Broader Community]
Helpwanted / Deferred Port Native Logging Component to Palm, Windows Mobile, Nokia and Sony Ericsson embedded systems (111019). [Theme: Embedded Development]
Helpwanted / Deferred Link Checking Tool based on Static Analysis Framework. Enable the static analysis framework in TPTP to check for broken links in documentation. TPTP project build should be able to run the check, produce a parse-able report, and send an email automatically to all plug-in owners whose documentation contains broken links. It is desirable to support adding additional rules for checking other documentation guidelines.(107856) [Theme: Appealing to the Broader Community]

 

TPTP Testing Tools Project Plan Items
Status Description
Committed / Done Graphical test results overview. A graphical top level summary of test results as well as certain level of details linked with the summary on overview page to be added to the current overview tab, from where user can easily navigate to details. (103549) [Theme: Simple to Use]
Committed / Done Navigate back to the test case from the test results. When verdicts or invocation event provides information about the test script file and line number of the invocation, Test Log Viewer should provide the function to navigate back to the code. This is especially useful when there are VP events. (103551) [Theme: Simple to Use]
Committed / Done Test Log viewer improved extensibility. Details view of execution event should display the properties contained by the events in the model and should also be extensible to allow customized properties. It should also be possible to add actions associated with certain execution event. An extension point can be defined to allow that. (103555) [Theme: Simple to Use]
Committed / Done Include a macro editor with the auto gui test suite editor. It is currently difficult to navigate through the macro of a test case. Users have reportedly copied and pasted the macro of a test case into a separate editor just so it is easier for them to edit it. The purpose of this feature is to provide better means for users to easily navigate through the macro of a test case. (110337) [Theme: Simple to Use]
Committed / Deferred Integrate the Manual Test View with the Eclipse TPTP workbench. This enhancement involves integrating the Manual Test View with the Eclipse TPTP workbench which requires the following: Port the Manual Test View to an embedded Workbench view in the TPTP Test perspective with no new functionality. Port the Manual Test View to a plug-in application (e.g. Rich Client Platform (RCP) or Generic Workbench compared to the IDE Workbench facilities defined in the org.eclipse.ui.ide plug-in) based on the OSGi Framework to provide extensibility via extension points. Provide a command line wrapper that emulates the Agent Controller environment for launching the Manual Test View independent of the Agent Controller. (121100) [Theme: Simple to Use]
Committed / Done Documention of the generic recording framework. Recording is one of the common starting points for creating a test for test tools. A generic recording facility can help provide a common UI interaction starting point of recording for users. The recording facility should allow other test types, recording protocols to leverage a common UI interaction (i.e., "Record a Test"). There should also be an update concerning the use of terminology to reflect usability feedback from users. (122949) [Theme: Simple to Use]
Committed / Done Support annotations for all ExecutionEvents. Since annotations for all ExecutionEvents are currently supported in the TPTP Test model (see class diagram for more details), this enhancement requires exposing this support to internal components and external users. That is, a schema with documentation is required for components to generate well-formed ExecutionEvents containing annotations coupled with providing support in the Test model loaders to consume ExecutionEvents with annotations. Furthermore, modifications to the Test Log view are required for external users to access annotations contained in ExecutionEvents from the UI. (76160) [Theme: Simple to Use]
Committed / Done Dynamic test asset deployment when test closure is not staticly definable. Enable dynamic test asset deployment when test closure is not statically definable Implement test service to allow retrieval of dependent classes or other files from workspace during test execution (87414) [Theme: Simple to Use]
Committed / Done Execution History Editor: Searching. Allow users to search an execution history by any of the visible attributes of a given execution event. Examples include time window, associated interaction fragment (test model element), etc. Search results should be displayed in the eclipse search view, and navigation should be provided from the search view back to the selected element in the execution history. Also allow extension point to register custom event types for searching (i.e. HTTP Request) (89341) [Theme: Simple to Use]
Committed / Done Improve usuability of the TPTP test reports. The report should have a summary to show which execution histories the attempted status (wedge in pie chart)? % of attempted and not attempted and links to each. The Test Suites should be refactored by platform, and a report should display them.Some sort of way to track which build or series of build that it was run on should be available. A consolidated lists of exceptions should be displayed (defects that are blocking test success and inconclusive results, both of those by test across the project). Add the bugzilla priority and resolution status so that you have one nice page, "this is a blocking issue & here's its bugzilla status". The hierarchy of test suites should be displayed (by project, also summaries of tests vs. the long detailed). A project health page that would combine bugzilla and test results: Summary of numbers by severity/priority. (109657) [Theme: Simple to Use]
Investigating / Deferred Test Execution and Agent Data Collection. Test Harness should be able to invoke user selectable agent data collectors when test is invoked on specified machines and associate the collected data results as with the test run. These choices, including which data should be collected, needs to be persisted (with some naming scheme), so that subsequent test runs can re-use the same data collection choices (or easily edit them). (75029) [Theme: Simple to Use]
Helpwanted / Deferred Automated Documentation Generation. The purpose of this feature is to allow users to automatically generate human-readable user instructions for a use case scenario that has been recorded. This will assist technical writers in ensuring that the most up-to-date instructions along with screen captures is shipped with the product. It's purpose is to also reduce translation costs by having the macro run with different language packs as opposed to requiring a 3rd party company to translate the same set of instructions into 8 or 9 sets of different languages. (110108) [Theme: Simple to Use]
Helpwanted / Deferred Support RCP applications for recording and playing back GUI test cases. The recording infrastructure needs to be separated from the playback infrastructure. The two need to be very loosely coupled Allow users to record and play back test cases for an RCP application The user experience must be very similar to how a test case is recorded and played back in the workbench (114159) [Theme: Simple to Use]
Helpwanted / Deferred Leverage Eclipse Contexts in the Test Perspective. Contexts provide support for the programmatic display (and possible removal) of views within the perspective. This is valuable when considering a mixed-test scenario within TPTP. Some test types may have additional views that are test-type-specific and not be relevant to other tests. Supporting contexts would allow the Test Perspective to display views relevant to a selected test type (e.g., selected in the Test Navigator) and hide irrelevant views. (83782) [Theme: Simple to Use]

 

TPTP Tracing and Profiling Tools Project Plan Items
Status Description
Committed / Done Divide the online help into documentation for users and documentation for consumers. The Foundation has asked TPTP to divide its on-line help into two categories: doc for users (developers who use TPTP to test & profile) and doc for consumers (extenders of TPTP). Both types of documentation remain in the plug-in format, but the consumer documentation should be shipped only in the SDK driver. The user documentation should remain in the binary production driver of TPTP. (109897) [Theme: Simple to Use]
Committed / Done ICU4J support in TPTP. Eclipse will incorporate and package ICU, however there are no packaging details as of yet. This will be in a plugin and intended for eclipse 3.2 platform. This enhancement applies to all TPTP UI, non test components. (120002) [Theme: Simple to Use]

 

TPTP Monitoring Tools Project Plan Items
Status Description
Committed / Done Provide mechanisms for adding and launching new statistical agents. Currently, the list of statistical agents that can be launched by TPTP is hardcoded. This feature will provide an extension point and the necessary infrastructure to allow third parties to implement new statistical agents and extend the monitoring capability of TPTP. (113166) [Theme: Design for Extensibility: Be a Better Platform]
Committed / Done Provide authentication when monitoring remote Windows systems. Currently the Windows Perfmon Agent is able to gather statistics from a remote system other than the one it is running on. This requires that the agent has the correct authentication to access the Perfmon data on the remote system. This can be accomplished by having the user map a network drive to the target system before starting the agent. This is an inconvenient extra step required to be performed by the user. This feature will add authentication capability to the Perfmon Agent so that it can connect to the remote system on behalf of the user. (118545) [Theme: Simple to Use]
Committed / Done Provide a new Symptom Database editor. A new symptom specification is being introduced with the Platorm Project feature 124699. Therefore a new symptom database editor is required to create symptoms in this new format. A conversion utiltity will also be provided to convert current symptom databases to the new specification. (64800) [Theme: Appealing to the Broader Community]
Committed / Partially Done Simplify Log Import User Experience. Currently, specifying multiple log files on a log import operation is difficult. The user must create a log set and add individual logs of various types to it. A support person may have received a zip file containing many logs of various types from a customer. They must unzip the file, determine which logs are of which types and create a log set for them to import them all at once. Another example of a difficult log import scenario is if a user wants to import multiple logs of the same type from different locations on the file system. Again a log set needs to be created and the individual log file need to be added to it as separate entries in the set. These log import scenarios will be simplified from a user perspective. If a zip file of log files is specified, the log set will be created automatically. The ability to specify multiple files of the same type on the Details pane will be added. (114818 and 115087) [Theme: Simple to Use]

Status: 115087 is done and deferred 114818 .

Committed / Done Improve performance of local log import. Currently for a local log import GLA parses the log file into producer CommonBaseEvent objects which are then serialized into XML and converted to consumer CommonBaseEvent objects when they are loaded into the EMF model. These conversions will be eliminated by creating a new GLA formatter that generates consumer CommonBaseEvent objects directly in the EMF model. The Log Import Wizard will modify the existing adapter configuration by replacing the existing formatter and outputter components with this new formatter component. (117062 and 117068) [Theme: Scaling Up]
Committed / Done Provide a log file parser so TPTP can import Java Logging XML log files. A GLA parser will be created for parsing Java Logging XML log files that can be integrated with the TPTP Log Import facility to enable import of this type of log file. (77318) [Theme: Appealing to the Broader Community]
Committed / Done Provide ability of GLA to automatically set CommonBaseEvent sequence number value. Currently a parser writer must write a static parser class in order to set CommonBaseEvent sequence number value correctly. This feature will enable the parser writer to specify a simple rule for sequence number that will cause GLA to set the sequence value correctly based on the creationTime value. (79579) [Theme: Simple to Use]
Committed / Done Provide ability of GLA to filter log data in the Sensor and Extractor. Parsing log data using regular expressions can be expensive. If the log data can be filtered before reaching the Parser component of GLA less regular expression parsing is required which will improve the performance of parsing log files with GLA. This feature will provide the infrastructure in GLA that will allow parser writers to provide filter specifications or a filter exit class to the Sensor or Extractor components of GLA to filter out log data. (79565) [Theme: Scaling Up]
Committed / Deferred Provide ability to programmatically modify GLA adapter configuration. Applications can package and make use of GLA run-time and adapter configuration files to parse log files. However, to use the packaged adapter files they must be modified. For example, the adapter configuration must specify the location and name of the log file to parse. Currently, the only way to do this programmatically is by using XML parsing libraries to read and modify these XML files. To simplify adapter configuration modification, this feature will extend the current org.eclipse.hyades.logging.adapter.Adapter API to provide methods to retrieve and modify the configuration before it is executed by GLA. (96433) [Theme: Design for Extensibility: Be a Better Platform]
Committed / Partially Done Simplify GLA Adapter Configuration Editor. Currently, creating a GLA adapter configuration file with the Adapter Configuration Editor to create a log parser is too complicated. It requires extensive knowledge of the GLA architecture, regular expressions, and the CommonBaseEvent schema. The Adapter Configuration Editor user interface will be simplified to make it easier to create a log parser based on a sample log file. (78319, 104487, and 106050) [Theme: Simple to Use]

Status: Deferred 78319. Completed 104487, and 106050.

Committed / Technical Preview Instrumenting Applications for Monitoring. The ability to monitor and manage existing Java applications can be difficult without manually modifying the source code to add logging or management instrumentation calls. To simplify this process, tooling will created to allow a user to automatically instrument their existing Java applications with code that generates Common Base Event events or Java Management Extensions (JMX) management calls based on filter criteria. The instrumentation will be performed using byte code insertion. Two methods of byte code insertion will be supported: probes using TPTP Probekit and aspects using AspectJ. (128731 and 130774) [Theme: Appealing to the Broader Community]
Committed / Technical Preview Managed Resource Explorer.In order to manage a resource, a user needs to interact with it via a user interface. A generic resource management user interface will be created to communicate with a management service in order to perform the management operations on the resources. Initially, the Java Management Extensions (JMX) service will be supported, but an extension point will be created to allow the user interface to communicate with other management services. (129450) [Theme: Appealing to the Broader Community]

Defects

In addition to the targeted features for this release, we plan to reduce the defect backlog. Defects are prioritized based on severity, age, and resource availability. We encourage users to report defects and we will do our best to fix them in priority order. We plan to reduce backlog of defects to under 350 defects (approximately 30% reduction relative to backlog at the end of  previous release).

See TPTP 4.2 Defects for a listing of already fixed defects, current defect targets and backlog.

Select "4.2 [Completed | nil| I1 | I2 | I3] bugs" tabs.