The purpose of this document is to enumeration the known requirements against the Hyades project in the next few quarters.
The intention is to provide Hyades GA level drivers quarterly, and this document will help communicate what the pressures are and how they are being prioritized. This document should not be taken as a commitment of all the things being developed. It should also not be assumed to be a complete or fixed list of requirements. The project is dynamic in many senses including the staffing and the requirement pressures.
Throughout the document you will see a notation like “T23(P1 1.3)” or “(P2 1.2)”. This is a reference to the theme the requirement is linked to “T23”, the priority assigned “(P2” and the target release “1.3)” for that priority. The objective is to describe the context of the requirement and provide a proper amount of support for each theme. Requirements with only a theme identified are not under consideration for the 1.1, 1.2 or 1.3 time frame.
Briefly, P1 means the release can not ship without the function, P2 means the function is highly desirable for the release and all effort will be made to do it, but it is not a stop ship function. P3 is basically a nice to have. Note the priority is not a staffing commitment by the core Hyades development team, and we are always looking for help and contribution.
The releases are intended to ship quarterly. 1.0 was targeted for the end of June 2003. 1.1 is targeted for the end of September 2003. 1.2 is targeted for December 2003, and 1.3 is March 2004.
At this time (summer 2003) the closer of both these standards is still underway. Until both of these standards are finalized Hyades will simply monitor them and feed back as appropriate. A commitment can not be made until closure is had on these standards, and in particular it is determined how best to get access to a UML2 model in the Eclipse environment. There are several discussions underway for projects in the UML2 area, but until something concrete and public is available Hyades will have to continue on its current path of building a wrapper service to abstract the needed structures, and maintain a façade over this area.
It often occurs that both a user and a tool developer need to modify code that has already been deployed. This could be for the purpose of adding a “System.out”, a logging event, or instrumentation to for the purpose of code coverage analysis among many other uses.
Hyades will provide a framework for doing this type of task, as well as a few tools that leverage this capability.
The base toolkit will provide a runtime component that can be used (at class load time for Java) to modify the byte stream of a class. There will also be an editor for creating or modifying the definitional “script” for a given probe.
For convenience of the probe writer there will also be a set of libraries available for the common tasks of logging and tracing that integrate well with the rest of Hyades.
In Hyades 1.0.2 a HTTP recorder and playback sample tool was provided to demonstrate how the Hyades model and data collection infrastructure could be used for this task. This was basically a generated URL blaster. In stages over 1.1 and 1.2 a more robust HTTP stress testing tool will be provided.
Support for SSL recording, data pool exploitation, and mixed user loads will all be provided.
In combination to the other data collection support in Hyades, this can be a very powerful tool when you are stressing and fixing your application.
Tools that generate code into Eclipse projects should set cross project dependencies automatically to allow for a simple build process. This means a close working resolution of this issue with the team support environment provided in Eclipse. In addition test projects should be able to refer to each other just as Java projects can, and the equivalent of build dependencies when dealing with the namespaces used in code generation for example.
A common use case of tests is to have them reused in slightly different execution environments. One thing that is required for this environment is that configurations need to be customizable for a specific deployment or run of the test. Not only does this imply that deployment time modifications need to be possible, it should be considered if for audit reasons those edits should be saved. In the audit scenario we recommend the change management system should be used and versions of the configuration saved. However temporary edits should be allowed by the Hyades user interfaces.
A feature that was targeted for 1.0.0 and did not make it was support for model loaders to be able to have their own extension points. The prime purpose for doing this is to provide a means for multiple handlers for a given fragment arrival. The current structure matches exactly no or one loader for each fragment type, however often there may be a need for further awareness of the event arrival. This may be before or after the data has been reflected in the model. A general notification scheme could be used, but it is felt that this would be too heavy a solution for this problem. Basically when a fragment arrives, loaders and other handlers need to be invoked in arbitrary order. Once a loader is about to run, or finished running another set of handlers need to be enabled. This should not require a resource save or any other state condition. Just the time situation of preload, load and post load time need to be trigger points. An example usage may be that as data arrives further stimulus needs to be sent to the execution environment. An automated tuning use case would be an example. A responsive stress testing tool would be another.
Another feature needed in the near term for Hyades is the ability to more dynamically control a test environment. The user needs to not only be able to run a test, but they also need to stop, pause and resume it. In addition batch executions need to be able to be dynamically attached to for monitor and control.
Performance analysis, test drivers and other tool scenarios supported by Hyades sometimes require the capturing of parameter values when recording or tracing. These may be needed for analysis or simply used in the playback of a recording.
A natural exploitation of being able to capture values during a trace is to use values during an execution. Although the support of capturing values and using of values can be separated, they have so much in common at the infrastructure level we will link these two capabilities together.
An important facet of using values at execution time is the notion of a data pool. Hyades needs to support set of data pools at execution time to allow for variable substitution of static and live data. This implies both a data storage facility for large volumes of static data, and runtime access to random data generators and other possible sources.
There is a requirement to allow editing environments running outside Eclipse to be able to interact with a Hyades project.
Although importing and exporting projects is naturally supported, there are no current plans to provide addition external tool integration beyond the basic plugin and resource integration provided by the base Eclipse workbench and the Hyades components themselves.
The general objective of Hyades is to handle more than Java. For example a more direct support for protocols like HTTP/S is needed to best exploit that type of record and playback. In addition languages such as C# are important players in a distributed environment.
Beyond the basic definitional recognition that these other languages needs to be supported, the execution environment needs to provide a native binding for these languages without any unneeded heavy cross language infrastructure. That said, fortunately the data control and communications layer of Hyades is native code with additional Java bindings. Some additional work is needed to provide equal work at the native level and this is the near term requirement on the execution environments that wish to directly drive with these non-Java language environments.
As we have grown the content and capability of the Hyades tools, the use of the profiling and test perspectives need to be re-visited. For example, what is the appropriate place to integrate code coverage views? In addition, linking performance profiling to debug views may be a natural way to integrate with the JDT for example.
Team support needs to be improved. Cross project references are facets of complexity in this problem area, however cross artifact associations are the problem area. This is amplified by the lack of artifact comparing tools. With structured resources using EMF we have a concise representation of the information, but this representation is tool friendly versus user friendly. Not only do we need to have these comparison tools, these tools need to deal with the implication of differences at the other end of an association.
For example the class path may be captured in the configuration, but if the test structure or SUT changes at a packaging level, the class path may need to be changed. This is the first step of doing impact analysis. Full impact analysis does not need to be support in the first pass, but it needs to be facilitated by the models and the user interfaces and extension points.
Another important concept that needs to be supported and exploited in Hyades is the notion of typed agents. This means that agent types and names are both captured in the model as well used to enable various user interface metaphors such as filter settings and agent selection. This implies more selective control in the user interface of the agents being used, plus visibility to the user what he agent actually does. The control is already possible; however it is not exploited by the UI or the agents.
Continued focus on speed and footprint will be required as well. The overall memory needs of the Hyades components needs to be continually reduced. Importantly at this time the tools and EMF itself work on a memory model of everything is live. This is not acceptable for tools that deal with large volumes of data, yet the problems in this area need to be hidden from the end user. The intention in Hyades is to make this scaling possible and allow the higher volume solutions become a value add vendors can plug in to the base Hyades tooling and services.
Finally there are always new realizations in the area of usability. A constant effort to continue improving in the area is key to project success. We have several requirements for increase ease of use and scalability for the very basic Log Analysis viewer and as with all requirements in this area, there will be incremental progress in each release.
Hyades already supports profiling data collection from the command line for cases where launch and/or attach from a workbench is not appropriate. The test execution and collection environment needs to provide same support. In other words, a user should be able to run a test suite and have all events log to a single or small number of files that can later be imported into the workbench.
Naturally the report generation capabilities in the test environment also need a “GUI less” way of being invoked as well. This will better enable large scale projects and well as easier reuse of the assets Hyades can create.
One typical quality metric associated with testing is coverage. This is sometimes practically limited to function level coverage, but is normally thought of as a line level or logic block level metric. This is a natural extension of the trace support provided in Hyades, but adds unique visualization and use cases to the data.
Hyades will need to provide complete infrastructure to handle all levels of coverage collection and reporting, but will need to provide at least a function/method level coverage tool as example function.
Execution engines need to be able to run other execution environments. In its simples form, a simple service to invoke Java implementations from other Java implementations would be okay. However this will lead to a discussion that will end up defining an important part of a generic testability interface.
Hyades definitely needs to work on defining both a generic interface to execution environments as well as to the testable interface of a SUT. Initially getting closure on the execution environment is appropriate and work to resolve the rest may be longer term.
Hyades provides a log monitor/import/analysis capability. Many tool providers will add their own log imports; however the current mechanism suggests a static parser be written. Many log formats are very predictable and can be parsed using basic regular expressions. A basic generic adaptor that is configurable for various pluggable data sensors, parsers and formatters will be provided in Hyades the ease the effort of adding more data sources.
In addition to recognizing basic log entries, a commonly logged piece of data is a stack dump and this can be more optimally captured than the current basic CBE.
Hyades provides a statistical model for storing numerical data over time. The intent was for this model to be used to store such things as system resources for viewing and analysis during load testing for example.
Now that we have a model, we need a data collection agent and a generic viewer to show trends graphically and a table view for the raw data. It has been suggested that we do a “perfmon” style data collector first as a sample data collector for this type of collection.
Hyades has developed a set of models for some domain specific usages. However the roots of Hyades are the linkage of these domains. It has already become apparent that we need to accelerate the linkage at the model level. For example we know intuitively there is at least a logical linkage between the behavior of a test case with the interactions captured in a trace, and we are working on this link in sync with the evolution of UML2.
Another important usage of the kinds of data collected in Hyades it the possible association of data in a log with data in a trace or a particular test execution. Another example is the capturing of statistical data during the run of a test or trace scenario.
These associations need to be made available in the Hyades models, but in a way that does not increase the cost of the models.
Often data collection can be done live and incrementally at execution time.
In fact this style of collection model has been avoided historically in order to deal with data compression immediately. Particularly in performance collection, the notion of collecting trace styled data has been avoided and the type of analysis desired was used to define compression techniques at collection time. Hyades has promoted trace style collection for example in order to simplify the collection agent. In effect this means the collection agent stays simple and sends trace data when ever it is collecting, and the data is filtered by the agent and the listening workbench where it is used to populate the model for various types of analysis, potentially avoiding re-runs of a scenario.
However because there are existing data collectors that locally aggregate, and there are data volume and disconnected scenarios that require compression of the data, Hyades needs to allow this data to be loaded without having to simulate tracing. The models are already capable of handling this, but the fragments that need to be loaded are not defined nor are the loaders provided.
One of the more difficult things to deal with in a large application is often the understanding how things actually flow once all put together. This is a critical piece of information for basic maintenance as well as more detailed performance tuning work.
Ensuring that each module developers write is correct, robust, and maintainable is an integral part of the process one should follow to deliver quality software. Static analysis techniques are commonly used to automate this process of reviewing code and verify that it complies with a set of rules. Rules typically range from readability to performance improvement, to best practices enforcement, to defect detection.
The goal of this framework is to provide a set of extension points for rules engines to plug onto Hyades. The purpose of these extension points is to provide a means to report results of such analysis as a test phase result. Additionally, the framework will provide a generic rule engine along with a rule schema and a basic set of rules definitions.
In addition to new scenario support, which will drive changes to the base framework there are enhancements to improve the existing user experience. For example once paging lists are leveraged in the models, the UI needs to be reworked to exploit this memory footprint optimization (P2 1.3).
The interface needs to exploit agent types and names to give the user more control.
The following are broken down to the various topical areas of the user interface code.
Log View and Analysis
Agent Control and selection
In general there is a need to provide a standard “best practices” way for vendors to extend the models without requiring the inclusion of their changes to be in the open source project, yet still allow data interoperability with other select vendors.
(T8) Models also need to have comparison tooling, and this may require some common services at the model level to make merge easier to manage.
(T4 (P2 1.1))Model loaders need to have a more flexible framework to execute in. Namely handlers need to be able to be registered dynamically for fragment arrival (pre-loader), load time (loaders), load complete. This should not be tied to resource state as there is already sufficient mechanisms in place on resource state change boundaries.
(T8) Continued performance tuning in the areas of footprint and speed, as well as low cost dynamic model extensibility is always important to continue focusing. A specific area of improvement will be to leverage the “paging list” support being developed in EMF which will allow unreferenced objects in list to be garbage collected (P2 1.2). Assuming paging lists are completed and exploited, making the addition step to optionally back the models in a relational database versus XMI file would be the next level of scalability (P1 1.3)
Have a way to display (unknown) nodes and associated actions in the Test Navigator.
Indeed, the Hyades model is candidate to be extended by a tool vendor. For example, the SUT object is generic and could be re-defined to a link to the source code under test and verifications on this test.
In that case, the test case would have two different children:
- a link on the source code
- a test verification
The associated tree could be something as:
|--Test case 1
| | |
| | |--assert1
| | |--assert2
| | |--
| | |
| | |--sourceFile1.java
| | |--sourceFile2.java
| | |--sourceFile3.java
| | |
|--Test case 2
|--Test case 3
A right click on a "verification" node could display a sub menu "test report".
So if I summarize:
- to have a way to display new nodes in the test navigator,
- to have a way to define action on these nodes.
First, I thought about a solution based on extension point, but if it exists something simpler it will be fine.