TPTP Testing Strategy Part 2 - Test Conventions and Best Practices
TPTP Testing Strategy Part 2 - Test Conventions and Best Practices
Title: TPTP Testing Strategy
Author: Alan Haggarty (haggarty@ca.ibm.com)
Show the Change History
Name Date Modified Sections
Alan Haggarty (haggarty@ca.ibm.com) June 12, 2008 Initial Creation

Table of Contents
1.0 TPTP Test Project Conventions
1.1 Naming Conventions
1.2 Test Project Detailed Structure
2.0 Test Creation Best Practices
2.1 Creating Manual Test Suites
2.2 Creating JUnit and JUnit Plug-in Test Suites
2.3 Creating AGR Test Suites
3.0 Test Execution Best Practices
3.1 Smoke Testing
4.0 TPTP Test Project Exceptions
5.0 Build Verification Test (BVT)
5.1 Reference Platform
5.2 Adding Tests
5.3 Adding Results
5.4 Installing and Executing
5.5 BVT Project Set File

1.0 TPTP Test Project Conventions

1.1 Naming Conventions
  1. All names for testing resources and folder names MUST adhere to the following rules:
    • Upper or lower case letter (for example, A-Z or a-z).
    • Digits (for example, 0 - 9).
    • Underscore (for example, _).
  2. Names must NOT contain any white space characters (for example, tab, space, and new line).
  3. Every name should begin with an upper or lower case letter (for example, A-Z or a-z). All other characters in the name should be upper or lower case letters, digits, or the underscore.
  4. Java keywords (for example, import, if, etc.) cannot be used for names of JUnit test resources.
  5. The name of the Deployment file (deployment files are often used for remote launch of test suites) may contain '.' characters. The name should be meaningful to the project and the host that it is associated with. For example, a local deployment file for the org.eclipse.hyades.use.cases folder can be org.eclipse.hyades.use.cases.localhost.deploy
  6. The name and directory structure containing the generated execution history file MUST be the same as it will be stored in CVS. If the generated execution history file is either moved or renamed, the linkage with the associated test suite will break thereby corrupting generated reports. In the event of an existing execution history in the same directory, TPTP Testing Tools will append a millisecond time stamp to the name of the new execution history file.
  7. Test suite names must start with the bugzilla component name associated with the plug-in or component under test.
    If the test suite contains several functions, you can either use the bugzilla component name as the name of the test suite or use the bugzilla component as a prefix.
  8. Multiple JRE or platform test suite names must end with the name, vendor, and version of the JRE or platform under test. These test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.

1.2 Test Project Detailed Structure
Name Purpose
deployment Directory containing all deployment configurations for executing the test suites in the test project.
gui Directory containing all AGR test suites for the test project.
gui/AllTests.testsuite AGR test suite to execute all lower level AGR test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1GUITests.testsuite (see Table 2).
gui/AllSmokeTests.testsuite AGR smoke test suite to execute all lower level AGR smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2GUITests.testsuite (see Table 2).
gui_results Directory containing all execution histories for automated GUI test suites for the test project.
junit_plugin Directory containing all JUnit Plug-in test suites for the test project.
junit_plugin/AllTests.testsuite JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitPluginTests.testsuite (see Table 2).
junit_plugin/AllSmokeTests.testsuite JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitPluginTests.testsuite (see Table 2).
junit_plugin_results Directory containing all execution histories for JUnit Plug-in test suites for the test project.
junit Directory containing all JUnit test suites for the test project.
junit/AllTests.testsuite JUnit test suite to execute all lower level JUnit test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitTests.testsuite (see Table 2).
junit/AllSmokeTests.testsuite JUnit smoke test suite to execute all lower level JUnit smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitTests.testsuite (see Table 2).
junit_results Directory containing all execution histories for JUnit test suites for the test project.
manual Directory containing all manual test suites for the test project.
manual/AllTests.testsuite Manual test suite to execute all lower level manual test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1ManualTests.testsuite (see Table 2).
manual/AllSmokeTests.testsuite Manual smoke test suite to execute all lower level manual smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2ManualTests.testsuite (see Table 2).
manual_results Directory containing all execution histories for manual test suites for the test project.
resources Directory containing any resources required to execute the test suites for the test project.
src Directory containing generated JUnit Java source code for the JUnit and JUnit Plug-in test suites and Java source code corresponding to verification hooks of AGR test suites for the test project.
Table 1: Required structure for contents of a test project.

Each of the subfolders contains a test project named org.eclipse.hyades.use.cases for testing functionality that crosses plug-in or component boundaries.

The test-results/platform/org.eclipse.hyades.tests test project contains all root-level test suites for TPTP. Table 2 details the required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.

Name Purpose
deployment Directory containing all deployment configurations for executing the root-level test suites in TPTP.
enablement Directory containing all the TPTP enablement tests.
TVT Directory containing all the TPTP Translation Verification Tests (TVT).
src Directory containing generated JUnit Java source code for the root-level JUnit and JUnit Plug-in test suites in TPTP and Java source code corresponding to verification hooks of root-level AGR test suites in TPTP.
TP1 Directory containing all root-level test suites to execute all lower level test suites during test pass 1 (TP1).
TP1/AllTP1GUITests.testsuite Root-level AGR test suite to execute all lower level AGR test suites during test pass 1
TP1/AllTP1JUnitPluginTests.testsuite Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during test pass 1
TP1/AllTP1JUnitTests.testsuite Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 1
TP1/AllTP1ManualTests.testsuite Root-level manual test suite to execute all lower level manual test suites during test pass 1
TP2 Directory containing all root-level test suites to execute all lower level test suites during test pass 2 (TP2).
TP2/AllTP2GUITests.testsuite Root-level AGR test suite to execute all lower level AGR test suites during test pass 2
TP2/AllTP2JUnitPluginTests.testsuite Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during test pass 2
TP2/AllTP2JUnitTests.testsuite Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 2
TP2/AllTP2ManualTests.testsuite Root-level manual test suite to execute all lower level manual test suites during test pass 2
BVT Directory containing all root-level test suites to execute all lower level test suites during build verification test (BVT).
BVT/AllBVTGUITests.testsuite Root-level AGR test suite to execute all lower level AGR test suites during build verification test
BVT/AllBVTJUnitPluginTests.testsuite Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during build verification tests
BVT/AllBVTJUnitTests.testsuite Root-level JUnit test suite to execute all lower level JUnit test suites during build verification tests
Table 2: Required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.

The execution history files (*.execution) generated when executing test suites are checked-in to the CVS branch for the release under test by the committer for the associated plug-in or component. For example, the execution history files generated from testing the current full release would be checked in to HEAD. Execution history files are checked in to the *_results folder for the type of test suite. For example, execution history files generated from executing a manual test suite would be checked in the manual_results folder. To delineate between test executions on multiple platforms, each *_results folder contains the following subfolders for each supported platform:

Supported platform subfolders may be optionally appended with the platform version and release:

Test execution reports are generated and checked in to CVS by the TPTP Project Lead or PMC representative at predetermined milestones in a test pass for public web access. Test execution reports are available at: www.eclipse.org/tptp >> Development Plans >> Test Pass Reports


2.0 Test Creation Best Practices

Based on the previous experiences of test resource creators and testers, the following general recommendations will assist testers on creating test cases. Section 2.1 - 2.3 go into more details about best practices for creating manual, JUnit, JUnit Plug-in, and AGR test suites.

  1. A test case should always have a specific purpose for testing a scenario. Having many test cases in a test suite does not necessarily ensure better quality. Each test case must be well thought out and have a specific purpose. Simply testing the basic functionality of a tool is often not enough to ensure high quality standards. Boundary cases are usually considered to be interesting test cases. As an example, consider the case of testing a function that removes entries from a table. The following test cases should be considered:
    • Removing the first entry (a boundary case)
    • Removing the second entry (tests basic functionality)
    • Removing the last entry (a boundary case)
  2. Avoid repeating the same set of steps in multiple test cases. If two test cases require the user to import a log file, then create three test cases. The first test case will only serve as setting the environment for successive test cases and the second and third test case will test the specific functionality in mind.
  3. This point provides a basic summary of the first two: apply the high cohesion principle to test cases. Cohesion means assigning specific responsibilities to test cases. A test case requiring a user to import a log file, profile an application, and use the Probekit has little common in its responsibilities and thus does not follow the high cohesion principle.
  4. Avoid unnecessarily complicating test cases that are designed to test functionality and not performance. For example, a test case that uses a class with 3 methods to test the Java Profiler is likely as good as a test case that uses a class with 10 methods.
  5. Comment test cases thoroughly in the description section of the editor including manual steps for what the test case is performing.
  6. If any resource (such as a datapool) needs to be modified before the test suite is executed, then include that as part of the description of the test suite. Any other important information that the tester should be aware of needs to be included in the description of the test suite.
  7. If an existing manual test case is being automated, then add '(DEPRECATED - Use )' to the beginning of the description of the manual test case.

2.1 Creating Manual Test Suites

Create small and terse manual test cases that cover well-defined partitions of functionality. Smaller sized test cases provide easier units of work for one tester to execute, more realistic reporting statistics and concise steps to reproduce defects.

  1. Create manual test suites with sufficient test cases for one tester to complete in less than an hour. Smaller sized test suites provide easier units of work dividing test suites between testers while decreasing the risk of lost execution data.
  2. In the event of manual test suites being executed by multiple testers, have the tester(s) tabulate the execution histories on paper and the tester with largest assigned test cases will record all the execution histories in the TPTP Manual Test Client.
  3. When structuring the internal organization as the associated test suite folder (for example, manual), create a summary test suite for the subfolder (for example,/manual/AllTests.testsuite). The summary test suite (for example,/manual/AllTests.testsuite) will reference all test suites within the current folder. This summary test suites is registered with the root-level manual test suite (for example, test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.
  4. Although the TPTP Manual Test Client permits recording more than one execution verdict per test case, it is recommended to only record one execution verdict. Otherwise, TPTP Testing Tools arbitrates the execution verdicts based on the following precedence rules (decreasing order) which may cause unintended execution histories:
    • Error
    • Fail
    • Inconclusive
    • Pass

2.2 Creating JUnit and JUnit Plug-in Test Suites
  1. Create small and terse unit test cases that cover individual methods or sequence of methods. Smaller sized test cases provide more realistic reporting statistics and concise steps to reproduce defects.
  2. When structuring the internal organization as the associated test suite folders (for example, junit and junit_plugin), create summary test suites for each subfolder (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite). The summary test suites (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite) will reference all test suites within the current folder. These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites (for example, test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite and test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.

2.3 Creating AGR Test Suites

The user is expected to have reviewed the available AGR User Guide before reading the recommendations below. The user guide is available on the document section linked from TPTP's web site.

  1. Ensure that a recording session is short. Avoid having a generated macro for a test case that is longer than 100 lines.
  2. As mentioned previously make sure that the set of test cases included in a test suite don't perform redundant operations. For example, if two test cases require launching a process, then launch the process once and use it for each of the test cases that follow. At the same time the test cases shouldn't be too dependent on the environment setup by previous test cases. A test case should easily be executable in quick mode without having to run many test cases that precede it. It becomes difficult to debug a test case when it is dependent on more than one test case that precedes it.
  3. Avoid using absolute paths in a test case. Absolute paths should be replaced with relative paths using one of the available static variables (for example, %testsuiteProjectLocation%). Other modifiable fields that are dependent on the environment that the test suite is launched in (for example, host name) should be linked to a datapool variable.
  4. Use a logger to make it convenient to debug verification hooks. See test-results/platform/org.eclipse.hyades.use.cases/src/org.eclipse.hyades.use.cases.auto.common/Logger.java
  5. Comment the test case thoroughly in the description section of the test case. Include all manual steps that the test case is performing.
  6. If any resource (such as a datapool) needs to be modified before the test suite is executed, then include that as part of the description of the test suite. Any other important information that the tester should be aware of needs to be included in the description of the test suite.
  7. If a manual test case is automated, then add '(DEPRECATED - Use <automated test suite name>)' to the beginning of the description of the manual test case.
  8. Avoid position-based recording as much as possible. If position-based recording must be used for a component, then try to include all such test cases of the component in a separate test suite. Clearly note the resolution that the tester should use as part of the description of the test suite.
  9. Verification hooks should verify the value of controls rather than their logical representation. For example, if a tree item represents object A, then verify the existence of the tree item by testing that its text field is the same as the name of object A as opposed to just simply testing for the existence of object A.

There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:

  1. The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. When structuring the internal organization as the associated test suite folder (for example, gui), create a summary test suite for the subfolder (for example,/gui/AllTests.testsuite). The summary test suite (for example,/gui/AllTests.testsuite) will reference all test suites within the current folder. This summary test suites is registered with the root-level gui test suite (for example, test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.
  2. The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated.
  3. A resource directory should be used to contain the resources required for the verification hooks of a test suite. The name of the directory should be <test-suite-name>.Resources (For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. This directory may contain any number of log files or any other files that the test suite depends on). If a datapool is being associated with a test suite, then include the datapool as part of this directory (see the AGR User Guide for more details).
  4. Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites

3.0 Test Execution Best Practices

3.1 Smoke Testing

Smoke testing, or patch/maintenance release testing, refers to partial regression test for testing the majority of the functionality in less time than a full test pass.

Smoke test suites should cover:

Smoke test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.


4.0 TPTP Test Project Exceptions

Some TPTP tests are not manual, AGR or TPTP JUnit/JUnit Plug-in tests as described above.

JVMPI Profiler Test Suites

The JVMTI Profiler tests (Platform.Agents.JVMPI.*) are run like regular JUnit tests but require manual configuration of the local host and installation and configuration of a test server on the remote host. This is documented in test-results/platform/org.eclipse.tptp.ac.testautomation/automation-files/notes/ACTestAutomation.doc.


5.0 Build Verification Test (BVT)

The common test infrastructure is used for the Build Verification Test.


5.1 Reference Platform

TPTP will use reference platforms. New code will be expected to be tested against the reference platform before being committed.

The build verification test will run on the reference platform.

The current reference platform for TPTP is: IBM Java 1.5 (latest SR) and Windows XP/x86.
The build verification tests are run against IBM Java 1.5 and 1.6, Sun Java 1.5 and 1.6 (latest SR), and Windows XP/x86.


5.2 Adding Tests

Tests can be added to the BVT testsuites which are:
org.eclipse.hyades.tests\BVT\AllBVTGUITests.testsuite
org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests.testsuite
org.eclipse.hyades.tests\BVT\AllBVTJUnitTests.testsuite

Tests for the individual JREs are contained under:
org.eclipse.hyades.tests\BVT\AllBVTGUITests_<JRE Vendor>_JRE_<version>.testsuite
org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests_<JRE Vendor>_JRE_<version>.testsuite
org.eclipse.hyades.tests\BVT\AllBVTJUnitTests_<JRE Vendor>_JRE_<version>.testsuite


5.3 Adding Results

BVT results are checked into CVS under:
org.eclipse.hyades.tests\BVT\AllBVTGUITests_<JRE Vendor>_JRE_<version>.execution
org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests_<JRE Vendor>_JRE_<version>.execution
org.eclipse.hyades.tests\BVT\AllBVTJUnitTests_<JRE Vendor>_JRE_<version>.execution

Previous results are overwritten with results from the latest build verification test run in CVS. A report can be found on the website at http://www.eclipse.org/tptp/home/downloads, selecting Development - <Release Version> >> <Build ID> >> Build Reports >> BVT Results. Results are also reported in the latest test pass report http://www.eclipse.org/tptp under >> Development Plan >> Releases Under Development >> Release <Release Version> >> Test Pass Reports. As a result, the build verification tests should be executed during the test pass in order for them to appear in the Test Pass Report.


5.4 Installing and Executing

All tests in the root level BVT testsuites should be run automatically every build.
Developers are expected to test locally before committing changes to CVS.

For setup and execution please see the BVT Readme at org.eclipse.tptp.platform.releng.tools/testautomation/readme.html


5.5 BVT Project Set File

The BVT project set file contains the components checked out during a BVT run. Use the project set file to setup a workspace containing the test execution environment.