TPTP Testing Strategy Part 1 - Creating and Executing TPTP Tests
TPTP Testing Strategy Part 1 - Creating and Executing TPTP Tests
Title: TPTP Testing Strategy
Author: Alan Haggarty (haggarty@ca.ibm.com)
Show the Change History
Name Date Modified Sections
Alan Haggarty (haggarty@ca.ibm.com) June 4, 2008 Initial Creation

Table of Contents
1.0 Preparation
1.1 Install the TPTP Testing Tools
1.2 Extracting the Test Resources
1.3 Structure of the Test Resources
2.0 Creating New TPTP Tests
3.0 Executing Tests
3.1 Creating Test Deployments
3.2 Executing Manual Test Suites
3.3 Executing JUnit and JUnit Plug-in Test Suites
3.4 Executing AGR Test Suites
3.5 Viewing Test Logs
3.6 Submitting Execution Histories
4.0 Help

1.0 Preparation

1.1 Install the TPTP Testing Tools
  1. Open the TPTP download site.
  2. Select the release of TPTP to be tested and download and unzip the TPTP plugins as well as the requirements listed in the “TPTP Plugins for Eclipse” section.
  3. Download and install the Manual Test Tools ("As Is Components" section).
  4. Download and install the BIRT Test Reports ("As Is Components" section).
  5. Download and install Automated GUI Recording (“As Is Components” section).
    Note: Remember to restart eclipse with the –clean option after installing the AGR
  6. If remote execution is required download and unzip the Agent Controller (“Agent Controller” section).
  7. The Installation Guide and Release Notes are located in the “Documentation” section.

1.2 Extracting the Test Resources

Test suites, cases, deployment configurations and execution histories are stored in the TPTP CVS repository (dev.eclipse.org:/cvsroot/tptp or www.eclipse.org/tptp >> CVS Repository), under the test-results module.

The test-results module contains subfolders for each of the four subprojects - monitoring, platform, test and trace - in the TPTP project. Each subfolder contains the test projects for that subproject.

  1. Check out the individual test projects under test-results/monitoring/, test-results/platform/, test-results/test/, or test-results/trace/ to the local workspace.
    For Example: test-results/platform/org.eclipse.hyades.use.cases
  2. To ensure that the local workspace contains the latest test projects, update at the beginning of each test pass and report generation.

1.3 Structure of the Test Resources

Test projects must follow a defined structure.

Please see Part 2 for the required structure.
Test Project Detailed Structure
Test Creation Best Practices


2.0 Creating New TPTP Tests

To create new TPTP Manual, JUnit, JUnit Plug-in and AGR test suites and cases please follow the testing documentation for each type of test. The location of the documentation can be found in the Help section of this document.

The TPTP Manual Test Case Generator may be used to generate the structure of manual test case descriptions based on the standardized structured manual test case format.

Please see Test Creation Best Practices in Part 2 for further information on creating effective TPTP test cases.


3.0 Executing Tests

3.1 Creating Test Deployments

Creating Test Deployments: Detailed steps for creating test deployments can be found in Eclipse Help >> Help Contents >> Testing Applications >> Creating a Test Deployment

Create a Test Launch Configuration

  1. Create a new launch configuration by selecting the entry Test from the list of configurations in Run >> Run Configurations
  2. Use the “Select test to run” pane to navigate to and select the test suite you wish to run.
  3. Select the deployment for running the test suite. By default the test will be deployed locally. Manual tests are always run in the local deployment.
  4. In the Test Logs tab override the default log configuration to match the desired log directory (for example plugin_results). Please refer to the Naming Conventions section of Part 2 for the names of result directories.

3.2 Executing Manual Test Suites
  1. Open the Test perspective.
  2. In the Test Navigator view, select the manual test suite to be executed.
  3. Launch the selected test suite by creating a Test configuration in the Run dialog
    • Under the Test tab select the test suite
    • Choose the results folder associated with the selected test suite and deployment platform
      Note: The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.
    • Click Run
  4. Launch a candidate workbench on the testing platform
  5. Execute each test case as specified in the TPTP Manual Test View on the local host.
  6. Record the result of the test case execution by capturing the status of the test case execution using the following explanation for TPTP Testing Tools Status options:
    TPTP Testing Tools Status Explanation
    Error Do not use.
    Fail The test case could be executed and failed due to a blocking defect.
    Inconclusive The test case was not applicable OR the test case could be executed and failed due to a non-blocking defect. A non-blocking defect is defined as a nice-to-have fix that does not degrade the core functionality being tested by the test case.
    Pass The test case could be executed and passed.
    For verdicts of Fail or Inconclusive please provide a defect number and explanation in the test log.

3.3 Executing JUnit and JUnit Plug-in Test Suites
  1. Open the Test perspective.
  2. In the Test Navigator view, select the JUnit test suite to be executed.
  3. Launch the selected test suite by creating a Test configuration in the Run dialog
    • Under the Test tab select the test suite
    • Select the deployment.
      To run a test remotely you will need to create and configure a new test deployment
      For JUnit plug-in tests you also may want to specify deployment on a different workbench location.
    • Choose the results folder associated with the selected test suite and deployment platform
      Note: The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.
    • Click Run
  4. An execution history will be generated according to the following explanation of TPTP Testing Tools Status options:
    TPTP Testing Tools Status Explanation
    Error The test case was or could not be executed due to a run time error.
    Fail The test case could be executed and failed due to a defect.
    Pass The test case could be executed and passed.
    The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.

3.4 Executing AGR Test Suites
  1. Ensure that the Agent Controller is installed and started.
  2. Open the Test perspective.
  3. In the Test Navigator view, select the Automated GUI Test to be executed.
  4. Execute the test suite by creating a launch configuration item.
    Important: Using the 'Quick Run' mode will not generate an execution history file. Make sure that a proper launch configuration is used.
  5. The runner will generate an execution history according to the following explanation of TPTP Testing Tools Status options:
    TPTP Testing Tools Status Explanation
    Error The test case was or could not be executed due to a run time error.
    Fail The test case could be executed and failed due to a defect.
    Pass The test case could be executed and passed.
    The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.

3.5 Viewing Test Logs

To open a test log file, use the Open File window, or double-click in the Test Navigator the file with the extension, .execution.

  1. From the File menu, click Open File.
  2. Browse to and select the test log file to open. (Test log files use the extension, .execution.)
  3. Click OK.

3.6 Submitting Execution Histories
  1. Execution histories should be checked-in to the CVS branch for the release under test. They are checked into a results folder under the associated plugin for the type of test executed (manual_results, junit_results, junit_plugin_results, gui_results). To delineate between test executions on multiple platforms, each results folder contains subfolders for the supported platforms.
    For Example: The results of executing the manual test org.eclipse.hyades.use.cases/manual/Profiling_and_Logging/Platform.Communication.Control_Channel_Remote_Profiling.Windows_IA32.testsuite should be generated to and checked into org.eclipse.hyades.use.cases/manual_results/Profiling_and_Logging/Windows_XP
  2. Testers who are not committers should email the execution history files to the committer for the associated component.
    Include the following in your email:
    • Name of tester if not same as sender
    • Deployment platform(s) including name, version and release.
    • Execution specific information including name, version and release.
    • TPTP driver build id and test pass identifier.

4.0 Help

General Help

TPTP General documentation can be found in the Eclipse Help under Help >> Help Contents >> Testing Applications
Getting Started with TPTP's Automated GUI Recorder
Getting Started with TPTP's Manual Test Tools
Getting Started with TPTP's Business Intelligence and Reporting Tools (BIRT) Test Reports

Creating Tests

Manual test cases and suites: Eclipse Help >> Help Contents >> Testing Applications >> Testing Manually
TPTP JUnit: Eclipse Help >> Help Contents >> Testing Applications >> Testing with JUnit
TPTP JUnit Plug-in: Eclipse Help >> Help Contents >> Testing Applications >> Testing plug-ins with JUnit
AGR: See the Documentation section of the AGR Getting Started Guide

More Resources

Eclipse Test & Performance Tools Platform Project Web Site
Documentation for TPTP extenders
TPTP Mailing Lists and Newsgroups