High-Level TPTP Test User Experience Design Document

 

 

 

 

 

 

 

 

 

 

 

 

 

History

Author

Date

Comments

Wayne Ho

2005-09-29

First draft.  Portions of the document written by Bianca Xue Jiang and Scott Schneider.

Wayne Ho

2005-10-21

Minor corrections from a review by Jerome Gout, Bianca Xue Jiang, and Scott Schneider.

 

 

 

 

 

 

 

 

 

 

Introduction

 

This document presents user experience designs for the TPTP Test project.  It presents high-level designs including existing designs as well as suggestions for future user experience design discussed during the TPTP Test User Experience Meetings (typically held on Thursdays at 10am ET).

 

The purpose of the TPTP Test User Experience Meetings is to promote good, cohesive, usable user experience for the TPTP Test Project.

 

As new designs are implemented and new user experience design directions are established, it is intended that this document be updated.  This document is intended to be a reference document for feature owners of user experience related features.  Designs described here are intended to filter down into feature design documents.

 

Test Navigator view

The Test Navigator provides the base functionality for managing tests of all types (e.g., URL, manual, performance, component tests, etc.) and test assets. 

 

The navigator offers a test-centric view of project resources by filtering and logically re-arranging tests and test assets.  Such logical representation of tests and test assets provides a simple, useful structure for test management.

 

Relevant enhancements include:

 

Provide Logical representation of test artifacts in Test Navigator:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=74962

 

Test Navigator Displays Execution Results with Overall Verdict Icon:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=90492

Test Navigator Organization

The Test Navigator supports multiple organizations of tests allowing users to see test scripts and test assets logically (i.e., by test type and test asset type per project) (Figure 1) or a resource-based organization (Figure 2).

 

The user can change the organization of the Test Navigator using toolbar buttons or the view drop-down menu.

 

Figure 1. Test Navigator, logical organization

 

Figure 2. Test Navigator, resource organization

Logical Test Navigator

The Logical Test Navigator organization displays test assets organized by projects.  Within each project, test scripts are organized by test types, that is, Manual Tests are grouped together, URL Tests are grouped together, etc.

 

Figure 3. Logical Test Navigator.

 

Test scripts are organized under Eclipse projects that are created by the user.  The physical structure of Eclipse Project provides the user with a simple top-level mechanism to organize test scripts and test assets.

 

Each project may contain any combination of test scripts (e.g., JUnit, Manual, URL Tests, or other test types defined by downstream products).  Different test types are organized under logical folders of test types of a specific type (e.g., a Functional Test type, a Component Test type).  Test type folders impose a standard organization by test type under an Eclipse project.  The test type folders appear in the Test Navigator beneath a project, ideally with special icons. 

 

When a user creates a test script, it is displayed under a corresponding test type folder, according to its type.  Creating a test script of a specific type causes the corresponding test type folder to appear.

 

Test scripts are often the final leaf node in the Test Navigator tree, but it is possible to display more granular items in the tree, such as in the case of a TPTP Manual Test’s test case objects.  It is recommended that further granularity objects (e.g., test steps or verification points) be displayed in the appropriate test editor or an Outline view.

Test Assets

Test assets such as datapools, locations, and deployments are also displayed in the Test Navigator.  Each asset type will appear under a logical folder (with the name of the test asset type) as a child of a project.  Test assets are displayed under the logical folder without taking into account the folder structure of the project (within the Eclipse workspace).

Recording

Common entry points for recording and consistent recording feedback can help promote a good, consistent user experience.

Generic Recording Facility (future)

Support for additional recorders in the TPTP Test Project is required.  A generic Recording facility would be help simplify the user experience by providing a common entry point for recording for users. 

 

For example, a user might wish to start recording via the recording toolbar.  A wizard UI could be displayed to the user allowing for the selection of recorder type.  The user could select the type of test (recording) to create.

 

A feature enhancement exists for a Generic Recording Facility:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=74926

Recording Feedback (future)

Enhancing the Recorder Control view to act as a common recording control and feedback view to support recording, pausing, and stopping recording can help to improve the user experience.

 

The view should support a consistent recording feedback mechanism while still providing flexibility to provide feedback targeted for a given target user (e.g., feedback for non-technical testers).

Datapool Editor

During the TPTP/Hyades 3.0 timeframe, a number of usability issues related to datapools were discovered.  The results came from a number of usability heuristic evaluations as well as usability studies.

 

Issues included:

 

Simplify Data-Entry: Data Entry/Populating Datapool

The New Datapool wizard was enhanced to allow users to define an initial dimension of the datapool (Bugzilla 76713).  This allows users to more quickly establish the size of the datapool and then begin entering data more quickly.

 

 

Keyboard shortcuts improvements

To help improve the productivity of users during data-entry, keyboard shortcuts have been improved and made more intuitive.  For example, the Enter key now traverses to the cell below the current cell and creates a new record if the next record does not exist (Bugzilla 74865).

 

The Tab key traverses to the right and prompts the user to add a new variable when the last column is reached (Bugzilla 89135).

Simplify Equivalence Class

The equivalence class concept is a somewhat advanced concept.  The original datapool design required users to create an equivalence class prior to being able to populate a datapool.

 

To simplify the creation of datapools, the equivalence class concept was simplified.  A new datapool is now created with a default Equivalence Class that appears as a separate data table page/tab.  Users that are not concerned about equivalence classes can simply perform their data editing in this data table.  Additional Equivalence Classes are presented in a separate data table tab of the editor (Bugzilla 73178).

 

Equivalence Classes and variables are managed on the overview page of the datapool editor. They are listed in Equivalence class and Variable sections respectively. Attributes of Equivalence Class or Variable can be viewed and edited via the generic Properties viewer of eclipse (Bugzilla 74864).

 

Equivalence Class and Variable names can also be edited in place in their own sections on the Overview page (Bugzilla 88953).

 

Discovering the data table

After a new datapool is created, the Data Table for default the equivalence class is opened instead of initially displaying the Overview tab.  The first cell has focus and users can immediately begin entering data.

 

 

Importing data

Additional flexibility for importing data into datapools has been added into TPTP.  It is now possible to import data from CSV file into an existing datapool (TPTP 4.0).

 

Future Datapool Enhancements

Datapool usability can continue to be improved through possible additional enhancements including:

 

Run/execution of a test

Running a test has also been found to be difficult for users.  There is a need to simplify the launching/running of tests that may be beyond the scope of the TPTP Test project.

 

However, there are aspects of test execution that can be improved within TPTP including the simplification of managing locations and deployments.

Deployments and Locations (future)

For some types of testers, the concept and requirement of deployments and locations (e.g., to run a test) is particularly difficult.

 

Users do not realize they must manually create deployments, locations, etc. (or generate the resources) in order to run a test even on the local machine. For some types of testers, they are not interested in creating resources in order to deploy/run a test.  They simply want to select the test to run, choose where to run it (e.g., locally or on a remote host) and then just run it.

 

If it is possible to remove the requirement for users to create and manage deployment resources (i.e., do that for the user under-the-covers), the user experience can be improved.

 

The task flow for executing a test locally has already been simplified as the user is no longer required to create a deployment or location file prior to execution.  However, further simplification for cases beyond localhost execution is still desirable.

 

·         Minimize the need to have users manually create deployments and locations prior to test execution; this might be done by improving the task flow (e.g., allowing a user to run a test and if a deployment/location is needed help the user create the required inputs)

·         Simplify the deployment and location wizard UIs

Test Results

Test Results involve three related aspects of user experience:

1)     Immediate feedback during the running/playback of a test: Playback/Run View.

2)     Test Reports: reports with common look & feel that are human-readable, provide the information necessary for each type of tester

3)     Test / Execution History Log

Execution Monitor / Playback View (future)

There is a need in the Test Perspective to provide immediate feedback to the user when a test is run.  Users want the ability to see whether their test is running, to see test results, to know if the test is following the appropriate path (e.g. is it running the script properly, is the test substituting data correctly?), etc.

Many users expected to see this feedback in a separate view during test execution (e.g. at the bottom of the perspective, stacked with the Recorder Control view).

It was also found during usability studies that users often did not discover the TPTP Test Log viewer.  This new view could provide links and integration between test execution feedback and test results in the form of the TPTP Test Log. 

 

To support the above, a common test execution monitor and control view could be created to provide a consistent way of displaying feedback and a way of controlling the test execution process from start to finish.  Different test types may display Test Run feedback information differently and the view could also support customization of the feedback.

 

This type of view can also be useful for remote execution use cases.

 

A TPTP feature is under consideration for the above issues:

 

·         Provide immediate feedback on test execution: https://bugs.eclipse.org/bugs/show_bug.cgi?id=74923

 

Some designs for this view have been considered and discussed during the TPTP Test User Experience Meetings.

 

A test execution monitor view can:

·         Provide a progress monitor (including integration with Eclipse jobs to display progress of test launch)

·         Display domain specific execution monitoring and control (of tests)

·         Provide monitoring and control of the entire test execution process from start to finish

 

One design solution consists of a split-pane view, with the first pane displaying a list of test executions and their status. The second pane displays details on the selected execution (from the 1st pane). From the details pane there are links to other test results (e.g., to the Test Logs, etc.)

 

 

 

Test Log / Execution History Viewer

The Test Log viewer allows users to navigate the test log.  User feedback on the Test Log Viewer suggested that usability enhancements were recommended.

 

Execution History models are often very large data models, and users may be

uninterested in much of the data contained in the model.  Instead, they often

would prefer easy navigation to the failed, inconclusive and error verdicts in

the model.

 

In TPTP 4.1, improved verdict navigation, via a toolbar is provided to allow users to navigate to the first/previous/next/last verdict event of a certain type (pass, fail, error, inconclusive) or all types which can be selected from the verdict type drop down menu on the toolbar.

 

https://bugs.eclipse.org/bugs/show_bug.cgi?id=89339

 

The toolbar allows the user to navigate to specific verdicts within the Events tree.  The following actions are available to the user via the toolbar:

 

 

 

Navigate to First button Navigate to First button – This action navigates to the first verdict event in the Events tree. The action will navigate to the verdict type specified by the Select the verdict type to navigate toolbar button drop-down menu.

 

Navigate to Previous button Navigate to Previous button – This action navigates to the previous verdict event in the Events tree. The action will navigate to the verdict type specified by the Select the verdict type to navigate toolbar button drop-down menu.

 

 Select the verdict type to navigate arrow – The button image shows the currently selected type of verdict for navigation. Click on this icon will navigate to the first verdict event of that type in the Events tree. Click on the arrow for the dropdown menu to change the type of verdict that you want to navigate through in the Events tree when used with the navigation buttons. The verdict types are:

Errorerror - Click to navigate only through the verdicts that are errors.

Fail fail - Click to navigate only through the failed verdicts.

Inconclusive inconclusive - Click to navigate only through the inconclusive verdicts.

Pass pass - Click to navigate only through the passed verdicts.

All all - Click to navigate through all verdicts.

 

Navigate to Next buttonNavigate to Next button – This action navigates to the next verdict event in the Events tree. The action will navigate to the verdict type specified by the Select the verdict type to navigate toolbar button drop-down menu.

 

Navigate to Last buttonNavigate to Last button – This action navigates to the last verdict event in the Events tree. The action will navigate to the verdict type specified by the Select the verdict type to navigate toolbar button drop-down menu.

 

Expand Selection buttonExpand Selection button – This action expands the selected part of the Events tree.

 

Collapse Selection buttonCollapse Selection button – This action collapses a selected part of the Events tree.

 

Horizontal orientation buttonHorizontal orientation button - Click to display information horizontally on the Events or Overview page.

 

Vertical orientation buttonVertical orientation button - Click to display information vertically on the Events or Overview page.

 

Future enhancements may include improving the usability of the Test Log viewer by providing customizable overview pages for specific target tester types, perhaps including a graphical summary etc.  Downstream products would be able to product useful overviews targeted for the specific user.

 

https://bugs.eclipse.org/bugs/show_bug.cgi?id=103549

 

Other usability enhancements to the Test Log viewer could include searching/sorting capabilities.

Terminology

Terminology consistency and simplicity also impacts the user experience.  During the TPTP Test User Experience meetings some terminology changes were discussed:

 

Renaming TPTP Test names to TPTP URL Test, TPTP JUnit Test, TPTP Manual Test.

·         It was agreed that surfacing the term "TPTP" was not desirable but for the time it is a necessary evil to allow differentiation between TPTP Tests and vendor test types if both must be exposed to the end-user

·         Only sample tools are to use the “TPTP” terms.  Platform items are not labeled TPTP (e.g., datapools are not labeled "TPTP datapools")

 

Some UI display names were renamed to simplify the terminology (Bugzilla 76849)

 

Some terminology confusion between the terms "Test Artifacts, "Artifact