Documentation Driven Testing








Goulwen Le Fur
Alex Lagarde (Morel)
Software Engineers, Obeo

My Problem

Developer of the EEF project

  • A presentation framework for EMF
  • Based on SWT
  • An Eclipse Project

As all software developers, need to create tests

  • Especially functional tests

GUI testing is never an easy task

Last stumbling block:

  • The need for an Open Source solution

First steps toward the solution

First event leading to the solution

  • SWTBot project creation

Allows creation of a Bot

  • Able to handle SWT GUIs


A fluent syntax enabling users to define the scenario performed by the bot

Great! I have my tests but...

Despite of the fluent syntax

Maintenance remains very complicated

I had a dream




A Test framework

Obeo is a software toolmaker!

So let's make a tooling
above SWTBot!

1. Simplify tests development _

First approach: Code generation



The idea


Maintenance is partly ensured by


With this solution

Hurray!

But this isn't the end of the story!




The increasing tests amount


The increasing functional coverage


=> a code generator hard to maintain


We need a plan B

Second approach: Tests modeling



3 required steps:


  1. 1. Create a DSL describing functional tests

  2. 2. Implement it with the Eclipse Modeling Project

  3. 3. Create a SWTBot interpreter for these test models

What does a Functional Test Model look like ?

Modeling Bot (I)



Modeling Bot (II)



Modeling Bot (III)



A Test model









Awesome but...

How can I use that?

Using a Treeview to create tests model isn't the best way...

2. Open the tooling to functional users _

Toward an accessible tooling



This kind of model is well suited for a textual syntax



Let's use the project to create a textual syntax for our test DSL

A Grammar for textual syntax


3. Understand intents behind tests _

Mylyn Intent

Keeping doc synchronized with Software is a burden

So you keep breaking specifications without realizing it




Clear lack of tooling to react to changes

That is exactly what Intent is about

One Intent Document mixing both

- Pure documentation zones

- Formal links zones

Chapter Standard EEF SWT Toolkit {

The SWT is composed of 7 main widgets:
* Text
* TextArea
* Checkbox
* Radio
* Combo
* Group
* Hbox

Section The Text widget {

The text widget allows users to edit _EAttributes_ by entering a String value.
The standard use case is to edit an _EAttribute_ as an *EString*.
For instance, in this scenario, an user is able do defines the firstname of conference participant using a text widget:
@M
@ref "org.eclipse.emf.eef.scenarios/models/setAttributeText.modelingbot"
M@
}
}

One Intent Document mixing both

- Pure documentation zones
Explanations in Natural language
Use the Textile syntax

Chapter Standard EEF SWT Toolkit {

The SWT is composed of 7 main widgets:
* Text
* TextArea
* Checkbox
* Radio
* Combo
* Group
* Hbox

Section The Text widget {

The text widget allows users to edit _EAttributes_ by entering a String value.
The standard use case is to edit an _EAttribute_ as an *EString*.
For instance, in this scenario, an user is able do defines the firstname of conference participant using a text widget:
@M
@ref "org.eclipse.emf.eef.scenarios/models/setAttributeText.modelingbot"
M@
}
}

One Intent Document mixing both

- Formal links zones
Reference to technical artifacts (Java, models, Manifests...)
Extensible behavior (display, hyperlinks, validation...)

Chapter Standard EEF SWT Toolkit {

The SWT is composed of 7 main widgets:
* Text
* TextArea
* Checkbox
* Radio
* Combo
* Group
* Hbox

Section The Text widget {

The text widget allows users to edit _EAttributes_ by entering a String value.
The standard use case is to edit an _EAttribute_ as an *EString*.
For instance, in this scenario, an user is able do defines the firstname of conference participant using a text widget:
@M
@ref "org.eclipse.emf.eef.scenarios/models/setAttributeText.modelingbot"
M@
}
}

4. Lessons learned

Lessons learned

1. SWTBot is cool, but tests can be hard to maintain & understand

2. Capturing scenarios through models instead of code makes test easier to maintain

3. Providing a textual syntax to edit those models improves user experience

4. Using Intent to tie doc, models & code makes sure intention is never lost

Improvements & Issues to tackle (I)

Conceptual concerns


1. Apply the Doc-Driven Testing approach on more use cases

2. Improve checking definition

Improvements & Issues to tackle (II)

Design concerns


1. Extensibility: allow to provide new Actions sets

2. Extensibility: what about non SWTBot-based tests? (Jubula, Q7...)

Improvements & Issues to tackle (III)

Tooling concerns: textual syntax

1. Need for an extensible grammar is it possible with Xtext ?

Tooling concerns: Mylyn Intent bridge

1. JUnit tests should be automatically generated (instead of using quick-fix)
2. Generated tests should be hidden: launch TestSuite directly from document
3. Improve junit failures: directly open related documentation

Thank You!