[ecf-dev] Automated test execution for ECF build based on autotestsuite

[Markus Alexander Kuppe <ecf-dev_eclipse.org@xxxxxxxxxxx>]

Hi Ted,

automated testing was one of yesterday's topics the ECF con call. Since 
I've integrated automated test execution into Versants build process 
recently, I'd like to share how it's done. You have probably an idea 
where it's best integrated into the ECF build.

I first looked into the Eclipse testing framework [1]. It's JUnit3 based 
and spawns of an Eclipse instance for test execution. It supports UI 
tests, headless tests as well as performance tests. Test execution is 
handled by Ant. It spawns an Eclipse instance and executes test suites. 
I find this way too complex and cumbersome. A developer simply wants to 
write a unit test. One doesn't want write/maintain xml that defines 
execution order. Especially since execution order shouldn't matter for 
unit tests.

Thus I continued to look for something that doesn't require additional 
xml next to the plain test cases and I found pluginbuilder autotestsuite 
(EPL) [2]. It consists of a bunch of OSGi bundles which come with an 
Equinox application. Basically one drops the unit tests + autotestsuite 
bundles into the Eclipse installation under test and starts the 
autotestsuite application (-application 
org.pluginbuilder.autotestsuite.application.AutoTestApplication). 
Autotestsuite then takes care of finding test cases by searching bundles 
and bundle classpaths based on regular expression. All test cases that 
are found are passed to a junit.framework.TestSuite and executed. Test 
results are collected by the junit xml formatter. Autotestsuite doesn't 
require lists of test cases nor hard coded junit.framework.TestSuites 
and IMO that's the beauty of it.

So how could the integration of autotestsuite in ECF look like?
1) Build test bundles. I'd suggest to create test features and build and 
package them independently. Like it is done for platform [3].
2) Deploy Eclipse SDK + ECF + ECF tests + autotestsuite to some temp 
folder on the build box, either by unzipping everything into features/ 
and plugins/ (<3.4), using the links/ folder (that's how Versant does 
it) or by using the p2 director.
3) Launch the autotestsuite application.
4) Collect the JUnit xml output and post process it (html reports, email 
notification...)

Cheers
Markus

[1] 
http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.test/testframework.html
[2] http://www.pluginbuilder.org/documentation/run-tests/
[3] 
http://download.eclipse.org/eclipse/downloads/drops/R-3.4-200806172000/download.php?dropFile=eclipse-Automated-Tests-3.4.zip