[aspectj-dev] Changes to AspectJ test suite...

["Adrian Colyer" <adrian_colyer@xxxxxxxxxx>]

To support automatic execution of the
full suite of AspectJ tests as part of the cruise-control builds, I've
made the following changes/additions to the test harness:

* I extended testing.drivers.Harness
with an additional output formatter so that it can produce junit XML formatted
test results. To run the Harness in this way by hand (you should never
need to), you can use

   - Harness -junit ajcTests.xml
 (runs all tests)

or 

  - Harness -cruisecontrol ajcTests.xml
(as above, but skips keywords knownLimitation and purejava... guess where
this is used...)

* tests/junitModules.xml has been updated.
The default target has been changed to test-all-but-ajc, which gives you
exactly the same behaviour as you are used to seeing when you ran ant against
this file. A new target "test-all" has been added which really
does: it runs all the junit tests, then runs ajcTests.xml with and without
-emacssym. The test results are all gathered up into the html output in
tests/bin/junitXMLOutput/html/index.html (same file as before, but now
the ajcTests results are included too). This creates a one-stop shop for
running tests before you commit too: run 

  ant -f junitModules.xml test-all


prior to making a commit.

* ***Jim*** To make the EajcModuleTests
run successfully outside of the Eclipse environment, I had to extend the
building of the classpath that is passed into the ajc compiler. In place
of simply "../runtime/bin", I now test for a defined system property
"aspectjrt.path" and add it to the classpath after ../runtime/bin.
When running the tests under cruisecontrol. we define this property to
point to the newly built aspectjrt.jar.  

* ***Erik*** We run the cruisecontrol
tests on a fast machine. It causes for the example the testFileSize() test
case to fail where it passes on slower machines (as per the email exchange
you've been having with Andy). We also see a whole bunch of failing BcWeaver
tests that only fail on our build machine (see below). These seem to be
failing because the expected line number information is not present in
the generated files in all the expected places (e.g. for the testAfterReturning
test, we're looking for: 

   |        
      LDC "hello world"   (line 9)

but we see

   |        
      LDC "hello world"
 
any clues as to why this may be happening?
We can't reproduce these failures running inside eclipse....

failure	testSizeChanges	EajcModuleTests
failure	testAfterReturning	BcweaverModuleTests
failure	testAfterReturningParam	BcweaverModuleTests
failure	testAfterReturningCheckcastParam	BcweaverModuleTests
failure	testAfterReturningConversionParam	BcweaverModuleTests
failure	testAfterThrowing	BcweaverModuleTests
failure	testAfterThrowingParam	BcweaverModuleTests
failure	testAfter	BcweaverModuleTests
failure	testAfterReturningArgs	BcweaverModuleTests
failure	testFancyAfterReturningArgs	BcweaverModuleTests
failure	testThrowing	BcweaverModuleTests
failure	testLots	BcweaverModuleTests
failure	testWeave	BcweaverModuleTests
failure	testAround	BcweaverModuleTests
failure	testAroundAll	BcweaverModuleTests
failure	testAroundAndOthers	BcweaverModuleTests
failure	testAroundAllAndOthers	BcweaverModuleTests
failure	testBefore	BcweaverModuleTests
failure	testStaticTjp	BcweaverModuleTests
failure	testBefore	BcweaverModuleTests
failure	testFancyId	BcweaverModuleTests
failure	testId	BcweaverModuleTests
failure	testHello	BcweaverModuleTests
failure	testFancyHello	BcweaverModuleTests
failure	testBefore	BcweaverModuleTests
failure	testBeforeCflow	BcweaverModuleTests
failure	testBeforePerThis	BcweaverModuleTests
failure	testPublic	BcweaverModuleTests
failure	testPrintln	BcweaverModuleTests
failure	testMumble	BcweaverModuleTests
failure	testFooBar	BcweaverModuleTests
failure	testGetOut	BcweaverModuleTests
failure	testAdd	BcweaverModuleTests
failure	testNot	BcweaverModuleTests
failure	testState	BcweaverModuleTests
failure	testNoResidueArgs	BcweaverModuleTests
failure	testCflowState	BcweaverModuleTests
failure	testTraceJar	BcweaverModuleTests
failure	testStaticTjp	BcweaverModuleTests
failure	testEnclosingStaticTjp	BcweaverModuleTests
failure	testTjp	BcweaverModuleTests
failure	testAroundTjp	BcweaverModuleTests
failure	testAround2Tjp	BcweaverModuleTests

* ***Wes*** here's the changes I made
in the Harness class, which you may want to cast an eye over:

  -  I extended the TestCompleteListener
with doStartSuite and doEndSuite methods because I needed to add custom
headers and footers to the report. These are called from runMain as it
begins and ends the processing of a suite.
  - I added a new JUnitXMLLogger
that extends TestCompleteListener, and add this to the feature set when
-logJUnit option is specified. 
 - I added the -junit and -cruisecontrol
aliases that configure the junit logger, and turn off all other output
(which otherwise interferes with the junit xml report output and causes
the junit report formatting task to fail).

-- Adrian
Adrian_Colyer@xxxxxxxxxx