Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [cdt-dev] junit tests

Related to this, and also as a part of the greater discussion of "what to do about the build system"...

The managed build core tests are extremely brittle, and a nightmare to maintain. Essentially what they do is create a project, then build it and compare the resulting generated makefiles to a set of benchmark makefiles contained with the test.

This doesn't work well for the following reasons:

1. As soon as you change *anything* that causes the generated makefiles to be different, no matter how slight, you have to go back and manually update thirty-someodd tests, by extracting the sample projects, building them, and putting the updated benchmark files into the appropriate zip file. This is a very time consuming process, and in the past this has acted as a barrier to making changes to MBS because either you spend days updating the testcases, or the tests become broken for an extended period of time.

2. The tests don't actually check to see if the build does what it's supposed to do, it only cares if the makefiles match. You might actually end up with a build that goes haywire and doesn't build anything, but so long as the makefile comes out as expected, the test passes.

3. The test projects are geared to particular toolchains. If we ever want to start actually checking if the build actually builds anything, we will have to make the tests flexible enough to build with the right toolchain for the platform on which the test is running.

4. There are no tests for the internal builder, which is the default now. What is more, you can't test the internal builder via the above method because it doesn't generate any makefiles to benchmark against. This means our primary use case for build isn't being exercised at all.

On top of all this, the tests behave inconsistently due to the concurrency issues in the build system. It is pretty random as to whether any particular test passes or not, and I find I have to rerun the tests at least once (if not several) times to either get a clean run, or to at least have seen different sets of failing tests in each run that indicate that in general the tests pass.

I am curious as to the opinons of the other committers, but personally I think that because of all the flaws above, these tests are not very useful anymore. Maybe we should stop running them as a part of the build.


Chris Recoskie
Team Lead, IBM CDT Team
IBM Toronto

Inactive hide details for "Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>"Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>

          "Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>
          Sent by: cdt-dev-bounces@xxxxxxxxxxx

          04/24/2008 11:28 AM

          Please respond to
          "CDT General developers list." <cdt-dev@xxxxxxxxxxx>


"CDT General developers list." <cdt-dev@xxxxxxxxxxx>



RE: [cdt-dev] junit tests

The only JUnits we run are the ones for the nightly builds. Everything
starts at the test.xml file in the org.eclipse.cdt.testing plug-in,
which lists the test suites that get run. Adding JUnits is as simple as
adding tests to those suites or adding new suites.

Never thought of doing a lastest link, might be a good idea. But
generally, I'm always looking to see when the build was done so I
wouldn't use a latest link.

Feel free to update the wiki with what you find. The people who set up
the test suite are all long gone (originally done by the testers at QNX
years ago).

BTW, we have a lot of test failures, especially in managedbuild. We
should be looking at why those are failing.


-----Original Message-----
From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx]
On Behalf Of Elena Laskavaia
Sent: Thursday, April 24, 2008 10:00 AM
To: CDT General developers list.
Subject: [cdt-dev] junit tests

I was search wiki page for any documentation regarding our junits, I
could not find anything...
I am looking for description on test environment, schedule of automated
tests and latest test results.
Along with documentation of creating new test, adding testing packages
into build etc.
Do we have anything like this?

I kind of can found some results following CDT nightly builds but there
is no link to "latest" so I cannot even bookmark it.
cdt-dev mailing list
cdt-dev mailing list

GIF image

GIF image

GIF image

Back to the top