| RE: [cdt-dev] junit tests |
Related to this, and also as a part of the greater discussion of "what to do about the build system"...
The managed build core tests are extremely brittle, and a nightmare to maintain. Essentially what they do is create a project, then build it and compare the resulting generated makefiles to a set of benchmark makefiles contained with the test.
This doesn't work well for the following reasons:
1. As soon as you change *anything* that causes the generated makefiles to be different, no matter how slight, you have to go back and manually update thirty-someodd tests, by extracting the sample projects, building them, and putting the updated benchmark files into the appropriate zip file. This is a very time consuming process, and in the past this has acted as a barrier to making changes to MBS because either you spend days updating the testcases, or the tests become broken for an extended period of time.
2. The tests don't actually check to see if the build does what it's supposed to do, it only cares if the makefiles match. You might actually end up with a build that goes haywire and doesn't build anything, but so long as the makefile comes out as expected, the test passes.
3. The test projects are geared to particular toolchains. If we ever want to start actually checking if the build actually builds anything, we will have to make the tests flexible enough to build with the right toolchain for the platform on which the test is running.
4. There are no tests for the internal builder, which is the default now. What is more, you can't test the internal builder via the above method because it doesn't generate any makefiles to benchmark against. This means our primary use case for build isn't being exercised at all.
On top of all this, the tests behave inconsistently due to the concurrency issues in the build system. It is pretty random as to whether any particular test passes or not, and I find I have to rerun the tests at least once (if not several) times to either get a clean run, or to at least have seen different sets of failing tests in each run that indicate that in general the tests pass.
I am curious as to the opinons of the other committers, but personally I think that because of all the flaws above, these tests are not very useful anymore. Maybe we should stop running them as a part of the build.
===========================
Chris Recoskie
Team Lead, IBM CDT Team
IBM Toronto
http://www.eclipse.org/cdt
"Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>
Sent by: cdt-dev-bounces@xxxxxxxxxxx 04/24/2008 11:28 AM
|
|
