Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [cdt-dev] junit tests

If the test was wrong why didn't you remove it rather than just commenting it out? Or why don't you add the proper assertions if you feel they are of value?
Sorry to troll, I am just unsure as to why you piped up about this molehill when Chris actually seemed to have a valid mountain (completely unmaintainable tests) earlier in this thread.

--
JohnC

Inactive hide details for Mike Kucera/Toronto/IBM@IBMCAMike Kucera/Toronto/IBM@IBMCA


          Mike Kucera/Toronto/IBM@IBMCA
          Sent by: cdt-dev-bounces@xxxxxxxxxxx

          24/04/2008 05:46 PM

          Please respond to
          "CDT General developers list." <cdt-dev@xxxxxxxxxxx>

To

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>

cc

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>, cdt-dev-bounces@xxxxxxxxxxx

Subject

RE: [cdt-dev] junit tests

How would anyone reliably find problems like this? If a test passed but the AST was actually slightly wrong there would be nothing to alert anyone of the problem. I guess in practice it hasn't been an issue though.

But I've actually noticed it a couple of times. An example is AST2CPPSpecTest.test10_4s2b(), which I have since commented out. In the spec this is actually an example of ill-formed code. Its a pure virtual function that has a body, which isn't allowed. But the test passes against the DOM parser, it parses as an initializer _expression_. Its not a serious problem by any means, but it does show that the spec tests aren't completely thorough.

Frankly, these tests seem like a quick copy-paste job. They don't test that the resulting AST is actually correct. Its possible that some bugs would have been detected sooner if these tests had been stronger.


Mike Kucera
Software Developer
IBM CDT Team, Toronto
mkucera@xxxxxxxxxx


Inactive hide details for John Camelon---04/24/2008 04:17:31 PM---I am curious how many regressions you found where the syntax John Camelon---04/24/2008 04:17:31 PM---I am curious how many regressions you found where the syntax and semantics of the test worked out without issue but that AST wa
                  John Camelon/Ottawa/IBM@IBMCA
                  Sent by: cdt-dev-bounces@xxxxxxxxxxx

                  04/24/2008 04:16 PM

Please respond to
"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
To

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
cc
Subject

RE: [cdt-dev] junit tests

I am curious how many regressions you found where the syntax and semantics of the test worked out without issue but that AST was incorrect.
--
JohnC

Inactive hide details for Mike Kucera/Toronto/IBM@IBMCAMike Kucera/Toronto/IBM@IBMCA
                                  Mike Kucera/Toronto/IBM@IBMCA
                                  Sent by: cdt-dev-bounces@xxxxxxxxxxx

                                  24/04/2008 02:52 PM


Please respond to
"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
To

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
cc

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>, cdt-dev-bounces@xxxxxxxxxxx
Subject

RE: [cdt-dev] junit tests

Many of the older parser tests also have the unfortunate property of not actually testing if things worked correctly. There are a bunch of tests that parse a string then just check for syntax errors and problem bindings in the resulting AST. So the parser could return complete nonsense but as long as there are no problem nodes the test will pass. In fact you could write a parser that trivially passes all these tests by just having it always return an empty AST.

Mike Kucera
Software Developer
IBM CDT Team, Toronto
mkucera@xxxxxxxxxx


Inactive hide details for Chris Recoskie---04/24/2008 12:10:04 PM---Related to this, and also as a part of the greater discussiChris Recoskie---04/24/2008 12:10:04 PM---Related to this, and also as a part of the greater discussion of "what to do about the build system"...
                                                                  Chris Recoskie/Toronto/IBM@IBMCA
                                                                  Sent by: cdt-dev-bounces@xxxxxxxxxxx

                                                                  04/24/2008 12:02 PM


Please respond to
"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
To

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
cc
Subject

RE: [cdt-dev] junit tests

Related to this, and also as a part of the greater discussion of "what to do about the build system"...

The managed build core tests are extremely brittle, and a nightmare to maintain. Essentially what they do is create a project, then build it and compare the resulting generated makefiles to a set of benchmark makefiles contained with the test.

This doesn't work well for the following reasons:

1. As soon as you change *anything* that causes the generated makefiles to be different, no matter how slight, you have to go back and manually update thirty-someodd tests, by extracting the sample projects, building them, and putting the updated benchmark files into the appropriate zip file. This is a very time consuming process, and in the past this has acted as a barrier to making changes to MBS because either you spend days updating the testcases, or the tests become broken for an extended period of time.

2. The tests don't actually check to see if the build does what it's supposed to do, it only cares if the makefiles match. You might actually end up with a build that goes haywire and doesn't build anything, but so long as the makefile comes out as expected, the test passes.

3. The test projects are geared to particular toolchains. If we ever want to start actually checking if the build actually builds anything, we will have to make the tests flexible enough to build with the right toolchain for the platform on which the test is running.

4. There are no tests for the internal builder, which is the default now. What is more, you can't test the internal builder via the above method because it doesn't generate any makefiles to benchmark against. This means our primary use case for build isn't being exercised at all.

On top of all this, the tests behave inconsistently due to the concurrency issues in the build system. It is pretty random as to whether any particular test passes or not, and I find I have to rerun the tests at least once (if not several) times to either get a clean run, or to at least have seen different sets of failing tests in each run that indicate that in general the tests pass.

I am curious as to the opinons of the other committers, but personally I think that because of all the flaws above, these tests are not very useful anymore. Maybe we should stop running them as a part of the build.


===========================

Chris Recoskie
Team Lead, IBM CDT Team
IBM Toronto

http://www.eclipse.org/cdt

Inactive hide details for "Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>"Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>
                                                                                                                                  "Schaefer, Doug" <Doug.Schaefer@xxxxxxxxxxxxx>
                                                                                                                                  Sent by: cdt-dev-bounces@xxxxxxxxxxx

                                                                                                                                  04/24/2008 11:28 AM


Please respond to
"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
To

"CDT General developers list." <cdt-dev@xxxxxxxxxxx>
cc
Subject

RE: [cdt-dev] junit tests

The only JUnits we run are the ones for the nightly builds. Everything
starts at the test.xml file in the org.eclipse.cdt.testing plug-in,
which lists the test suites that get run. Adding JUnits is as simple as
adding tests to those suites or adding new suites.

Never thought of doing a lastest link, might be a good idea. But
generally, I'm always looking to see when the build was done so I
wouldn't use a latest link.

Feel free to update the wiki with what you find. The people who set up
the test suite are all long gone (originally done by the testers at QNX
years ago).

BTW, we have a lot of test failures, especially in managedbuild. We
should be looking at why those are failing.

Cheers,
Doug.

-----Original Message-----
From: cdt-dev-bounces@xxxxxxxxxxx [
mailto:cdt-dev-bounces@xxxxxxxxxxx]
On Behalf Of Elena Laskavaia
Sent: Thursday, April 24, 2008 10:00 AM
To: CDT General developers list.
Subject: [cdt-dev] junit tests

I was search wiki page for any documentation regarding our junits, I
could not find anything...
I am looking for description on test environment, schedule of automated
tests and latest test results.
Along with documentation of creating new test, adding testing packages
into build etc.
Do we have anything like this?

I kind of can found some results following CDT nightly builds but there
is no link to "latest" so I cannot even bookmark it.
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx

https://dev.eclipse.org/mailman/listinfo/cdt-dev
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx

https://dev.eclipse.org/mailman/listinfo/cdt-dev
(See attached file: pic08097.gif)
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx

https://dev.eclipse.org/mailman/listinfo/cdt-dev
(See attached file: pic08097.gif)
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx

https://dev.eclipse.org/mailman/listinfo/cdt-dev
(See attached file: pic31958.gif)(See attached file: pic08097.gif)
_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx

https://dev.eclipse.org/mailman/listinfo/cdt-dev
(See attached file: pic31958.gif)(See attached file: pic08097.gif)_______________________________________________
cdt-dev mailing list
cdt-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cdt-dev

GIF image

GIF image

GIF image

GIF image

GIF image

GIF image

Attachment: pic31958.gif
Description: GIF image

Attachment: pic08097.gif
Description: GIF image


Back to the top