Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [cdt-dev] CDT regression tests
Doug,
 
Thanks for the pointers on the scripts.
 
What do the nightly scripts do for a DISPLAY? I have been trying to use Xvfb to create a virtual display and this seems to mostly work but I have come across one test that seems to hang with the virtual display but not with a real display.
 
With regards to the number of tests being run. When I get into eclipse and ask it to "Run all tests in the selected project, package or source folder" and select org.eclipse.cdt.core.tests, it shows the following numbers in the JUnit window(after a partial run): Runs:  913/12276. I am taking this to mean there are over 12,000 tests that it is going to run.
 
I exported the test run history (showing 12276 test cases) and the results file for the nightly tests showing 4577 tests. I also did some filtering of the files to try to isolate the test names. The nightly tests run 5 packages so in some ways I would expect the nightly tests to be higher, not smaller. In fact, there are tests that only show up in the nightly tests and they seem to be in other packages like the org.eclipse.cdt.ui.tests. But there are a lot of tests that have duplicate entries. The number of unique names is higher in the nightly tests. I am just trying to figure out what the numbers mean.
 
It seems like either the interactive tests are taking longer than they need to by running the same test mulitple times or the nightly tests are not running some tests that they should be.Either way it doesn't seem like a good combination.
 
 
 
 

John Pruitt
630-971-6430

 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Schaefer, Doug
Sent: Thursday, October 09, 2008 1:51 PM
To: CDT General developers list.
Subject: RE: [cdt-dev] CDT regression tests

1. The nighly build runs the regression tests. The releng scripts drive it. They are in the org.eclipse.cdt.releng project under org.eclipse.cdt/all. They run on the build.eclipse.org server which is an 8P IBM ppc64 box running Linux.
 
2. You probably aren't running the same tests. I've never run a suite that has 11,000 tests in it, but there are some dormant suites in our test plug-ins.
 
Cheers,
Doug.
 
 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Pruitt, John
Sent: Thursday, October 09, 2008 2:01 PM
To: CDT General developers list.
Subject: [cdt-dev] CDT regression tests

I would like to try again with getting information about the CDT nightly builds and regression tests.
 
I have two main questions:
 
1. Where are the scripts and commands used to do the nightly CDT builds and run the nightly regression tests? What host platform are these run on?
 
2. In 3.3, when the regression tests are run interactively, there are over 11,000 tests. In the nightly tests, there are only about 4,000. Why is there such a difference?
 
Thanks.
 

John Pruitt
630-971-6430

 

From: cdt-dev-bounces@xxxxxxxxxxx [mailto:cdt-dev-bounces@xxxxxxxxxxx] On Behalf Of Pruitt, John
Sent: Thursday, October 02, 2008 10:11 AM
To: cdt-dev@xxxxxxxxxxx
Subject: [cdt-dev] CDT nightly builds

Where can I find information about how the CDT nightly builds are done? Are these script based? What java compiler is currently being used? How are the tests run?
 
I have tried looking in CVS and have found some old files that are out-of-date. Where are the current files?
 
I am currently working with Europa and downloaded the eclipse-Automated-Tests-3.3.2.zip. It contains a runtests script for linux that doesn't seem to have any chance of working so I suspect this zip file is outdated or is only used on Windows systems. Nonetheless, I have modified the runtests script and some of the other files and have something that runs some CDT tests but I have a couple of problems.
 
Sometimes the tests seem to hang. Particularly in a call to a addr2line and c++filt programs. If I kill those programs, then the tests proceed.
 
The results are slightly different but I am not using exactly the same versions as shown in the nightly builds so I am not comparing apples to apples. I do get some many failures and just a couple of errors. I assume at least one of the errors is due to me killing the addr2line and c++filt programs.
 
The number of tests in my batch run matches well with the official nightly runs but this number seems considerably different from running the tests within Eclipse. The number of tests in the batch runs is 4021 while the nightly build shows 3931. I am getting one additional package for org.eclipse.cdt.managedbuilder.tests.suite (93 additional tests) and the number of tests I show for org.eclipse.cdt.core.suite is slightly less (3247 versus 3250).
 
When run within eclipse and picking org.eclipse.cdt.core.tests, it says it is running 11683 tests. Considerably more than the 3247 shown in the batch run.
 
Is it possible to run all the tests in batch mode?
 
Where are the scripts and files used to build and test CDT on a nightly basis?
 
Thanks.
 
 
 

John Pruitt
630-971-6430

 

Back to the top