Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages

Here is what I would recommend…

 

Setup one test job that is a small subset of your tests. Have it triggered by completion of the build. That will give you a quick smoke test to flush out any obvious issues.

 

Setup another test job that is all of your tests. Trigger it manually when you want to “promote” a build at whatever the appropriate frequency is for your project. If the entire test suite is too large, consider creating several jobs. Alternatively, multi-configuration jobs (it’s a type of a job – selectable at creation time only) is an extremely powerful construct. It allows you to configure a matrix with any number of axis that you define. You can have an axis for different platforms you support, an axis for your components, etc. Each cell in that matrix runs separately, but you provide only one job definition. This means your tests can run in parallel if executors are available and some cells can fail or be aborted without affecting the rest of the matrix. Hudson tracks completion of the matrix and you get a nice overview at the end. If you want to run multiple test jobs, Hudson has a merge feature that allows you to bring it all together for the purpose of promotion, etc. In my experience, the approach of breaking up a large test set using multiple jobs is more difficult to configure and maintain than using one multi-configuration job.

 

To configure a test job, you check “this build is parameterized” then add a “run parameter”. Specify your build job for the “project”. This way, when the test job is invoked, an environment variable will be set with a URL to the build run that it should use. You can use that URL to download build artifacts that you can use to run tests.

 

You may be tempted to setup the long test job to run on a timer (nightly, weekly, etc.). That almost works with existing Hudson facilities. The gotcha is that when a timer kicks of a job parameterized with a run parameter, it grabs the latest run parameter without regard for whether that run has completed (it could be running still) or failed. There is an outstanding enhancement request on Hudson to add “Good Run Parameter” option. In the meantime, I would recommend limiting yourself to kicking off test jobs after a build job or triggering them manually.

 

Hope that is helpful.

 

- Konstantin

 

 

From: cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] On Behalf Of Miles Parker
Sent: Thursday, September 30, 2010 2:55 PM
To: Cross project issues
Subject: Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages

 

 

On Sep 30, 2010, at 1:35 PM, Konstantin Komissarchik wrote:

 

For running tests, you need very little I/O power and good amount of CPU power. Since tests are more likely to fubar a node and thereby cause interference to other jobs, I recommend creating a node/VM per-core as long as memory allows and using only one Hudson executor per node/VM.

 

This brings up the issue of UI testing. It makes a lot of sense to separate those out. I've got some SWTBot tests that could conceivably take up a lot of time..in fact I have't been putting too much thought into how to get them running hosted just because of potential resource use. Any special recommendations on good strategies for UI?

 

One other related issue. If a test fails, we probably should't be promoting I'd think -- not even nightly. Has anyone given thought to how this might be accomplished?

 

-Miles


Back to the top