Just a few random tips for Hudson stability based
on our
experience…
Build and test should be split into separate
jobs. Tests should
never be run on the same nodes as builds. Tests (especially
those that are
UI-dependent or launch external processes) are a lot more
fragile than the
build. They have higher probability of fubaring a node. It
can be very
difficult to tell what fubared a node if you are running
many executors per
node. Test and build jobs also place very different demands
on the hardware,
which implies that you’d like to configure the node VMs
differently.
For running builds, you need as much I/O power as
possible and
relatively little CPU power. It works pretty well to create
beefier build nodes
and crank up the number of executors. In terms of figuring
out the number of
executors to use, I recommend figuring out how many
concurrent builds the I/O
system can handle without more than 50% slowdown (as
compared to running only
one build process). Then assign one core for every two
executors. The remaining
cores and RAM can be used to run test nodes as those will
barely touch the
disk.
For running tests, you need very little I/O power
and good amount
of CPU power. Since tests are more likely to fubar a node
and thereby cause
interference to other jobs, I recommend creating a node/VM
per-core as long as
memory allows and using only one Hudson executor per
node/VM.
- Konstantin
The more important question is why does
this particular job
take 7 plus hours to run?
Dave
On 09/30/2010 12:50 PM, Denis Roy wrote:
Trip, in theory that would be a good idea.
But many
builds are launched from an SCM trigger, so the load naturally
follows the work
days.
Denis
On 09/30/2010 12:32 PM, Trip Gilman wrote:
Do you think it
might be helpful to set up
some type of calendar that at least tracks when people have
their various
builds scheduled to run? This might cut down on some of the
rush hour
effect to the servers.
Trip
On 9/30/10 9:58 AM, "Denis Roy" <denis.roy@xxxxxxxxxxx>
wrote:
Yesterday
the slaves has eight executors each, and we were having SSH
issues. So I followed
Apache's lead and reduced the executors per slave.
Even when only one job is running, it seems to get spaced
out and no
longer works, even if the slave and master are still in
harmony (pun not
intended).
At this point I'm grasping at straws.
On 09/30/2010 10:43 AM, David Carver wrote:
Current
backlog on Hudson doesn't seem to be Master, but the
Hudson-Slave1 and
Hudson-Slave2 servers. The number of executors on these
machines is way to low for the number of jobs we have. I
noticed that the
number of executors were greatly reduced yesterday.
Dave
On 09/30/2010 07:33 AM, Denis Roy wrote:
On
09/30/2010 10:25 AM, David M Williams wrote:
But,
we can't all run all jobs on master.
Well, why not. Since the master is virtualized, why don't
we goose
it up and crank up the executor threads and see how it goes?
Maybe then
we'll all be able to get some work done?
Is anyone game?
From: Matthias
Sohn <matthias.sohn@xxxxxxxxxxxxxx>
<mailto:matthias.sohn@xxxxxxxxxxxxxx>
To: Cross
project
issues <cross-project-issues-dev@xxxxxxxxxxx>
<mailto:cross-project-issues-dev@xxxxxxxxxxx>
Date: 09/30/2010
10:11 AM
Subject: [cross-project-issues-dev]
cbi-papyrus-0.7-nightly
build is taking
ages
Sent by: cross-project-issues-dev-bounces@xxxxxxxxxxx
This
build seems to take ages, according to Hudson it started Sep
30, 2010 3:11:46 AM
<https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/>
and
now we have Sep 30, 2010 10:08:33 AM, anything wrong
here or is this some massive
compile
or test job ?
https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console>
I
am just wondering since I am waiting since hours for my few
minutes egit build
job
to
grab a free worker thread.
--
Matthias_______________________________________________
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev