| [cross-project-issues-dev] RE : cbi-papyrus-0.7-nightly build is taking ages |
Hi all, I'm the guy in charge of the Papyrus releng. All apologizes for the 7 or 8 hours builds, but this happened when the hudson server was buggy (version 1.378 perhaps?!). Usually, Papyrus job only takes a couple of minutes (between 15 and 20 min). I do not understand really well why the 2 builds took so much time (build id 1104 and 1105). The new ones, 1106 and 1107, seems to have nice execution time, I have not changed anything from my side. The Papyrus job is a very simple one, it only compiles and builds the p2 repository. It does not do any junit test, it does not also have xvnc option activated. Whatever, to be more 'server-friendly', I've changed the timer for the job: it will build fewer times in the day. As a remark: I do not know why the hudson web interface does not propose anymore a list of available build nodes (main, hudson slaves, etc.), but only a text field. Is this the effect of newer versions or some modified configuration in hudson? Thanks for Denis and other guys' job, you must now have white hair and no more nails I guess... Rémi -------- Message d'origine-------- De: cross-project-issues-dev-bounces@xxxxxxxxxxx de la part de Denis Roy Date: jeu. 30/09/2010 22:44 À: Cross project issues Objet : Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages Thanks for this. Separating build and test seems to be what Apache is doing as well. They seem to have a fairly strict policy about which nodes are used for which types of job. We'll have to look further into doing something similar. For now, it seems that with 1.376 we've returned to some form of normalcy. Thanks for everyone's help here. My sanity was taking a beating :) Denis On 09/30/2010 04:35 PM, Konstantin Komissarchik wrote: > > Just a few random tips for Hudson stability based on our experience... > > Build and test should be split into separate jobs. Tests should never > be run on the same nodes as builds. Tests (especially those that are > UI-dependent or launch external processes) are a lot more fragile than > the build. They have higher probability of fubaring a node. It can be > very difficult to tell what fubared a node if you are running many > executors per node. Test and build jobs also place very different > demands on the hardware, which implies that you'd like to configure > the node VMs differently. > > For running builds, you need as much I/O power as possible and > relatively little CPU power. It works pretty well to create beefier > build nodes and crank up the number of executors. In terms of figuring > out the number of executors to use, I recommend figuring out how many > concurrent builds the I/O system can handle without more than 50% > slowdown (as compared to running only one build process). Then assign > one core for every two executors. The remaining cores and RAM can be > used to run test nodes as those will barely touch the disk. > > For running tests, you need very little I/O power and good amount of > CPU power. Since tests are more likely to fubar a node and thereby > cause interference to other jobs, I recommend creating a node/VM > per-core as long as memory allows and using only one Hudson executor > per node/VM. > > - Konstantin > > *From:* cross-project-issues-dev-bounces@xxxxxxxxxxx > [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] *On Behalf Of > *David Carver > *Sent:* Thursday, September 30, 2010 1:03 PM > *To:* cross-project-issues-dev@xxxxxxxxxxx; Denis Roy > *Subject:* Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly > build is taking ages > > The more important question is why does this particular job take 7 > plus hours to run? > > Dave > > On 09/30/2010 12:50 PM, Denis Roy wrote: > > Trip, in theory that would be a good idea. But many builds are > launched from an SCM trigger, so the load naturally follows the work days. > > Denis > > > > On 09/30/2010 12:32 PM, Trip Gilman wrote: > > Do you think it might be helpful to set up some type of calendar that > at least tracks when people have their various builds scheduled to > run? This might cut down on some of the rush hour effect to the servers. > > Trip > > > On 9/30/10 9:58 AM, "Denis Roy" <denis.roy@xxxxxxxxxxx> wrote: > > Yesterday the slaves has eight executors each, and we were having > SSH issues. So I followed Apache's lead and reduced the executors per > slave. > > Even when only one job is running, it seems to get spaced out and no > longer works, even if the slave and master are still in harmony (pun > not intended). > > At this point I'm grasping at straws. > > > > > On 09/30/2010 10:43 AM, David Carver wrote: > > Current backlog on Hudson doesn't seem to be Master, but the > Hudson-Slave1 and Hudson-Slave2 servers. The number of executors on > these machines is way to low for the number of jobs we have. I > noticed that the number of executors were greatly reduced yesterday. > > Dave > > On 09/30/2010 07:33 AM, Denis Roy wrote: > > On 09/30/2010 10:25 AM, David M Williams wrote: > > But, we can't all run all jobs on master. > > > Well, why not. Since the master is virtualized, why don't we goose > it up and crank up the executor threads and see how it goes? Maybe > then we'll all be able to get some work done? > > Is anyone game? > > > > > > > > > From: Matthias Sohn <matthias.sohn@xxxxxxxxxxxxxx> > <mailto:matthias.sohn@xxxxxxxxxxxxxx> > To: Cross project issues <cross-project-issues-dev@xxxxxxxxxxx> > <mailto:cross-project-issues-dev@xxxxxxxxxxx> > Date: 09/30/2010 10:11 AM > Subject: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is > taking ages > Sent by: cross-project-issues-dev-bounces@xxxxxxxxxxx > > ------------------------------------------------------------------------ > > > > > This build seems to take ages, according to Hudson it started _Sep 30, > 2010 3:11:46 AM_ > <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/> > and now we have Sep 30, 2010 10:08:33 AM, anything wrong here or is > this some massive > compile or test job ? > > _https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console_ > <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console> > > > I am just wondering since I am waiting since hours for my few minutes > egit build job > to grab a free worker thread. > > -- > Matthias_______________________________________________ > > > > _______________________________________________ > cross-project-issues-dev mailing list > cross-project-issues-dev@xxxxxxxxxxx <mailto:cross-project-issues-dev@xxxxxxxxxxx> > https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev >
<<winmail.dat>>