Hudson: Slow Parsing POMs or low performance for production [message #1165775] |
Fri, 01 November 2013 14:31 |
Sergey Nikitin Messages: 3 Registered: November 2013 |
Junior Member |
|
|
Hi!
We have performance problem using Hudson as a central continuous integration server for 200+ projects. Do you have any ideas how to improve performance?
Problem: we using Hudson as a ci-server for 200+ build projects (5-10 simultaneous jobs). The hardware is quite adequate: 16 CPUs + 100 GB RAM, and it works quite fast, except two points:
- slow project configuration saving (when you create it or change ): 4+ minutes;
- slow Parsing POM step during build process: 10+ minutes.
We have done some tests on the same server: if you start Hudson with several build projects it works very fast, but when you increase number of build projects (even if you don't start them) the project configuration saving and Parsing POM step becomes slower and slower....
We have found cause of the problem in Hudson dependencies, look at the mails below.
But since 2011, when the problem was found, nothing changed. We don't have an option in Hudson to turn off dependences calculation or do anything else to improve performance.
Any ideas?
===================== mails from web ===============
From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: [Hudson] "Parsing POMs" can be very slow
Date Sun, 23 Jan 2011 18:06:11 GMT
Benson,
Right now, when there is a change in Axiom, Hudson will also trigger
builds of Neethi, Woden, Axis2, Sandesha2 and Rampart. This is what we
want because it allows us to ensure non-regression in downstream
projects. This relies on Hudson calculating the dependencies
automatically. Disabling this would mean that we have to find an
alternative configuration to achieve the same. So, yes, interjob
dependencies are needed, and disabling them should only be a last
resort. I like Niklas' proposal to run the recalculation of these
dependencies as a cron job.
Andreas
On Sun, Jan 23, 2011 at 18:21, Benson Margulies <bimargulies@gmail.com> wrote:
> Niklas,
>
> I'm a very rusty hudson dev. So I'm reduced to the following question:
> do we really need interjob hudson dependencies at Apache (the
> downstream/upstream business). If all the jobs are set to turn this
> off, does it still calculate?
>
> Another view of this would be to look to splitting up hudson, and
> maybe fixing this might be less work than that.
>
> --benson
>
>
> On Sun, Jan 23, 2011 at 11:15 AM, Niklas Gustavsson
> <niklas@protocol7.com> wrote:
>> Hi
>>
>> Some of you might have noticed that Hudson can take a long time on
>> Maven builds in the step called "Parsing POMs". In fact, some builds
>> will time-out due to this step taking 15 minutes or more. Thing is, as
>> part of this step, Hudson will do the problematic rebuild of its
>> dependency graph. This is the same problem causing saving project
>> configuration taking forever. We have previously reported this here:
>> issues.hudson-ci.org/browse/HUDSON-7535
>>
>> I've now added a comment on the issue describing the additional
>> problem of builds taking a long time.
>>
>> I think we got some Hudson devs around here, anyone knows a workaround
>> for this problem? The problem at hand is line 748 in
>> MavenModuleSetBuild.java. I'm considering patching Hudson to remove
>> these calls on saving projects/starting builds and instead rebuilding
>> the graph as a cron job every hour or so. Would that work (given the
>> dependency graph will obviously be broken for some short times)?
>>
>> /niklas
[Updated on: Wed, 06 November 2013 07:35] Report message to a moderator
|
|
|
|
|
|
Re: Hudson: Slow Parsing POMs or low performance for production [message #1220064 is a reply to message #1202130] |
Mon, 09 December 2013 12:54 |
Sergey Nikitin Messages: 3 Registered: November 2013 |
Junior Member |
|
|
Hi, Winston!
Thank you for help! 50% of time leak is closed: Maven3 + Freestyle job configuration works significantly faster and don't slow down the build process by Parsing POM step!
Do you have any ideas about other 50% of great time leak of Hudson production ci-server (see the details below)?
Now our Hudson ci-server has 300+Jobs and I get more and more complaints from developers for Job's configuration saving time. As I wrote before, it was 4+ minutes, when we had 200+ jobs, but now, when we have 300+Jobs, time for configuration saving is 30 min 07 sec for every Job - I have measured it today - extremely slow...
Configuration saving time independent from Job's details. Every our new Job or changed Job just saves its own configuration for 30 min 07 sec now and configuration saving time continue to grow with number of Hudson Jobs.
Certainly we can search a production solution for ci-server, but do we have any chance to improve job's configuration saving time in Hudson, any ideas?
[Updated on: Mon, 09 December 2013 12:58] Report message to a moderator
|
|
|
|
Powered by
FUDForum. Page generated in 0.02662 seconds