Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Hudson » Hudson: Slow Parsing POMs or low performance for production(Hudson: Slow Parsing POMs or low performance for production)
Hudson: Slow Parsing POMs or low performance for production [message #1165775] Fri, 01 November 2013 10:31 Go to next message
Sergey Nikitin is currently offline Sergey Nikitin
Messages: 3
Registered: November 2013
Junior Member
Hi!
We have performance problem using Hudson as a central continuous integration server for 200+ projects. Do you have any ideas how to improve performance?
Problem: we using Hudson as a ci-server for 200+ build projects (5-10 simultaneous jobs). The hardware is quite adequate: 16 CPUs + 100 GB RAM, and it works quite fast, except two points:
- slow project configuration saving (when you create it or change ): 4+ minutes;
- slow Parsing POM step during build process: 10+ minutes.
We have done some tests on the same server: if you start Hudson with several build projects it works very fast, but when you increase number of build projects (even if you don't start them) the project configuration saving and Parsing POM step becomes slower and slower....

We have found cause of the problem in Hudson dependencies, look at the mails below.

But since 2011, when the problem was found, nothing changed. We don't have an option in Hudson to turn off dependences calculation or do anything else to improve performance.

Any ideas?

===================== mails from web ===============

From Andreas Veithen <andreas.veit...@gmail.com>
Subject Re: [Hudson] "Parsing POMs" can be very slow
Date Sun, 23 Jan 2011 18:06:11 GMT
Benson,

Right now, when there is a change in Axiom, Hudson will also trigger
builds of Neethi, Woden, Axis2, Sandesha2 and Rampart. This is what we
want because it allows us to ensure non-regression in downstream
projects. This relies on Hudson calculating the dependencies
automatically. Disabling this would mean that we have to find an
alternative configuration to achieve the same. So, yes, interjob
dependencies are needed, and disabling them should only be a last
resort. I like Niklas' proposal to run the recalculation of these
dependencies as a cron job.

Andreas

On Sun, Jan 23, 2011 at 18:21, Benson Margulies <bimargulies@gmail.com> wrote:
> Niklas,
>
> I'm a very rusty hudson dev. So I'm reduced to the following question:
> do we really need interjob hudson dependencies at Apache (the
> downstream/upstream business). If all the jobs are set to turn this
> off, does it still calculate?
>
> Another view of this would be to look to splitting up hudson, and
> maybe fixing this might be less work than that.
>
> --benson
>
>
> On Sun, Jan 23, 2011 at 11:15 AM, Niklas Gustavsson
> <niklas@protocol7.com> wrote:
>> Hi
>>
>> Some of you might have noticed that Hudson can take a long time on
>> Maven builds in the step called "Parsing POMs". In fact, some builds
>> will time-out due to this step taking 15 minutes or more. Thing is, as
>> part of this step, Hudson will do the problematic rebuild of its
>> dependency graph. This is the same problem causing saving project
>> configuration taking forever. We have previously reported this here:
>> issues.hudson-ci.org/browse/HUDSON-7535
>>
>> I've now added a comment on the issue describing the additional
>> problem of builds taking a long time.
>>
>> I think we got some Hudson devs around here, anyone knows a workaround
>> for this problem? The problem at hand is line 748 in
>> MavenModuleSetBuild.java. I'm considering patching Hudson to remove
>> these calls on saving projects/starting builds and instead rebuilding
>> the graph as a cron job every hour or so. Would that work (given the
>> dependency graph will obviously be broken for some short times)?
>>
>> /niklas

[Updated on: Wed, 06 November 2013 02:35]

Report message to a moderator

Re: Hudson: Slow Parsing POMs or low performance for production [message #1181498 is a reply to message #1165775] Mon, 11 November 2013 11:45 Go to previous messageGo to next message
Winston Prakash is currently offline Winston Prakash
Messages: 412
Registered: August 2011
Location: Fremont, CA USA
Senior Member
Hi Sergey,

Can you please answer few questions

-What version of Hudson are you using?
-What Maven builder are you using?
--Deprecated legacy Maven Job type
--Recommended Freestyle project with Maven3 Builder?

Thanks.


Winston Prakash
Eclipse Hudson team
Re: Hudson: Slow Parsing POMs or low performance for production [message #1194199 is a reply to message #1181498] Mon, 18 November 2013 06:37 Go to previous messageGo to next message
Sergey Nikitin is currently offline Sergey Nikitin
Messages: 3
Registered: November 2013
Junior Member
Hi, Winston,

Thank you for your involvement. I was on vacation, that's why I delayed my answer, but I still hope to find a solution to improve performance of Hudson ci-server.

Answers to your questions:

-What version of Hudson are you using?
Answer: 2.2.1

-What Maven builder are you using?
Answer: 2.2.1

--Deprecated legacy Maven Job type
Answer: most of our projects have type Maven 2/3 (Legacy)

--Recommended Freestyle project with Maven3 Builder?
Answer: we use basically projects Maven 2/3 (Legacy) with Maven v2.2.1 and few Freestyle Projects with Maven2 (Legacy) steps

We also use Nested View Plugin with many pages (Group job views into multiple levels instead of one big list of tabs) for departments and projects. Can it be the origin of the slow Parsing POM step and slow Job configuration saving?

Thanks.
Re: Hudson: Slow Parsing POMs or low performance for production [message #1202130 is a reply to message #1194199] Thu, 21 November 2013 21:53 Go to previous messageGo to next message
Winston Prakash is currently offline Winston Prakash
Messages: 412
Registered: August 2011
Location: Fremont, CA USA
Senior Member
Legacy Maven 2/3 project is notoriously slow. That is why we developed Maven3 plugin separately. I will highly recommend you to use the Freestyle project with Maven3 builder. Several people have noticed vast improvement.

Winston Prakash
Eclipse Hudson team
Re: Hudson: Slow Parsing POMs or low performance for production [message #1220064 is a reply to message #1202130] Mon, 09 December 2013 07:54 Go to previous messageGo to next message
Sergey Nikitin is currently offline Sergey Nikitin
Messages: 3
Registered: November 2013
Junior Member
Hi, Winston!

Thank you for help! 50% of time leak is closed: Maven3 + Freestyle job configuration works significantly faster and don't slow down the build process by Parsing POM step!

Do you have any ideas about other 50% of great time leak of Hudson production ci-server (see the details below)?

Now our Hudson ci-server has 300+Jobs and I get more and more complaints from developers for Job's configuration saving time. As I wrote before, it was 4+ minutes, when we had 200+ jobs, but now, when we have 300+Jobs, time for configuration saving is 30 min 07 sec for every Job - I have measured it today - extremely slow...

Configuration saving time independent from Job's details. Every our new Job or changed Job just saves its own configuration for 30 min 07 sec now and configuration saving time continue to grow with number of Hudson Jobs.

Certainly we can search a production solution for ci-server, but do we have any chance to improve job's configuration saving time in Hudson, any ideas?

[Updated on: Mon, 09 December 2013 07:58]

Report message to a moderator

Re: Hudson: Slow Parsing POMs or low performance for production [message #1220113 is a reply to message #1220064] Mon, 09 December 2013 14:59 Go to previous message
Winston Prakash is currently offline Winston Prakash
Messages: 412
Registered: August 2011
Location: Fremont, CA USA
Senior Member
Hi Sergey, can you please read the details in this bug (which is fixed)

https://bugs.eclipse.org/bugs/show_bug.cgi?id=422275

It sounds very similar


Winston Prakash
Eclipse Hudson team
Previous Topic:Hudson 3.1.1 CR release
Next Topic:Release Candidate for 3.1.1. Is Available
Goto Forum:
  


Current Time: Sat Aug 30 22:28:22 EDT 2014

Powered by FUDForum. Page generated in 0.01694 seconds