Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Hudson » Performance problems
Performance problems [message #1070606] Thu, 18 July 2013 10:02 Go to next message
Fabrizio Giudici is currently offline Fabrizio GiudiciFriend
Messages: 21
Registered: May 2013
Location: Italy
Junior Member

Since a couple of weeks my Hudson is experiencing some performance problems that I can't understand. At first I thought I had a disk that was going to fault (it's part of a RAID 1 setup, and in my (limited) understanding of this stuff a disk unit that experience troubles could slow down the overall performance). After a few investigation I'm less and less sure of this explanation, so - while I can't find out why the problem arose all of a sudden, without changes in my configuration - I'm now trying to first focus on Hudson performance.

In short, a bunch of nightly jobs that started at 3 AM and were mostly completed within 8/9 AM now take until 1/2 PM - from the previous 5/6 hours we're now at 10/11, thus the time doubled.

Some numbers first:

* I have about 30 projects (all Maven)
* For each project, a "Compile from Scratch" job is started, trying to compile from a clean repo
* For each project, a "Metrics" job is started, which compiles with QA activated
* For each project, an archival project is started, which just collects the result of Metrics and archives them.

For what concerns the server, I copy the specs from my provider:

* Ubuntu 12.04 x64
* Intel Core i7-920 Quad Core
* 2x 750 GB SATA II HDD in a RAID arrangement

Hudson is configured to have 6 parallel workers.

I've been monitoring the thing with top and iotop. The "Compile from Scratch" are very slow, but this is expected as they are mostly network-bound, having to download from scratch all the Maven artifacts from the internet (not from a local proxy, as the other jobs). When they are running, the load is pretty low (about 3) and the machine responsive. When Metrics jobs are active, the load is around 20/30 (with peaks of 40/50) and the matchine is mostly unresponsive. The CPU usage is very low (way below 10%) and iotop shows that the machine is disk bound. While this is expected, given the very nature of the jobs, I only see peaks of a few MB/sec in the disk I/O - SATA II disks are quite obsolete, still they should warranty at least one magnitudo better speed. All the 8GB of RAM are used, with just a few of swap memory used (about 250k) and iotop report absolutely no swapping.

Looking at the logs of Jobs, I see something like this:

11:06:45 Started by upstream job "Zephyr_Metrics" build number 34
11:06:47 Restoring workspace from build #108 of project Zephyr_Compile_and_Test_and_Deploy_Local_Snapshot
11:42:24 [JDK_1.7.0_21] $ /home/tomcat/Workspaces/Hudson/tools/Maven_3.0.4/bin/mvn ...

That is, it took more than 30 minutes just to retrieve the workspace from a previous jobs. During this phase, I could see Java threads on I/O with speeds of 150/200 Kb/sec - I'd expect that since they are just reading a workspace, they should be just reading from disk - why so slow?

Remember, my server could be obsolete, but up to a couple of weeks ago Hudson was 2x faster - what could be happened?

Thanks for any hint.

[Updated on: Thu, 18 July 2013 10:23]

Report message to a moderator

Re: Performance problems [message #1070757 is a reply to message #1070606] Thu, 18 July 2013 15:50 Go to previous messageGo to next message
Bob Foster is currently offline Bob FosterFriend
Messages: 85
Registered: January 2013
I have more questions than answers at this point. The first question to ask is, what changed from a couple of weeks ago? Answers that might explain a slowdown include: Changed Hudson version. Nothing, Hudson has not been restarted in the past couple of weeks. Etc. What Hudson version are you running? What are its memory limits and what is current usage?

Re: Performance problems [message #1070768 is a reply to message #1070757] Thu, 18 July 2013 16:05 Go to previous message
Fabrizio Giudici is currently offline Fabrizio GiudiciFriend
Messages: 21
Registered: May 2013
Location: Italy
Junior Member
Well, I think that we can suspend the topic - my first impression was indeed right, now I'm sure I've got a hardware error with a disk, and it would explain everything. I'll have the disk replaced and let you know.
Previous Topic:JAVA source code and Hudson
Next Topic:${WORKSPACE} resolution
Goto Forum:

Current Time: Wed May 24 21:44:07 GMT 2017

Powered by FUDForum. Page generated in 0.04069 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software