Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] tycho-gmp.gmf.tooling builds consumes enormous amount of memory


Right now I'm not talking about memory leak, but memory consumption.

Hudson has a tendency to keep the info about entire artifacts of a build. For example, if a job contains JUnit test results then this info is kept in the memory. Currently Hudson is holding about 360 jobs with 6500 builds and 510,000 JUnit Case results in the memory. Average memory consumed by each JUnit Test case is about 200kb. So net memory consumption by JUnit results should be about 100 MB, however memory profiler reports 250 MB of memory occupied by Test cases.

I noticed that the job hudson-test-harness (which is maintained by me) has the debug turned on. Because of this the Junit Test standard outputs has about 15-20 MB of debug results and this entire result was in the memory. Just 10 JUnit Case results occupy about 150-200 MB of memory. Now I turned off the debug. However, old Junit test results are still in the memory. So next time when Hudson restarts it should consume less memory (about 150 MB less). I noticed some other jobs too have large JUnit standard output. Reducing those output should reduce over all Hudson memory consumption. I will try to list those jobs. The ideal solution would be to fix in Hudson to load JUnit Test results lazily, but that is a long term solution.

Similarly, the job "tycho-gmp.gmf.tooling" is producing huge amount of maven artifacts which are also kept in the memory (about 128 MB). I'm not sure exactly what it is, could be JavaDoc as pointed out by Mickael. I'm trying to find from tycho team if we could reduce those maven artifacts in the build (temporarily at least), until we find proper solution in Hudson itself.

Of course we need to fix the memory leaks, however the quick solution is to reduce these huge memory footprints for now in a easy way.

- Winston

On 2/22/12 10:32 AM, Denis Roy wrote:

Hudson is a very important part of our infrastructure.  If you've found
a plugin (or a specific build job) that is obviously leaking memory
(thus making our Hudson instance slow) I suggest we disable that plugin
and/or job immediately and open a bug to request it be fixed.

In this case, is the problem with a specific plugin, or is it within the
job?  I can't tell from the words that are written below.



On 02/22/2012 11:37 AM, Winston Prakash wrote:

I saw your TODO note in the code about the reference.  I'm not worried
about that.  I'm wondering about the amount of memory it takes to keep
the parsed artifacts info.  Only builds of this job takes this much
memory. Could this be because of maven-Javadoc-plugin as mentioned by

When it comes to memory leak, Stapler and XStream are the worst
enemies I have to fight with :-)

- Winston

On 2/21/12 9:08 PM, Jason Dillon wrote:
IIRC the build state is only in a soft ref when loading from disk,
but there is a hard-ref when a new build is run, so its retained
until the server reboots. The project work at sonatype was terminated
before this and many other issues could be resolved. There needed to
be some additional api added to the xref bits to allow a reference to
build state once built to be transitioned from a hard to soft ref.


On Tuesday, February 21, 2012 at 8:51 PM, Winston Prakash wrote:

Hi Jason D./Stewart,

As per request from Eclipse foundation, I'm analyzing performance of
their Hudson instance. I noticed that tycho-gmp.gmf.tooling
builds are occupying about 120 MB of memory. What is special about
this job. The memory is specifically consumed by maven3 plugin data
model. When I try to look at the maven3 structure UI
in Hudson it spins for ever. See the tree in the attached image. Any
idea what is causing this huge memory consumption? BTW, only builds
of this job seem to occupying this much memory. Other job builds
consume much less memory (<   10 MB).

A memory leak in Stapler causes GC from collecting these huge build
related objects and eventually Hudson dies running out of memory.

- Winston

cross-project-issues-dev mailing list

Back to the top