[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [cross-project-issues-dev] tycho-gmp.gmf.tooling builds consumes enormous amount of memory
|
Denis,
Right now I'm not talking about memory leak, but memory consumption.
Hudson has a tendency to keep the info about entire artifacts of a
build. For example, if a job contains JUnit test results then this info
is kept in the memory. Currently Hudson is holding about 360 jobs with
6500 builds and 510,000 JUnit Case results in the memory. Average memory
consumed by each JUnit Test case is about 200kb. So net memory
consumption by JUnit results should be about 100 MB, however memory
profiler reports 250 MB of memory occupied by Test cases.
I noticed that the job hudson-test-harness (which is maintained by me)
has the debug turned on. Because of this the Junit Test standard
outputs has about 15-20 MB of debug results and this entire result was
in the memory. Just 10 JUnit Case results occupy about 150-200 MB of
memory. Now I turned off the debug. However, old Junit test results are
still in the memory. So next time when Hudson restarts it should consume
less memory (about 150 MB less). I noticed some other jobs too have
large JUnit standard output. Reducing those output should reduce over
all Hudson memory consumption. I will try to list those jobs. The ideal
solution would be to fix in Hudson to load JUnit Test results lazily,
but that is a long term solution.
Similarly, the job "tycho-gmp.gmf.tooling" is producing huge amount of
maven artifacts which are also kept in the memory (about 128 MB). I'm
not sure exactly what it is, could be JavaDoc as pointed out by Mickael.
I'm trying to find from tycho team if we could reduce those maven
artifacts in the build (temporarily at least), until we find proper
solution in Hudson itself.
Of course we need to fix the memory leaks, however the quick solution is
to reduce these huge memory footprints for now in a easy way.
- Winston
On 2/22/12 10:32 AM, Denis Roy wrote:
Winston,
Hudson is a very important part of our infrastructure. If you've found
a plugin (or a specific build job) that is obviously leaking memory
(thus making our Hudson instance slow) I suggest we disable that plugin
and/or job immediately and open a bug to request it be fixed.
In this case, is the problem with a specific plugin, or is it within the
job? I can't tell from the words that are written below.
Thanks,
Denis
On 02/22/2012 11:37 AM, Winston Prakash wrote:
Jason,
I saw your TODO note in the code about the reference. I'm not worried
about that. I'm wondering about the amount of memory it takes to keep
the parsed artifacts info. Only builds of this job takes this much
memory. Could this be because of maven-Javadoc-plugin as mentioned by
Mickael.
When it comes to memory leak, Stapler and XStream are the worst
enemies I have to fight with :-)
- Winston
On 2/21/12 9:08 PM, Jason Dillon wrote:
IIRC the build state is only in a soft ref when loading from disk,
but there is a hard-ref when a new build is run, so its retained
until the server reboots. The project work at sonatype was terminated
before this and many other issues could be resolved. There needed to
be some additional api added to the xref bits to allow a reference to
build state once built to be transitioned from a hard to soft ref.
--jason
On Tuesday, February 21, 2012 at 8:51 PM, Winston Prakash wrote:
Hi Jason D./Stewart,
As per request from Eclipse foundation, I'm analyzing performance of
their Hudson instance. I noticed that tycho-gmp.gmf.tooling
(https://hudson.eclipse.org/hudson/view/Tycho%20+%20Maven/job/tycho-gmp.gmf.tooling)
builds are occupying about 120 MB of memory. What is special about
this job. The memory is specifically consumed by maven3 plugin data
model. When I try to look at the maven3 structure UI
(https://hudson.eclipse.org/hudson/view/Tycho%20+%20Maven/job/tycho-gmp.gmf.tooling/136/maven/?)
in Hudson it spins for ever. See the tree in the attached image. Any
idea what is causing this huge memory consumption? BTW, only builds
of this job seem to occupying this much memory. Other job builds
consume much less memory (< 10 MB).
A memory leak in Stapler causes GC from collecting these huge build
related objects and eventually Hudson dies running out of memory.
- Winston
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
