|Re: [cross-project-issues-dev] tycho-gmp.gmf.tooling builds consumes enormous amount of memory|
Denis, Right now I'm not talking about memory leak, but memory consumption.Hudson has a tendency to keep the info about entire artifacts of a build. For example, if a job contains JUnit test results then this info is kept in the memory. Currently Hudson is holding about 360 jobs with 6500 builds and 510,000 JUnit Case results in the memory. Average memory consumed by each JUnit Test case is about 200kb. So net memory consumption by JUnit results should be about 100 MB, however memory profiler reports 250 MB of memory occupied by Test cases.
I noticed that the job hudson-test-harness (which is maintained by me) has the debug turned on. Because of this the Junit Test standard outputs has about 15-20 MB of debug results and this entire result was in the memory. Just 10 JUnit Case results occupy about 150-200 MB of memory. Now I turned off the debug. However, old Junit test results are still in the memory. So next time when Hudson restarts it should consume less memory (about 150 MB less). I noticed some other jobs too have large JUnit standard output. Reducing those output should reduce over all Hudson memory consumption. I will try to list those jobs. The ideal solution would be to fix in Hudson to load JUnit Test results lazily, but that is a long term solution.
Similarly, the job "tycho-gmp.gmf.tooling" is producing huge amount of maven artifacts which are also kept in the memory (about 128 MB). I'm not sure exactly what it is, could be JavaDoc as pointed out by Mickael. I'm trying to find from tycho team if we could reduce those maven artifacts in the build (temporarily at least), until we find proper solution in Hudson itself.
Of course we need to fix the memory leaks, however the quick solution is to reduce these huge memory footprints for now in a easy way.
- Winston On 2/22/12 10:32 AM, Denis Roy wrote:
Winston, Hudson is a very important part of our infrastructure. If you've found a plugin (or a specific build job) that is obviously leaking memory (thus making our Hudson instance slow) I suggest we disable that plugin and/or job immediately and open a bug to request it be fixed. In this case, is the problem with a specific plugin, or is it within the job? I can't tell from the words that are written below. Thanks, Denis On 02/22/2012 11:37 AM, Winston Prakash wrote:Jason, I saw your TODO note in the code about the reference. I'm not worried about that. I'm wondering about the amount of memory it takes to keep the parsed artifacts info. Only builds of this job takes this much memory. Could this be because of maven-Javadoc-plugin as mentioned by Mickael. When it comes to memory leak, Stapler and XStream are the worst enemies I have to fight with :-) - Winston On 2/21/12 9:08 PM, Jason Dillon wrote:IIRC the build state is only in a soft ref when loading from disk, but there is a hard-ref when a new build is run, so its retained until the server reboots. The project work at sonatype was terminated before this and many other issues could be resolved. There needed to be some additional api added to the xref bits to allow a reference to build state once built to be transitioned from a hard to soft ref. --jason On Tuesday, February 21, 2012 at 8:51 PM, Winston Prakash wrote:Hi Jason D./Stewart, As per request from Eclipse foundation, I'm analyzing performance of their Hudson instance. I noticed that tycho-gmp.gmf.tooling (https://hudson.eclipse.org/hudson/view/Tycho%20+%20Maven/job/tycho-gmp.gmf.tooling) builds are occupying about 120 MB of memory. What is special about this job. The memory is specifically consumed by maven3 plugin data model. When I try to look at the maven3 structure UI (https://hudson.eclipse.org/hudson/view/Tycho%20+%20Maven/job/tycho-gmp.gmf.tooling/136/maven/?) in Hudson it spins for ever. See the tree in the attached image. Any idea what is causing this huge memory consumption? BTW, only builds of this job seem to occupying this much memory. Other job builds consume much less memory (< 10 MB). A memory leak in Stapler causes GC from collecting these huge build related objects and eventually Hudson dies running out of memory. - Winston_______________________________________________ cross-project-issues-dev mailing list cross-project-issues-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
Back to the top