| Re: [hudson-dev] Suggestions for improving Hudson performance |
Hi
Any update on below?
From my investigation it seems we already have #1, but I would like to add another suggestion
Replace any homegrown multi threading solutions like OneShotEvent with the java standard libraries
Best regards
Henrik
Hi Henrik,
Thanks, I'll go through this and answer in detail. Mean while you might want to read the threads at Eclipse cross project where we are discussing about Hudson performance.
- Winston
On 2/22/12 1:33 PM, Henrik Lynggaard Hansen wrote:Hi Before I get to the actual suggestions, here are some numbers from our Hudson installation to give you a sense of scale I used http://wiki.hudson-ci.org/display/HUDSON/Monitoring. Perhaps winston or someone can provide similar from the internal oracle instances or the eclipse instances? Number of jobs: around 600, but properly one around 400 active at the moment Number of builds kept: 30.000 Number of slaves:60 (many of these are host specific nodes which are inactive for parts of the day) Number of http requests per minute (daytime): average around 1400-1600 with peaks of 2500 Number of http request served in a day: 730.000 measured but we had to reboot Hudson because it was hanging so I would estimate around 800.000-900.000 would be more realistic I had to disable the moniroting plugin again because it has too big overhead and we would need to reboots during office hours. I am still working on getting some stats on which URLs are being hit the most. This installation is not fully sized, we will need to grow it significantly as we add more projects. Anyways onwards to my suggestions: 1. Create a static folder served by the container: Move all static content (help pages, logo etc.) to a separate folder and exclude that URL pattern from stabler. This way it should be served by the container and much better cached, or proxied 2. Don't generate semi-static content: It looks like semi-static content such as RSS feeds are generated using jelly templates on each request which seems quite expensive.(I suspect rss request make up the bulk of my 730.000 requests). Suggest we move to a model where such resources are written to disk when it changes which is far less often than it is requested. We then make a fast path in stabler where the first thing it does is look into a map of URL, and if the URL is present in the map the file on disk is served directly instead of going through stabler "views". Plugins could even take part in this be writing files to disk and registering the URL to file mapping via a service. I suspect this could speed up things quite a bit since plugins could write all sorts of resources that only needs to be calculated once a build is done like change logs, test reports, dependency graphs etc. (we need to figure out something with the dynamic sidebar but still) (if someone can give me a pointer to the good starting points in the code I would like to give this a go) 3. Don't reserve a thread per executor. As soon as a slave goes on line there is created a executor thread for each executor. There really is no need for this, just create the threads as needed 4. Revisit master slave handshaking I am not sure how master/slave communications are initiated, but I think it might be different based on the slave type. SSH slaves appear to keep using the SSH channel opened during startup, while JNLP slaves "phone home" to a server socket but both reserve a thread on the master. I suggest we make use of a server socket approach on all slave types as a default and use the "master calls slave" as fallback. This will allow us to use NIO and a thread pool instead of reserved threads on the master. 5. Pre-compile / pre-assemble jelly? I don't know if it is possible but right now the jelly files are re-parsed on request (as far as I can tell), is there a technique where we can precompile sush files or perhaps just pre-resolve all the includes and build a jelly file per page? Best regards Henrik