Hi all, hopefully someone can help me with this.
I'm seeing my Jetty stand-alone servers chew up available memory under load
(-Xmx4g eventually all gets used). Based on Java Object histograms, it
seems like the problem has to do with Jetty, or how I have Jetty configured.
Here's the info I have:
Server: Jetty Hightide 7.0.0 (problem occurred before upgrading, tho, previously 6.1.11).
Setup: Stand-alone, running one production Java webapp: Spring 2.5, JSP
Load: ~20-250 req./second over the course of a day, distributed across three server instances on separate machines
Histogram shows large number of Jetty objects being retained at the request level:
# cat /tmp/histo.out | grep jetty
5: 1683568 121216896 org.eclipse.jetty.http.HttpFields$Field
6: 1370718 98691696 org.eclipse.jetty.io.View
14: 550942 39667824 org.eclipse.jetty.io.ByteArrayBuffer
15: 98505 29945520 org.eclipse.jetty.server.Request
18: 98505 18124920 org.eclipse.jetty.server.nio.SelectChannelConnector$2
22: 197325 14207400 org.eclipse.jetty.io.View$CaseInsensitive
23: 98505 14184720 org.eclipse.jetty.http.HttpParser
28: 98505 12608640 org.eclipse.jetty.http.HttpGenerator
29: 116775 12144600 org.eclipse.jetty.server.session.HashSessionManager$Session
30: 98505 11032560 org.eclipse.jetty.io.nio.SelectChannelEndPoint
31: 98505 9456480 org.eclipse.jetty.server.Response
32: 98505 8668440 org.eclipse.jetty.http.HttpURI
33: 98068 8629984 org.eclipse.jetty.server.HttpConnection$Output
38: 197010 7880400 org.eclipse.jetty.http.HttpFields
41: 98505 7092360 org.eclipse.jetty.io.nio.SelectChannelEndPoint$IdleTask
45: 98505 6304320 org.eclipse.jetty.server.AsyncContinuation
46: 70473 6201624 org.eclipse.jetty.io.BufferCache$CachedBuffer
58: 98557 3153824 org.eclipse.jetty.util.Utf8StringBuilder
59: 98515 3152480 org.eclipse.jetty.util.MultiMap
60: 98505 3152160 org.eclipse.jetty.server.HttpConnection$RequestHandler
72: 98502 2364048 org.eclipse.jetty.util.AttributesMap
87: 24008 1152384 org.eclipse.jetty.servlet.ServletHandler$CachedChain
101: 11734 563232 org.eclipse.jetty.server.CookieCutter
(--truncated at 10K instance count--)
There are no comparable/suspicious counts for my-domain-level objects.
Based on a session-scoped object, there appear to be about 29K active
sessions when this snapshot was taken, which seems a bit high, but not
inconceivable, given our traffic type (Facebook apps). There are <100
active requests present at the time of the snapshot.
Seems like maybe old, fulfilled requests are not getting GCed because they are
hanging around in the idle pool? Any ideas?
Thanks in advance!
