| Re: [jetty-users] Jetty 7 memory leak |
|
We will try
to get a test going to see if we can duplicate the memory situation.
Just for grins, we did try the Sun 1.7 JVM and memory still looks the
same there so it is possible that it is not JVM related. Hopefully the
test will show something.
Tony Greg Wilkins wrote: Tony, what needs to be done is to run the SslSelectChannelEndPoint on it's own and stream data through it endlessly and see if the problem can be reproduced in a simpler test case. We have an EndPointTest that could be used as the basis of this. Could you have a stab at creating this test? It will probably be next week before we can get somebody onto this. cheers On 23 September 2010 12:30, Tony Thompson <Tony.Thompson@xxxxxxxxxxxxxx> wrote:ok, we have been running with Jetty 7.2 for about 18 hours now and this is what I have found. The memory leak still appears to be there but, it is different. It appears that maybe Jetty is not holding on to anything anymore and possibly it has shifted to a bug in the JVM (?). We have been comparing heap dumps today and the retained byte arrays do continue to grow with a GC root that looks like this: +-------------------------------------------------------------------------------------- | +-------------------------------------------------------------------------------------- | +---byte[16665] | | | +---buf of com.sun.net.ssl.internal.ssl.InputRecord | | | +---r of com.sun.net.ssl.internal.ssl.AppInputStream | | | +---input of com.sun.net.ssl.internal.ssl.SSLSocketImpl | | | +---referent of java.lang.ref.Finalizer | | | +---prev of java.lang.ref.Finalizer | | | +---prev of java.lang.ref.Finalizer | | | +---prev of java.lang.ref.Finalizer (and on like that...truncated for brevity) We are also still experiencing high CPU utilization but it seems to be better as well. When I say better, I just mean the server has been usable. Before it would get to a point where it wouldn't respond to requests any more. Any suggestions at this point? Tony Jan Bartel wrote: Tony, We're getting read to ship 7.2 out the door. It would be great if you could try the latest 7.2-SNAPSHOT out: https://oss.sonatype.org/content/groups/jetty-with-staging/org/eclipse/jetty/jetty-distribution/7.2.0-SNAPSHOT/jetty-distribution-7.2.0-20100922.104837-6.zip thanks, Jan On 23/09/10 08:55, Tony Thompson wrote: This particular issue happens with both 7.1.4 and 7.1.6. We were attempting to upgrade to 7.1.6 to see if it fixed the issue but it has not. Of course, you saw my other post with the high utilization issues so it has been a battle. Tony Jan Bartel wrote: Tony, From your other posts, you seem to be upgrading to 7.1.6 - can you confirm whether this is happening also on 7.1.6 or only 7.1.4? thanks Jan On 23/09/10 01:08, Tony Thompson wrote: I am also seeing this same issue with the 1.6.0_21 JVM on a 32 bit Linux server so it is not because of the older JVM. I still need some help with this one. Thanks Tony Tony Thompson wrote: I am running Jetty 7.1.4.v20100610 on a Linux server with this JVM: Java HotSpot(TM) 64-Bit Server VM; 1.6.0; 1.6.0-b105; mixed mode The server will run for a period of time and then run out of memory and crash. I have bumped the number of file handles so the server can run longer and I could get some heap dumps to compare. What I have found is literally millions of byte arrays hanging around in memory. As you can see below, it appears to be related to SSL. I know I am on an older JVM so it could be a JVM issue but, I found this which could indicate that maybe Jetty is not shutting things down properly: http://stackoverflow.com/questions/1623676/suns-java-ssl-implementation-is-leaking-memory Not quite my exact situation but similar. Can anyone provide some insight? Hoping the server can last until a reboot tonight.... Tony Paths from "GC Roots" to "Object byte[] #00002AAAB9C201E0" +-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------+ | Name | Retained Size | Shallow Size | +-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------+ | +---byte[16665] | 16,688 | 16,688 | | | | | | | +---buf of com.sun.net.ssl.internal.ssl.EngineInputRecord | 17,792 | 96 | | | | | | | +---inputRecord of com.sun.net.ssl.internal.ssl.SSLEngineImpl | 29,496 | 192 | | | | | | | +---_engine of org.eclipse.jetty.io.nio.SslSelectChannelEndPoint | 53,568 | 184 | | | | | | | +---attachment of sun.nio.ch.SelectionKeyImpl | 64 | 64 | | | | | | | +---value of java.util.HashMap$Entry | 72 | 48 | | | | | | | +---[506] of java.util.HashMap$Entry[1024] | 50,696 | 8,216 | | | | | | | +---table of java.util.HashMap | 50,760 | 64 | | | | | | | +---fdToKey of sun.nio.ch.EPollSelectorImpl [Stack Local] | 51,184 | 120 | | | | | | | +---<local variable> of java.lang.Thread [Stack Local, Thread] "qtp1891328597-44 - Acceptor0 c@xxxxxxxxxxxxxxxxxxxxxxxxxx:443" | 1,216 | 168 | +-----------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------+ This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from Stoneware, Inc. may be monitored. _______________________________________________ jetty-users mailing list jetty-users@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/jetty-users This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from Stoneware, Inc. may be monitored. _______________________________________________ jetty-users mailing list jetty-users@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/jetty-users This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from Stoneware, Inc. may be monitored. This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from Stoneware, Inc. may be monitored. _______________________________________________ jetty-users mailing list jetty-users@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/jetty-users_______________________________________________ jetty-users mailing list jetty-users@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/jetty-users This message (and any associated files) is intended only for the
use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from Stoneware, Inc. may be monitored. |