[jetty-users] OutOfMemory errors

[radiatejava <radiatejava@xxxxxxxxx>]

We are using Jetty 8.1.3 with JRE 1.6 that is in production. I know
these versions are old but right now this is in govt org and it is not
likely to be upgraded this year.

Problem is about once a month JVM is reporting OutOfMemory error and
the application dies. This is urgent for us. Heap dump does not show
anything suspicious. I am putting the stack trace here. Any help is
appreciated.

Here is the reported error:
2015-03-11 13:00:08,081 qtp31782389-6377     WARN  jetty.io.nio
                               - handle failed
java.lang.OutOfMemoryError
    at sun.security.pkcs11.wrapper.PKCS11.C_DecryptUpdate(Native Method)
    at sun.security.pkcs11.P11Cipher.implUpdate(P11Cipher.java:549)
    at sun.security.pkcs11.P11Cipher.engineUpdate(P11Cipher.java:464)
    at javax.crypto.Cipher.update(DashoA13*..)
    at com.sun.net.ssl.internal.ssl.CipherBox.decrypt(Unknown Source)
    at com.sun.net.ssl.internal.ssl.InputRecord.decrypt(Unknown Source)
    at com.sun.net.ssl.internal.ssl.EngineInputRecord.decrypt(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(Unknown Source)
    at javax.net.ssl.SSLEngine.unwrap(Unknown Source)
    at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519)
    at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:396)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:188)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
    at java.lang.Thread.run(Unknown Source)
2015-03-11 13:00:08,084 qtp31782389-6404     WARN  jetty.io.nio
                               - handle failed
java.lang.OutOfMemoryError: (class:
sun/security/pkcs11/P11TlsRsaPremasterSecretGenerator)
    at sun.security.pkcs11.SunPKCS11$P11Service.newInstance0(SunPKCS11.java:934)
    at sun.security.pkcs11.SunPKCS11$P11Service.newInstance(SunPKCS11.java:892)
    at javax.crypto.KeyGenerator.a(DashoA13*..)
    at javax.crypto.KeyGenerator.<init>(DashoA13*..)
    at javax.crypto.KeyGenerator.getInstance(DashoA13*..)
    at com.sun.net.ssl.internal.ssl.JsseJce.getKeyGenerator(Unknown Source)
    at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.generateDummySecret(Unknown
Source)
    at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.polishPreMasterSecretKey(Unknown
Source)
    at com.sun.net.ssl.internal.ssl.RSAClientKeyExchange.<init>(Unknown Source)
    at com.sun.net.ssl.internal.ssl.ServerHandshaker.processMessage(Unknown
Source)
    at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
    at com.sun.net.ssl.internal.ssl.Handshaker$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at com.sun.net.ssl.internal.ssl.Handshaker$DelegatedTask.run(Unknown Source)
    at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:370)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:188)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
    at java.lang.Thread.run(Unknown Source)


And the top level (objects occupying the largest memory) histogram is like this:

Class Name                                                         |
Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------------
org.eclipse.jetty.server.ssl.SslSelectChannelConnector @ 0x81581498|
       192 |    14,677,920 |     51.87%
|- org.eclipse.jetty.http.HttpBuffersImpl @ 0x815a4248             |
        88 |    14,673,288 |     51.85%
  |- org.eclipse.jetty.io.PooledBuffers @ 0x8157af50              |
       56 |    14,607,160 |     51.62%
---------------------------------------------------------------------------------------------------------------