| [jetty-dev] Troubles with embedded Jetty Session scavenging |
Hi everybody, I’ve been using Jetty for quite a while and a few months ago we decided to switch from the standard start.jar-launcher to our own. Main reasons are greatly improved deployments (startup) time by using a own largly simplified hot-WebApp deployer and automatic SSL certificate updates through our own letsencrypt-client. Having troubles with the Session HouseKeeper: Using simple, standard in-memory sessions. Scavenging starts and initially works fine. I can debug it and see that it collects expired sessions. But as soon as I let it run on productive systems for a few days at some point in time it stops and all expired sessions remain in memory which leads to OutOfMemory-errors after a few weeks. Works fine with the original start.jar-launcher, experienced this problem only with our custom server. Using jetty 9.4.7 and 9.4.8, running on Oracle JDK 9.0.1 (was the same with 9.0) on Windows 10, various Linux distros and macOS 10.13.2 . I’m initializing Server / Session Manager this way … server = new org.eclipse.jetty.server.Server(); DefaultSessionIdManager session = new DefaultSessionIdManager(server); HouseKeeper houseKeeper = new HouseKeeper(); houseKeeper.setIntervalSec(30); session.setSessionHouseKeeper(houseKeeper); server.setSessionIdManager(session); Have not found a "Stopped scavenging" or any other clue in the logs. Any ideas, what I’m doing wrong or how to narrow down the issue? Best regards, Paul And the server dump … org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED += QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED | +- 401 qtp674483268-401 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method) | +- 402 qtp674483268-402 IDLE TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method) | +- 221 qtp674483268-221 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) | +- 18 qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3 | +- 20 qtp674483268-20 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) | +- 400 qtp674483268-400 RESERVED TIMED_WAITING @ java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method) | +- 21 qtp674483268-21 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) | +- 22 qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} ACCEPTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) prio=3 | +- 23 0:0:0:0:0:0:0:1 2018-01-12 15:36:35.055 GET /About RUNNABLE @ java.base/java.lang.Thread.getStackTrace(Thread.java:1654) | +- 78 qtp674483268-78 SELECTING RUNNABLE @ java.base@9.0.1/sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) | +- jobs += org.eclipse.jetty.server.session.DefaultSessionIdManager@1de47212[worker=node0] - STARTED | +? org.eclipse.jetty.server.session.HouseKeeper@501792b1[interval=33000, ownscheduler=true] - STARTED += ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED | +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED | +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED | += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@b7dd107 - STARTED | | +- java.base@9.0.1/jdk.internal.misc.Unsafe.park(Native Method) | | +- java.base@9.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) | | +- java.base@9.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2104) | | +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1131) | | +- java.base@9.0.1/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:848) | | +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1092) | | +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) | | +- java.base@9.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) | | +- java.base@9.0.1/java.lang.Thread.run(Thread.java:844) | +- org.eclipse.jetty.io.ArrayByteBufferPool@7caa80e4 | += HttpConnectionFactory@cb0ed20[HTTP/1.1] - STARTED | | +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]} | += SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED | | += ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED | | += org.eclipse.jetty.io.ManagedSelector@4a61c4ef id=0 keys=0 selected=0 actions=0 - STARTED | | | += EatWhatYouKill@62e4c0e7/SelectorProducer@1d4ff7ab/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED | | | +- SelectorProducer@1d4ff7ab | | | +- keys | | | | +- sun.nio.ch.WindowsSelectorImpl@3f46ce65 keys=0 | | | +- actions | | += org.eclipse.jetty.io.ManagedSelector@5a02a8a4 id=1 keys=1 selected=0 actions=0 - STARTED | | += EatWhatYouKill@1e02ae9b/SelectorProducer@7c92e05f/PRODUCING/ReservedThreadExecutor@268f5895{s=1/4,p=0}@SelectorManager@ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} - STARTED | | +- SelectorProducer@7c92e05f | | +- keys | | | +- sun.nio.ch.WindowsSelectorImpl@2bdff7b9 keys=1 | | | +- SelectionKey@67dc299b{i=0}->SocketChannelEndPoint@168b474c{/0:0:0:0:0:0:0:1:52677<->/0:0:0:0:0:0:0:1:80,OPEN,fill=-,flush=-,to=119/30000}{io=0/0,kio=0,kro=1}->HttpConnection@5d013a2c[p=HttpParser{s=END,0 of -1},g=HttpGenerator@593eb554{s=START}]=>HttpChannelOverHttp@1c3c3017{r=1,c=false,a=DISPATCHED,uri=//localhost/4p/About?b9=} | | +- actions | += HTTP2CServerConnectionFactory@2fd66ad3[h2c] - STARTED | | +- HttpConfiguration@18c5cd7d{32768/8192,8192/8192,https://:443,[]} | +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:80] | +- qtp674483268-22-acceptor-0@2c9c6f53-ServerConnector@5c86a017{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:80} += ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED | +~ org.eclipse.jetty.server.Server@3aefe5e5[9.4.z-SNAPSHOT] - STARTED | +~ QueuedThreadPool@qtp674483268{STARTED,8<=10<=200,i=2,q=0} - STARTED | += org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@4b9e255 - STARTED | +- org.eclipse.jetty.io.ArrayByteBufferPool@2cc73a0f | += SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED | | += ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED | | += org.eclipse.jetty.io.ManagedSelector@6832fc2f id=0 keys=0 selected=0 actions=0 - STARTED | | | += EatWhatYouKill@3c9696e/SelectorProducer@16cb2c5a/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED | | | +- SelectorProducer@16cb2c5a | | | +- keys | | | | +- sun.nio.ch.WindowsSelectorImpl@5f633211 keys=0 | | | +- actions | | += org.eclipse.jetty.io.ManagedSelector@a9d5c10 id=1 keys=0 selected=0 actions=0 - STARTED | | += EatWhatYouKill@75311ae8/SelectorProducer@3d7b9f39/PRODUCING/ReservedThreadExecutor@63cff0bb{s=0/4,p=0}@SelectorManager@ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} - STARTED | | +- SelectorProducer@3d7b9f39 | | +- keys | | | +- sun.nio.ch.WindowsSelectorImpl@5c443958 keys=0 | | +- actions | += SslConnectionFactory@d6da883{SSL->alpn} - STARTED | | += SslContextFactory@642a7222[provider=null,keyStore=file:///C:/Entw/btkrt/etc/keystore,trustStore=null] trustAll=false | | +- Protocol Selections | | | +- Enabled (size=3) | | | | +- TLSv1 | | | | +- TLSv1.1 | | | | +- TLSv1.2 | | | +- Disabled (size=2) | | | +- SSLv2Hello - ConfigExcluded:'SSLv2Hello' | | | +- SSLv3 - JreDisabled:java.security, ConfigExcluded:'SSLv3' | | +- Cipher Suite Selections | | +- Enabled (size=13) | | | +- SSL_RSA_WITH_3DES_EDE_CBC_SHA | | | +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA | | | +- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 | | | +- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 | | | +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA | | | +- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 | | | +- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 | | | +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA | | | +- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 | | | +- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 | | | +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA | | | +- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 | | | +- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 | | +- Disabled (size=69) | | +- SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- SSL_DHE_DSS_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified | | +- SSL_DHE_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_DH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- SSL_DH_anon_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_RSA_EXPORT_WITH_DES40_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_RSA_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- SSL_RSA_WITH_NULL_MD5 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified | | +- SSL_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 - ConfigExcluded:'.*DSS.*', ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_128_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_128_GCM_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_256_CBC_SHA256 - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_DH_anon_WITH_AES_256_GCM_SHA384 - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDHE_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDHE_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_ECDSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_ECDH_RSA_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_ECDH_anon_WITH_AES_128_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_ECDH_anon_WITH_AES_256_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_ECDH_anon_WITH_NULL_SHA - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | | +- TLS_EMPTY_RENEGOTIATION_INFO_SCSV - ConfigIncluded:NotSpecified | | +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- TLS_KRB5_WITH_3DES_EDE_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigIncluded:NotSpecified | | +- TLS_KRB5_WITH_3DES_EDE_CBC_SHA - JreDisabled:java.security, ConfigIncluded:NotSpecified | | +- TLS_KRB5_WITH_DES_CBC_MD5 - JreDisabled:java.security, ConfigExcluded:'.*MD5.*', ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- TLS_KRB5_WITH_DES_CBC_SHA - JreDisabled:java.security, ConfigExcluded:'.*_DES.*', ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_128_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_128_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_128_GCM_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_256_CBC_SHA - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_256_CBC_SHA256 - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_AES_256_GCM_SHA384 - ConfigIncluded:NotSpecified | | +- TLS_RSA_WITH_NULL_SHA256 - JreDisabled:java.security, ConfigExcluded:'.*NULL.*', ConfigIncluded:NotSpecified | += ALPNServerConnectionFactory@45afc369{[alpn],HTTP/1.1,[]} - STARTED | += HTTP2ServerConnectionFactory@70beb599[h2] - STARTED | | +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]} | += HttpConnectionFactory@4e41089d[HTTP/1.1] - STARTED | | +- HttpConfiguration@7dd64052{32768/8192,8192/8192,https://:443,[SecureRequestCustomizer@35528b51]} | +- sun.nio.ch.ServerSocketChannelImpl[/0:0:0:0:0:0:0:0:443] | +- qtp674483268-18-acceptor-0@ee4204f-ServerConnector@3fa247d1{SSL,[ssl, alpn, h2, http/1.1]}{0.0.0.0:443} += com.bdm.bwebserver.Server$2@190bae5a - STARTED | +> webapps | +- /bc - STARTED | +- /bdm - STARTED += org.eclipse.jetty.server.handler.ErrorHandler@67eb0042 - STARTED +> jdk.internal.loader.ClassLoaders$AppClassLoader@504bae78 +- jdk.internal.loader.ClassLoaders$PlatformClassLoader@17ed40e0 |