Hi,
I found there's connections leak in jetty. Environment: CentOS 6 x86_64, JDK 7u25 x86_64, jetty-9.1.0.
When alive connections were growing, I got several stack trace:
jstack at 16:04
"Scheduler-225690927" prio=10 tid=0x00007fcd682cc800 nid=0x5002 runnable [0x00007fcdb1a0c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
- locked <0x0000000609e0cb98> (a java.lang.Object)
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:136)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:104)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:30)
at org.eclipse.jetty.server.HttpInput.getNextContent(HttpInput.java:110)
at org.eclipse.jetty.server.HttpInput.consumeAll(HttpInput.java:245)
- locked <0x0000000608f9bd20> (a org.eclipse.jetty.server.HttpInputOverHTTP)
at org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:327)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:401)
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:238)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1159)
at org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:440)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:92)
at my.web.async.BasicAsyncListener.onTimeout(BasicAsyncListener.java:34)
at org.eclipse.jetty.server.HttpChannelState.expired(HttpChannelState.java:396)
at org.eclipse.jetty.server.HttpChannelState$AsyncTimeout.run(HttpChannelState.java:699)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- <0x00000005b0043ad0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
jstack at 16:12
"Scheduler-225690927" prio=10 tid=0x00007fcd682cc800 nid=0x5002 runnable [0x00007fcdb1a0c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:321)
- locked <0x0000000609e0cbc8> (a java.lang.Object)
- locked <0x0000000609e0cb98> (a java.lang.Object)
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:136)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:104)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:30)
at org.eclipse.jetty.server.HttpInput.getNextContent(HttpInput.java:110)
at org.eclipse.jetty.server.HttpInput.consumeAll(HttpInput.java:245)
- locked <0x0000000608f9bd20> (a org.eclipse.jetty.server.HttpInputOverHTTP)
at org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:327)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:401)
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:238)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1159)
at org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:440)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:92)
at my.web.async.BasicAsyncListener.onTimeout(BasicAsyncListener.java:34)
at org.eclipse.jetty.server.HttpChannelState.expired(HttpChannelState.java:396)
at org.eclipse.jetty.server.HttpChannelState$AsyncTimeout.run(HttpChannelState.java:699)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- <0x00000005b0043ad0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
jstack at 16:53
"Scheduler-225690927" prio=10 tid=0x00007fcd682cc800 nid=0x5002 runnable [0x00007fcdb1a0c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:321)
- locked <0x0000000609e0cbc8> (a java.lang.Object)
- locked <0x0000000609e0cb98> (a java.lang.Object)
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:136)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:104)
at org.eclipse.jetty.server.HttpInputOverHTTP.nextContent(HttpInputOverHTTP.java:30)
at org.eclipse.jetty.server.HttpInput.getNextContent(HttpInput.java:110)
at org.eclipse.jetty.server.HttpInput.consumeAll(HttpInput.java:245)
- locked <0x0000000608f9bd20> (a org.eclipse.jetty.server.HttpInputOverHTTP)
at org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:327)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:401)
at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:238)
at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1159)
at org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:440)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:92)
at my.web.async.BasicAsyncListener.onTimeout(BasicAsyncListener.java:34)
at org.eclipse.jetty.server.HttpChannelState.expired(HttpChannelState.java:396)
at org.eclipse.jetty.server.HttpChannelState$AsyncTimeout.run(HttpChannelState.java:699)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- <0x00000005b0043ad0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Scheduler thread was always locked on an instance of org.eclipse.jetty.server.HttpInputOverHTTP (address is 0x0000000608f9bd20)
I looked at org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:327)
line325 else if (_parser.inContentState() && _generator.isPersistent())
line326 // Complete reading the request
line327 _channel.getRequest().getHttpInput().consumeAll();
I'm not sure what's wrong, clients will not use persistent connections for now, as a workaround.
Any ideas?
Thanks
