Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-dev] Jetty 9.3.15 stopped accepting incoming connections (too many open files?)

hi all,

our production system running Jetty 9.3.15 this morning stopped accepting incoming connection producing the following suspicious log lines:


10:26:52.121|WARN |        o.e.j.s.HttpInput||
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[ ........................... removed application log lines ......................................]

10:43:30.148|WARN |       o.e.j.h.HttpParser||Illegal character 0x16 in state=START for buffer HeapByteBuffer@23931b7a[p=1,l=265,c=8192,r=264]={\x16<<<\x03\x01\x01\x04\x01\x00\x01\x00\x03\x03Xx\xA1B\xF3h\x9a...\x03\x02\x03\x03\x02\x01\x02\x02\x02\x03\x00\x0f\x00\x01\x01>>>,application/xml;...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
10:43:30.148|WARN |       o.e.j.h.HttpParser||bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@6d081ff2{r=0,c=false,a=IDLE,uri=null}
10:43:30.974|WARN |o.e.j.s.AbstractConnector||
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)



We have only just recently upgraded the system to Jetty 9.3.15 and with the previous build we never saw anything like this happening.

And the load on the system wasn't high at all.

I am letting you guys know in case some of you is aware of changes made that could cause this kind a problem.

A simple bounce of the system resolved the problem.


I am now considering a roll-back to the previous Jetty release while I try to gather more data on what happened.


If the problem re-occurs I am planning to attempt to diagnose it by dumping the threads and also by running unix's "lsof"on the jetty process to determine the number of open files.

thanks,
Michele




Back to the top