[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jetty-users] Problem with ClosedByInterruptException
|
Hi,
On Sat, Nov 5, 2016 at 12:37 AM, John Gardiner Myers
<jgmyers@xxxxxxxxxxxxxx> wrote:
> I've been having problems with some Jetty client (embedded) instances
> encounter frequent ClosedByInterruptException errors attempting to query a
> different service.
>
> I have four instances in our testing environment that have gotten into this
> state. Three are Jetty 9.3.7, one is Jetty 9.3.6. All are either Java 8u72
> or 8u73.
>
> The stack trace is:
>
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:753)
> at
> org.eclipse.jetty.io.SelectorManager.finishConnect(SelectorManager.java:337)
> at
> org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:341)
> at
> org.eclipse.jetty.io.ManagedSelector.access$900(ManagedSelector.java:56)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:278)
> at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:170)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume
> .java:162)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> Looking at AbstractInterruptibleChannel.end(boolean) for 8u92, I see what
> appears to be a bug in NIO. Once an AbstractInterruptibleChannel interrupts
> on a given Thread, all subsequent calls to #end(boolean) from that thread
> will throw ClosedByInterruptException. The line "interrupted = null;" in
> that method has no effect and probably should have been "this.interrupted =
> null;"
>
> I'm not sure what the source of interruptions are, but I do see
> TimeoutException("DNS timeout " + getTimeout() + " ms") exceptions from
> within SocketAddressResolver.Async.resolve() being logged.
>
> So my suspicion is that DNS timeouts might be causing worker threads to get
> into a state where they cannot complete opening connections.
>
> I'd appreciate any suggestions for how to diagnose the problem further or
> otherwise proceed.
While looking again at this, I don't think it's a big problem, and
JDK-8065720 is probably indeed harmless.
So the DNS lookup fails, so the thread is interrupted and a
TimeoutException reported to the application.
If and when the DNS finally resolves, it will find that *for that
particular socket* the AbstractInterruptibleChannel.interrupted field
is set and throw ClosedByInterruptException.
Although the field is not reset, that socket instance is now dead, and
it was dead since the TimeoutException failed it.
So I don't think that the same pooled thread could affect other sockets.
I think what you were experiencing were just DNS failures.
My guess is that the notification of the TimeoutException raced with
the ClosedByInterruptException, so sometimes you see one, sometimes
the other.
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.