[jetty-users] Problem with ClosedByInterruptException

[John Gardiner Myers <jgmyers@xxxxxxxxxxxxxx>]

I've been having problems with some Jetty client (embedded) instances 
encounter frequent ClosedByInterruptException errors attempting to query 
a different service.

I have four instances in our testing environment that have gotten into 
this state. Three are Jetty 9.3.7, one is Jetty 9.3.6. All are either 
Java 8u72 or 8u73.

The stack trace is:

java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:753)
	at org.eclipse.jetty.io.SelectorManager.finishConnect(SelectorManager.java:337)
	at org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:341)
	at org.eclipse.jetty.io.ManagedSelector.access$900(ManagedSelector.java:56)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:278)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:170)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume
.java:162)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

Looking at AbstractInterruptibleChannel.end(boolean) for 8u92, I see 
what appears to be a bug in NIO. Once an AbstractInterruptibleChannel 
interrupts on a given Thread, all subsequent calls to #end(boolean) from 
that thread will throw ClosedByInterruptException. The line "interrupted 
= null;" in that method has no effect and probably should have been 
"this.interrupted = null;"

I'm not sure what the source of interruptions are, but I do see 
TimeoutException("DNS timeout " + getTimeout() + " ms") exceptions from 
within SocketAddressResolver.Async.resolve() being logged.

So my suspicion is that DNS timeouts might be causing worker threads to 
get into a state where they cannot complete opening connections.

I'd appreciate any suggestions for how to diagnose the problem further 
or otherwise proceed.