| [tyrus-dev] Sporadic client side disconnects |
I'm using Tyrus client to connect to multiple WebSocket servers (around 10 connections) in parallel. The servers publish a lot of messages (hundreds of millions of messages per day). At some point, one of the connections begins to experience client side disconnects. The `onClose` handlers is invoked with close reason `CloseReason[1006,Closed abnormally.]`. The `onError` handler is not invoked. After the error, the connection is re-established using the `ReconnectHandler`. Couple of facts regarding those disconnects/reconnects: - It's always the connection to the same server that has issues. All other connections behave as expected. - When I estalish the connection to just that one server that is having issues then no disconnects take place - The first disconnect occurs after about an hour after the start of the application - Then disconnects happen every few minutes I also have the stack trace from the onError handler: 2020-11-16 16:23:03,347 3985311 [Grizzly(2)] INFO MyEndpoint - java.lang.Throwable at MyEndpoint.onClose(MyEndpoint.scala:185) at org.glassfish.tyrus.core.TyrusEndpointWrapper.onClose(TyrusEndpointWrapper.java:1235) at org.glassfish.tyrus.core.TyrusWebSocket.onClose(TyrusWebSocket.java:110) at org.glassfish.tyrus.core.ProtocolHandler.close(ProtocolHandler.java:481) at org.glassfish.tyrus.core.TyrusWebSocket.close(TyrusWebSocket.java:244) at org.glassfish.tyrus.client.TyrusClientEngine$2$1.close(TyrusClientEngine.java:613) at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter$CloseTask.execute(GrizzlyClientFilter.java:470) at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:91) at org.glassfish.tyrus.container.grizzly.client.TaskProcessor.processTask(TaskProcessor.java:68) at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleClose(GrizzlyClientFilter.java:197) at org.glassfish.grizzly.filterchain.ExecutorResolver$4.execute(ExecutorResolver.java:50) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:248) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:181) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:121) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:99) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) at org.glassfish.grizzly.nio.NIOConnection.preClose(NIOConnection.java:819) at org.glassfish.grizzly.nio.transport.TCPNIOConnection.preClose(TCPNIOConnection.java:78) at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:557) at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:249) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(TCPNIOTransport.java:596) at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:59) at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:133) at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:919) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:248) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:181) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:121) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:99) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:34) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) at java.base/java.lang.Thread.run(Thread.java:834) Here is (roughly) the code I'm using import jakarta.websocket._ class MyEndpoint extends Endpoint with LazyLogging { def onOpen(session: Session, config: EndpointConfig): Unit = { session.addMessageHandler(new MessageHandler.Whole[String] { def onMessage(message: String): Unit = { // handle message } override def onError(session: Session, thr: Throwable): Unit = { logger.error(s"error occurred", thr) super.onError(session, thr) } override def onClose(session: Session, closeReason: CloseReason): Unit = { logger.error(s"closing due to $closeReason") logger.info(ExceptionUtils.getStackTrace(new Throwable())) super.onClose(session, closeReason) } } } } |