Re: [jetty-users] HttpChannel.java:802 java.io.EOFException: reset

[Simone Bordet <simone.bordet@xxxxxxxxx>]

Hi,

On Tue, Sep 27, 2022 at 12:35 AM Glen Peterson
<glen.k.peterson@xxxxxxxxx> wrote:
>
> Thank you for Jetty.  It is wonderful!  I especially like configuring it in code.
>
> Running Jetty 11.0.11 (possibly also saw this with 11.0.9??)
>
> Our server seems to have partially or completely hung on Aug 19 and again today (Sept 26 2022).  I don't fully understand why.  Right before the "hang" I see a bunch of errors in the logs like:
>
> o.e.j.s.HttpChannel [HttpChannel.java:802] handleException /cov/25.png java.io.EOFException: reset

Are you using HTTP/2?
This is typically reported by the server when the client aborts the request.

> We had adequate disk space (only 16% full), CPUs mostly idle.  I forgot to check memory consumption.  Docker container (adoptopenjdk/openjdk11:alpine) seemed to be running fine.
>
> I was able to reproduce this on my own machine (Ubuntu 18.04) running through IntelliJ, I think with Java 11.  I did it by requesting maybe 3 or 4 thousand images from a singe HTML page.  Sometimes I'd have to click on the page 2 or 3 times to get the hang.
>
> Setting Jetty to DEBUG produced a lot of output, but nothing obviously useful.  EXCEPT: it slowed down the system significantly and wouldn't hang!
>
> Tried connecting jconsole to the running app.  First time loading the Images screen for block-images, all good, but went up very quickly to 146 live threads.  Second time, it looked like a garbage collection happened, and the CPU usage dropped to zero.  See attachment from 4:40PM.  Out of ideas on that for now.
>
> Using Chrome dev tools, in the network tab I can see a request sent , but no response received from the server.

This is normal if the client sent a RST_STREAM for that request, which
appears to be your case.

> On the server side, I see the request sent and even processed (meaning the end of my servlet handle() method, but somehow, nothing is returned by Jetty.  The request should have been written to buffer, and maybe even some of that buffer returned by Jetty by then, but nothing is returned.  Hmm... It looks like maybe the server is hung?
>
> After a minute or two, Chrome's console gave me over 1K errors along the lines of:
> GET https://my.site.com:8443/img/3024.png net::ERR_FAILED Images:3049
> Chrome appears to have crashed too, which doesn't help.
>
> I thought I could do various things on my end to lighten the load, or not goof things up, but nothing I tried made any difference.

Can you take a thread dump via JVM tools (jstack), or better a server
dump as explained here:
https://www.eclipse.org/jetty/documentation/jetty-11/operations-guide/index.html#og-troubleshooting-dump

-- 
Simone Bordet
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz