Re: [jetty-users] Changes in Jetty socket close behavior from 9.2 -> 9.4

[Tommy Becker <twbecker@xxxxxxxxx>]

Thanks for your response. I managed to snag a tcp dump of what's going on in this scenario. From what I can see the sequence of events is the following. Recall that our Jetty server is fronted by a Varnish cache.

1) Varnish sends the headers and initial part of the content for a large POST.

2) On the Jetty server, we use a streaming parser and begin validating the content.

3) We detect a problem with the content and throw an exception that results in a 400 Bad Request to the client (via JAX-RS exception mapper)

4) An ACK is sent for the segment containing the 400 error.

5) The Jetty server sends a FIN.

6) An ACK is sent for the FIN

7) Varnish sends another segment that continues the content from #1.

8) The Jetty server sends a RST.

In the server logs, we see an Early EOF from our JAX-RS resource that is parsing the content. This all seems pretty ok from the Jetty side, and it certainly seems like Varnish is misbehaving here (I'm thinking it may be this bug https://github.com/varnishcache/varnish-cache/issues/2332).  But I'm still unclear as to why this started after our upgrade from Jetty 9.2 -> 9.4. Any thoughts?

On Thu, Sep 20, 2018 at 5:14 AM Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

Hi,

On Wed, Sep 19, 2018 at 4:06 PM Tommy Becker <twbecker@xxxxxxxxx> wrote:
>
> Hi folks,
> We recently upgraded our application from Jetty 9.2 to 9.4. After doing so, we've noticed a lot of Early EOF stacktraces happening on POST requests. Of note, our application is fronted by a Varnish cache, which holds connections to the Jetty backend open indefinitely, even though we have the Jetty idle timeout configured to 50s. We're aware this is sub optimal, but it has been this way for some time, and worked fine under Jetty 9.2. To be clear, we do not have any evidence, or even believe that Jetty is doing anything wrong here per-se. But I understand that a fair bit of work happened in Jetty in this area (timeout/socket close) between 9.2 and 9.4 and I'm trying to get a high level description of what changed in an attempt to explain the behavior we're seeing now. Any info you could provide would be appreciated!
>

There have been many changes between Jetty 9.2 and 9.4 in many places.
As an example, we have improved and made more compliant/strict HTTP
parsing, and that may be the cause of your POSTs not going through.

The best thing you can do to figure out this issue is to grab a
network trace via Wireshark on the Jetty host, along with possibly a
stack trace (that you did not report and you don't even say if it's on
client or on server), and possibly server DEBUG logs.

The network trace will be important to know who's closing the
connection. Stack traces and logs to know why it was closed.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users