Re: [jetty-users] Changes in Jetty socket close behavior from 9.2 -> 9.4

[Thomas Becker <twbecker@xxxxxxxxx>]

Thanks so much again for your response, this is great information. What you say makes sense, but I now see I failed to mention the most critical part of this problem. Which is that the client never actually sees the 400 response we are sending from Jetty. When varnish sees the RST, it considers the backend request failed and returns 503 Service Unavailable to the client, effectively swallowing our application’s response. We can pursue a solution to this on the Varnish side, but in the interim I’m guessing there is no way to configure this behavior in Jetty?

On Sep 25, 2018, at 4:28 PM, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

Hi,

On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <twbecker@xxxxxxxxx> wrote:

Update: we setup an environment with the old Jetty 9.2 code and this does not occur. 9.2 does not send the FIN in #5 above, and seems happy to receive the rest of the content, despite having sent a response already.

On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <twbecker@xxxxxxxxx> wrote:

Thanks for your response. I managed to snag a tcp dump of what's going on in this scenario. From what I can see the sequence of events is the following. Recall that our Jetty server is fronted by a Varnish cache.

1) Varnish sends the headers and initial part of the content for a large POST.
2) On the Jetty server, we use a streaming parser and begin validating the content.
3) We detect a problem with the content and throw an exception that results in a 400 Bad Request to the client (via JAX-RS exception mapper)
4) An ACK is sent for the segment containing the 400 error.
5) The Jetty server sends a FIN.
6) An ACK is sent for the FIN
7) Varnish sends another segment that continues the content from #1.
8) The Jetty server sends a RST.

In the server logs, we see an Early EOF from our JAX-RS resource that is parsing the content. This all seems pretty ok from the Jetty side, and it certainly seems like Varnish is misbehaving here (I'm thinking it may be this bug https://github.com/varnishcache/varnish-cache/issues/2332).  But I'm still unclear as to why this started after our upgrade from Jetty 9.2 -> 9.4. Any thoughts?

This is normal.
In Jetty 9.4 we are more aggressive in closing the connection because
we don't want to be at the mercy of a possible nasty client sending us
GiB of data when we know the application does not want to handle them.
Varnish behavior is correct too: it sees the FIN from Jetty but does
not know that Jetty does not want to read until it tries to send more
content and gets a RST.
At that point, it should relay the RST (or FIN) back to the client.

So you have 2 choices: you catch the exception during your validation,
and finish to read (and discard) the content in the application; or
you ignore the early EOFs in the logs.
I don't think that those early EOFs are logged above DEBUG level, is
that correct?

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users