| Re: [jetty-users] Changes in Jetty socket close behavior from 9.2 -> 9.4 |
It is more about how the response was generated and less about the response code itself.If the application throws and exception to Jetty during request handling, we now always make the connection non persistent before trying to send a response. If the request input is terminated early or is not fully consumed and would block, then we also abort the connection.Interesting that you say we don't set the Connection: close header. There is actually no requirement to do so as the server can close a connection at any time, but I thought we would do so as a courtesy.... checking....cheersOn Wed, 26 Sep 2018 at 10:25, Tommy Becker <twbecker@xxxxxxxxx> wrote:Thanks Greg. Just so I’m clear, what does Jetty key on to know whether to close the connection? Just the 4xx/5xx response code? I’m trying to understand the difference between this case and the “normal unconsumed input” case you describe. Also, I did notice that Jetty does not set the Connection: close header when it does this, is that intentional?_______________________________________________On Sep 25, 2018, at 6:37 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:Thomas,There is no configuration to avoid this behaviour. If jetty sees and exception in the application it will send the 400 and close the connection.However, as Simone says, your application can be setup to avoid this situation by catching the exception and consuming any input. You can do this in a filter that catches Throwable, it can then check the request input stream (and/or reader) for unconsumed input and read & discard to end of file. If the response is not committed, it can then send a 400 or any other response that you like.Just remember that this may make your application somewhat vulnerable to DOS attacks as it will be easy to hold a thread in that filter slowly consuming data. I would suggest imposing a total time and total data limit on the input consumption.Note that for normal unconsumed input, jetty 9.4 does make some attempt to consume it... but if the reading of that data would block, it gives up and closes the connection, as there is no point blocking for data that will be discarded.regardsOn Wed, 26 Sep 2018 at 07:35, Thomas Becker <twbecker@xxxxxxxxx> wrote:Thanks so much again for your response, this is great information. What you say makes sense, but I now see I failed to mention the most critical part of this problem. Which is that the client never actually sees the 400 response we are sending from Jetty. When varnish sees the RST, it considers the backend request failed and returns 503 Service Unavailable to the client, effectively swallowing our application’s response. We can pursue a solution to this on the Varnish side, but in the interim I’m guessing there is no way to configure this behavior in Jetty?_______________________________________________On Sep 25, 2018, at 4:28 PM, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:Hi,
On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <twbecker@xxxxxxxxx> wrote:
Update: we setup an environment with the old Jetty 9.2 code and this does not occur. 9.2 does not send the FIN in #5 above, and seems happy to receive the rest of the content, despite having sent a response already.
On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <twbecker@xxxxxxxxx> wrote:
Thanks for your response. I managed to snag a tcp dump of what's going on in this scenario. From what I can see the sequence of events is the following. Recall that our Jetty server is fronted by a Varnish cache.
1) Varnish sends the headers and initial part of the content for a large POST.
2) On the Jetty server, we use a streaming parser and begin validating the content.
3) We detect a problem with the content and throw an exception that results in a 400 Bad Request to the client (via JAX-RS exception mapper)
4) An ACK is sent for the segment containing the 400 error.
5) The Jetty server sends a FIN.
6) An ACK is sent for the FIN
7) Varnish sends another segment that continues the content from #1.
8) The Jetty server sends a RST.
In the server logs, we see an Early EOF from our JAX-RS resource that is parsing the content. This all seems pretty ok from the Jetty side, and it certainly seems like Varnish is misbehaving here (I'm thinking it may be this bug https://github.com/varnishcache/varnish-cache/issues/2332). But I'm still unclear as to why this started after our upgrade from Jetty 9.2 -> 9.4. Any thoughts?
This is normal.
In Jetty 9.4 we are more aggressive in closing the connection because
we don't want to be at the mercy of a possible nasty client sending us
GiB of data when we know the application does not want to handle them.
Varnish behavior is correct too: it sees the FIN from Jetty but does
not know that Jetty does not want to read until it tries to send more
content and gets a RST.
At that point, it should relay the RST (or FIN) back to the client.
So you have 2 choices: you catch the exception during your validation,
and finish to read (and discard) the content in the application; or
you ignore the early EOFs in the logs.
I don't think that those early EOFs are logged above DEBUG level, is
that correct?
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users--_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users--