[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jetty-users] Changes in Jetty socket close behavior from 9.2 -> 9.4
|
Makes sense, I wondered if that might be the case. I'm still curious how/where it decides to close the connection if it doesn't see the exception.
Not sure if this is the case, but the "Connection: close" header cannot be added by Jetty if the HttpServletResponse.isCommitted() == true before using something like sendError().
We definitely do not see one. But I'm still a bit confused as to how Jetty is determining that it wants to close the connection. Although our JAX-RS resource throws an exception, that exception should be handled by the JAX-RS runtime and not propagated to Jetty (We have ExceptionMappers defined for everything). So our application is generating the response and not Jetty itself.
Thomas,
I see a connection:close header when the connection is closed after sending a 400 or 500.
However, nothing is sent if the connection is closed due to an abort while giving up reading unconsumed content, which can happen before/during/after a response hence we keep that simple.
So are you sure you are seeing a 400/500 response without connection:close ?
It is more about how the response was generated and less about the response code itself.
If the application throws and exception to Jetty during request handling, we now always make the connection non persistent before trying to send a response. If the request input is terminated early or is not fully consumed and would block, then we also abort the connection.
Interesting that you say we don't set the Connection: close header. There is actually no requirement to do so as the server can close a connection at any time, but I thought we would do so as a courtesy.... checking....
cheers
Thanks Greg. Just so I’m clear, what does Jetty key on to know whether to close the connection? Just the 4xx/5xx response code? I’m trying to understand the difference between this case and the “normal unconsumed input” case you describe. Also, I did notice that Jetty does not set the Connection: close header when it does this, is that intentional?
Thomas,
There is no configuration to avoid this behaviour. If jetty sees and exception in the application it will send the 400 and close the connection.
However, as Simone says, your application can be setup to avoid this situation by catching the exception and consuming any input. You can do this in a filter that catches Throwable, it can then check the request input stream (and/or reader) for unconsumed input and read & discard to end of file. If the response is not committed, it can then send a 400 or any other response that you like.
Just remember that this may make your application somewhat vulnerable to DOS attacks as it will be easy to hold a thread in that filter slowly consuming data. I would suggest imposing a total time and total data limit on the input consumption.
Note that for normal unconsumed input, jetty 9.4 does make some attempt to consume it... but if the reading of that data would block, it gives up and closes the connection, as there is no point blocking for data that will be discarded.
regards
Thanks so much again for your response, this is great information. What you say makes sense, but I now see I failed to mention the most critical part of this problem. Which is that the client never actually sees the 400 response we are sending from Jetty. When varnish sees the RST, it considers the backend request failed and returns 503 Service Unavailable to the client, effectively swallowing our application’s response. We can pursue a solution to this on the Varnish side, but in the interim I’m guessing there is no way to configure this behavior in Jetty?
Hi,On Tue, Sep 25, 2018 at 8:34 PM Tommy Becker <twbecker@xxxxxxxxx> wrote:
Update: we setup an environment with the old Jetty 9.2 code and this does not occur. 9.2 does not send the FIN in #5 above, and seems happy to receive the rest of the content, despite having sent a response already.
On Tue, Sep 25, 2018 at 10:01 AM Tommy Becker <twbecker@xxxxxxxxx> wrote:
Thanks for your response. I managed to snag a tcp dump of what's going on in this scenario. From what I can see the sequence of events is the following. Recall that our Jetty server is fronted by a Varnish cache.
1) Varnish sends the headers and initial part of the content for a large POST.
2) On the Jetty server, we use a streaming parser and begin validating the content.
3) We detect a problem with the content and throw an exception that results in a 400 Bad Request to the client (via JAX-RS exception mapper)
4) An ACK is sent for the segment containing the 400 error.
5) The Jetty server sends a FIN.
6) An ACK is sent for the FIN
7) Varnish sends another segment that continues the content from #1.
8) The Jetty server sends a RST.
In the server logs, we see an Early EOF from our JAX-RS resource that is parsing the content. This all seems pretty ok from the Jetty side, and it certainly seems like Varnish is misbehaving here (I'm thinking it may be this bug https://github.com/varnishcache/varnish-cache/issues/2332). But I'm still unclear as to why this started after our upgrade from Jetty 9.2 -> 9.4. Any thoughts?
This is normal.In Jetty 9.4 we are more aggressive in closing the connection becausewe don't want to be at the mercy of a possible nasty client sending usGiB of data when we know the application does not want to handle them.Varnish behavior is correct too: it sees the FIN from Jetty but doesnot know that Jetty does not want to read until it tries to send morecontent and gets a RST.At that point, it should relay the RST (or FIN) back to the client.So you have 2 choices: you catch the exception during your validation,and finish to read (and discard) the content in the application; oryou ignore the early EOFs in the logs.I don't think that those early EOFs are logged above DEBUG level, isthat correct?-- Simone Bordet----http://cometd.orghttp://webtide.comDeveloper advice, training, services and supportfrom the Jetty & CometD experts._______________________________________________jetty-users mailing listjetty-users@xxxxxxxxxxxTo change your delivery options, retrieve your password, or unsubscribe from this list, visithttps://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
--
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxxTo change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
--
--
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
