Re: [jetty-users] Configuring separate keep-alive and idle timeouts to i

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Fri, Sep 6, 2024 at 8:45 PM Steven Schlansker via jetty-users
<jetty-users@xxxxxxxxxxx> wrote:
>
> Happy Friday Jetty friends,
>
> We run a Jetty based web application behind Google Cloud Classic Load Balancer
> We experience a small, but ongoing, volume of 502 errors from the load balancer - "backend_connection_closed_before_data_sent_to_client"
>
> Investigating via various internet searches[1], my current hypothesis is a race between the LB and Jetty -
> with an open keep-alive connection (the default / always), the LB is sending a request to Jetty, and at the same time Jetty is
> closing the connection due to idle timeout. Then the LB gets the closed connection, reports 502, and Jetty does not receive the
> request at all since the socket is closed.
>
> Digging into the LB configuration, I find this gem [2]:
>
> > The backend keepalive timeout is fixed at 10 minutes (600 seconds) and cannot be changed. The load balancer's backend keepalive timeout should be less than the keepalive timeout used by software running on your backends. This avoids a race condition where the operating system of your backends might close TCP connections with a TCP reset (RST). Because the backend keepalive timeout for the load balancer is not configurable, you must configure your backend software so that its HTTP keepalive (TCP idle) timeout value is greater than 600 seconds.
>
> After emitting a stream of unprintable curse words - 10 minute non-configurable timeout? - I start to investigate following their recommendation, and increase the Jetty keep-alive timeout.
> They recommend directives like apache: "KeepAliveTimeout 620" or nginx: "keepalive_timeout 620s;"
>
> However, as far as I can tell, Jetty does not distinguish between keepalive and idle timeouts.
>
> I am hoping to avoid increasing the "normal" idle timeout. If a client connects to us, sends one byte, and then loses internet - we want to shut the connection reasonably quickly.
> I fear if we increase this too much we open ourselves to easily run into denial of service situations.

For Jetty this is not a problem, and it likely won't even occupy more memory.
Jetty will wait threadlessly until more bytes will arrive, exactly
like it would do if no bytes have arrived.

> However, if there is no active request, and we are just in a keep-alive state, then we would like to configure that timeout to be >10m as required by our cloud provider.
>
> Is it possible with Jetty to have separate short during-request idle timeout, and long no-request keepalive timeout?
> If not, would this be relatively straightforward to add? Any other suggestions for possible fixes?

It's currently not supported, and I would just rely on the idle timeout for now.
Open an issue with the cloud provider -- "cannot be changed" seems
like "who needs more than 640 KiB of RAM".

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.