Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] FW: avoiding earlyEOF

On 6/20/2018 11:23 PM, Robben, Bert wrote:
> The problem that we face is that we regularly see IOExceptions
> exceptions occurring in the communication between these components. See
> stacktrace below. These IOExceptions always contain the earlyEOF. A long
> time ago I already posted a similar message on this forum (see
> https://dev.eclipse.org/mhonarc/lists/jetty-users/msg07965.html). I
> followed the advice mentioned, upgraded to the latest version and
> explicitly set a different idleTimeout on both the server and the client.

I'm not an expert by any means.  Starting off with that in case I say
something wrong.

I come from the Apache Solr project, which uses Jetty.  We see
EOFException quite a lot on the solr-user mailing list.

In my experience with Solr, this is almost always caused by the client
disconnecting (usually due to TCP socket timeout) before the server has
completed the request, because the user has set their socket timeout too
low on the client.  When the server finally does try to respond,
EOFException is logged, because the connection is gone.  If the client
is the one logging the exception, then the server may have closed the
connection, followed by the client trying to send more data, and failing.

I am not seeing any direct mention of what you are setting the idle
timeout to, but I do see this in your message:  "so it’s clear to me
that nothing is idle for a second".  In my opinion, setting socket or
idle timeouts to low numbers is asking for problems.  Going as low as
one second will be extremely likely to lead to timeout issues.

I do understand the value of having these timeouts, but the timeout
needs to be significantly longer than you expect the requests to
actually take, because there may be situations where requests do take
longer than you expect them to.

Java software is prone to experiencing noticeable GC pauses, especially
as the heap size grows.  I have seen pauses of 10-15 seconds happen with
an 8GB heap unless garbage collection is extensively tuned to avoid full
GC.  For Solr, an 8GB heap could actually be quite small.

If there is no GC tuning beyond setting which collector to use, even the
G1 collector, which is Oracle's best option for low-pause collection,
will not avoid full GCs effectively.  It is the full GC that causes the
most problems with pauses.  GC tuning is very much an art form, and
settings that work well for one application may produce awful results on
another.

Let's say that you expect all requests to complete in 10 milliseconds or
less.  So you set your timeout to 1 second, thinking that's always going
to be plenty of time.  But then your application fills up its 2GB heap
right in the middle of handling one of those requests, and the resulting
garbage collection pauses the JVM for two seconds.  The entity at the
other end of the connection is going to give up and close the connection
before the program experiencing the GC pause can respond.  Tuning
garbage collection to reduce GC pauses is certainly a good idea, but if
the timeout were 10 seconds instead of one second, it probably would not
have had any problem.

Thanks,
Shawn


Back to the top