Re: [jetty-users] Jetty 8.0.0 - 8.0.4 not closing connections using Sele

[Mathijs Vogelzang <mathijs@xxxxxxxxxxxxxx>]

Hi Jan,

This is on Ubuntu server 11.04 (Linux 2.6.35.4). I have explicitly
checked (and try to set lower) parameters that should influence this,
such as
/proc/sys/net/ipv4/tcp_fin_timeout, but this didn't help. As soon as I
stop the jetty server, the connections stay around for about 15
seconds,
and then all disappear, this led me to believe there's still something
in jetty that's holding on to the connections.
My current workaround is to send back "Connection: close" with all
requests, which seems to work, but I'd really like to keep connections
open for
a short while in case more requests are sent by the client, and close
them when they're idle for a while (for testing I've set maxIdleTime
to 15000).

It is also not 100% of requests for which this happen, the relevant
debug log for the following connection shows that it's properly closed
after 15 seconds:

265436 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.Server  - REQUEST /api/pb on
org.eclipse.jetty.server.nio.SelectChannelConnector$SelectChannelHttpConnection@3bc6d11a@204.232.---.---:80<->62.140.---.---:-----
265436 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.handler.ContextHandler  - scope null||/api/pb
@ o.e.j.s.ServletContextHandler{/api,null}
265436 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.handler.ContextHandler  - context=/api||/pb @
o.e.j.s.ServletContextHandler{/api,null}
265436 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.servlet.ServletHandler  - servlet /api|/pb|null ->
scm.adserver.ProtobufServiceServlet-928946006
265436 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.servlet.ServletHandler  - chain=null
265438 [qtp161514210-14 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.Server  - RESPONSE /api/pb  200
...
280993 [qtp161514210-11 Selector0] DEBUG org.eclipse.jetty.io.nio  -
destroyEndPoint
SCEP@xxxxxxxxxxxxxxxxxx.channels.SocketChannel[closed][o=false
d=false,io=1,w=true,rb=false,wb=false]
280994 [qtp161514210-11 Selector0] DEBUG
org.eclipse.jetty.server.HttpConnection  - closed
org.eclipse.jetty.server.nio.SelectChannelConnector$SelectChannelHttpConnection@3bc6d11a@204.232.---.---:80<->62.140.---.---:------

But in other cases just nothing happens, even though the initial
request looks exactly the same:

167947 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.Server  - REQUEST /api/pb on
org.eclipse.jetty.server.nio.SelectChannelConnector$SelectChannelHttpConnection@5918cb3a@204.232.---.---:80<->68.55.--.--:45290
167947 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.handler.ContextHandler  - scope null||/api/pb
@ o.e.j.s.ServletContextHandler{/api,null}
167948 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.handler.ContextHandler  - context=/api||/pb @
o.e.j.s.ServletContextHandler{/api,null}
167948 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.servlet.ServletHandler  - servlet /api|/pb|null ->
scm.adserver.ProtobufServiceServlet-928946006
167948 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.servlet.ServletHandler  - chain=null
167949 [qtp161514210-16 - /api/pb?action=UpdateCheckRequest] DEBUG
org.eclipse.jetty.server.Server  - RESPONSE /api/pb  200

(and this connection is in FIN_WAIT2 hours later still). Any other
diagnostic information that would be useful or things I can try apart
from Connection: close ?

Mathijs

On Mon, Oct 31, 2011 at 7:52 AM, Jan Bartel <janb@xxxxxxxxxxx> wrote:
> Mathijs,
>
> What OS is this running on? Seems like it might be one that does not
> have a timeout for FIN_WAIT_2 in the tcp stack, so the sockets might be
> building up and up until you reach the limit.
>
> Is this problem seen with persistent connections? If so, have you
> tried closing the connection on each request?
>
> Finally, do you have any log output that mentions problems with the
> endpoint/connection?
>
> thanks
> Jan
>
>
> On 31 October 2011 02:49, Mathijs Vogelzang <mathijs@xxxxxxxxxxxxxx> wrote:
>> Hi,
>> I'm running an embedded jetty server which has fairly high numbers
>> ofconnections from mobile phones, which oftensuddenly lose connections
>> without properly shutting them down.I noticed that on my server, over
>> time it's building up lots ofconnections that are in FIN_WAIT2 state.
>> This was the casewith 8.0.0, and also with the latest version 8.0.4. I
>> tried both withopenJDK 6 and Sun JDK 6 (all on a linux 64 bit
>> machine).Using a thread dump, I see that jetty is
>> maintainingSelectChannelEndPoint s for these hundreds of idle
>> connections,even though I'm setting maxIdleTime to 15000 (i.e. 15
>> seconds). Someof the _idleTimestamp values are hours old already,but
>> the connection doesn't seem to be cleaned up.Is there some setting I'm
>> overlooking or is this a bug in jetty?
>> Best regards,Mathijs Vogelzang
>> _______________________________________________
>> jetty-users mailing list
>> jetty-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>