We use Jetty (9.2.1) for a high throughput application which responds with a 307 redirect to a Expect 100-continue request. Our average response time seen on the access logs is around 1-4ms. However one off response received by our client is seen to be around 3-5 seconds which is quite high for us. (Say 10 in 40k requests every 5 mins)
Following such request if we take a tcpdump on our host, we notice that although application (from access logs) took 1ms, the tcpdump shows response was returned in 5 seconds. Is it possible that with Jetty's NIO connector the bytes aren't getting flushed fast enough to TCP buffers. We run on a 24 core machine which defaults to 24 selector threads.
We run with following params:
file_descriptor_limit: 1000000
http_accept_count: 4096
idleTimeout: 120000
java_opts: -verbose:gc -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDetails -XX:+PrintGCDateStamps
lowResources__idleTimeout: 500
lowResources__maxConnections: 25000
lowResources__maxLowResourcesTime: 1000
maxThreads: 1000
max_heap_size_mb: 2048
min_heap_size_mb: 2048
Appreciate any pointers on what could be going wrong here. Also are there options to enable a trace on running system, to see what happens on connector side and validate if all selector threads polling channels were busy.
Regards
Monish
