Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-users] Jetty http2 client request hangs indefinitely


Hi,

   We are trying to implement a http2 jetty client which makes a multiplexed connection with our remote servers. Our program is as follows,

Security.addProvider(new OpenSSLProvider());//Conscrypt for ALPN support in jdk8
SslContextFactory sslContextFactory = new SslContextFactory.Client(true);
sslContextFactory.setProvider("Conscrypt");

sslContextFactory.setProtocol("TLSv1.3");//Use TLSv1.3
HTTP2Client http2Client = new HTTP2Client();
http2Client.setConnectTimeout(5000);
http2Client.setIdleTimeout(30000);

org.eclipse.jetty.client.HttpClient http2HttpClient = new org.eclipse.jetty.client.HttpClient(new HttpClientTransportOverHTTP2(http2Client), sslContextFactory);
http2HttpClient.setConnectTimeout(5000);
http2HttpClient.setIdleTimeout(30000);
http2HttpClient.start();
http2HttpClient.addBean(sslContextFactory);
http2HttpClient.start();

org.eclipse.jetty.client.api.Request request = http2HttpClient.POST(locationurl+locationposturl);
request.header(HttpHeader.CONTENT_TYPE, "application/json");//No I18N
request.header("Pragma", "no-cache");
request.header("Custom", customheaderName);
request.header("Secure", secureHeader);
request.header("protocol", "h2");
request.content(new StringContentProvider(myXMLRequest,"utf-8"));

request.timeout(420, TimeUnit.SECONDS);
request.idleTimeout(30, TimeUnit.SECONDS);

ContentResponse response = request.send();
if (response != null && (response.getStatus() >= 200 && response.getStatus() < 400)) {
int statuscode=response.getStatus();
byte [] wbresponse = response.getContent();
}

We use synchronous calls and conscrypt(security provider) to support ALPN in jdk8 and use TLSv1.3. we are able to make http2 requests successfully to our remote locations. but we face a problem where most of the threads hangs indefinitely until we restart our servers. we are using tomcat 9.x at both ends and each remote servers are behind dedicated HAproxy(used as h2/ssl terminator).

Client : Security provider used: Conscrypt-2.1.0, JAVA: JDK 8, tomcat: 9.0.21, Jetty: 9.4.19

Server: JAVA: JDK 11, tomcat: 9.0.21, Haproxy: 2.0

I checked jetty docs for possible timeout and we have configured all timeout but still we don't know why threads hung indefinitely. We have thread dump for the same,

34534543535345435(DNS) - Tue Aug 13 04:40:06 PDT 2019" #1081 prio=5 os_prio=0 tid=0x00007f93e40d3000 nid=0xd820 waiting on condition [0x00007f91c7cfa000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000d9648be0> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:100)
at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:683)
at com.adventnet.webmon.util.LocationTestUtil.getLocationResponse(LocationTestUtil.java:430)
at com.adventnet.webmon.util.LocationTestUtil.getLocationResponse(LocationTestUtil.java:219)
at com.adventnet.webmon.conf.server.ConfDataCollection.doMultiLocationDC(ConfDataCollection.java:3192)
at com.adventnet.webmon.conf.server.ConfDataCollection.getResponseData(ConfDataCollection.java:2703)
at com.adventnet.webmon.conf.server.ConfDataCollection.invokeDataCollection(ConfDataCollection.java:479)
at com.adventnet.webmon.conf.task.ConfMonitorTask.run(ConfMonitorTask.java:676)
at com.zoho.scheduler.JobWrapper.invokeRunnableJob(JobWrapper.java:116)
at com.zoho.scheduler.instrument.WrappedJobWrapper.invokeRunnableJob(WrappedJobWrapper.java:34)
at com.zoho.scheduler.JobWrapper.executeJob(JobWrapper.java:89)
at com.zoho.scheduler.JobWrapper.run(JobWrapper.java:47)
at com.adventnet.taskengine.inmemory.TaskExecutor$2.call(TaskExecutor.java:113)
at com.adventnet.sas.scheduler.inmemory.WrappedSASExecutorListener.call(WrappedSASExecutorListener.java:33)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The biggest problem is it does not happen in test grids, its only happening in production servers.

Kindly help to fix this problem

TIA

--
With Regards,
Santhosh Kumar J

Back to the top