Re: [jetty-users] Jetty http2 client request hangs indefinitely

[Santhosh Kumar <santhosh89j@xxxxxxxxx>]

Hi,

>This is typically way too large. Is there any reason you want this
large total timeout?

We have operations that may 5-10 mins, however threads hung indefinitely even beyond this timeout and even shorter timeouts like 180 seconds are also hanging.

>>From where are you using the client? From a remote client, or from
within a web application deployed in Tomcat?

Yes our client runs in an application deployed inside tomcat which makes calls to remotely hosted servers.

>With "restart our servers" you mean restart the remote servers that
the client calls?

Not the remote servers, we restart appserver where jettyclient is running.

>A schematic diagram of client, load balancer, servers, etc will be
great to understand how is the system.

https://imgur.com/bxV3BA9

>The thread dump does not show much.
You will have better luck registering HttpClient instances in JMX and
the performing a HttpClient dump via JMX.
This is how you do it:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.20.v20190813/jetty-client/src/test/java/org/eclipse/jetty/client/jmx/HttpClientJMXTest.java
Once you are blocking again, you connect with a JMX console, find the
HttpClient MBean, and invoke its dump() operation to get the
HttpClient internal state.
Then attach it here for analysis.

I cannot use JMX in live production servers, so I just called httpclient.dump() while it was hanging using a local servlet call,  FYKI we instantiate a static httpclient and it will be used  to establish connection with many remote servers  in a multithreaded environment. PFA links below for client dumps

https://pastebin.com/N017wph4

https://pastebin.com/k21hzLwL

>Do you have the same problem if you don't use OpenSSL, but the
standard JDK TLS implementation?

Our framework does not support JDK 9 or above so we are dependent on conscrypt(OPENSSL) for alpn support in jdk8.

Kidnly let us know if any other details are required 

On Wed, Aug 21, 2019 at 9:29 PM Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

Hi,

On Wed, Aug 21, 2019 at 11:42 AM Santhosh Kumar <santhosh89j@xxxxxxxxx> wrote:
> We are trying to implement a http2 jetty client which makes a multiplexed connection with our remote servers. Our program is as follows,

[snip]

The code is correct, apart perhaps for this:

> request.timeout(420, TimeUnit.SECONDS);

This is typically way too large. Is there any reason you want this
large total timeout?

> We use synchronous calls and conscrypt(security provider) to support ALPN in jdk8 and use TLSv1.3. we are able to make http2 requests successfully to our remote locations. but we face a problem where most of the threads hangs indefinitely until we restart our servers.

You have to explain this a bit better.
>From where are you using the client? From a remote client, or from
within a web application deployed in Tomcat?
With "restart our servers" you mean restart the remote servers that
the client calls?
A schematic diagram of client, load balancer, servers, etc will be
great to understand how is the system.

> I checked jetty docs for possible timeout and we have configured all timeout but still we don't know why threads hung indefinitely. We have thread dump for the same,

The thread dump does not show much.
You will have better luck registering HttpClient instances in JMX and
the performing a HttpClient dump via JMX.
This is how you do it:
https://github.com/eclipse/jetty.project/blob/jetty-9.4.20.v20190813/jetty-client/src/test/java/org/eclipse/jetty/client/jmx/HttpClientJMXTest.java
Once you are blocking again, you connect with a JMX console, find the
HttpClient MBean, and invoke its dump() operation to get the
HttpClient internal state.
Then attach it here for analysis.

Do you have the same problem if you don't use OpenSSL, but the
standard JDK TLS implementation?

--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users

--

With Regards,
Santhosh Kumar J