Re: [jetty-users] EXTERNAL: Re: jetty client delivers incomplete respons

["Robben, Bert" <Bert.Robben@xxxxxxxxxxxxx>]

Thanks for the reply.

 

Since we actively kill the server ourselves (as part of our test), I’m not surprised at all connections are closed prematurely. These exceptions are all expected and what I want.

 

The problem is that for the other request that the test did, there was NO EXCEPTION at all. From the client’s point-of-view, the request finished correctly. The problem was that the response was incomplete. My question is if it is normal behavior for the client to sometimes only get a subset of the response data with NO exceptions being thrown at all.

 

Since this test failed only once in many runs, I think there is some race condition involved.

 

Bert

 

From: jetty-users <jetty-users-bounces@xxxxxxxxxxx> On Behalf Of Joakim Erdfelt
Sent: Thursday, 14 April, 2022 6:08 PM
To: JETTY user mailing list <jetty-users@xxxxxxxxxxx>
Subject: EXTERNAL: Re: [jetty-users] jetty client delivers incomplete response without exception?

 

CAUTION: This email originated from outside of the company. Do not click links or open attachments unless you recognize the sender and know the content is safe.

 

> Caused by: java.io.IOException: Connection reset by peer

>                  at sun.nio.ch.IOUtil.write(IOUtil.java:148)

 

This means the remote side closed the connection prematurely during a write operation.

This is fishy, and I would consider it a red-flag that the remote isn't following the HTTP spec properly.

 

You can verify if all of the data was legitimately sent over the network using a tracing tool like wireshark.

You can also determine how that data was sent (with Content-Length, using Transfer-Encoding: chunked, compressed or not, or using super old school HTTP/1.0 techniques of raw bytes and closure of the connection).

 

Joakim Erdfelt / joakim@xxxxxxxxxxx

 

 

On Thu, Apr 14, 2022 at 10:47 AM Robben, Bert via jetty-users <jetty-users@xxxxxxxxxxx> wrote:

Hey,

 

I had a test failure (race condition, saw it only once) where a jetty client makes a request on a jetty server and only gets an incomplete response back. The jetty client does an asynchronous POST and consumes the response using an InputStreamResponseListener.

 

                try (InputStream inputStream = listener.getInputStream()) {

                    long dataReceived = 0;

                    byte[] buffer = new byte[…];

                    int read = inputStream.read(buffer);

                    while (read >= 0) {

                        dataReceived += read;

                        read = inputStream.read(buffer);

                    }

 

This loop ends without exceptions. However, the amount of data in the response (dataReceived) is > 0, but a lot smaller than expected which makes the test fail.

 

Important to know is that at the moment of this failure, the server was killed (as part of the test). Other requests that were ongoing get exceptions like the ones below. I understand and fully expect these exceptions. What I don’t understand is why the request that was consuming its response didn’t get any.

 

Is it wrong from me to assume I’ll always get some exception in case the server is killed?

 

I assume that our server-side resource is working as it should (so it completes when the entire response has been sent to the output); so this one doesn’t count.

 

Note that the server does NOT fill in the CONTENT_LENGTH of the response; as such the client doesn’t know when the stream will end. Is it possible that the client interprets closure of the response stream sometimes as “ok, this is the end” and doesn’t throw exceptions?

 

 

Here are some of the other exceptions from other (but different) requests:

 

[2022-04-12 03:09:49,154] [ERROR] [] [http-requests] [] - 1649732989067,87,GET,http://test-product:8080/test-product/v1/clustersingleton/clusterinfo/clusterInfoProvider?id=554d75d7-5c16-47e4-95e9-c0b110d1ec40-3484&replyAfter=110,EOFException,"HttpConnectionOverHTTP@48aa14a1::SocketChannelEndPoint@4c95996c{l=/10.233.90.85:35120,r=test-product/10.233.45.8:8080,ISHUT,fill=-,flush=-,to=85/29123}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@48aa14a1(l:/10.233.90.85:35120 <-> r:test-product/10.233.45.8:8080,closed=false)=>HttpChannelOverHTTP@9971e9a(exchange=HttpExchange@20c3f0a6{req=HttpRequest[GET /test-product/v1/clustersingleton/clusterinfo/clusterInfoProvider HTTP/1.1]@38d4f703[TERMINATED/null] res=HttpResponse[null 0 null]@586a478c[PENDING/null]})[send=HttpSenderOverHTTP@69b8e1d5(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@56417a84{s=START}],recv=HttpReceiverOverHTTP@4932f3a0(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]"

 

                                at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:376)

                                at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:181)

                                at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:131)

                                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

                                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

                                at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1614)

 

and

 

                java.io.IOException: org.eclipse.jetty.io.EofException

                                at org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:151)

…

                Caused by: java.io.IOException: Connection reset by peer

                                at sun.nio.ch.IOUtil.write(IOUtil.java:148)

 

This test failure was exceptional; I suspect some race condition on the server.

 

Note: I was using Jetty 9.4.43.v20210629 in this test. (I’ve now upgraded to 9.4.46)

 

Thanks,

 

Bert

The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users

The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.