Re: [jetty-users] EXTERNAL: Re: jetty client delivers incomplete respons

["Robben, Bert" <Bert.Robben@xxxxxxxxxxxxx>]

> The client knows how many bytes from the `Content-Length` header; if it sees less, it produces an EOF exception.
> If the content is chunked, it expects the terminating chunk; if it does not see it, it produces an EOF exception.

Our server-side resource does not set the Content-Length header. It doesn't do this because it has no idea up-front what the size will be. It traverses a huge data structure and writes that in json-form to the response stream. The data structure is never in-memory in its entirety because the total size can go > 1 Gb. As such, the resource also writes the json piece-by-piece.

In pseudo-code, that would go more or less like this

response.write('[');
for (int part = 0; part < someBigNumber; part++) {
   if (part != 0) response.write(',');
   objectMapper.writeAsJson(producePart(part), response);
}
response.write(']');
response.close();

The real code is more complicated than shown here because it does all of this asynchronously, but it should still give a good idea of what is going on.

I can imagine that if the connection is cut somewhere in the middle of some tcp package that we'll see this as an IOEXception. If the connection is cut at a moment that no data is being transferred (because the server-side resource is e.g. still producing some part) then I can understand that the client is not aware of a problem and doesn't throw.

Would that make sense?

Thanks,

Bert

-----Original Message-----
From: Simone Bordet <sbordet@xxxxxxxxxxx>
Sent: Tuesday, 19 April, 2022 7:03 PM
To: Robben, Bert <Bert.Robben@xxxxxxxxxxxxx>; JETTY user mailing list <jetty-users@xxxxxxxxxxx>
Subject: Re: [jetty-users] EXTERNAL: Re: jetty client delivers incomplete response without exception?

Hi,

On Tue, Apr 19, 2022 at 9:08 AM Robben, Bert <Bert.Robben@xxxxxxxxxxxxx> wrote:
>
> Hi,
>
> I know that the response was incomplete because the test expects a certain amount of bytes (it knows this because it knows what the server should be returning) and it gets only a fraction of the expected amount. In other words, the client received some response, but only a part of it.
>
> This looks to be very much a race condition. This test is part of our daily build so it runs a lot. But this particular failure I've seen only once so far. So I'm afraid I cannot reproduce.
>
> How does the http client know that more data is supposed to come? I can understand that it detects that the connection is closed, but how can it make a difference between "this is it, you got your complete response" and "there is a problem, server needed to send more stuff but there was for some reason a connection issue"?
>

The client knows how many bytes from the `Content-Length` header; if it sees less, it produces an EOF exception.
If the content is chunked, it expects the terminating chunk; if it does not see it, it produces an EOF exception.

So yes could be a Jetty bug, or an application bug (e.g. your application expects just one chunk of content, but when it receives 2 it loses one), that's why we would need some evidence in the form of DEBUG logs.

> We do run in k8s, so there are some hops (k8s services) in between the jetty client and the jetty web server. I have no idea if these can also play a role here.

It could. We don't have a particularly good experience with k8s, although it seems more stable with TCP than with UDP.
Again, no way to tell unless there is evidence (although this would require network traces to be sure).

--
Simone Bordet
----
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcometd.org%2F&amp;data=05%7C01%7Cbert.robben%40fisglobal.com%7C6b9a6bc7a77240137ccf08da22268dbd%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637859846664537035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=oE1wL9ZCbx2Iu1eMnfPUsWWvmdpMPONzKC5cAs5vZFA%3D&amp;reserved=0
https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwebtide.com%2F&amp;data=05%7C01%7Cbert.robben%40fisglobal.com%7C6b9a6bc7a77240137ccf08da22268dbd%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637859846664537035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=OxP8lJHv%2BXNoGW%2BV3DMEVzAnO%2BXglP9ZnvbtzBSWZz4%3D&amp;reserved=0
Developer advice, training, services and support from the Jetty & CometD experts.
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.