[jetty-users] Debugging random HTTP 400 responses

["Stephan Brosinski" <s@xxxxxxxxxxxxx>]

Hi,

I'm stuck on debugging a strange issue were Jetty is responding with HTTP 400 and my application is never seeing those requests.

These errors only started to appear once I removed Nginx which was running in front of that app. A soon as I re-enable Nginx, the errors disappear. So I'm assuming Nginx does some sort of sanitizing and once that is not happening anymore, Jetty sees requests it usually doesn't. Apart from Nginx, there is also an Envoy running in that cluster, and client requests make it successfully through Envoy. I can see the HTTP 400 responses in the Envoy and Jetty request log, but not in our app's request log.

Nginx seemed to have a config which effectively disabled keep-alive (it send a Connection: close Header to Jetty), so I'm also looking at possible keep-alive issues (e.g. input streams not being read fully).

- It's a Java 21 Spring Boot 3.4.2 app with an embedded Jetty 12.0.16
- The majority of requests are fine, a few a day are rejected by Jetty with HTTP 400
- Seems to be mostly POST and PUT requests
- The same requests against the same APIs are successful in 99,9% of the cases
- Added a custom request logger to also log all HTTP headers, which look fine.
- It's running with uri-compliance=LEGACY, since I need to support a few APIs with encoded URIs
- I've experimented which changing HTTP compliance, but it did seem to make a difference and it's at the default setting now
- I've added a compliance violation listener and added logging to it, it didn't show any violations.
- In order to debug possible issues with the app not completely reading request bodies, I've introduced a filter which would read and cache the complete request body, that made no difference.

I've enabled the debug log, it shows this message from HttpConnection:

caught exception HttpConnection@4ea4c5b1::SocketChannelEndPoint@189e06a4[{l=null,r=null,CLOSED,fill=-,flush=-,to=0/360000}{io=0/0,kio=-1,kro=-1}]->[HttpConnection@4ea4c5b1[p=HttpParser{s=CLOSE,0 of -1},g=HttpGenerator@5c5c58c0{s=START}]=>HttpChannelState@42d875c8{handling=null, handled=false, send=SENDING, completed=false, request=null}] HttpChannelState@42d875c8{handling=null, handled=false, send=SENDING, completed=false, request=null}
org.eclipse.jetty.io.RuntimeIOException: Parser is terminated

This only shows up in the debug log, so I'm assuming it's a situation Jetty can deal with somehow. 
Then I've enabled the same debug log for the version with Nginx that doesn't return any HTTP 400 and I also have that log output there. So it's nothing that changed between the two versions.

Also, I've found no way so far to reproduce it locally. Only seems to happen with live traffic.

I'm kind of running out of ideas of what else I could try to identify the issue.
Would you have any other ideas?

Thank you,
Stephan