Re: [jetty-users] Flow control in AsyncMiddleManServlet

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Tue, Feb 19, 2019 at 2:11 AM Raoul Duke <rduke496@xxxxxxxxx> wrote:
> to try to zone in on the above I'm just trying to understand the basic flow control workflow.  lets say I had (say) 1000 clients all sending large HTTP PUTs to the proxy concurrently with large files (10MB, say).
>
> my assumptions are as follows (please correct any that are wrong as that would be a big help):
> * a single socket read of $chunk_size will be performed on each client connection to the extent of the max read size.  (which seems to be 4K by default but correct me if I'm wrong)

Correct.

Note that if your clients send concurrent requests, Jetty at the proxy
will allocate a number of threads to serve those requests
concurrently, where the number of threads depends on the thread pool
configuration.
So if you have the default thread pool of 200 threads, and you have
100 concurrent clients, Jetty will allocate 100 threads (or so) to
serve the concurrent requests.
Note also that AsyncMiddleManServlet is completely asynchronous, so if
those requests upload large files, Jetty will read the request content
asynchronously.
This means that if the clients are "slow" to send content, Jetty will
return idle threads to the thread pool and only use threads when there
is content available.
Think of this as Jetty will use the minimum number of threads to read
the requests but with the largest parallelism possible.

> * each read from above would  then have to be written to the upstream connection (which in my case will have SSL enabled) on a one-to-one basis i.e. one read of $chunk_size will have to be then written to the upstream /before/ the next read can happen on the client socket
> * /OR/ is it the case that many reads can happen on the client socket which are then buffered / queued in the proxy meaning a fast sender and slow upstream can end up with bloat at this level?  if this is the case - then which configuration parameters will infleuence this?

AsyncMiddleManServlet will read a chunk of content, then pass the pair
(chunk, callback) to the application, then stops reading (for that
request).
It is the application that decides _when_ to complete the callback.
When the callback is completed, Jetty will resume reading.
The application can pass the callback to the proxy-to-server side for
the write, and the callback will be completed when the write is
completed.
This allows you to completely control the backpressure towards the client.
If you just want to forward the content, then you just pass the
(chunk, callback) pair to the writing side and you're good: the write
side will be slower to write because of TLS, and the reads will be
slowed down via backpressure, all the way to the client.
If you need to transform the content, then you need to manage the
callback completion yourself at the application level (e.g. maybe
buffer a couple of (chunk, callback) pairs, then transform them, then
write them as a single unit, and when the write is finished succeed
both callbacks).

> * in the case of these reads/writes it looks like jetty has a pool of threads but I assume the thread is only occupied for the length of time the I/O operation takes and is not in any sense blocked on one connection.

Correct, AsyncMiddleManServlet is completely asynchronous.

>  so even if there was only one thread for handling upstream connection  it could still handle (say) 100 client connections by polling for reads/writes as something like libevent would do in the C world.

Behavior WRT to upstream depends on the configuration of HttpClient.
In particular, its thread pool and its maxConnectionsPerDestination.
HttpClient will open at most maxConnectionsPerDestination towards the
server, so the parallelism to can obtain towards one server will
depend on that setting.
On the receiving side (receiving responses from the upstream servers),
HttpClient uses the same mechanism used by Jetty server: it will try
to read concurrently from as many connections as possible without
blocking - within the HttpClient thread pool limits.

Bottom line is that AsyncMiddleManServlet is completely asynchronous,
will try to be as parallel as possible but without ever blocking, and
that backpressure can be controlled by the application (and by default
there is no buffering but backpressure is always applied).

Hope this helped.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.