Re: [jetty-users] understanding selector threads in Jetty

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Fri, May 2, 2014 at 1:51 PM, Roland Dragoi <roland.dragoi@xxxxxxxxx> wrote:
> Hi,
>
> I am using the embedded Jetty web server (9.13), for which I configure the
> maximum number of threads in the thread pool (maxThreads).
> By looking into the Jetty code, I noticed that Jetty uses a formula for
> determining the minimum allowed value for maxThreads = 1 + noConnectors*(1 +
> noCores/16 + noCores).

The QTP has a default of 200 threads, no formula.
I can't find a place in the code where Jetty uses the formula you report.

> That is, for every configured connector, Jetty allocates a number of 1 +
> noCores/16 acceptor threads and a number of noCores selector threads.

Correct.

> My problem is that if I use a value for the maxThread property close to the
> minimum accepted by Jetty, the browser blocks when loading the page.

You are misconfiguring the thread pool.

> By monitoring my application on a machine with 80 cores, I noticed that in
> the Jetty thread pool there are 6 acceptor threads (of which only one is
> running at any given time) and 80 selector threads (all of them are running
> continuously in the selector loop). If I configure the maxThreads to the
> minimum accepted of 87, I am only left with 1 thread to actually do the work
> of handling the request and the browser blocks, as many requests stay
> pending. Of course, I could just increase the maxThreads value until it
> works, but the memory resources are limited. In my case I only use one
> connector, but if I will later add a second connector for HTTPS, the number
> of selector threads will double. Also, Jetty uses by default a number of 200
> for the maxThreads property, but I imagine that on a machine with even more
> cores, my problem will become the default.
> I noticed it is possible to also configure the number of acceptors and
> selectors for each connector and I intend to try that, but I would like to
> understand first what they do.

Acceptor threads accepts new connections from client. They run a loop
on a blocking accept() call to accept connections.
With a box with 80 cores, which is not typical, you are recommended to
manually configure the number of acceptors and selectors, since the
guesses we make are probably off.

The right number of acceptor threads is determined by the server load
and the traffic shape, in particular by connection open/close rate.
The higher this rate, the more acceptors you want.

> My question for you is what is exactly that the selector threads do and if
> such a great number of them is really necessary.

Selector threads manage events on connected sockets.

Every time a socket connects, the acceptor threads accepts the
connection and assign the socket to a selector, chosen in a round
robin fashion.
The selector is responsible to detect activity on that socket (I/O
events such as read availability and write availability) and signal
that event to other code in Jetty that will handle the event.
One thread runs one selector. This is basic async I/O using selectors.

If you have a very busy server (say 100k connected clients at any time
or more), you want to "spread" those clients among many selectors, so
that each selector will handle a portion of the connected clients and
be faster in responding to client activity.
Each selector has a limit of 64k connections, so you want to take that
into account.

Sizing the number of selector threads is therefore a function of the
load on the server, the number of cores and traffic shape (how often
clients are active on a connection).
For many many connections, very very active, you want a high number of
selectors.
For many many connections, but not very active, fewer selectors are
typically needed.
For 5k clients for normal HTML pages you can easily go by with 1 selector.

> If I were to decrease their number, is there a recommended ratio between the
> selector threads and the worker threads?

It's not possible to tell without having more information about your
server load and traffic shape.

My recommendation is to leave everything at default, and Jetty will just work.

You did not say how much memory you have on the box, but it is
typically unlikely that hit memory resource limits.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.