[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jetty-dev] Trying to make sense of slow server response times
|
Thanks for your help!
Basically it looks like there are three different 'types' of results: On
some machines there is practically no overhead (measureSumArrayHttp is
hardly any slower than measureSumArray, i.e. the server response is
almost as fast as the raw array computation). On others there is some
constant overhead, like 7ms or so. In the third case the overhead is
proportional to the array size and causes a x2 slow-down of the worker
method.
We have seen servers with the same CPU that show different behavior. So
far it looks like we can only reproduce the problematic second and third
cases on AMD processors (AMD Ryzen 9 5950X, L1/2/3: 512K/8M/64M, 16
cores). It also looks like we can switch between the second and third
case by going from Debian 10 (constant overhead) to Debian 11 (overhead
proportional to array size). All Intel machines we tried seem
unproblematic so far, but we also saw an AMD laptop processor that had
no problems (an AMD Ryzen 7 PRO 5850U) using Ubuntu 21.10.
So far we did not find any patterns of the hardware specifications that
might be the (sole) cause, especially since we see different behavior
using the same hardware. It might be a software issue or a combination
of both the hardware specs and the software we use. Then again I don't
really understand what kind of software changes (other than the JDK) can
affect the performance like this. We are not talking about a slowdown of
a few percent. Basically we do a seemingly trivial task (calculate an
array's sum) that is x2 slower when it is run behind an HttpServlet
using Jetty!
Are there any Jetty settings we could try to gain more information about
the cause of the problem? Or maybe use the Jetty API in some way to
exclude certain features that might be the cause of this?
I did not try to compare context switches yet. Can you recommend any
resources that explain how to do this?
On 08.05.22 10:16, Thomas Becker wrote:
Hi,
first of allL good problem description and summary.
Could you add some details on CPU architecture and other hardware
differences on the machines you‘ve used?
I‘d first try to find some patterns about the CPU architecture (number of
cores, cpu L1/L2/L3 cache sizes, memory bandwith between main memory etc.).
Did you also monitor and compare context switches, etc.?
Finding the root cause of such problems is usually hard.
Cheers,
Thomas
On Sun 8. May 2022 at 10:03, <easbar.mail@xxxxxxxxxx> wrote:
Dear Jetty Devs and Community,
hopefully I've come to the right place to ask for help with the
following issue. I am puzzled by the server response times I am
measuring using a Jetty instance. I reduced the issue such that now
there are only two HttpServlets. One just returns a constant number and
serves as a baseline. The other calculates and returns the sum of all
integers in an array. I also measure the time the actual summation takes.
On some machines everything seems fine: The response time of the servlet
that does the summation is roughly equal to the time the summation is
expected to take. Querying the server only adds a minimal overhead.
But on others there is significant overhead (the servlet response is
much slower than the actual summation), and on some the overhead is even
proportional to the size of the array!
You can find the benchmark, instructions how to run it and a more
detailed explanation of my results (in README.md) here:
https://github.com/easbar/jetty_jmh_benchmark/commit/f32ca12256021c69589a90f460097887ac802740.
I am referencing to this specific commit to avoid confusion in case
there will be changes to the benchmark later.
I have tried using profilers and JVM flags like `-XX:+PrintCompilation`
to find out what causes this slow down, but the server response times I
am measuring still remain a complete mystery to me. They are very
reproducible, i.e. whenever I run the benchmark I pretty much get the
same result, but (qualitatively) different results for different
machines. I also tried different JDK versions (8,11,17), and Jetty 9&11,
but that does not seem to change anything.
Frankly speaking I don't really know what else to try to get to the
bottom of this, which is why I came here.
Any kind of help would be highly appreciated, e.g.
* running the benchmark and posting your results
* looking at the results in README.md and giving some advice what I
could do to debug this, or thinking about what *could* be the cause for
such behavior
Thanks in advance!
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-dev
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-dev
