|Re: [jetty-dev] Trying to make sense of slow server response times|
Thanks for your help!Basically it looks like there are three different 'types' of results: On some machines there is practically no overhead (measureSumArrayHttp is hardly any slower than measureSumArray, i.e. the server response is almost as fast as the raw array computation). On others there is some constant overhead, like 7ms or so. In the third case the overhead is proportional to the array size and causes a x2 slow-down of the worker method.
We have seen servers with the same CPU that show different behavior. So far it looks like we can only reproduce the problematic second and third cases on AMD processors (AMD Ryzen 9 5950X, L1/2/3: 512K/8M/64M, 16 cores). It also looks like we can switch between the second and third case by going from Debian 10 (constant overhead) to Debian 11 (overhead proportional to array size). All Intel machines we tried seem unproblematic so far, but we also saw an AMD laptop processor that had no problems (an AMD Ryzen 7 PRO 5850U) using Ubuntu 21.10.
So far we did not find any patterns of the hardware specifications that might be the (sole) cause, especially since we see different behavior using the same hardware. It might be a software issue or a combination of both the hardware specs and the software we use. Then again I don't really understand what kind of software changes (other than the JDK) can affect the performance like this. We are not talking about a slowdown of a few percent. Basically we do a seemingly trivial task (calculate an array's sum) that is x2 slower when it is run behind an HttpServlet using Jetty!
Are there any Jetty settings we could try to gain more information about the cause of the problem? Or maybe use the Jetty API in some way to exclude certain features that might be the cause of this?
I did not try to compare context switches yet. Can you recommend any resources that explain how to do this?
On 08.05.22 10:16, Thomas Becker wrote:
Hi, first of allL good problem description and summary. Could you add some details on CPU architecture and other hardware differences on the machines you‘ve used? I‘d first try to find some patterns about the CPU architecture (number of cores, cpu L1/L2/L3 cache sizes, memory bandwith between main memory etc.). Did you also monitor and compare context switches, etc.? Finding the root cause of such problems is usually hard. Cheers, Thomas On Sun 8. May 2022 at 10:03, <easbar.mail@xxxxxxxxxx> wrote:Dear Jetty Devs and Community, hopefully I've come to the right place to ask for help with the following issue. I am puzzled by the server response times I am measuring using a Jetty instance. I reduced the issue such that now there are only two HttpServlets. One just returns a constant number and serves as a baseline. The other calculates and returns the sum of all integers in an array. I also measure the time the actual summation takes. On some machines everything seems fine: The response time of the servlet that does the summation is roughly equal to the time the summation is expected to take. Querying the server only adds a minimal overhead. But on others there is significant overhead (the servlet response is much slower than the actual summation), and on some the overhead is even proportional to the size of the array! You can find the benchmark, instructions how to run it and a more detailed explanation of my results (in README.md) here: https://github.com/easbar/jetty_jmh_benchmark/commit/f32ca12256021c69589a90f460097887ac802740. I am referencing to this specific commit to avoid confusion in case there will be changes to the benchmark later. I have tried using profilers and JVM flags like `-XX:+PrintCompilation` to find out what causes this slow down, but the server response times I am measuring still remain a complete mystery to me. They are very reproducible, i.e. whenever I run the benchmark I pretty much get the same result, but (qualitatively) different results for different machines. I also tried different JDK versions (8,11,17), and Jetty 9&11, but that does not seem to change anything. Frankly speaking I don't really know what else to try to get to the bottom of this, which is why I came here. Any kind of help would be highly appreciated, e.g. * running the benchmark and posting your results * looking at the results in README.md and giving some advice what I could do to debug this, or thinking about what *could* be the cause for such behavior Thanks in advance! _______________________________________________ jetty-dev mailing list jetty-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-dev_______________________________________________ jetty-dev mailing list jetty-dev@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-dev
Back to the top