Hi,
I’ve been trying to get to the bottom of what I believe is a performance problem in a Jetty application. Unfortunately, I’ve hit a road block and could use some help. Googling hasn’t shed any light on the issue. Neither has stackoverflow.
Basically, I’m seeing very high CPU load without a lot of throughput. To quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, and some GC tuning.
We’re using a custom load generation tool to find the point at which the server falls over. However, we’re hitting 90% CPU burn with loads well over 2 while handling ~25 concurrent transactions/second (throughput was 63 tps with ~400ms response times). That just doesn’t seem right (way too low).
In profiling the JVM regardless of load (low or under siege), we can see clearly that 99% of our CPU time is spent in one of two places:
sun.nio.ch.SelectorImpl.select() at 66%
sun.nio.ch.ServerSocketChannelImpl.accept() at 33%
with all of that time being consumed by the acceptor/selector threads dumped below in equal parts.
In looking at the thread dump, we have many threads that look like this:
"qtp665633643-88" - Thread t@88
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <2ef1b67e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
And I see the acceptor/selectors like this:
"qtp665633643-83 Selector0" - Thread t@83
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <7b18010e> (a sun.nio.ch.Util$2)
- locked <23bb2f20> (a java.util.Collections$UnmodifiableSet)
- locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
"qtp1586177080-28 Acceptor0 SelectChannelConnector@0.0.0.0:80" - Thread t@28
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
- locked <58341b55> (a java.lang.Object)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
"qtp1586177080-27 Selector0" - Thread t@27
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <be7202b> (a sun.nio.ch.Util$2)
- locked <2761d665> (a java.util.Collections$UnmodifiableSet)
- locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Thread.java:722)
Locked ownable synchronizers:
- None
Not a single thread in BLOCKED state.
I know I only have a single acceptor configured when I have two processors, but when I do configure two acceptors, I see the two acceptor threads blocking each other (one RUNNABLE while the other BLOCKED and switch back and forth after 10s of seconds).
Any help is appreciated, and I apologize for the long email.
Regards,
Devon Lazarus, Internet Services Group
Sonos, Inc.