Re: [jetty-users] jetty qtp thread many

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Tue, Feb 21, 2017 at 5:22 AM, alias <524839460@xxxxxx> wrote:
> I started solr, the default use jetty container, but I look through jmx,
> found a lot of jetty qtp thread, why?
> When I do stress tests, Total waiting is getting bigger
>
> Name: qtp1209271652-17
> Status:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39bc254b
> on TIMED_WAITING
> Total number of blocks: 51, Total waiting: 269
>
> Stack trace:
> Sun.misc.Unsafe.park (Native Method)
> Java.util.concurrent.locks.LockSupport.parkNanos (LockSupport.java:215)
> Java.util.concurrent.locks.AbstractQueuedSynchronizer $
> ConditionObject.awaitNanos (AbstractQueuedSynchronizer.java:2078)
> Org.eclipse.jetty.util.BlockingArrayQueue.poll (BlockingArrayQueue.java:392)
> Org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll
> (QueuedThreadPool.java:546)
> Org.eclipse.jetty.util.thread.QueuedThreadPool.access $ 800
> (QueuedThreadPool.java:47)
> Org.eclipse.jetty.util.thread.QueuedThreadPool $ 3.run
> (QueuedThreadPool.java:609)
> Java.lang.Thread.run (Thread.java:745)

This stack trace means that the thread is idle in the Jetty thread pool.


> --------------- The following is also the pressure test when the crawl
>
>
> Status:
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57e34b9e
> on TIMED_WAITING
> Total number of blocks: 1, Total number of waiting: 814
>
> Stack trace:
> Sun.misc.Unsafe.park (Native Method)
> Java.util.concurrent.locks.LockSupport.parkUntil (LockSupport.java:256)
> Java.util.concurrent.locks.AbstractQueuedSynchronizer $
> ConditionObject.awaitUntil (AbstractQueuedSynchronizer.java:2120)
> Org.apache.http.pool.PoolEntryFuture.await (PoolEntryFuture.java:136)
> Org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking
> (AbstractConnPool.java:306)
> Org.apache.http.pool.AbstractConnPool.access $ 000
> (AbstractConnPool.java:64)
> Org.apache.http.pool.AbstractConnPool $ 2.getPoolEntry
> (AbstractConnPool.java:192)
> Org.apache.http.pool.AbstractConnPool $ 2.getPoolEntry
> (AbstractConnPool.java:185)
> Org.apache.http.pool.PoolEntryFuture.get (PoolEntryFuture.java:107)
> Org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection
> (PoolingClientConnectionManager.java:208)
> Org.apache.http.impl.conn.PoolingClientConnectionManager $ 1.getConnection
> (PoolingClientConnectionManager.java:195)
> Org.apache.http.impl.client.DefaultRequestDirector.execute
> (DefaultRequestDirector.java:423)
> Org.apache.http.impl.client.AbstractHttpClient.doExecute
> (AbstractHttpClient.java:882)
> Org.apache.http.impl.client.CloseableHttpClient.execute
> (CloseableHttpClient.java:82)
> Org.apache.http.impl.client.CloseableHttpClient.execute
> (CloseableHttpClient.java:55)
> Org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod
> (HttpSolrClient.java:498)
> Org.apache.solr.client.solrj.impl.HttpSolrClient.request
> (HttpSolrClient.java:262)
> Org.apache.solr.client.solrj.impl.HttpSolrClient.request
> (HttpSolrClient.java:251)
> Org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest
> (LBHttpSolrClient.java:435)
> Org.apache.solr.client.solrj.impl.LBHttpSolrClient.request
> (LBHttpSolrClient.java:387)
> Org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest
> (HttpShardHandlerFactory.java:275)
> Org.apache.solr.handler.component.HttpShardHandler.lambda $ submit $ 0
> (HttpShardHandler.java:198)
> Org.apache.solr.handler.component.HttpShardHandler $$ Lambda $ 107 /
> 1277830087.call (Unknown Source)
> Java.util.concurrent.FutureTask.run (FutureTask.java:266)
> Java.util.concurrent.Executors $ RunnableAdapter.call (Executors.java:511)
> Java.util.concurrent.FutureTask.run (FutureTask.java:266)
> Org.apache.solr.common.util.ExecutorUtil $ MDCAwareThreadPoolExecutor.lambda
> $ execute $ 0 (ExecutorUtil.java:229)
> Org.apache.solr.common.util.ExecutorUtil $ MDCAwareThreadPoolExecutor $$
> Lambda $ 9 / 572523235.run (Unknown Source)
> Java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1142)
> Java.util.concurrent.ThreadPoolExecutor $ Worker.run
> (ThreadPoolExecutor.java:617)
> Java.lang.Thread.run (Thread.java:745)

This stack trace shows that the thread is blocked waiting for a
connection from the HttpClient thread pool.
If you are doing load tests or something similar, the client is
under-configured.

It's all normal from the Jetty point of view.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.