Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jet

[Jan Bartel <janb@xxxxxxxxxxx>]

And when you open the bug, please make sure to state which operating
system and version you are using.

thanks
Jan

On 24 November 2013 09:55, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:
> Hi,
>
> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <rototor@xxxxxxxxxx> wrote:
>> Hello everybody,
>>
>> I´m using jetty as standalone embedded web server behind a nginx and it is
>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>> to 9.1.0.v20131115.
>>
>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>> still have the same leaks. Native memory leaks mean, that the process memory
>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>> But to get there the server process has to run for about two weeks.
>>
>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>> be 7 GB.
>>
>> First i thought that the many threads started/stopped in the thread pool are
>> the problem. I specified a idle time of -1, and still many threads are
>> restarted. Also some answers on the web about DirectByteBuffer OOM
>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>
>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>> at some peak times many buffers are used and never freed.  On the other side
>> this doesn’t really fit with the allocated memory. At the moment the server
>> has this direct memory allocations (according to the MBeans info):
>>
>> java.nio
>> BufferPool
>> name=direct
>> -  Count 384
>> -  MemoryUsed 2728821147
>> -  Name direct
>> -  ObjectName java.nio:type=BufferPool,name=direct
>> -  TotalCapacity 2728821147
>>
>> That would mean an medium buffer size of about 7 MB. But the
>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>> kb sized buffers by default ?!
>>
>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>
>> This numbers are after the process is running for about 32 hours. I´ve got
>> about 320 request/minute at peak times. All this numbers are according to
>> the embedded JavaMelody monitoring.
>>
>> The JVM arguments are:
>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>> JDK 1.7.0_45
>>
>> The thread pool is configured this way:
>>
>> tp = new QueuedThreadPool();
>> tp.setMaxThreads(256);
>> tp.setMinThreads(5);
>> tp.setIdleTimeout(-1);
>> tp.setName("Http Server Thread Pool");
>> tp.setDaemon(true);
>>
>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>> hour does not help. Also the i´ve got no other GC problems.
>>
>> Any ideas what could cause this problems?
>> Why are threads restarted in the pool even when i specify a idle timeout of
>> -1? I also regulary schedule some own runnables on the thread pool - but
>> they should not cause this problems, should they? They do in 99.99% of all
>> cases not throw any exceptions.
>
> Seems like a bug.
> I would keep the ThreadPool at default configuration for now, to have
> less variables in the system.
>
> Would you be able to replace usage of ArrayByteBufferPool with
> MappedByteBufferPool and see if the problem persist ?
>
> Also, please file an issue about this, it really looks from JMX that
> there is a problem.
>
> Thanks !
>
> --
> Simone Bordet
> ----
> http://cometd.org
> http://webtide.com
> http://intalio.com
> Developer advice, training, services and support
> from the Jetty & CometD experts.
> Intalio, the modern way to build business applications.
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users



-- 
Jan Bartel <janb@xxxxxxxxxxx>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'