Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » How to properly limit net4j's thread pool
How to properly limit net4j's thread pool [message #1795103] Fri, 14 September 2018 14:50 Go to next message
Linuxhippy Mising name is currently offline Linuxhippy Mising nameFriend
Messages: 36
Registered: July 2009
Member
Hi,

In a real-world net4j/cdo application, we see occasional thread-spikes. While the application typically runs well below <150 threads total, sometimes we see spikes up to ~2000 threads - which is already a bit concerning, keeping in mind the spikes seem random.

I found out those threads are created by net4j, however when looking at the thread-dump they are already lurking in the ThreadPool, just to be released soon after.

To limit those spikes, this is what I currently do during setup:
			ExecutorService o = ExecutorServiceFactory.get(container);
			try {
				Method coreMethod = ThreadPoolExecutor.class.getMethod("setCorePoolSize", new Class[] { int.class});
				Method maxMethod = ThreadPoolExecutor.class.getMethod("setMaximumPoolSize", new Class[] { int.class});
				Proxy.getInvocationHandler(o).invoke(o, coreMethod, new Object[] {1});
				Proxy.getInvocationHandler(o).invoke(o, maxMethod, new Object[] {2});
			} catch (Throwable thrbl) {
				throw new RuntimeException(thrbl);
			}


Therefore my question:
* Is it harmful to limit net4j's ThreadPool to a certain size?
* Is there a proper way, instead of using ugly reflection trickeria?

Thank you in advance
Re: How to properly limit net4j's thread pool [message #1795121 is a reply to message #1795103] Sat, 15 September 2018 08:16 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6444
Registered: July 2009
Senior Member
I don't think that limiting the thread pool size to something less than Integer.MAX_VALUE is harmful per se. Of course you'll need to test your configuration.

If you use your own IManagedContainer you can contribute a custom TransportInjector instead of the default one. If instead you want to continue to use the default IPluginContainer.INSTANCE, you should modify it very early in the bootstrap phase of your client or server:

    List<IElementProcessor> postProcessors = IPluginContainer.INSTANCE.getPostProcessors();
    for (Iterator<IElementProcessor> it = postProcessors.iterator(); it.hasNext();)
    {
      IElementProcessor elementProcessor = it.next();
      if (elementProcessor instanceof TransportInjector)
      {
        it.remove();
        break;
      }
    }

    TransportInjector transportInjector = new TransportInjector()
    {
      @Override
      protected ExecutorService getExecutorService(IManagedContainer container)
      {
        return (ExecutorService)container.getElement(ExecutorServiceFactory.PRODUCT_GROUP, ExecutorServiceFactory.TYPE, "10:1000:60");
      }
    };

    postProcessors.add(transportInjector);


If you submit a bugzilla I can also add some system properties to override the default values (type and description) of the thread pool and the buffer provider...


Re: How to properly limit net4j's thread pool [message #1796491 is a reply to message #1795121] Mon, 15 October 2018 08:40 Go to previous messageGo to next message
Linuxhippy Mising name is currently offline Linuxhippy Mising nameFriend
Messages: 36
Registered: July 2009
Member
Hello Eike,

Thanks a lot for your suggestion and the confirmation that limiting the thread pool's size should be harmless.

I tried to dig a little deeper and injected my own ThreadPool implementation which prints a stack trace in case a certain threshold is reached, and it seems the thread spikes we see are linked to View invalidation.
All situations with many threads in the pool (>500) show the following stack trace:
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1329)
at DebugThreadPool.newThread(DebugThreadPool.java:192)
at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:612)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.net4j.util.lifecycle.LifecycleUtil$Delegator.invoke(LifecycleUtil.java:454)
at com.sun.proxy.$Proxy5.execute(Unknown Source)
at org.eclipse.net4j.util.concurrent.ExecutorWorkSerializer.startWork(ExecutorWorkSerializer.java:98)
at org.eclipse.net4j.util.concurrent.ExecutorWorkSerializer.addWork(ExecutorWorkSerializer.java:64)
at org.eclipse.emf.internal.cdo.session.CDOSessionImpl$Invalidator.scheduleInvalidations(CDOSessionImpl.java:1869)
at org.eclipse.emf.internal.cdo.session.CDOSessionImpl$Invalidator.reorderInvalidations(CDOSessionImpl.java:1861)
at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidate(CDOSessionImpl.java:1100)
at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.handleCommitNotification(CDOSessionImpl.java:964)
at org.eclipse.emf.cdo.internal.net4j.protocol.CommitNotificationIndication.indicating(CommitNotificationIndication.java:39)
at org.eclipse.emf.cdo.internal.net4j.protocol.CDOClientIndication.indicating(CDOClientIndication.java:74)
at org.eclipse.net4j.signal.Indication.doExtendedInput(Indication.java:57)
at org.eclipse.net4j.signal.Signal.doInput(Signal.java:377)
at org.eclipse.net4j.signal.Indication.execute(Indication.java:51)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:283)
at org.eclipse.net4j.signal.Signal.run(Signal.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Should I file a bug report about this issue, too?

Best regards, Clemens
Re: How to properly limit net4j's thread pool [message #1796800 is a reply to message #1796491] Fri, 19 October 2018 05:41 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6444
Registered: July 2009
Senior Member
I'd need more details to judge whether your "thread spikes" should be considered a bug. How is the server configured? How many clients are connected concurrently and what are they doing? How many threads were in the thread pool before the clients started doing that and how many threads are newly created? Do the spikes you observe cause any other noticeable problems/exceptions?

Re: How to properly limit net4j's thread pool [message #1797248 is a reply to message #1796800] Mon, 29 October 2018 06:42 Go to previous message
Linuxhippy Mising name is currently offline Linuxhippy Mising nameFriend
Messages: 36
Registered: July 2009
Member
Hello Eike,

Thanks a lot for your ongoing help/advice for using CDO!
I am not exactly sure how the load looks like CDO is exposed to - actually there are just a few users, but it might be some external batch jobs cause the spikes.
I've limited the thread-pool's size to 250 and haven't experienced any issues, so I'll leave it as-is for now.

Please find the bug report about a configureable CDO thread pool here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=540552

Thanks again and best regards, Clemens
Previous Topic:Separate workflow project for emf code generation
Next Topic:Duplicate references
Goto Forum:
  


Current Time: Wed Sep 18 23:53:35 GMT 2019

Powered by FUDForum. Page generated in 0.01413 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top