Skip to main content



      Home
Home » Modeling » TMF (Xtext) » Standalone setup parsing threadsafety
Standalone setup parsing threadsafety [message #1799992] Tue, 18 December 2018 11:41 Go to next message
Eclipse UserFriend
I have a question regarding standalone setup and thread-safety of parsing.

Here's the code I use for parsing String input:
Injector injector = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration();
IResourceFactory resourceFactory = injector.getInstance(IResourceFactory.class);
Provider<XtextResourceSet> xtextResourceSetProvider = injector.getInstance(new Key<Provider<XtextResourceSet>>() {});

XtextResourceSet resourceSet = xtextResourceSetProvider.get();

Resource resource = resourceFactory.createResource(URI.createURI(UUID.randomUUID() + ".mydsl"));
resourceSet.getResources().add(resource);
resource.load(new StringInputStream(body), null);
final EObject astRoot = resource.getContents().isEmpty() ? null : resource.getContents().get(0);


Question is - what of this can be reused with regard to thread safety? Are IResourceFactory and Provider<XtextResourceSet> thread-safe?
I suppose they should be, but I'm not sure.

Also is XtextResourceSet thread safe? I have doubts about that one, but maybe someone has at least a general idea whether it should be thread safe or not.
Re: Standalone setup parsing threadsafety [message #1799993 is a reply to message #1799992] Tue, 18 December 2018 11:48 Go to previous messageGo to next message
Eclipse UserFriend
Xtext and EMF are not thread safe per se.
Re: Standalone setup parsing threadsafety [message #1800002 is a reply to message #1799993] Tue, 18 December 2018 13:24 Go to previous messageGo to next message
Eclipse UserFriend
Hi

What do you want thread-safety to make safe?

If you actually want to make Xtext parsing multi-threaded then you need to do probably significant research and redesign.

But if you just want to do multiple parses, make sure you use a new thread per parse and more generally for each EMF resource creation. Only consider allowing multi-thread access for immutable Resources and their contents. Google EMF thread safety for a variety of recommendations on making EMF's marginal thread safety useful.

Regards

Ed Willink
Re: Standalone setup parsing threadsafety [message #1800099 is a reply to message #1800002] Wed, 19 December 2018 17:40 Go to previous messageGo to next message
Eclipse UserFriend
Ok, I've listened to you guys and created a commons pool for my parsers that would provide me with a separate parser every time I need it, using the code above (creating new instance of Injector and so on).

But I've run into a very strange problem.

I'm also having Xtext Servlet in the same project - to provide web editor (obviously). And then after editing in web is done, I want to submit final data for parsing - so I call pool which instantiates me the MyDslStandaloneSetup, creates Injector and the rest of the parsing stack (see code above).

After I do that, Xtext Servlet stops working.
Caused by: java.lang.ArrayIndexOutOfBoundsException: 13825
	at com.google.inject.internal.asm.$ClassReader.readClass(Unknown Source) ~[guice-3.0.jar:na]
	at com.google.inject.internal.asm.$ClassReader.accept(Unknown Source) ~[guice-3.0.jar:na]
	at com.google.inject.internal.asm.$ClassReader.accept(Unknown Source) ~[guice-3.0.jar:na]
	at com.google.inject.internal.util.$LineNumbers.<init>(LineNumbers.java:62) ~[guice-3.0.jar:na]
	at com.google.inject.internal.util.$StackTraceElements$1.apply(StackTraceElements.java:36) ~[guice-3.0.jar:na]
	at com.google.inject.internal.util.$StackTraceElements$1.apply(StackTraceElements.java:33) ~[guice-3.0.jar:na]
	at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:549) ~[guice-3.0.jar:na]
	... 51 common frames omitted


This is puzzling to say the least.
I was supposed to have completely independent instances of parsers, and why would they in any way affect XtextServlet is beyond me.

Does anyone have a slightliest idea what is happening here?

[Updated on: Wed, 19 December 2018 17:41] by Moderator

Re: Standalone setup parsing threadsafety [message #1800110 is a reply to message #1800099] Thu, 20 December 2018 00:57 Go to previous messageGo to next message
Eclipse UserFriend
This looks like improper Java 8 support of guice 3 and bad error messages in its exception handling

So it seems you somewhere have a binding issue
Re: Standalone setup parsing threadsafety [message #1800112 is a reply to message #1800110] Thu, 20 December 2018 00:59 Go to previous messageGo to next message
Eclipse UserFriend
https://github.com/eclipse/xtext-core/issues/393
Re: Standalone setup parsing threadsafety [message #1800113 is a reply to message #1800112] Thu, 20 December 2018 01:06 Go to previous messageGo to next message
Eclipse UserFriend
Hi

A pool of parsers is worth a try, but you run significant risks that cached context may cause one 'read-only' usage to conflict obscurely with another 'read-write' usage. You may need a fresh parser each time.

Regards

Ed Willink
Re: Standalone setup parsing threadsafety [message #1800117 is a reply to message #1800112] Thu, 20 December 2018 03:43 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Thu, 20 December 2018 05:59
https://github.com/eclipse/xtext-core/issues/393


I'm not sure what you're trying to say here - Guice3 doesn't work with Java8, and since Xtext is using Guice3 Xtext also doesn't work with Java8?

That's quite absurd.

I'm not sure if you realize that it is not the Guice that's the culprit here - the exception occurs not in the Guice but in one of the components it tries to instantiate, supposedly in an instance initializer block or in static initializer. Guice is just bad in providing stacktrace for such errors.
Re: Standalone setup parsing threadsafety [message #1800118 is a reply to message #1800117] Thu, 20 December 2018 03:45 Go to previous messageGo to next message
Eclipse UserFriend
no i say: with java 8 the guice error messages suck. => you likely have some binding errors or errors during injection and guice fails to give you a proper error message
Re: Standalone setup parsing threadsafety [message #1800120 is a reply to message #1800110] Thu, 20 December 2018 04:14 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Thu, 20 December 2018 05:57
you somewhere have a binding issue

I don't have a binding issue - XtextServlet has a binding issue.
None of the code I wrote does any Guice bindings - I'm using Xtext generated Guice modules. I think I've posted the code which should make it pretty clear.

Christian Dietrich wrote on Thu, 20 December 2018 08:45
you likely have some binding errors or errors during injection and guice fails to give you a proper error message

Yes, Guice is not reporting errors properly. But that's beside the point.

Question is - why are the errors occurring at all.

Again, I'm using the binding code generated by Xtext, not a code I wrote on my own.

[Updated on: Thu, 20 December 2018 04:15] by Moderator

Re: Standalone setup parsing threadsafety [message #1800121 is a reply to message #1800120] Thu, 20 December 2018 04:24 Go to previous messageGo to next message
Eclipse UserFriend
P.S. Here are the errors occurring in Guice found out during debug:
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.IContentAssistParser was bound.
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.internal.Lexer annotated with @com.google.inject.name.Named(value=org.eclipse.xtext.ui.editor.contentassist.antlr.internal.Lexer.CONTENT_ASSIST) was bound.
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.IContentAssistParser was bound.
No implementation for java.util.concurrent.ExecutorService was bound.
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.IContentAssistParser was bound.
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.internal.Lexer annotated with @com.google.inject.name.Named(value=org.eclipse.xtext.ui.editor.contentassist.antlr.internal.Lexer.CONTENT_ASSIST) was bound.
No implementation for org.eclipse.xtext.ide.editor.contentassist.antlr.IContentAssistParser was bound.
No implementation for java.util.concurrent.ExecutorService was bound.
No implementation for org.eclipse.xtext.web.server.generator.IContentTypeProvider was bound.


Basically it seems when I do
 Injector injector = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration(); 

This somehow breaks existing MyDslWebSetup's injector which is created for XtextServlet:

		super.init();
		Injector injector = new MyDslWebSetup().createInjectorAndDoEMFRegistration();
		disposableRegistry = injector.getInstance(DisposableRegistry.class);


I have no idea why/how they affect each other - are there some static fields overwritten?

Should I just create one instance of MyDslWebSetup then and reuse it at all times?

What's happening?
Re: Standalone setup parsing threadsafety [message #1800122 is a reply to message #1800121] Thu, 20 December 2018 04:26 Go to previous messageGo to next message
Eclipse UserFriend
Note: the code for instantiating web setup comes from MyDslServlet.xtend that Xtext has generated:
/*
 * generated by Xtext 2.16.0
 */
package x.mvmn.xtext.mydsl

import javax.servlet.annotation.WebServlet
import org.eclipse.xtext.util.DisposableRegistry
import org.eclipse.xtext.web.servlet.XtextServlet

/**
 * Deploy this class into a servlet container to enable DSL-specific services.
 */
@WebServlet(name = 'XtextServices', urlPatterns = '/xtext-service/*')
class MyDslServlet extends XtextServlet {
	
	DisposableRegistry disposableRegistry
	
	override init() {
		super.init()
		val injector = new MyDslWebSetup().createInjectorAndDoEMFRegistration()
		disposableRegistry = injector.getInstance(DisposableRegistry)
	}
	
	override destroy() {
		if (disposableRegistry !== null) {
			disposableRegistry.dispose()
			disposableRegistry = null
		}
		super.destroy()
	}
	
}

[Updated on: Thu, 20 December 2018 04:27] by Moderator

Re: Standalone setup parsing threadsafety [message #1800124 is a reply to message #1800122] Thu, 20 December 2018 04:38 Go to previous messageGo to next message
Eclipse UserFriend
P.S. Oh yes, I see them now - there are some static fields used in MyDslStandaloneSetupGenerated generated by Xtext, which MyDslStandaloneSetup extends (and MyDslWebSetup extends MyDslStandaloneSetup):
- Resource.Factory.Registry.INSTANCE
- EPackage.Registry.INSTANCE
- IResourceServiceProvider.Registry.INSTANCE
- MyDslPackage.eINSTANCE
- XtextPackage.eINSTANCE
These store shared instances of IResourceFactory and IResourceServiceProvider
	public void register(Injector injector) {
		if (!EPackage.Registry.INSTANCE.containsKey("http://www.mvmn.x/mydsl")) {
			EPackage.Registry.INSTANCE.put("http://www.mvmn.x/mydsl", MyDslPackage.eINSTANCE);
		}
		IResourceFactory resourceFactory = injector.getInstance(IResourceFactory.class);
		IResourceServiceProvider serviceProvider = injector.getInstance(IResourceServiceProvider.class);
		
		Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put("mydsl", resourceFactory);
		IResourceServiceProvider.Registry.INSTANCE.getExtensionToFactoryMap().put("mydsl", serviceProvider);
	}

	@Override
	public Injector createInjectorAndDoEMFRegistration() {
		// register default ePackages
		if (!Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().containsKey("ecore"))
			Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put(
				"ecore", new EcoreResourceFactoryImpl());
		if (!Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().containsKey("xmi"))
			Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put(
				"xmi", new XMIResourceFactoryImpl());
		if (!Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().containsKey("xtextbin"))
			Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put(
				"xtextbin", new BinaryGrammarResourceFactoryImpl());
		if (!EPackage.Registry.INSTANCE.containsKey(XtextPackage.eNS_URI))
			EPackage.Registry.INSTANCE.put(XtextPackage.eNS_URI, XtextPackage.eINSTANCE);

		Injector injector = createInjector();
		register(injector);
		return injector;
	}
	
	public Injector createInjector() {
		return Guice.createInjector(new MyDslRuntimeModule());
	}


So whenever I create new instance of MyDslWebSetup or MyDslStandalone setup - I still get those instances shared, so I don't have threadsafety guarantees whatever I do.

Why does Xtext generate such a thing?

[Updated on: Thu, 20 December 2018 04:56] by Moderator

Re: Standalone setup parsing threadsafety [message #1800129 is a reply to message #1800124] Thu, 20 December 2018 05:06 Go to previous messageGo to next message
Eclipse UserFriend
still dont get your flow:
do the creation of injector on one thread.
do parsing on multiple threads
if there are other places used that are not thread safe like serialization etc.
make sure the caches are filled on a single thread before going multithreaded
Re: Standalone setup parsing threadsafety [message #1800130 is a reply to message #1800124] Thu, 20 December 2018 05:07 Go to previous messageGo to next message
Eclipse UserFriend
P.S. After considering this all once again I conclude that Xtext web setup IS or at least MUST BE thread safe because it is used by XtextServlet, and, as you all know, Servlets by definition operate in multithreaded environment where each request comes in a separate thread.

I will continue investigating, but so far I suppose as long as I don't have multi-threaded access to shared opf instances of Resource (and possibly ResourceSet) I should be fine because XtextServlet does that too. Or at least my code won't be more broken than XtextServlet (which I hope is not broken).

It's a pity I have to figure this all out by myself the hard way. This forum is really not helpful.

[Updated on: Thu, 20 December 2018 05:08] by Moderator

Re: Standalone setup parsing threadsafety [message #1800131 is a reply to message #1800130] Thu, 20 December 2018 05:13 Go to previous messageGo to next message
Eclipse UserFriend
please open a issue on github.com/eclipse/xtext-web with your findings
Re: Standalone setup parsing threadsafety [message #1800133 is a reply to message #1800130] Thu, 20 December 2018 05:40 Go to previous messageGo to next message
Eclipse UserFriend
Mykola Makhin wrote on Thu, 20 December 2018 05:07
I conclude that Xtext web setup IS or at least MUST BE thread safe


HI

I'm not sure whether you are too naive of too optimistic.

Code is only threadsafe if it is designed, coded and tested to be threadsafe; possibly by running in a single thread sandbox. I very much doubt that XtextServlet fullfills any of the foregoing.

Providing non-trivial threadsafe functionality is very very hard, and almost certainly requires the designers to be motivated from the outset to use disciplined idioms universally, probably aided by autogeneration. Testing thread safety is close to impossible without a disciplined autogeneration that is provably rather than accidentally threadsafe.

Given that EMF itself is not thread safe, and that extended EMF, e.g. EMF + OCL delegates, is definitely not threadsafe, there is clearly no way that Xtext designers could be expected to make even the slightest effort to make Xtext fully threadsafe.

If you want to use Xtext in a threadsafe production, rather than research, context, you should instrument all Xtext's caches to confirm that no changes occur after you have done what you think is appropriate to warm them up. You might need to use Java reflection to discover all the caches by computation rather than by manual enumeration.

Regards

Ed Willink
Re: Standalone setup parsing threadsafety [message #1800136 is a reply to message #1800133] Thu, 20 December 2018 06:05 Go to previous messageGo to next message
Eclipse UserFriend
Ed Willink wrote on Thu, 20 December 2018 10:40
Mykola Makhin wrote on Thu, 20 December 2018 05:07
I conclude that Xtext web setup IS or at least MUST BE thread safe


HI

I'm not sure whether you are too naive of too optimistic.

I'm being realistic here. Xtext claims to have web integration, using XtextServlet. Servlets must be thread safe because they are used in multi-threaded environment (servlet container) where every HTTP request is processed in a separate thread. This is a definition of servlet.

Which of these facts do you think is false? Are you implying that Xtext developers lie when they say it has web integration since XtextServlet is flawed because it's not threadsafe?

XtextServlet can either be thread-safe, or not a properly written servlet (simply speaking, broken). It's either of the two, can't be both. If XtextServlet is not thread safe - it means XtextServlet is broken.

But so far I did not find it to be broken - it works perfectly fine for me (as long as I reuse an instance of Injector created by MyDslWebSetup.createInjectorAndDoEMFRegistration method but keep access to XtextResourceSet/Resource single-threaded/synchronized) - and it is being used by multiple threads by definition and in practice.

Quoting the servlet API javadoc:
https://docs.oracle.com/javaee/7/api/javax/servlet/http/HttpServlet.html
Quote:
Servlets typically run on multithreaded servers, so be aware that a servlet must handle concurrent requests and be careful to synchronize access to shared resources. Shared resources include in-memory data such as instance or class variables and external objects such as files, database connections, and network connections.

[Updated on: Thu, 20 December 2018 06:18] by Moderator

Re: Standalone setup parsing threadsafety [message #1800138 is a reply to message #1800136] Thu, 20 December 2018 06:33 Go to previous messageGo to next message
Eclipse UserFriend
Mykola Makhin wrote on Thu, 20 December 2018 06:05
a servlet must handle concurrent requests and be careful to synchronize access to shared resources


Hi

I'm not familiar with servlets but I can well imagine that it provides an approximation to what I earlier referred to a a single thread sandbox. But as you can see from your quote, it is possible for a servlet to be badly coded. An XtextServlet might well be threadsafe, but that in no way proves it is. Most software has bugs, all manually coded threadsafe software has bugs. Xtext itself is certainly not threadsafe just because a an XtextServlet might be.

Regards

Ed Willink
Re: Standalone setup parsing threadsafety [message #1800139 is a reply to message #1800138] Thu, 20 December 2018 06:36 Go to previous message
Eclipse UserFriend
as i said: please open a bug with your observations/usecase
Previous Topic:Serialize DSL model from with out Xtext Web project classes
Next Topic:Building a map collection
Goto Forum:
  


Current Time: Mon Jun 16 09:47:18 EDT 2025

Powered by FUDForum. Page generated in 0.08396 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top