Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Parallel Serialization
Parallel Serialization [message #1833941] Wed, 28 October 2020 14:40 Go to next message
L R is currently offline L RFriend
Messages: 12
Registered: July 2019
Junior Member
Our project is translating from one Xtext grammar to another. As part of this translation, we perform a many to many translation as one single operation. We have found that the serialization takes a long time to process. To improve this, we were considering performing serialization in parallel. Unfortunately, we have been seeing various exceptions in the serialization that we attribute to parallel operation.

We found the recent blog post: https://blogs.itemis.com/en/how-the-code-generation-process-in-xtext-can-be-executed-parallely. This post was regarding code generation execution in parallel. This included mentioning that EMF is not thread-safe and to take special care regarding proxy resolution.

We also found a couple of old open issues related to this topic:
https://github.com/eclipse/xtext-core/issues/203
https://github.com/eclipse/xtext-core/issues/906

The recommendation at the end of 203 was to run a static initialization in a single thread before running the actual serialization. While the recommendation at the end of 906 was to create a custom GrammarProvider and install the adapters upfront.

In our scenario, we generated the ASTs serially and the resulting grammar does not have proxy elements. Each AST has its own Resource, but they all share the same injected ResourceSet. However, even after running an initial serialization serially, we occasionally still receive a serialization exception. Is there guidance on how much serialization needs to be performed by the static initialization?

I have prototyped a custom GrammarProvider, similar to the 906 solution, but haven't tried it enough to see if it resolves this issue.

Overall, is there guidance on how to safely perform serialization in parallel?
Re: Parallel Serialization [message #1833942 is a reply to message #1833941] Wed, 28 October 2020 14:42 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13976
Registered: July 2009
Senior Member
i have doubts serialization is threadsafe
=> i fear there is no guidance


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Parallel Serialization [message #1833943 is a reply to message #1833941] Wed, 28 October 2020 14:55 Go to previous messageGo to next message
Karsten Thoms is currently offline Karsten ThomsFriend
Messages: 747
Registered: July 2009
Location: Dortmund, Germany
Senior Member

Since the target is again a target DSL code, why not simply generate the target with model2text, i.e. code generation?
I doubt that parallelization of the serialization is doable with reasonable effort. Sorry, we can't offer guide on that approach, we did not do it that way before.
Re: Parallel Serialization [message #1833957 is a reply to message #1833943] Wed, 28 October 2020 20:24 Go to previous message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7162
Registered: July 2009
Senior Member
Hi

Parallel GrammarProvder.. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=460978

In Xtext 2.8, an evolution to Xcore.ecore was mishandled so that it became a breaking change to the *.xtextbin, preventing OCL and QVTd maintaining compatibility. I therefore replaced the XXX.xtextbin reader with its mandatory new feature, by an auto-generated XXXGrammarResource that builds a grammar singleton via new-feature-tolerant Java code; faster and more compatible. Since the underlying GrammarResource is a boring singleton, I suspect that one or perhaps two "synchronized" keywords could make the GrammarProvider threadsafe.

See https://git.eclipse.org/r/plugins/gitiles/ocl/org.eclipse.ocl/+/refs/heads/master/plugins/org.eclipse.ocl.xtext.oclinecore/src-gen/org/eclipse/ocl/xtext/oclinecore/OCLinEcoreGrammarResource.java for an example

Parallel Serializer. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=564265

A new declarative formatter for Xtext also provides a serializer that is driven by static auto-generated metadata. It too probably only needs a couple of outer "synchronized" keywords to be threadsafe. Further, the internal algorithm is a tree descent for which the descents to children could be parallelized. Parallelizing the final M2T StringBuilder would also be possible if the ultimate StringBuilder deferred concatenating trees till almost done. This is still work in somewhat slow progress.


Regards

Ed Willink
Previous Topic:Adding generated objects to the index
Next Topic:JVM Model Infer Append String to XBlockExpression
Goto Forum:
  


Current Time: Tue Jun 15 01:44:10 GMT 2021

Powered by FUDForum. Page generated in 0.01994 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top