Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Xtext Resource Storage(how to write fully resolved and computed resources to disk )
Xtext Resource Storage [message #1841177] Wed, 05 May 2021 17:01 Go to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
I read from Sven Efftinge's blog (http://blog.efftinge.de/2015/05/the-future-of-xtext.html), that "Xtext support storage Resource - Xtext can write fully resolved and computed resources to disk in a binary format, so it can later be loaded again quickly" This can be very helpful for a DSL project with thousands of model files. We can avoid to re-parse and link the resources again and again. The same idea can also be found in Mark Chrisiaens's blog (https://insights.sigasi.com/tech/xtext-resource-caching-loading-resources-5-times-faster/), which was published in 2012.
Unfortunately I did not find any documents or videos introduce how to do this.
Can someone tell me how can I use the storage resource in Xtext?

[Updated on: Wed, 05 May 2021 17:02]

Report message to a moderator

Re: Xtext Resource Storage [message #1841178 is a reply to message #1841177] Wed, 05 May 2021 17:07 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
Hi, can you give more details about your exact usecase
i am not sure if the storage aware feature solves anything for you

besides that binding

StorageAwareResourceDescriptionManager as IResourceDescriptionManager
and StorageAwareResource as XtextResource

bindings for IResourceStorageFacade to ResourceStorageFacade
and AbstractFileSystemAccess2 to EclipseResourceFileSystemAccess2 should do the trick
(maybe some more needed)

for reading you can steer with the SourceLevelURIsAdapter if resources should be loaded via storage or not (happens automatically during build)

public class MyDslUiModule extends AbstractMyDslUiModule {

	public MyDslUiModule(AbstractUIPlugin plugin) {
		super(plugin);
	}


//.....
	
	public Class<? extends IResourceDescription.Manager> bindIResourceDescription$Manager() {
		return StorageAwareResourceDescriptionManager.class;
	}
	
	public Class<? extends XtextResource> bindXtextResource() {
		return StorageAwareResource.class;
	}
	public Class<? extends IResourceStorageFacade> bindIResourceStorageFacade() {
		return ResourceStorageFacade.class;
	}
	public Class<? extends AbstractFileSystemAccess2> bindAbstractFileSystemAccess2() {
		return EclipseResourceFileSystemAccess2.class;
	}
}


you may also have to do some corrective customization like org.eclipse.xtext.xbase.resource.BatchLinkableResourceStorageFacade.getSourceContainerURI(StorageAwareResource) / org.eclipse.xtext.resource.persistence.ResourceStorageFacade.computeOutputPath(StorageAwareResource) so that the storages land in a folder

e.g.

		@Override
		protected String computeOutputPath(StorageAwareResource resource) {
			URI u = resource.getURI();
			if (u.isPlatformResource()) {
				URI base = u.trimFileExtension().trimSegments(u.segmentCount()-3);
				
				URI binaryStorageURI = getBinaryStorageURI(resource.getURI());
				String result = binaryStorageURI.deresolve(base)
						.path();
				return result;
				
			}
			return super.computeOutputPath(resource);
		}


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Thu, 06 May 2021 14:52]

Report message to a moderator

Re: Xtext Resource Storage [message #1841179 is a reply to message #1841178] Wed, 05 May 2021 19:09 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7268
Registered: July 2009
Senior Member
Hi

I'm not sure you really need to use an Xtext facility. Binary storage is an EMF option, so you just need to set the correct save options when saving your parsed Resource. I've never used them, but you should find that the reverse just works automatically. Interactively you could add a Save-As-Binary action unless Xtext already does. For thousands of files you probably want to write a little tool/builder to load one or more XML files and store them in binary format.

Regards

Ed Willink
Re: Xtext Resource Storage [message #1841181 is a reply to message #1841178] Wed, 05 May 2021 19:33 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
Thanks Christian for the quick answer.
We have our own DSL, which is developed by Xtext, to do the data modelling.
The models are saved in the Git repository. Users are working on different Git branches. Every time when the user switch to another Git branch, the cross-references and many other parsed information are broken. User has to use "Project-Clean" in Eclipse to re-parse all the models again. This takes with current model size around 20 minutes.
The idea is, is that possible to persist parsed information in disk . When the model git branch is switched, the already pared models can be directly get from the disk. and not need to re-parse all the models again.
Re: Xtext Resource Storage [message #1841182 is a reply to message #1841181] Wed, 05 May 2021 19:42 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7268
Registered: July 2009
Senior Member
Hi

I strongly recommend that you do not persist binary files in GIT unless you can guarantee never to evolve your DSL, since an evolution may easily break binary files that refer to feature "4" rather than feature "name", so if you insert a new feature at "3" all the subsequent ones are off by one. Crazy EMF crash.

Exactly this happened with *.xtextbin [1], although fortuitously the new feature was a last feature so in that case the incompatibility was only in one direction.

You can probably manage to make your textual DSL upward compatible so if you check in *.dsl to GIT and have a builder that refreshes *.dslbin from *.dsl, reparsing will only happen after changes. Your downstream tooling can try to open *.dslbin and fall back to *.dsl.

Regards

Ed Willink

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=460978
Re: Xtext Resource Storage [message #1841183 is a reply to message #1841182] Wed, 05 May 2021 20:30 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
The storages won't speed up clean builds but incremental ones and open editors with many transitive dependencies only.
So you would have to give it a try
If you change 90% of the files by a switch branch profit will be likely zero
Same if you just have a hand full of deps for the files that change


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Resource Storage [message #1841184 is a reply to message #1841183] Wed, 05 May 2021 20:32 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
And you should also check why switching branches brakes things
The same problem will be there with the storages

Same you should check why clean build takes 20 mins


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Wed, 05 May 2021 20:35]

Report message to a moderator

Re: Xtext Resource Storage [message #1841207 is a reply to message #1841184] Thu, 06 May 2021 13:54 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
Thanks Christian and Ed for your helpful suggestions.
1. switch git branch break the models
I tried today again to find out the reason, why after switching git branch, the model need to be cleaned. I think I find out something.
Because we are using git command or the git client application (GitExtensions) to work with git. Every time when we switch the git branch. The changed files are not synchronized in Eclipse Model Explorer. Only when I refresh the folder in eclipse model explorer, the changed files are synchronized.
I tried with the git inside eclipse (EGit), with it after switching git branch, the files in Model Explorer are updated and the automatically build run successfully.
So it seems that it's a problem of Eclipse working together with third party git application

2. why the clean build takes 20 minutes
it was my type error. most time the clean build can be finished around 10 minutes. We have some time consuming validations with the models, and totally there are around 5000 models. These models have also many transitive dependencies with each other. So I think the clean build with 10 minutes is still acceptable.
But I will try to use the storaged resource to accelerate the incremental build.

One more question is, does Xtext persistent the parse result somewhere in the disk. I ask this, because even the computer restarted and when I reopen the models, the cross reference and other things still works. If the parsing result were saved in the memory, this should not happen. Please correct me if I am wrong.

Re: Xtext Resource Storage [message #1841210 is a reply to message #1841207] Thu, 06 May 2021 14:26 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
you may check auto refresh or refresh on access in your workspace preferences.

xtext will just store the Xtext in index in a file called builder.state in a subfolder of .metadata in the workspace.
this index data then will be used when you continue with the workspace.

with the storage face you will get a for a src-gen/.somefile.mydslbin somefile.mydsl for a somefile.mydsl (path is calculated by org.eclipse.xtext.resource.persistence.ResourceStorageFacade.computeOutputPath(StorageAwareResource))




Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Resource Storage [message #1841216 is a reply to message #1841210] Thu, 06 May 2021 16:28 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
With the suggestion of Christian, I changed the refresh setting in eclipse.
It was only the option "Refresh on access" selected. Now I also selected another option "Refresh using native hooks or polling". Bingo, it works well with the third party git client now!!!! Thanks so much Christian.

The "builder.state" file I could not find under .metadata folder and its sub folders. I only found some history.index files, which is under .metadata/.plugins/org.eclipse.core.resources/.project/<xtext project name>/.indexes, I don't think these files are generated by xtext.
I'm using xtext version 2.19.
Re: Xtext Resource Storage [message #1841218 is a reply to message #1841216] Thu, 06 May 2021 18:23 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
The file is only there after you shutdown and before you start eclipse

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Resource Storage [message #1841234 is a reply to message #1841178] Fri, 07 May 2021 09:55 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
hi Christian,

I tried to bind the required classes.
There are some warning message in the import part of codes.
Warning - Discouraged access: The type StorageAwareResourceDescriptionManager is not accessible due to restriction on required project com.xxxxxx.model.ui

Can I ignore them?
Re: Xtext Resource Storage [message #1841235 is a reply to message #1841234] Fri, 07 May 2021 11:11 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
yes (it means that the api might change/break with a future release)
i assume you can live with it.


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Resource Storage [message #1841244 is a reply to message #1841235] Fri, 07 May 2021 14:55 Go to previous messageGo to next message
Juewei Jin is currently offline Juewei JinFriend
Messages: 17
Registered: July 2009
Junior Member
hi christian,
finally the storable resources are working. I saw the "dslbin" files are created.
I want to make clear, what is the difference between these "dslbin" files and the "builder.state" file. As you said, the "builder.state"is the index file. do the "dslbin" files contains more information than the "builder.state" file?

another question is, do I need to add some codes to tell xtext when to load the dslbin file, when not? if yes, how can I do this?
Re: Xtext Resource Storage [message #1841249 is a reply to message #1841244] Fri, 07 May 2021 19:19 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14068
Registered: July 2009
Senior Member
The builder state is just "the index"
It says there are these objects if that types with these names in these files

The mydsl bin are a binary version of the ast and nodemodel

The mydsl build are use automatically if you don't opt out using SourceLevelURIsAdapter
You can observe it debugging the storageawareresource class


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Previous Topic:Initialization of member JvmFields in constructor of a parent JvmGenericType
Next Topic:Cannot import an Existing Maven Projetct in a Run As Eclipse Aplication
Goto Forum:
  


Current Time: Mon Sep 27 20:15:43 GMT 2021

Powered by FUDForum. Page generated in 0.02124 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top