Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » persistent cache for cross-references
persistent cache for cross-references [message #815113] Wed, 07 March 2012 08:48 Go to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 430
Registered: July 2009
Location: Gothenburg
Senior Member
Hi!

For larger files, it takes a while to create all cross-references the first time one is required. I wonder if there is some mechanism that I missed and that allows one to save the xref index database and reuse it whenever the file is reopened (if it hasn't changed, of course). Or at least if there is anything that can start indexing in the background when the file is opened in an editor.

If no such thing exists, is it something that might be of interest as a part of the framework?

best regards,
Vlad
Re: persistent cache for cross-references [message #815147 is a reply to message #815113] Wed, 07 March 2012 09:37 Go to previous messageGo to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 430
Registered: July 2009
Location: Gothenburg
Senior Member
Actually, the current cache that throws away all data for even the smallest change is also very slow for large files. It wouldn't be as big a problem if it wasn't being recreated with a builder participant that locks the file for further changes: making a change and trying to save blocks the UI until the cache is regenerated.

So a related question would be if it would be possible to parse files incrementally (in most cases, a change doesn't affect the parse tree for the code before it) and/or to rebuild the cache only for the affected parts of the code (if possible to find them, of course).

Foremost I'd like to know if you think it's possible at all without rewriting xtext from scratch or something like that. I suppose the parsing part depends on what antlr can do, but a smarter cache might be doable.

best regards,
Vlad
Re: persistent cache for cross-references [message #846710 is a reply to message #815147] Mon, 16 April 2012 16:33 Go to previous messageGo to next message
Knut Wannheden is currently offline Knut WannhedenFriend
Messages: 298
Registered: July 2009
Senior Member
Hi Vlad,

I don't think I quite understand what you are trying to do. Can you be
more specific on what the problem is you are trying to solve or exactly
what operations don't perform well?

There is in fact a persisted cache of reference descriptions. That is
part of what the builder produces. You can use the
ResourceDescriptionsProvider to access the reference descriptions for a
particular resource. Note that this will normally only include external
cross references (i.e. references pointing to objects in other resources).

I'd also like to point out that Xtext already tries to do partial
reparsing using ANTLR. That works and is also quite fast. But Xtext
still needs to recreate all reference descriptions, because even
references originating from source code earlier in the file may now be
invalidated.

Regards,

--knut

On 3/7/12 10:37, Vlad Dumitrescu wrote:
> Actually, the current cache that throws away all data for even the
> smallest change is also very slow for large files. It wouldn't be as big
> a problem if it wasn't being recreated with a builder participant that
> locks the file for further changes: making a change and trying to save
> blocks the UI until the cache is regenerated.
>
> So a related question would be if it would be possible to parse files
> incrementally (in most cases, a change doesn't affect the parse tree for
> the code before it) and/or to rebuild the cache only for the affected
> parts of the code (if possible to find them, of course).
>
> Foremost I'd like to know if you think it's possible at all without
> rewriting xtext from scratch or something like that. I suppose the
> parsing part depends on what antlr can do, but a smarter cache might be
> doable.
> best regards,
> Vlad
>
Re: persistent cache for cross-references [message #846850 is a reply to message #846710] Mon, 16 April 2012 20:14 Go to previous messageGo to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 430
Registered: July 2009
Location: Gothenburg
Senior Member
Hi Knut,

Since posting the question I did find some of the existing caching mechanisms, but my problem is that it's not enough in the worst case... I also admit mixing together the parsing with the linking stage, which increased the confusion.

I've been working on other parts of the code, so I don't have hard examples handy, but with large files there are still very noticeable delays (UI blocks because the builder locks the file) after making changes in the file. What users point out is that similarly large Java files behave much better in the editor.

My line of thought was that with some knowledge about the language structure, both parsing and linking can be optimized a lot. For example, if the top level of the language looks like (very simplified)

Module: Forms*;
Form: ID Body; // Body defines no referenceable entities

then after a change in a Body, if the Body still parses we know we don't have to relink the rest of the module.

Maybe it is something like this you meant by "tries to do partial reparsing", but I'd think it is difficult for the general case without hints from the language definition.

Thanks!
best regards,
Vlad
Re: persistent cache for cross-references [message #847312 is a reply to message #846850] Tue, 17 April 2012 07:21 Go to previous messageGo to next message
Knut Wannheden is currently offline Knut WannhedenFriend
Messages: 298
Registered: July 2009
Senior Member
Hi Vlad,

On 4/16/12 10:14 PM, Vlad Dumitrescu wrote:
>
> I've been working on other parts of the code, so I don't have hard examples
> handy, but with large files there are still very noticeable delays (UI blocks
> because the builder locks the file) after making changes in the file. What
> users point out is that similarly large Java files behave much better in the
> editor.
>

That is true. JDT's builder is indeed very fast. I don't think it is a level
of performance Xtext will reach anytime soon.

> My line of thought was that with some knowledge about the language structure,
> both parsing and linking can be optimized a lot. For example, if the top level
> of the language looks like (very simplified)
>
> Module: Forms*;
> Form: ID Body; // Body defines no referenceable entities
>
> then after a change in a Body, if the Body still parses we know we don't have
> to relink the rest of the module.
> Maybe it is something like this you meant by "tries to do partial reparsing",
> but I'd think it is difficult for the general case without hints from the
> language definition.
>

The Eclipse automatic build is triggered as soon as resource changes are
detected. A typical case is when a file is saved in the editor. The Eclipse
auto build job will then acquire a workspace lock which isn't released until
it's finished which means that all resources are locked (for writing and thus
saving) in the meantime. I believe that is what your users notice: It may not
be possible to save a file and then save it again only a moment later without
having to wait for the build to finish.

Unfortunately the builder doesn't know *what exactly* changed in the changed
files. It is only handed a list of all changed files. Thus it cannot easily
perform any partial parsing or linking.

But if you make sure that the resource descriptions only contain the exported
objects which can be referenced from other resources, then you can at least
make sure that the builder doesn't unnecessarily rebuild other dependent
sources. That will reduce the time the builder is running. Have a look at
extending IDefaultResourceDescriptionStrategy for this.

Regards,

--knut
Re: persistent cache for cross-references [message #847374 is a reply to message #847312] Tue, 17 April 2012 08:34 Go to previous message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 430
Registered: July 2009
Location: Gothenburg
Senior Member
Thank you very much, Knut!

Configuring IDefaultResourceDescriptionStrategy may help, I will try it.

regards,
Vlad
Previous Topic:fileContentsToString ?
Next Topic:Formatting multi-line comments?
Goto Forum:
  


Current Time: Mon Jul 13 19:12:29 GMT 2020

Powered by FUDForum. Page generated in 0.02225 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top