Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Running the Xtext machinery outside the builder
Running the Xtext machinery outside the builder [message #985515] Thu, 15 November 2012 13:09 Go to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 322
Registered: July 2009
Location: Gothenburg
Senior Member
Hi!

I want to try to improve the experience for large projects by disabling the Xtext builder and instead have the builder feed the resource change events to a queue, which a background thread can use as input and feed them to the Xtext indexer. This also provides a place where to hook in pre-built indexes, for libraries and such.

We are accepting that while this isn't finished, the IDE functionality will be incomplete.

My question is: is there any technical detail in Xtext that might prevent this from working?

best regards,
Vlad
Re: Running the Xtext machinery outside the builder [message #985735 is a reply to message #985515] Thu, 15 November 2012 21:22 Go to previous messageGo to next message
Jan Koehnlein is currently offline Jan KoehnleinFriend
Messages: 658
Registered: July 2009
Senior Member
But the builder is already running in background and queueing/clustering
resources to be build, isn't it?

Also note that a whole bunch of services are relying of the index to be
uptodate and consistent, e.g. linking, refactoring, find references...

I'd hook in pre-build indexes with a customized IGlobalScopeProvider.

Am 15.11.12 14:09, schrieb Vlad Dumitrescu:
> Hi!
>
> I want to try to improve the experience for large projects by disabling
> the Xtext builder and instead have the builder feed the resource change
> events to a queue, which a background thread can use as input and feed
> them to the Xtext indexer. This also provides a place where to hook in
> pre-built indexes, for libraries and such.
>
> We are accepting that while this isn't finished, the IDE functionality
> will be incomplete.
> My question is: is there any technical detail in Xtext that might
> prevent this from working?
>
> best regards,
> Vlad
>


--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com
Re: Running the Xtext machinery outside the builder [message #985740 is a reply to message #985735] Thu, 15 November 2012 21:51 Go to previous messageGo to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 322
Registered: July 2009
Location: Gothenburg
Senior Member
Hi Jan,

Jan Kohnlein wrote on Thu, 15 November 2012 22:22
But the builder is already running in background and queueing/clustering
resources to be build, isn't it?


Yes, it does that, but in the meanwhile the workspace is locked and then the user can't do very much. It is important in may case, since the build takes so long.

Quote:
Also note that a whole bunch of services are relying of the index to be
uptodate and consistent, e.g. linking, refactoring, find references...


My hope is that they only need the index to be updated and consistent if they are to provide correct information. If the user is aware that while building, there might be inconsistent info/behaviour, then I believe it can mitigate the pain. Operations like refactoring or quickfixes should be disabled.

That said, I tried to put in practice this idea and it's not easy to get the right context, having Guice and OSGi to play nice... Maybe it won't be possible to do it, but I feel I have to try, as I am running out of options.

Even if I can convince you (the Itemis team) to work some magic (for the community or as a commercial service), I begin to think that it still won't be something done in a few weeks or so.

Thanks!

best regards,
Vlad

[Updated on: Thu, 15 November 2012 21:51]

Report message to a moderator

Re: Running the Xtext machinery outside the builder [message #985743 is a reply to message #985740] Thu, 15 November 2012 22:03 Go to previous messageGo to next message
Jan Koehnlein is currently offline Jan KoehnleinFriend
Messages: 658
Registered: July 2009
Senior Member
Have you done some profiling on what exactly takes so long? If it is
really the indexing, maybe you can implement a better strategy.

Am 15.11.12 22:51, schrieb Vlad Dumitrescu:
> Hi Jan,
>
> Jan Kohnlein wrote on Thu, 15 November 2012 22:22
>> But the builder is already running in background and
>> queueing/clustering resources to be build, isn't it?
>
>
> Yes, it does that, but in the meanwhile the workspace is locked and then
> the user can't do very much. It is important in may case, since the
> build takes so long.
>
> [/quote]Also note that a whole bunch of services are relying of the
> index to be uptodate and consistent, e.g. linking, refactoring, find
> references... [/quote]
>
> My hope is that they only need the index to be updated and consistent if
> they are to provide correct information. If the user is aware that while
> building, there might be inconsistent info/behaviour, then I believe it
> can mitigate the pain. Operations like refactoring or quickfixes should
> be disabled.
>
> That said, I tried to put in practice this idea and it's not easy to get
> the right context, having Guice and OSGi to play nice... Maybe it won't
> be possible to do it, but I feel I have to try, as I am running out of
> options.
>
> Even if I can convince you (the Itemis team) to work some magic (for the
> community or as a commercial service), I begin to think that it still
> won't be something done in a few weeks or so.
> Thanks!
>
> best regards,
> Vlad
>


--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com
Re: Running the Xtext machinery outside the builder [message #985747 is a reply to message #985743] Thu, 15 November 2012 22:49 Go to previous messageGo to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 322
Registered: July 2009
Location: Gothenburg
Senior Member
Jan Kohnlein wrote on Thu, 15 November 2012 23:03
Have you done some profiling on what exactly takes so long? If it is
really the indexing, maybe you can implement a better strategy.


Yes, I profiled it. Maybe I'm using the wrong words here. By 'indexing' I refer to validation and linking. I don't have any validation other than the default one, but the linking is what takes time. From what I can see, it is heavier than linear with the number of files/links.

I also tried to remove all cross-references from the grammar and of course it's faster, but it's still too slow for the large projects. That's why I thought about the "run it outside the builder" idea.

Thanks for your help!

best regards,
Vlad
Re: Running the Xtext machinery outside the builder [message #985935 is a reply to message #985747] Fri, 16 November 2012 18:01 Go to previous messageGo to next message
Jan Koehnlein is currently offline Jan KoehnleinFriend
Messages: 658
Registered: July 2009
Senior Member
If there are really no more cross-references, linking should be a no-op.
And the Antlr parser is usually very fast.

Is it an Xbase language? Linking can be seriously speeded up by adding
some explicit types, especially to closure parameters. Or with the
completely overhauled typesystem implementation that is planned for the
next release early next year.

If validation is slow, it's worth looking at closer, as you can really
write validation rules such that complexity explodes.


Am 15.11.12 23:49, schrieb Vlad Dumitrescu:
> Jan Kohnlein wrote on Thu, 15 November 2012 23:03
>> Have you done some profiling on what exactly takes so long? If it is
>> really the indexing, maybe you can implement a better strategy.
>
>
> Yes, I profiled it. Maybe I'm using the wrong words here. By 'indexing'
> I refer to validation and linking. I don't have any validation other
> than the default one, but the linking is what takes time. From what I
> can see, it is heavier than linear with the number of files/links.
>
> I also tried to remove all cross-references from the grammar and of
> course it's faster, but it's still too slow for the large projects.
> That's why I thought about the "run it outside the builder" idea.
>
> Thanks for your help!
>
> best regards,
> Vlad
>


--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com
Re: Running the Xtext machinery outside the builder [message #985967 is a reply to message #985935] Fri, 16 November 2012 21:19 Go to previous messageGo to next message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 322
Registered: July 2009
Location: Gothenburg
Senior Member
Hi Jan,

The language is Erlang, so it has nothing to do with Xbase. This also explains why there is already a huge base of code that needs to be parsed and analyzed.

In a project the size that I need to handle, all jobs that need to go through all the files will be slow (for an operation in an IDE). This is unavoidable. What makes it unacceptable for our users is that the project is locked and no changes can be saved. If this part is solved, it will buy us some time to optimize the linking and validation.

I've spent the day investigating my idea for wrapping the Xtext builder in a non-locking job, but it hasn't been designed with this use case in mind and it's not as easy as I had hoped Smile

best regards,
Vlad
Re: Running the Xtext machinery outside the builder [message #987124 is a reply to message #985967] Fri, 23 November 2012 15:56 Go to previous messageGo to next message
Knut Wannheden is currently offline Knut WannhedenFriend
Messages: 296
Registered: July 2009
Senior Member
Hi Vlad,

I think you will probably find that this won't work well. I think there
is just too much that can go wrong. E.g. what happens if the user edits
a file that was already built or even deletes a file which the builder
has queued, etc.

I think you are then probably better of by investigating the efforts
that were made on tickets
https://bugs.eclipse.org/bugs/show_bug.cgi?id=358714 and
https://bugs.eclipse.org/bugs/show_bug.cgi?id=359824. Note that these
are certainly not for the faint-hearted as they will require some
fiddling (overriding Guice module for shared state).

Regards,

--knut

On 11/16/12 10:19 PM, Vlad Dumitrescu wrote:
> Hi Jan,
>
> The language is Erlang, so it has nothing to do with Xbase. This also
> explains why there is already a huge base of code that needs to be
> parsed and analyzed.
>
> In a project the size that I need to handle, all jobs that need to go
> through all the files will be slow (for an operation in an IDE). This is
> unavoidable. What makes it unacceptable for our users is that the
> project is locked and no changes can be saved. If this part is solved,
> it will buy us some time to optimize the linking and validation.
>
> I've spent the day investigating my idea for wrapping the Xtext builder
> in a non-locking job, but it hasn't been designed with this use case in
> mind and it's not as easy as I had hoped :)
> best regards,
> Vlad
>
Re: Running the Xtext machinery outside the builder [message #987381 is a reply to message #987124] Mon, 26 November 2012 11:30 Go to previous message
Vlad Dumitrescu is currently offline Vlad DumitrescuFriend
Messages: 322
Registered: July 2009
Location: Gothenburg
Senior Member
Thanks, Knut.

What I hope would happen if there is an edit while the index is built is that there will be some transient errors, that will go away when the notification for the new change is processed.

Realistically, in the ~15 minutes at the start of his session (when xtext will do the heavy build) a user will probably checkout a couple of files from VCS and make changes to a couple more. That should not make Xtext utterly confused, in my mind.

regards,
Vlad
Previous Topic:Accessing model context in quickfix
Next Topic:Assignments and function output have same sintax
Goto Forum:
  


Current Time: Tue Dec 23 01:40:26 GMT 2014

Powered by FUDForum. Page generated in 0.09762 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software