Home » Modeling » TMF (Xtext) » endless Loop with extern lexer
| | | |
Re: endless Loop with extern lexer [message #1160450 is a reply to message #1159015] |
Tue, 29 October 2013 06:11 |
Henrik Lindberg Messages: 2509 Registered: July 2009 |
Senior Member |
|
|
On 2013-28-10 9:37, Chris Ainsley wrote:
> Hi Henrik,
>
>
> I've looked at geppetto before as an example of integrating custom
> lexers and it very much helped to get my DSL working in Xtext, and
> everything is working, with the one exception of content assist within
> non empty quoted text.
>
> I understand that the Lexer APIs are different between the three
> different lexers used by XText but I was of the understanding that there
> was no problem in making a stateful custom lexer (until now).
>
> I posted source code of the problem into the
> http://www.eclipse.org/forums/index.php/t/566064/ thread.
>
> If there is an obvious problem with my lexer to cause such a crash, then
> it should be found within this project. My assumption at the moment is
> that my stateful lexer cannot return back the correct token if provided
> a partial character stream (as it would be in the wrong state), and that
> the content assist code is providing a partial fragment of my DSL, and
> somehow entering a deadly loop of no escape when no tokens can be matched.
>
> If Xtext assumes a stateless lexer for content assist then there is
> nothing I can do to resolve this problem. Is there anyone on the Xtext
> team that can confirm or refute such a requirement?
>
> Chris
I think the lexers are always context free; in that they will be given
any snippet of source text - however, regions are also involved in this,
It cannot for instance safely assume that it can start at any position
(i.e. in the middle of a string).
IIRC I had to also tweak the region calculations.
The Geppetto lexer keeps track of previous significant token, but has
a special case of being fed something partial (no prior token seen).
Hope that helps figuring out what is going on.
Regards
- henrik
|
|
| | |
Re: endless Loop with extern lexer [message #1166583 is a reply to message #1164391] |
Sat, 02 November 2013 03:26 |
Chris Ainsley Messages: 78 Registered: March 2010 Location: UK |
Member |
|
|
Hi Henrik,
There is so much specialisation in the Gepetto codebase, its hard for me to understand what exactly is crashing Xtext. I still don't understand how the partition mapper relates to code completion at all, and looking at the Gepetto code, it becomes even more confusing as I don't exactly know how your class overrides are affecting the default behaviour of XText (as I don't know how the default behaviour fully works due to loose coupling and undocumented methods).
I suppose the point I'm making is that I started to destroy my codebase trying to merge in snippets extracted from Gepetto without being able to understand what the intent of the code is (as there is no real documentation about this low level code anywhere). I had to revert all changes just to get the code back to the previous known crashing state.
I truly thank you for your assistance in this matter, but it may be quicker to look at the sample project I created ( in this thread and to suggest direct remedial action. Without detailed knowledge of the inner workings of antlr and xtext, I can't adapt your Gepetto specific lexer workarounds to the general case (it seems like you are solving far more than this type of issue in your own code).
I can't appreciate how busy you must be, but maybe if you could contact me to solve this single isolated issue, then I think that would the only possible chance of me getting a working editor - ever. If you would assist me, I would take responsibility for documenting the solution, so that other people having the same problem (and I am not the first) would be able to understand it in laymans terms.
Sorry for my insufficient understanding here. Up until this point I had a good grasp of custom lexing but it seems that the interface between the lexer and regions just got the better of me and I'm stuck with a program I can't fix.
Chris
[Updated on: Sat, 02 November 2013 03:27] Report message to a moderator
|
|
|
Re: endless Loop with extern lexer [message #1166630 is a reply to message #1166583] |
Sat, 02 November 2013 04:13 |
Henrik Lindberg Messages: 2509 Registered: July 2009 |
Senior Member |
|
|
On 2013-02-11 4:26, Chris Ainsley wrote:
>
> Hi Henrik,
>
> There is so much specialisation in the Gepetto codebase, its hard for me
> to understand what exactly is crashing Xtext. I still don't understand
> how the partition mapper relates to code completion at all, and looking
> at the Gepetto code, it becomes even more confusing as I don't exactly
> know how your class overrides are affecting the default behaviour of
> XText (as I don't know how the default behaviour fully works due to
> loose coupling and undocumented methods).
>
> I suppose the point I'm making is that I started to destroy my codebase
> trying to merge in snippets extracted from Gepetto without being able to
> understand what the intent of the code is (as there is no real
> documentation about this low level code anywhere). I had to revert all
> changes just to get the code back to the previous known crashing state.
>
It gets complex quickly and most of Geppetto (except the parts under the
xtext namespace) where never intended to be reusable outside of
Geppetto. The parts under xtext were made as generic as possible - but I
also had deadlines for Geppetto to consider.
> I truly thank you for your assistance in this matter, but it may be
> quicker to look at the sample project I created ( in
> http://www.eclipse.org/forums/index.php/t/566064/ thread and to suggest
> direct remedial action.
>
I am sorry, but that will just take too much time before I am able to
get to the point where it is possible to pin-point the particular
problem you are seeing.
I spent many hours (days and weeks actually) debugging what was going on
in Geppetto. (Use of the debugger is really the only way to learn the
low level behavior, as you point out - there really is no documentation
at that level).
> I truly thank you for your assistance here, but without detailed
> knowledge of the inner workings of antlr and xtext, I can't adapt your
> Gepetto specific lexer workarounds to the general case (it seems like
> you are solving far more than this type of issue in your own code).
>
Yes, there are many issues that I worked around. They do not all relate
to use of an external lexer but rather the reasons why certain things
had to be solved in the lexer.
> I fully appreciate you are busy, but if you could contact me to solve
> this single isolated issue, then I think that would the only possible
> chance of me getting a working editor - ever. If you would assist me, I
> would take responsibility for documenting the solution, so that other
> people having the same problem (and I am not the first) would be able to
> understand it in laymans terms.
>
> Sorry for my insufficient understanding here. Up until this point I had
> a good grasp of custom lexing but it seems that the interface between
> the lexer and regions just got the better of me and I'm stuck with a
> program I can't fix.
>
I had the same feeling a couple of times when I got stuck. I can try
helping with advice.
The regions play an important role as IIRC they define demarcation of
positions the lexer will be given when doing partial parsing. The lexer
itself is really not involved until it is given something to lex - so
regions matter to the higher levels.
OTOH - I have no clue if that is the problem you are seeing or not.
I would look at the stacktrace for your NPE - and set a breakpoint at
some point higher up in the call stack and then just step through down
to the NPE. (I know it is painful and at first you will be stepping
through lots of low level code. Once you get familiar with the
completely uninteresting parts you can filter so that the debugger skips
those parts.
Let's see, what else...
Since xtext generates multiple lexers from the grammar and uses the
terminal rules as they are written in the grammar for these auxiliary
lexers they will (naturally) be based on the wrong terminals (even if
you have terminal rules in your grammar that are close, the generated
lexer will still be wrong. I did not want to write multiple external
lexers so I wrapped the one and only in a class that adapted it so it
can function as "the lexer" for all the lexing jobs (highlighting, etc).
That part of Geppetto should not be too hard to figure out.
I am currently working on other things than Geppetto (busy on the actual
language Geppetto is an IDE for) and it will be some time (weeks to
months) before I am going to be working on Geppetto again - meanwhile we
(Cloudsmith) have transitioned all the code to Puppet Labs and I am
currently not set up properly with a dev environment for the
transitioned code (everything moved to a new namespace). Once I am
working on Geppetto actively again it is easier for me to answer
questions etc. Meanwhile there will be plenty of IIRC remarks :-)
If you are in a hurry, I suggest contacting Itemis and buying some
consulting from them.
Another thing you could do is to try to run and debug Geppetto. Set a
breakpoint in the region calculation code and see from where it gets
called. Then find the corresponding place in your project.
Although it requires work to setup it is fairly well automated with
Buckminster and there are instructions (although they may be behind -
but should you want to try this approach we (me and Thomas Hallgren)
would help you out since those instructions should be up to date. (If
everything works ok it should take you an hour or so to set that up).
You can probably find the corresponding class in your project without
debugging Geppetto. There should be enough clues what it extends, where
it is mentioned in guice module configurations etc.
Once you started debugging and getting a better understanding of how the
various parts work it is also easier to ask specific questions about
what is going on, and this makes it far more likely that you get good
answers in this forum. (That is how I learned).
I hope some of those pointers may help you, and again I am sorry that I
don't have the bandwidth to dig deeper into your code.
Regards
- henrik
|
|
| |
Re: endless Loop with extern lexer [message #1169451 is a reply to message #1166636] |
Mon, 04 November 2013 02:04 |
Chris Ainsley Messages: 78 Registered: March 2010 Location: UK |
Member |
|
|
Henrik Lindberg wrote on Sat, 02 November 2013 13:16Doh, you were seeing an endless loop, not a NPE.
So start with a breakpoint someplace safe and step towards the void...
To find that place try invoking the code completion for a sample
language and break in code you know gets called from the code completion
- look at the stack there, and then do that for your own language.
- henrik
Thanks for taking the time for a long and reasoned response, and of course I understand you don't have the bandwidth to take on anything like this.
After dozens of hours spent on working on this from multiple angles - the debugger hasn't helped me with this thus far; without anyone familiar with the low level code to assist I think "the definition of insanity" applies here.
Consultancy is probably the answer to this but I'm not currently in a position to be able to acquire any more consultancy, so I guess I'll leave the issue here for now and document that the users should avoid auto-complete within quoted strings. I'm beat.
Thanks for all your help again Henrik.
[Updated on: Mon, 04 November 2013 02:09] Report message to a moderator
|
|
|
Goto Forum:
Current Time: Thu Apr 25 08:00:51 GMT 2024
Powered by FUDForum. Page generated in 0.04776 seconds
|