Eclipse Community Forums: TMF (Xtext) » How to use BacktrackingLexerDocumentTokenSource

Help

Home

Home » Modeling » TMF (Xtext) » How to use BacktrackingLexerDocumentTokenSource

Show: Today's Messages :: Show Polls :: Message Navigator

How to use BacktrackingLexerDocumentTokenSource [message #1772427]

Sat, 09 September 2017 07:05

Waqas Ilyas

Messages: 80
Registered: July 2009

Member

This topic is a continuation of the problem discussed here:
https://www.eclipse.org/forums/index.php/m/1769726/

The conclusion on that topic was that I needed to use a custom lexer with Xtext.

I have started down the road to using JFlex lexer with my Xtext grammar (utilizing https://github.com/TypeFox/xtext-jflex). I have been so far able to hook up the fragment, and define my own jflex based lexer rules. On opening the editor for the first time the integration succeeds. But if I edit something things start to fall apart.

I think the reason is that since my lexer is context based we cannot scan for tokens incrementally. My lexer needs to be "backtracking". And as it seems, the XtextDocument instance created for my file should get a BacktrackingLexerDocumentTokenSource, instead of a DocumentTokenSource instance. But I can't figure out how to achieve that. Any ideas?

Here is the language definition in the workflow (mwe2) I am using:

		language = StandardLanguage {
			name = "org.xtext.example.CustomLexer"
			fileExtensions = "clx"

			parserGenerator = {
				debugGrammar = true
				combinedGrammar = false
				options = {
					backtrack = true
					backtrackLexer = true
				}
			}
			serializer = {
				generateStub = false
			}
			validator = {
				// composedCheck = "org.eclipse.xtext.validation.NamesAreUniqueValidator"
			}
			
			fragment = typefox.JFlexFragment {}
		}

The backtrack and backtrackLexer options don't seem to affect anything. And I couldn't find how DocumentTokenSource is currently bound to the XtextDocument instance, or how to override that binding.

Report message to a moderator

Re: How to use BacktrackingLexerDocumentTokenSource [message #1772428 is a reply to message #1772427]

Sat, 09 September 2017 07:12

Christian Dietrich

Messages: 14729
Registered: July 2009

Senior Member

simpy add a binding of DocumentTokenSource to the backtracking one does not work? similar to https://github.com/eclipse/xtext-xtend/blob/c224e8f71fdb5ec2ebf65e61001f35f8ac8059be/org.eclipse.xtend.ide/src/org/eclipse/xtend/ide/XtendUiModule.xtend#L390

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

[Updated on: Sat, 09 September 2017 07:15]

Report message to a moderator

Re: How to use BacktrackingLexerDocumentTokenSource [message #1772515 is a reply to message #1772428]

Mon, 11 September 2017 19:33

Waqas Ilyas

Messages: 80
Registered: July 2009

Member

I tried finding the binding for this in the runtimemodule, and also the uimodule generated class but didn't see one. Where can I add this binding? My *UiModule class does not extend XtendUiModule

I am using Xtext 2.10.0 by the way, should I be able to do this in that version?

Report message to a moderator

Re: How to use BacktrackingLexerDocumentTokenSource [message #1772518 is a reply to message #1772515]

Mon, 11 September 2017 19:54

Christian Dietrich

Messages: 14729
Registered: July 2009

Senior Member

Nooo extend. Simply add a binding wherever you like.
That is what the modules are for.
If you need a binding simply add it

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Day Job: https://www.everest-systems.com

Report message to a moderator

Re: How to use BacktrackingLexerDocumentTokenSource [message #1772523 is a reply to message #1772518]

Tue, 12 September 2017 00:11

Waqas Ilyas

Messages: 80
Registered: July 2009

Member

Yeah I need to read up more on Guice and Xtext integration. It is working now though, no more errors. Thanks a lot for that.

However, I see now that even though the editor's DocumentTokenSource is behaving properly, but the generated parser is using "partial parsing" that sometimes emits errors. Though these errors are not hurting right now but I wanted to see if I can do something about it. I can't seem to solve this by setting "partialParsing = false". Not sure what this option does. How can one set AbstractAntlrParser.partialParser to null?

I can perhaps bind IPartialParserHelper with TokenSequencePreservingPartialParsingHelper, although I still need to look into it if it would help.

And I also can't seem to understand what is the affect of following options "partialParsing", "backtrack" and "backtrackLexer":

			parserGenerator = {
				debugGrammar = true
				combinedGrammar = false
				partialParsing = false
				
				options = {
					backtrack = true
					backtrackLexer = true
				}
			}

Can you point me to any documentation?

Report message to a moderator