Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Python-like indentation and editor problems
Python-like indentation and editor problems [message #1015898] Mon, 04 March 2013 03:38 Go to next message
Shavarsh Odajavan is currently offline Shavarsh Odajavan
Messages: 7
Registered: March 2013
Junior Member
Hi, i'm writing a DSL which handles blocks in a (modified) Python-like way. A block starts with a newline followed by a number of tabs, and each tab added is one more nested level.
I'm using the AbstractSplittingTokenSource class (IndentTokenSource.java file) to handle indents/dedents and everything works but sometimes i'm having trouble with the editor.
In particular, the artificial indent/dedent tokens have to be set in their attributes (text, type, line, char position in line, start index, stop index) to create a consistent node model but they are "invisible" to the editor. This causes the editor to behave in a strange way (fake error highlighting even if no error is reported by the console, shifted highliting for errors, outline shifted highliting etc.).
My opinion is these problems come out because artificial tokens are overlapped with other tokens.
I would like to ask if someone has figured out how to get this working normally.

Project can be found in the attachment

Thanks in advance
Re: Python-like indentation and editor problems [message #1052910 is a reply to message #1015898] Wed, 01 May 2013 05:24 Go to previous messageGo to next message
Mark Butler is currently offline Mark Butler
Messages: 2
Registered: May 2013
Junior Member
I am also interested in writing an indentation based language like Shavarsh. Thanks Sharvash for posting such a complete example. Unfortunately I can't get it the example, or a number of others, to work.

I've been looking back through the Eclipse email lists, and also seen this project
http : // code.google.com/a/eclipselabs.org/p/todotext/

(sorry not to post full URLs, but this forum won't let me)

Ralf Ebert's code for Todotext mentioned above is no longer available from Github but
I discovered it is in the Eclipse test repository.

http : // eclipse.googlesource.com/tmf/org.eclipse.xtext/+/master/tests/org.eclipse.xtext.tests/src/org/eclipse/xtext/testlanguages/indent/

I also found the original thread about the work
http://www.eclipse.org/forums/index.php/m/556885/

and the Eclipse issue raised about it
https://bugs.eclipse.org/bugs/show_bug.cgi?format=multiple&id=329624

I also found an alternative implementation here
http : // eclipse-snippets.googlecode.com/svn/trunk/xtext-indenting/org.xtext.example.mydsl/src/org/xtext/example/

I have tried all three versions but I cannot get them to work. In the issue Ralf raised, he said there was a problem due to "partial parsing" because the indent sensitive language needs a stateful lexer. He then proposed a series of patches, but looking at the XText code. When I run the lexer in the debugger, it does seem to be doing some kind of partial parsing, and this leads to the fake errors that Sharvash describes.

I could not find the files that Ralf patches to solve this problem, so I wonder if those changes seem long gone? Can anybody confirm is this the case?

There is interest in parsing indentation based grammars as the question has come up on Stackoverflow -
http : // stackoverflow.com/questions/7167834/xtext-grammar-for-language-with-significant-semantic-whitespace

Other tools like Parboiled do support this "out-the-box"
https : // github.com/sirthias/parboiled/wiki/Indentation-Based-Grammar

so it would be great if XText could do the same.

thanks,

Mark
Re: Python-like indentation and editor problems [message #1052940 is a reply to message #1052910] Wed, 01 May 2013 09:45 Go to previous messageGo to next message
Shavarsh Odajavan is currently offline Shavarsh Odajavan
Messages: 7
Registered: March 2013
Junior Member
Hello Mark, the working (with some bugs) implementation i'm using right now for my language is the one you called "alternative implementation". It needs some changes to get it work, at least with my Xtext version (2.3.1) but it does its job in the end. Before that, i also tried using AbstractSplittingTokenSource class to handle indents/dedents (the class documentation says it's used to achieve an indentation based language) but i found out there were problems and i think a bit of work has to be done in order to get it working properly (to put it simple, you have a corrupted tree which leads to editor exceptions).
If you need help just write.
Re: Python-like indentation and editor problems [message #1053195 is a reply to message #1052940] Fri, 03 May 2013 01:10 Go to previous messageGo to next message
Mark Butler is currently offline Mark Butler
Messages: 2
Registered: May 2013
Junior Member
Hi Sharvarsh,

How did you modify IndentingParser?

http :// eclipse-snippets.googlecode.com/svn/trunk/xtext-indenting/org.xtext.example.mydsl/src/org/xtext/example/indenting/IndentingParser.java

AbstractAntlrParser seems to have changed, before it needed an implementation of parse, now it needs an implementation of createParser?

thanks,

Mark
Re: Python-like indentation and editor problems [message #1053203 is a reply to message #1053195] Fri, 03 May 2013 02:15 Go to previous messageGo to next message
Shavarsh Odajavan is currently offline Shavarsh Odajavan
Messages: 7
Registered: March 2013
Junior Member
Exactly, you need to remove the implemented parse method and implement a few methods like createLexer, createTokenStream and createParser. See below for what i did. Also don't forget to edit getDefaultRuleName() returning your specific entry rule. Hope it helps.

@Override
	protected TokenSource createLexer(CharStream stream) {
		Lexer lexer = new IndentingLexer(stream);
		lexer.setCharStream(stream);
		return lexer;
	}

	@Override
	protected XtextTokenStream createTokenStream(TokenSource tokenSource) {
		XtextTokenStream stream = new XtextTokenStream(tokenSource, getTokenDefProvider());
		stream.setInitialHiddenTokens("RULE_WS", "RULE_ML_COMMENT", "RULE_SL_COMMENT");
		return stream;
	}
	
	@Override
	protected AbstractInternalAntlrParser createParser(XtextTokenStream stream) {
		InternalFallParser parser = new InternalFallParser(stream, getGrammarAccess());
		return parser;
	}
Re: Python-like indentation and editor problems [message #1063055 is a reply to message #1053203] Tue, 11 June 2013 19:15 Go to previous messageGo to next message
Farshad T is currently offline Farshad T
Messages: 2
Registered: June 2013
Junior Member

Hi Shavarsh,

Have you fixed the problem with your indentation aware lexer?

Re: Python-like indentation and editor problems [message #1064564 is a reply to message #1015898] Wed, 19 June 2013 18:32 Go to previous message
Farshad T is currently offline Farshad T
Messages: 2
Registered: June 2013
Junior Member
Create the new commonToken with empty text and do no change the start end index of the existing tokens. This way you will get a partial working solution.

Basically, each time lexing/parsing is done on the whole text in a file, it works correctly. However, when the lexing/parsing is done a limited part of the text the editor get confused. It seems that in this conditions the index of each token is calculated automatically from the AST tree and mistakenly shifted.

I think the solution to this is to adjust the start end indexes manually , probable by providing a TreeAdaptor. However, I am not able to engage my TreeAdaptor to the generated plug-in. Please let me know if you could resolve this.
Previous Topic:Xtext starts the generator of the inherited plugin.
Next Topic:Cyclic resolution of lazy links
Goto Forum:
  


Current Time: Thu Aug 21 22:05:14 EDT 2014

Powered by FUDForum. Page generated in 0.02926 seconds