Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » missing RULE_BEGIN if using whitespace-aware
missing RULE_BEGIN if using whitespace-aware [message #1707374] Fri, 04 September 2015 10:48 Go to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
I am trying to use xtext to create python like grammar via whitespace-aware.
However, if I run this and test it, I get missing RULE_BEGIN at indented text


Re: missing RULE_BEGIN if using whitespace-aware [message #1707400 is a reply to message #1707374] Fri, 04 September 2015 13:29 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
can you share your implementations of AbstractIndentationTokenSource and a sample model as well

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707404 is a reply to message #1707400] Fri, 04 September 2015 13:59 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
The problem is the split of nl and ws.
Can you file a ticket


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707407 is a reply to message #1707404] Fri, 04 September 2015 14:11 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
(the problematic point is org.eclipse.xtext.parser.antlr.AbstractIndentationTokenSource.computeIndentation(String)

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707410 is a reply to message #1707407] Fri, 04 September 2015 14:42 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Christian Dietrich wrote on Fri, 04 September 2015 14:11
(the problematic point is org.eclipse.xtext.parser.antlr.AbstractIndentationTokenSource.computeIndentation(String)


	protected int computeIndentation(String text) {
		int result = 0;
		for(int i = text.length() - 1; i>=0; i--) {
			char c = text.charAt(i);
			if (c == '\n' || c == '\r') {
				return result;
			}
			if (c == '\t') {
				result+=getTabWidth();
			} else {
				result++;
			}
		}
		return -1;
	}


Is what I have in that class.

Is that a bug in the xtext? If so, can I fix it and inject the correct one?

[Updated on: Fri, 04 September 2015 14:46]

Report message to a moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707412 is a reply to message #1707410] Fri, 04 September 2015 14:49 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
I am note sure if it a bug. Please file a ticket. Anyway.. Why dont you simply throw away and use a ws rule that allows newline space and tab

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707419 is a reply to message #1707412] Fri, 04 September 2015 15:17 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Christian Dietrich wrote on Fri, 04 September 2015 14:49
I am note sure if it a bug. Please file a ticket. Anyway.. Why dont you simply throw away and use a ws rule that allows newline space and tab

Wouldn't that be incorrect? I need newline as a separate token, but now it would match whitespace as well or only some.
Re: missing RULE_BEGIN if using whitespace-aware [message #1707423 is a reply to message #1707419] Fri, 04 September 2015 15:33 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
You can increase decrease indention only after a newline. This is
Why xtext looks for these newline. Can you give it a try with a simplyfied language


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707434 is a reply to message #1707423] Fri, 04 September 2015 17:05 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Christian Dietrich wrote on Fri, 04 September 2015 15:33
You can increase decrease indention only after a newline. This is
Why xtext looks for these newline. Can you give it a try with a simplyfied language

Alright, tried it out, works fine, but now the grammar is invalid python because I am missing new line token Sad
Re: missing RULE_BEGIN if using whitespace-aware [message #1707438 is a reply to message #1707434] Fri, 04 September 2015 17:29 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
i am not firm in python. can you give a example that fails?
can you please file a bug that explains your problem and requirement and why you need to separate
NL and WS


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707442 is a reply to message #1707438] Fri, 04 September 2015 18:03 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Well any block with more than one statement

def test():
    print "hello, world"
    return 1


return 1 will cry about expecting newline, because statement has new line as token to separate statements

Do you have a link where to put the bug report?

[Updated on: Fri, 04 September 2015 18:04]

Report message to a moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707444 is a reply to message #1707442] Fri, 04 September 2015 18:11 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13939
Registered: July 2009
Senior Member
https://bugs.eclipse.org/bugs/enter_bug.cgi?product=TMF&component=Xtext

the following works for me ?!?

grammar org.xtext.example.mydsl3.MyDsl hidden (WS, SL_COMMENT)

generate myDsl "http://www.xtext.org/example/mydsl3/MyDsl"
import "http://www.eclipse.org/emf/2002/Ecore" as ecore


file_input:
	{file_input}
 ( los+=stmt )*
 ;
 
 if_stmt:
 'if' tests+=test ':' suite+=suite
;

test:
	value="true" | value="false"
;
 

suite:
 simple_stmt=suiteblock
;

suiteblock:
	BEGIN 
		stmts+=stmt+ 
	END
;

stmt:
	print_stmt | if_stmt
;

print_stmt:
	PRINT vlaue=STRING
;

 
terminal BEGIN: 'synthetic:BEGIN';
terminal END: 'synthetic:END';
terminal PRINT: 'print';

terminal STRING: 
			'"' ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|'"') )* '"'? |
			"'" ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|"'") )* "'"? |
			'"""' -> '"""'?;


terminal SL_COMMENT: '#' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS: ('\r'|'\n'|'\t'|' ')+;



if true:
    print "x"
    print "x"
print "y"



Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: missing RULE_BEGIN if using whitespace-aware [message #1707447 is a reply to message #1707444] Fri, 04 September 2015 18:34 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Christian Dietrich wrote on Fri, 04 September 2015 18:11
https://bugs.eclipse.org/bugs/enter_bug.cgi?product=TMF&component=Xtext

the following works for me ?!?

grammar org.xtext.example.mydsl3.MyDsl hidden (WS, SL_COMMENT)

generate myDsl "http://www.xtext.org/example/mydsl3/MyDsl"
import "http://www.eclipse.org/emf/2002/Ecore" as ecore


file_input:
	{file_input}
 ( los+=stmt )*
 ;
 
 if_stmt:
 'if' tests+=test ':' suite+=suite
;

test:
	value="true" | value="false"
;
 

suite:
 simple_stmt=suiteblock
;

suiteblock:
	BEGIN 
		stmts+=stmt+ 
	END
;

stmt:
	print_stmt | if_stmt
;

print_stmt:
	PRINT vlaue=STRING
;

 
terminal BEGIN: 'synthetic:BEGIN';
terminal END: 'synthetic:END';
terminal PRINT: 'print';

terminal STRING: 
			'"' ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|'"') )* '"'? |
			"'" ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|"'") )* "'"? |
			'"""' -> '"""'?;


terminal SL_COMMENT: '#' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS: ('\r'|'\n'|'\t'|' ')+;



if true:
    print "x"
    print "x"
print "y"



There are other problematic parts where new line is required. For instance empty lines at module level throw error right now. Also, for instance decorators require new line

@testdec
def x():
    pass

[Updated on: Fri, 04 September 2015 19:07]

Report message to a moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707478 is a reply to message #1707447] Sat, 05 September 2015 05:56 Go to previous messageGo to next message
Peter Vanusanik is currently offline Peter VanusanikFriend
Messages: 7
Registered: June 2015
Junior Member
Submitted the bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=476690

Do you think I can fix it on my own and inject fixed tokenizer?
Re: missing RULE_BEGIN if using whitespace-aware [message #1840169 is a reply to message #1707478] Thu, 08 April 2021 04:29 Go to previous message
wings cu is currently offline wings cuFriend
Messages: 14
Registered: January 2021
Junior Member
Do you solved your problem? I have the same problem in my dsl.
Previous Topic:Keywords available only in specific cases
Next Topic:Customize Scoping for inheritance
Goto Forum:
  


Current Time: Fri May 14 11:40:06 GMT 2021

Powered by FUDForum. Page generated in 0.23282 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top