Skip to main content



      Home
Home » Modeling » TMF (Xtext) » missing RULE_BEGIN if using whitespace-aware
missing RULE_BEGIN if using whitespace-aware [message #1707374] Fri, 04 September 2015 06:48 Go to next message
Eclipse UserFriend
I am trying to use xtext to create python like grammar via whitespace-aware.
However, if I run this and test it, I get missing RULE_BEGIN at indented text


Re: missing RULE_BEGIN if using whitespace-aware [message #1707400 is a reply to message #1707374] Fri, 04 September 2015 09:29 Go to previous messageGo to next message
Eclipse UserFriend
can you share your implementations of AbstractIndentationTokenSource and a sample model as well
Re: missing RULE_BEGIN if using whitespace-aware [message #1707404 is a reply to message #1707400] Fri, 04 September 2015 09:59 Go to previous messageGo to next message
Eclipse UserFriend
The problem is the split of nl and ws.
Can you file a ticket
Re: missing RULE_BEGIN if using whitespace-aware [message #1707407 is a reply to message #1707404] Fri, 04 September 2015 10:11 Go to previous messageGo to next message
Eclipse UserFriend
(the problematic point is org.eclipse.xtext.parser.antlr.AbstractIndentationTokenSource.computeIndentation(String)
Re: missing RULE_BEGIN if using whitespace-aware [message #1707410 is a reply to message #1707407] Fri, 04 September 2015 10:42 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Fri, 04 September 2015 14:11
(the problematic point is org.eclipse.xtext.parser.antlr.AbstractIndentationTokenSource.computeIndentation(String)


	protected int computeIndentation(String text) {
		int result = 0;
		for(int i = text.length() - 1; i>=0; i--) {
			char c = text.charAt(i);
			if (c == '\n' || c == '\r') {
				return result;
			}
			if (c == '\t') {
				result+=getTabWidth();
			} else {
				result++;
			}
		}
		return -1;
	}


Is what I have in that class.

Is that a bug in the xtext? If so, can I fix it and inject the correct one?

[Updated on: Fri, 04 September 2015 10:46] by Moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707412 is a reply to message #1707410] Fri, 04 September 2015 10:49 Go to previous messageGo to next message
Eclipse UserFriend
I am note sure if it a bug. Please file a ticket. Anyway.. Why dont you simply throw away and use a ws rule that allows newline space and tab
Re: missing RULE_BEGIN if using whitespace-aware [message #1707419 is a reply to message #1707412] Fri, 04 September 2015 11:17 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Fri, 04 September 2015 14:49
I am note sure if it a bug. Please file a ticket. Anyway.. Why dont you simply throw away and use a ws rule that allows newline space and tab

Wouldn't that be incorrect? I need newline as a separate token, but now it would match whitespace as well or only some.
Re: missing RULE_BEGIN if using whitespace-aware [message #1707423 is a reply to message #1707419] Fri, 04 September 2015 11:33 Go to previous messageGo to next message
Eclipse UserFriend
You can increase decrease indention only after a newline. This is
Why xtext looks for these newline. Can you give it a try with a simplyfied language
Re: missing RULE_BEGIN if using whitespace-aware [message #1707434 is a reply to message #1707423] Fri, 04 September 2015 13:05 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Fri, 04 September 2015 15:33
You can increase decrease indention only after a newline. This is
Why xtext looks for these newline. Can you give it a try with a simplyfied language

Alright, tried it out, works fine, but now the grammar is invalid python because I am missing new line token Sad
Re: missing RULE_BEGIN if using whitespace-aware [message #1707438 is a reply to message #1707434] Fri, 04 September 2015 13:29 Go to previous messageGo to next message
Eclipse UserFriend
i am not firm in python. can you give a example that fails?
can you please file a bug that explains your problem and requirement and why you need to separate
NL and WS
Re: missing RULE_BEGIN if using whitespace-aware [message #1707442 is a reply to message #1707438] Fri, 04 September 2015 14:03 Go to previous messageGo to next message
Eclipse UserFriend
Well any block with more than one statement

def test():
    print "hello, world"
    return 1


return 1 will cry about expecting newline, because statement has new line as token to separate statements

Do you have a link where to put the bug report?

[Updated on: Fri, 04 September 2015 14:04] by Moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707444 is a reply to message #1707442] Fri, 04 September 2015 14:11 Go to previous messageGo to next message
Eclipse UserFriend
https://bugs.eclipse.org/bugs/enter_bug.cgi?product=TMF&component=Xtext

the following works for me ?!?

grammar org.xtext.example.mydsl3.MyDsl hidden (WS, SL_COMMENT)

generate myDsl "http://www.xtext.org/example/mydsl3/MyDsl"
import "http://www.eclipse.org/emf/2002/Ecore" as ecore


file_input:
	{file_input}
 ( los+=stmt )*
 ;
 
 if_stmt:
 'if' tests+=test ':' suite+=suite
;

test:
	value="true" | value="false"
;
 

suite:
 simple_stmt=suiteblock
;

suiteblock:
	BEGIN 
		stmts+=stmt+ 
	END
;

stmt:
	print_stmt | if_stmt
;

print_stmt:
	PRINT vlaue=STRING
;

 
terminal BEGIN: 'synthetic:BEGIN';
terminal END: 'synthetic:END';
terminal PRINT: 'print';

terminal STRING: 
			'"' ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|'"') )* '"'? |
			"'" ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|"'") )* "'"? |
			'"""' -> '"""'?;


terminal SL_COMMENT: '#' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS: ('\r'|'\n'|'\t'|' ')+;



if true:
    print "x"
    print "x"
print "y"

Re: missing RULE_BEGIN if using whitespace-aware [message #1707447 is a reply to message #1707444] Fri, 04 September 2015 14:34 Go to previous messageGo to next message
Eclipse UserFriend
Christian Dietrich wrote on Fri, 04 September 2015 18:11
https://bugs.eclipse.org/bugs/enter_bug.cgi?product=TMF&component=Xtext

the following works for me ?!?

grammar org.xtext.example.mydsl3.MyDsl hidden (WS, SL_COMMENT)

generate myDsl "http://www.xtext.org/example/mydsl3/MyDsl"
import "http://www.eclipse.org/emf/2002/Ecore" as ecore


file_input:
	{file_input}
 ( los+=stmt )*
 ;
 
 if_stmt:
 'if' tests+=test ':' suite+=suite
;

test:
	value="true" | value="false"
;
 

suite:
 simple_stmt=suiteblock
;

suiteblock:
	BEGIN 
		stmts+=stmt+ 
	END
;

stmt:
	print_stmt | if_stmt
;

print_stmt:
	PRINT vlaue=STRING
;

 
terminal BEGIN: 'synthetic:BEGIN';
terminal END: 'synthetic:END';
terminal PRINT: 'print';

terminal STRING: 
			'"' ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|'"') )* '"'? |
			"'" ( '\\' . /* ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') */ | !('\\'|"'") )* "'"? |
			'"""' -> '"""'?;


terminal SL_COMMENT: '#' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS: ('\r'|'\n'|'\t'|' ')+;



if true:
    print "x"
    print "x"
print "y"



There are other problematic parts where new line is required. For instance empty lines at module level throw error right now. Also, for instance decorators require new line

@testdec
def x():
    pass

[Updated on: Fri, 04 September 2015 15:07] by Moderator

Re: missing RULE_BEGIN if using whitespace-aware [message #1707478 is a reply to message #1707447] Sat, 05 September 2015 01:56 Go to previous messageGo to next message
Eclipse UserFriend
Submitted the bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=476690

Do you think I can fix it on my own and inject fixed tokenizer?
Re: missing RULE_BEGIN if using whitespace-aware [message #1840169 is a reply to message #1707478] Thu, 08 April 2021 00:29 Go to previous message
Eclipse UserFriend
Do you solved your problem? I have the same problem in my dsl.
Previous Topic:Keywords available only in specific cases
Next Topic:Customize Scoping for inheritance
Goto Forum:
  


Current Time: Sun Jul 06 23:13:42 EDT 2025

Powered by FUDForum. Page generated in 0.08720 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top