Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Blank *not* being hidden (ignored) despite rules demanding it
Blank *not* being hidden (ignored) despite rules demanding it [message #1771312] Wed, 23 August 2017 21:59 Go to next message
David BlackFriend
Messages: 33
Registered: June 2017
Member
Hi,

I have run into a weird situation.

Despite the lines:
grammar org.instrumenteddreams.lang.Sable hidden(WS, ML_COMMENT, SL_COMMENT)

...

terminal WS:							(' ' | '\u000A' | '\u000D' | '\u0009' | '\u000B' | '\u000C' | '\u0000')+;



I am having errors when attempting to parse the following line:
'     let a = 5'

The leading whitespaces are (correctly) ignored, but the whitespace after the let word is not, and it is causing the following error:
java.lang.AssertionError: [XtextSyntaxDiagnostic: null:1 no viable alternative at input' ']

I've tried the following:

  • Specify the hidden tokens explicitely at a rule-level
  • Delete and re-generate the parser

but none of that worked.

Am I missing anything?

Thanks.

[Updated on: Wed, 23 August 2017 22:02]

Report message to a moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771315 is a reply to message #1771312] Wed, 23 August 2017 23:36 Go to previous messageGo to next message
Miguel Jimenez is currently offline Miguel JimenezFriend
Messages: 38
Registered: July 2015
Member
Hi David,

You may reuse the "Terminals" grammar that comes with Xtext. It already defines what a whitespace is. I tried it with your example
and it worked just fine. Here is the code:

My grammar:
grammar demo.Demo with org.eclipse.xtext.common.Terminals
generate demo "http://www.Demo.demo"

Model:
	variables+=VariableDecl*;
	
VariableDecl:
	'let' name=ID '=' value=INT;


If you don't want to use it, you can define what a WS is in your grammar and hide it from the parser:

grammar demo.Demo hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore

generate demo "http://www.Demo.demo"

Model:
	variables+=VariableDecl*;
	
VariableDecl:
	'let' name=ID '=' value=INT;

terminal ID  		: '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
terminal INT returns ecore::EInt: ('0'..'9')+;
terminal ML_COMMENT	: '/*' -> '*/';
terminal SL_COMMENT 	: '//' !('\n'|'\r')* ('\r'? '\n')?;
terminal WS			: (' '|'\t'|'\r'|'\n')+;
terminal ANY_OTHER: .;


It works as well.
I also tried your definition of the terminal WS in the previous code and it also works. So, you may want to provide more information.

I'm using Xtext 2.10.0
Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771386 is a reply to message #1771315] Thu, 24 August 2017 14:12 Go to previous messageGo to next message
David BlackFriend
Messages: 33
Registered: June 2017
Member
Hi Miguel,

thanks for your prompt reply.

My grammer is very large (about 1500 lines) and in addition part of a project which I cannot make public at the moment, so I cannot upload it here, unfortunately.

I've tried to reproduce the error on a small, testing purposes grammar by moving to it the rules I consider meaningful in this case. It turns out though that everything goes well when on this small grammar - I haven't been able to reproduce the error.

I'm now trying to comment out all the regions in my large grammar that are not directly related to this problematic rules and see what happens.

Any other idea about how to debug or approach this problem?

[Updated on: Thu, 24 August 2017 14:34]

Report message to a moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771391 is a reply to message #1771386] Thu, 24 August 2017 15:39 Go to previous messageGo to next message
David BlackFriend
Messages: 33
Registered: June 2017
Member
OK, I've found the problem by reducing the use cases to a minimum.

The following grammar ilustrates my mistake:
grammar org.instrumenteddreams.lang.Sable hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore

generate sable "http://www.instrumenteddreams.org/lang/Sable"

Sable hidden(WS, ML_COMMENT, SL_COMMENT):
	source+=Statement*;


Statement hidden(WS, ML_COMMENT, SL_COMMENT):
	((Declaration)) =>(';'?);

Declaration hidden(WS, ML_COMMENT, SL_COMMENT):
	ConstantOrVariableDeclaration;

ConstantOrVariableDeclaration hidden(WS, ML_COMMENT, SL_COMMENT):
	(ConstantDeclaration);

ConstantDeclaration hidden(WS, ML_COMMENT, SL_COMMENT):
	KEYWORD_LET IDENTIFIER '=' LITERAL_DECIMAL;
	
terminal fragment DECIMAL_DIGIT: 							'0'..'9';
terminal fragment IDENTIFIER_HEAD:
	('a'..'z') | ('A'..'Z') |
	'_' |
	'\u00A8' | '\u00AA' | '\u00AD' | '\u00AF' | ('\u00B2'..'\u00B5') | ('\u00B7'..'\u00BA')	|
	('\u00BC'..'\u00BE') | ('\u00C0'..'\u00D6') | ('\u00D8'..'\u00F6') | ('\u00F8'..'\u00FF') |
	('\u0100'..'\u02FF') | ('\u0370'..'\u167F') | ('\u1681'..'\u180D') | ('\u180F'..'\u1DBF') |
	('\u1E00'..'\u1FFF') |
	('\u200B'..'\u200D') | ('\u202A'..'\u202E') | ('\u203F'..'\u2040') | '\u2054' | ('\u2060'..'\u206F') |
	('\u2070'..'\u20CF') | ('\u2100'..'\u218F') | ('\u2460'..'\u24FF') | ('\u2776'..'\u2793') |
	('\u2C00'..'\u2DFF') | ('\u2E80'..'\u2FFF') |
	('\u3004'..'\u3007') | ('\u3021'..'\u302F') | ('\u3031'..'\u303F') | ('\u3040'..'\uD7FF') |
	('\uF900'..'\uFD3D') | ('\uFD40'..'\uFDCF') | ('\uFDF0'..'\uFE1F') | ('\uFE30'..'\uFE44') |
	('\uFE47'..'\uFFFD');
terminal fragment IDENTIFIER_CHARACTER:
	DECIMAL_DIGIT | 
	('\u0300'..'\u036F') | ('\u1DC0'..'\u1DFF') | ('\u20D0'..'\u20FF') | ('\uFE20'..'\uFE2F') |
	IDENTIFIER_HEAD;
terminal fragment IMPLICIT_PARAMETER_NAME: 					'$' DECIMAL_DIGIT+;
terminal fragment DECIMAL_LITERAL_CHARACTER:				DECIMAL_DIGIT | '_';
terminal fragment DECIMAL_LITERAL_CHARACTERS:				DECIMAL_LITERAL_CHARACTER+;

terminal LITERAL_DECIMAL:				DECIMAL_DIGIT DECIMAL_LITERAL_CHARACTERS?;	
terminal KEYWORD_LET: 					'let';
terminal CHAR_FROM_U000E_TO_UUU21: 		('\u000E'..'\u0021');
terminal IDENTIFIER:
	(IDENTIFIER_HEAD IDENTIFIER_CHARACTER*) |
	('`'(IDENTIFIER_HEAD IDENTIFIER_CHARACTER*)'`') |
	IMPLICIT_PARAMETER_NAME;
terminal ML_COMMENT: 					'/*' -> '*/';
terminal SL_COMMENT: 					'//' !('\n'|'\r')* ('\r'? '\n')?;
terminal WS:							(' ' | '\u000A' | '\u000D' | '\u0009' | '\u000B' | '\u000C' | '\u0000')+;




The problem is with the line:
terminal CHAR_FROM_U000E_TO_UUU21: 		('\u000E'..'\u0021');

Because it defines as a terminal blanks (the u+20 character). But blanks are also told to be hidden by the grammar, as seen above in diverse hidden clauses.

So it's my fault that I've specified inconsistent treatment of blanks.

Thanks for your help anyway!

[Updated on: Thu, 24 August 2017 15:41]

Report message to a moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771404 is a reply to message #1771391] Thu, 24 August 2017 17:52 Go to previous message
Miguel Jimenez is currently offline Miguel JimenezFriend
Messages: 38
Registered: July 2015
Member
I'm glad you could find the problem David. I've found that these kind of grammar issues are much easier to solve when you focus on a minimum example rather than the whole thing.
By the way, that name (CHAR_FROM_U000E_TO_UUU21) is a bit hard to decipher, a more descriptive name would be helpful in future situations!
Previous Topic:Using ParseHelper outside of the test project
Next Topic:Reference ecore model from other ecore model?
Goto Forum:
  


Current Time: Fri Jul 20 12:49:19 GMT 2018

Powered by FUDForum. Page generated in 0.03987 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top