Eclipse Community Forums: TMF (Xtext) » Blank *not* being hidden (ignored) despite rules demanding it

Home

Home » Modeling » TMF (Xtext) » Blank *not* being hidden (ignored) despite rules demanding it

Blank *not* being hidden (ignored) despite rules demanding it [message #1771312]

Wed, 23 August 2017 17:59

Eclipse User

Hi,

I have run into a weird situation.

Despite the lines:

grammar org.instrumenteddreams.lang.Sable hidden(WS, ML_COMMENT, SL_COMMENT)

...

terminal WS:							(' ' | '\u000A' | '\u000D' | '\u0009' | '\u000B' | '\u000C' | '\u0000')+;

I am having errors when attempting to parse the following line:

'     let a = 5'

The leading whitespaces are (correctly) ignored, but the whitespace after the let word is not, and it is causing the following error:

java.lang.AssertionError: [XtextSyntaxDiagnostic: null:1 no viable alternative at input' ']

I've tried the following:

Specify the hidden tokens explicitely at a rule-level
Delete and re-generate the parser

but none of that worked.

Am I missing anything?

Thanks.

[Updated on: Wed, 23 August 2017 18:02] by Moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771315 is a reply to message #1771312]

Wed, 23 August 2017 19:36

Eclipse User

Hi David,

You may reuse the "Terminals" grammar that comes with Xtext. It already defines what a whitespace is. I tried it with your example
and it worked just fine. Here is the code:

My grammar:

grammar demo.Demo with org.eclipse.xtext.common.Terminals
generate demo "http://www.Demo.demo"

Model:
	variables+=VariableDecl*;
	
VariableDecl:
	'let' name=ID '=' value=INT;

If you don't want to use it, you can define what a WS is in your grammar and hide it from the parser:

grammar demo.Demo hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore

generate demo "http://www.Demo.demo"

Model:
	variables+=VariableDecl*;
	
VariableDecl:
	'let' name=ID '=' value=INT;

terminal ID  		: '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
terminal INT returns ecore::EInt: ('0'..'9')+;
terminal ML_COMMENT	: '/*' -> '*/';
terminal SL_COMMENT 	: '//' !('\n'|'\r')* ('\r'? '\n')?;
terminal WS			: (' '|'\t'|'\r'|'\n')+;
terminal ANY_OTHER: .;

It works as well.
I also tried your definition of the terminal WS in the previous code and it also works. So, you may want to provide more information.

I'm using Xtext 2.10.0

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771386 is a reply to message #1771315]

Thu, 24 August 2017 10:12

Eclipse User

Hi Miguel,

thanks for your prompt reply.

My grammer is very large (about 1500 lines) and in addition part of a project which I cannot make public at the moment, so I cannot upload it here, unfortunately.

I've tried to reproduce the error on a small, testing purposes grammar by moving to it the rules I consider meaningful in this case. It turns out though that everything goes well when on this small grammar - I haven't been able to reproduce the error.

I'm now trying to comment out all the regions in my large grammar that are not directly related to this problematic rules and see what happens.

Any other idea about how to debug or approach this problem?

[Updated on: Thu, 24 August 2017 10:34] by Moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771391 is a reply to message #1771386]

Thu, 24 August 2017 11:39

Eclipse User

OK, I've found the problem by reducing the use cases to a minimum.

The following grammar ilustrates my mistake:

grammar org.instrumenteddreams.lang.Sable hidden(WS, ML_COMMENT, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore

generate sable "http://www.instrumenteddreams.org/lang/Sable"

Sable hidden(WS, ML_COMMENT, SL_COMMENT):
	source+=Statement*;


Statement hidden(WS, ML_COMMENT, SL_COMMENT):
	((Declaration)) =>(';'?);

Declaration hidden(WS, ML_COMMENT, SL_COMMENT):
	ConstantOrVariableDeclaration;

ConstantOrVariableDeclaration hidden(WS, ML_COMMENT, SL_COMMENT):
	(ConstantDeclaration);

ConstantDeclaration hidden(WS, ML_COMMENT, SL_COMMENT):
	KEYWORD_LET IDENTIFIER '=' LITERAL_DECIMAL;
	
terminal fragment DECIMAL_DIGIT: 							'0'..'9';
terminal fragment IDENTIFIER_HEAD:
	('a'..'z') | ('A'..'Z') |
	'_' |
	'\u00A8' | '\u00AA' | '\u00AD' | '\u00AF' | ('\u00B2'..'\u00B5') | ('\u00B7'..'\u00BA')	|
	('\u00BC'..'\u00BE') | ('\u00C0'..'\u00D6') | ('\u00D8'..'\u00F6') | ('\u00F8'..'\u00FF') |
	('\u0100'..'\u02FF') | ('\u0370'..'\u167F') | ('\u1681'..'\u180D') | ('\u180F'..'\u1DBF') |
	('\u1E00'..'\u1FFF') |
	('\u200B'..'\u200D') | ('\u202A'..'\u202E') | ('\u203F'..'\u2040') | '\u2054' | ('\u2060'..'\u206F') |
	('\u2070'..'\u20CF') | ('\u2100'..'\u218F') | ('\u2460'..'\u24FF') | ('\u2776'..'\u2793') |
	('\u2C00'..'\u2DFF') | ('\u2E80'..'\u2FFF') |
	('\u3004'..'\u3007') | ('\u3021'..'\u302F') | ('\u3031'..'\u303F') | ('\u3040'..'\uD7FF') |
	('\uF900'..'\uFD3D') | ('\uFD40'..'\uFDCF') | ('\uFDF0'..'\uFE1F') | ('\uFE30'..'\uFE44') |
	('\uFE47'..'\uFFFD');
terminal fragment IDENTIFIER_CHARACTER:
	DECIMAL_DIGIT | 
	('\u0300'..'\u036F') | ('\u1DC0'..'\u1DFF') | ('\u20D0'..'\u20FF') | ('\uFE20'..'\uFE2F') |
	IDENTIFIER_HEAD;
terminal fragment IMPLICIT_PARAMETER_NAME: 					'$' DECIMAL_DIGIT+;
terminal fragment DECIMAL_LITERAL_CHARACTER:				DECIMAL_DIGIT | '_';
terminal fragment DECIMAL_LITERAL_CHARACTERS:				DECIMAL_LITERAL_CHARACTER+;

terminal LITERAL_DECIMAL:				DECIMAL_DIGIT DECIMAL_LITERAL_CHARACTERS?;	
terminal KEYWORD_LET: 					'let';
terminal CHAR_FROM_U000E_TO_UUU21: 		('\u000E'..'\u0021');
terminal IDENTIFIER:
	(IDENTIFIER_HEAD IDENTIFIER_CHARACTER*) |
	('`'(IDENTIFIER_HEAD IDENTIFIER_CHARACTER*)'`') |
	IMPLICIT_PARAMETER_NAME;
terminal ML_COMMENT: 					'/*' -> '*/';
terminal SL_COMMENT: 					'//' !('\n'|'\r')* ('\r'? '\n')?;
terminal WS:							(' ' | '\u000A' | '\u000D' | '\u0009' | '\u000B' | '\u000C' | '\u0000')+;

The problem is with the line:

terminal CHAR_FROM_U000E_TO_UUU21: 		('\u000E'..'\u0021');

Because it defines as a terminal blanks (the u+20 character). But blanks are also told to be hidden by the grammar, as seen above in diverse hidden clauses.

So it's my fault that I've specified inconsistent treatment of blanks.

Thanks for your help anyway!

[Updated on: Thu, 24 August 2017 11:41] by Moderator

Re: Blank *not* being hidden (ignored) despite rules demanding it [message #1771404 is a reply to message #1771391]

Thu, 24 August 2017 13:52

Eclipse User

I'm glad you could find the problem David. I've found that these kind of grammar issues are much easier to solve when you focus on a minimum example rather than the whole thing.
By the way, that name (CHAR_FROM_U000E_TO_UUU21) is a bit hard to decipher, a more descriptive name would be helpful in future situations!

Previous Topic:	Using ParseHelper outside of the test project
Next Topic:	Reference ecore model from other ecore model?

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 06:37:55 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter