Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Hidden tokens are replaced by a whitespace
Hidden tokens are replaced by a whitespace [message #1700838] Tue, 07 July 2015 14:04 Go to next message
Erika Baena is currently offline Erika BaenaFriend
Messages: 5
Registered: July 2015
Junior Member
Hi,

The attached folder contains 3 projects generated with Xtext that allows to have a reproducible example.

I have a short grammar that allows to write paragraphs delimited by begin and end. Comments should be accepted everywhere, that's why they are hidden.

grammar org.xtext.example.commentws.CommentWs
hidden(ML_COMMENT)
...

CwFile:
	{CwFile}
	elems+=Elem*;

Elem:
	'begin' SEPARATOR content=ID SEPARATOR 'end' SEPARATOR;

SEPARATOR:
	LINE_RETURN;

ID:
	ALPHABETIC+ (WS+ ALPHABETIC+)*;

terminal ML_COMMENT:
	'/*'->'*/';

terminal ALPHABETIC:
	'a'..'z' | 'A'..'Z';

terminal WS:
	' ' | '\t';

terminal LINE_RETURN:
	'\r'? '\n';


Reproductible example

The `tests` module contain a unit test that should success.

The following element content field (Elem.content) should be "some text" but I got "so me text".

begin
so/* a comment that should be ignored */me text
end


Installation details
* Eclipse IDE for Java developers 4.4.2
* Xtext Complete SDK 2.8.1

Is there a way to change this behaviour? Should I make a bug report?
Re: Hidden tokens are replaced by a whitespace [message #1700885 is a reply to message #1700838] Tue, 07 July 2015 20:56 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
the problem is the value converter. instead of the default IDValueConverter you should use something like

org.eclipse.xtext.xtext.XtextValueConverters.GrammarID()


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Hidden tokens are replaced by a whitespace [message #1700942 is a reply to message #1700885] Wed, 08 July 2015 11:04 Go to previous messageGo to next message
Erika Baena is currently offline Erika BaenaFriend
Messages: 5
Registered: July 2015
Junior Member
The value converter I should use is the result of a method call.
Is there a way to simply inject that or should I create a new class that implements IValueConverter and that uses an XtextValueConverters?
Re: Hidden tokens are replaced by a whitespace [message #1700943 is a reply to message #1700942] Wed, 08 July 2015 11:08 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14665
Registered: July 2009
Senior Member
try add a binding for AbstractIDValueConverter

if that does not work
Sublass DefaultTerminalConverters and override the method ID()


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Hidden tokens are replaced by a whitespace [message #1703577 is a reply to message #1700943] Mon, 03 August 2015 10:17 Go to previous message
Erika Baena is currently offline Erika BaenaFriend
Messages: 5
Registered: July 2015
Junior Member
The binding for AbstractIDValueConverter doesn't solve the issue but subclassing DefaultTerminalConverters does the job.

Thanks!
Previous Topic:How could i parse string and link its model with my files in eclipse project?
Next Topic:Extending Java Grammar
Goto Forum:
  


Current Time: Wed Apr 24 19:56:10 GMT 2024

Powered by FUDForum. Page generated in 0.06544 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top