Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Unclear (or problematic?) hidden tokens related behaviour(Clarifications regarding hidden tokens)
Unclear (or problematic?) hidden tokens related behaviour [message #1838020] Sat, 13 February 2021 10:54 Go to next message
Petru-Florin Mihancea is currently offline Petru-Florin MihanceaFriend
Messages: 4
Registered: February 2021
Junior Member
Hi.

Consider the next grammar. Note that it is just a toy grammar to present my question/problem.

grammar com.Test hidden(WS)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate test "Test"

X:
{X} (Y /*WS?*/ '/' Y) ;

Y hidden() :
ID ('->'+ ID)* '->'* ;

terminal ID:
'x';

terminal WS:
('\t' | ' ' | '\r' | '\n')+ ;

x/x is recognised ok.
x /x produces a parse error - no viable alternative at input ' '

If I uncomment the WS? part in rule X, the parsing is successful. It somehow looks that the space is seen during the processing of Y (where the space is relevant due to hidden() declaration at Y) but remains relevant when returning to X (where the space should be ignored due to the hidden declaration at the grammar level). Could somebody explain to me this behaviour? Do I miss something here?

Petru Florin Mihancea

ps: I use xtext 2.22.0
Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838026 is a reply to message #1838020] Sat, 13 February 2021 18:27 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7224
Registered: July 2009
Senior Member
Hi

Normally WS is hidden and that is it, so Xtext takes care of all whitespace.

However you have chosen to handle some whitespace explicitly. It seems to me inevitable that you and the tooling will both get confused and probably differently confused.

You will avoid problems by preferably leaving all whitespace to Xtext else do all whitespace explicitly.

Regards

Ed Willink
Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838061 is a reply to message #1838026] Mon, 15 February 2021 07:04 Go to previous messageGo to next message
Petru-Florin Mihancea is currently offline Petru-Florin MihanceaFriend
Messages: 4
Registered: February 2021
Junior Member
Hi. I agree, xtext should take care of WS(s) without any intervention / explicit handling. Unfortunately, it doesn't in this case! Please note that WS? is inside a comment in rule X, so there isn't any explicit occurence of WSs. As I explained in the initial message, I put that comment just to show that x /x should be recognised by the generated purser but it isn't. It is recognised only if I explicitly add that WS? (i.e., I uncomment that part of rule X), which is of course not desired. Since we both agree that xtext should handle by itself all WS tokens, it starts smelling like an issue.

Regards,
Petru Mihancea
Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838068 is a reply to message #1838061] Mon, 15 February 2021 10:01 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 7224
Registered: July 2009
Senior Member
Hi

Yes. Sorry the various * syntaxes confused me.

However in "Y hidden()" you have specified irregular control of whitespace. in a datatype rule. Better to have a terminal rule that directly parses your character sequences without backtracking a multi-'->' not followed by an ID.

Y: 'x' ('->'+ 'x'?)*

Regards

Ed Willink

[Updated on: Mon, 15 February 2021 19:24]

Report message to a moderator

Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838091 is a reply to message #1838068] Mon, 15 February 2021 19:54 Go to previous message
Petru-Florin Mihancea is currently offline Petru-Florin MihanceaFriend
Messages: 4
Registered: February 2021
Junior Member
Hi,

As I said, this is just a toy grammar emphasising the issue in a simpler way :). The grammar I am working on is much, much more complex. Anyway, the problem here is: does xtext behave correctly in the context of the exemplified grammar? hidden clause is perfectly valid for a datatype rule according to the xtext documentation (also mentioned as an explicit distinguishing characteristic of a data type rule when compared to a terminal). I am more and more tempted to say that this looks like a bug.

Regards,
Petru Mihancea
Previous Topic:EMF - adding new model objects programmatically
Next Topic:Derivation from OpenApi Specification
Goto Forum:
  


Current Time: Wed Aug 04 10:30:21 GMT 2021

Powered by FUDForum. Page generated in 0.02329 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top