Eclipse Community Forums: TMF (Xtext) » Unclear (or problematic?) hidden tokens related behaviour

Home » Modeling » TMF (Xtext) » Unclear (or problematic?) hidden tokens related behaviour(Clarifications regarding hidden tokens)

Unclear (or problematic?) hidden tokens related behaviour [message #1838020]

Sat, 13 February 2021 05:54

Eclipse User

Hi.

Consider the next grammar. Note that it is just a toy grammar to present my question/problem.

grammar com.Test hidden(WS)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate test "Test"

X:
{X} (Y /*WS?*/ '/' Y) ;

Y hidden() :
ID ('->'+ ID)* '->'* ;

terminal ID:
'x';

terminal WS:
('\t' | ' ' | '\r' | '\n')+ ;

x/x is recognised ok.
x /x produces a parse error - no viable alternative at input ' '

If I uncomment the WS? part in rule X, the parsing is successful. It somehow looks that the space is seen during the processing of Y (where the space is relevant due to hidden() declaration at Y) but remains relevant when returning to X (where the space should be ignored due to the hidden declaration at the grammar level). Could somebody explain to me this behaviour? Do I miss something here?

Petru Florin Mihancea

ps: I use xtext 2.22.0

Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838026 is a reply to message #1838020]

Sat, 13 February 2021 13:27

Eclipse User

Hi

Normally WS is hidden and that is it, so Xtext takes care of all whitespace.

However you have chosen to handle some whitespace explicitly. It seems to me inevitable that you and the tooling will both get confused and probably differently confused.

You will avoid problems by preferably leaving all whitespace to Xtext else do all whitespace explicitly.

Regards

Ed Willink

Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838061 is a reply to message #1838026]

Mon, 15 February 2021 02:04

Eclipse User

Hi. I agree, xtext should take care of WS(s) without any intervention / explicit handling. Unfortunately, it doesn't in this case! Please note that WS? is inside a comment in rule X, so there isn't any explicit occurence of WSs. As I explained in the initial message, I put that comment just to show that x /x should be recognised by the generated purser but it isn't. It is recognised only if I explicitly add that WS? (i.e., I uncomment that part of rule X), which is of course not desired. Since we both agree that xtext should handle by itself all WS tokens, it starts smelling like an issue.

Regards,
Petru Mihancea

Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838068 is a reply to message #1838061]

Mon, 15 February 2021 05:01

Eclipse User

Hi

Yes. Sorry the various * syntaxes confused me.

However in "Y hidden()" you have specified irregular control of whitespace. in a datatype rule. Better to have a terminal rule that directly parses your character sequences without backtracking a multi-'->' not followed by an ID.

Y: 'x' ('->'+ 'x'?)*

Regards

Ed Willink

[Updated on: Mon, 15 February 2021 14:24] by Moderator

Re: Unclear (or problematic?) hidden tokens related behaviour [message #1838091 is a reply to message #1838068]

Mon, 15 February 2021 14:54

Eclipse User

Hi,

As I said, this is just a toy grammar emphasising the issue in a simpler way :). The grammar I am working on is much, much more complex. Anyway, the problem here is: does xtext behave correctly in the context of the exemplified grammar? hidden clause is perfectly valid for a datatype rule according to the xtext documentation (also mentioned as an explicit distinguishing characteristic of a data type rule when compared to a terminal). I am more and more tempted to say that this looks like a bug.

Regards,
Petru Mihancea

Previous Topic:	EMF - adding new model objects programmatically
Next Topic:	Derivation from OpenApi Specification

Goto Forum:

-=] Back to Top [=-

Current Time: Tue May 13 14:29:06 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter