Eclipse Community Forums: TMF (Xtext)

Help

Home

Home » Modeling » TMF (Xtext) » Terminal confusion

Show: Today's Messages :: Show Polls :: Message Navigator

Terminal confusion [message #760144]

Thu, 01 December 2011 13:34

Andreas Graf

Messages: 211
Registered: July 2009

Senior Member

Dear Xtext-Gurus,

in which order are terminals matched during the parsing?

I am running into the following problem. My grammar both has possibilities to specify multiplicities

Multiplicity: "[" lower=INT ('..' (upper=INT|unbounded?='*'))? "]";

as well as NUMBER (used at some other place)

    terminal NUMBER returns ecore::EBigDecimal:
	('-')?('0'..'9')+ ('.' ('0'..'9')+);

Before introducing NUMBER, multiplicities worked fine with a syntax like this :

 [2..3]

However, this now results in:

Multiple markers at this line
- required (...)+ loop did not match anything at character '.'
- extraneous input '.' expecting RULE_INT

and I have to add extra whitespace.


 [2..3]

I have now idea why it gets confused, because NUMBER should not match anything in that statement.

Any ideas?

Thanks,

Andreas

Report message to a moderator

Re: Terminal confusion [message #760147 is a reply to message #760144]

Thu, 01 December 2011 13:39

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Hi,

terminals are *greedy*. As soon as "2." is read, the lexer goes on trying to match NUMBER and fails. This is why you would define number as a datatype rule rather than a terminal rule.

Alex

Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext@itemis.de

Report message to a moderator

Re: Terminal confusion [message #760148 is a reply to message #760147]

Thu, 01 December 2011 13:41

Andreas Graf

Messages: 211
Registered: July 2009

Senior Member

Thanks, that helps. So greedy in that context means not only that the regexp is greedy, but that ANTLR/Xtext does not try any other combinations.

Report message to a moderator

Re: Terminal confusion [message #760163 is a reply to message #760148]

Thu, 01 December 2011 13:59

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Indeed. Lexing is done before parsing, independent of the rest of the grammar (i.e. just terminal rules and keywords). As long as the token can be made "longer", the lexer goes on. And as there is a rule that fits the prefix "2.", it is chosen even if it is a dead end.

This is why you try to reduce the number of terminal rules to the absolute minimum and do as much using datatype rules as possible.

Alex

Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext@itemis.de

Report message to a moderator

Re: Terminal confusion [message #760167 is a reply to message #760144]

Thu, 01 December 2011 14:08

Sebastian Zarnekow

Messages: 3118
Registered: July 2009

Senior Member

Andreas,

please use a data type rule to model NUMBERs.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 01.12.11 14:34, schrieb Andreas Graf:
> Dear Xtext-Gurus,
>
> in which order are terminals matched during the parsing?
>
> I am running into the following problem. My grammar both has
> possibilities to specify multiplicities
>
> Multiplicity: "[" lower=INT ('..' (upper=INT|unbounded?='*'))? "]";
>
>
> as well as NUMBER (used at some other place)
>
>
> terminal NUMBER returns ecore::EBigDecimal:
> ('-')?('0'..'9')+ ('.' ('0'..'9')+);
>
>
> Before introducing NUMBER, multiplicities worked fine with a syntax like
> this :
>
> [2..3]
>
> However, this now results in:
>
> Multiple markers at this line
> - required (...)+ loop did not match anything at character '.'
> - extraneous input '.' expecting RULE_INT
>
>
> and I have to add extra whitespace.
>
> [2..3]
>
>
> I have now idea why it gets confused, because NUMBER should not match
> anything in that statement.
>
> Any ideas?
>
> Thanks,
>
> Andreas
>

Report message to a moderator

Re: Terminal confusion [message #760213 is a reply to message #760167]

Thu, 01 December 2011 15:51

Andreas Graf

Messages: 211
Registered: July 2009

Senior Member

I used datatype rule for NUMBER, but that still gets into conflicts with the terminal INT. I tried to replace INT in my grammar with DINT, but that then conflicts with the parts where I reference an XExpression.

XExpression internally uses INT, which is not datatype rule (shouldn't it be, then?)

Redefining XIntLiteral to use my DINT results in an ambiguous grammar Sad

Did anybody already manage to integrate NUMBERs with INTs in their grammars (and I need the ".." syntax for multiplicities

Regards,

Andreas

Report message to a moderator

Re: Terminal confusion [message #760216 is a reply to message #760213]

Thu, 01 December 2011 15:56

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

what does your NUMBER rule lool like?
make sure it does not start with
"terminal NUMBER..." because then it is a terminal rule. It should look something like
NUMBER returns ecore::EBigDecimal: ('-')?('0'..'9')+ ('.' ('0'..'9')+)?;

Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext@itemis.de

Report message to a moderator

Re: Terminal confusion [message #760309 is a reply to message #760216]

Thu, 01 December 2011 20:05

Sebastian Zarnekow

Messages: 3118
Registered: July 2009

Senior Member

Or probably

Number returns EBigDecimal hidden(): '-'? INT ('.' INT)?;

Just to be picky :-)

Andreas, a terminal can usually not conflict with a data type rule. You
have to make sure to not use Number and INT in an alternative.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 01.12.11 16:56, schrieb Alexander Nittka:
> what does your NUMBER rule lool like?
> make sure it does not start with
> "terminal NUMBER..." because then it is a terminal rule. It should look
> something like
> NUMBER returns ecore::EBigDecimal: ('-')?('0'..'9')+ ('.' ('0'..'9')+)?;

Report message to a moderator

Re: Terminal confusion [message #760333 is a reply to message #760216]

Thu, 01 December 2011 22:35

Andreas Graf

Messages: 211
Registered: July 2009

Senior Member

Hi Alexander,

is "('0'..'9')" supposed to work in data type rules? For me it seemed to only work for terminals and I had to use "'0'|'1'|'2' etc." in data type rules.

Andreas

Report message to a moderator