Eclipse Community Forums: TMF (Xtext) » [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well?

Help

Home

Home » Modeling » TMF (Xtext) » [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well?

Show: Today's Messages :: Show Polls :: Message Navigator

[Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219328]

Tue, 03 December 2013 11:07

Wayne Campbell

Messages: 11
Registered: June 2013

Junior Member

Hi,

I am currently devloping a grammar for a distributed file format. Each of these files has its own grammar, therefore my model looks like this:

...
terminal PLUS: '+';
terminal MINUS: '-';
terminal DOT: '.';
terminal ID : ('^'|'$'|DOT)? ('a'..'z'|'A'..'Z'|'_') (('a'..'z'|'A'..'Z'|'_'|'0'..'9'|PLUS|MINUS)|DOT)*;
...

Model: 
  AFile | BFile;

AFile:
  'Entity' '{'
  ...
  'name' '=' name=ID
  ... ;

BFile: ABCRecord | DEFRecord | TOPRecord;

ABCRecord: ... ;
DEFRecord: ... ;
TOPRecord: 'TOP' valA=INT valB=INT valC=INT ';' ;

Everything worked well until I found out that the following construct is legal:

file A:

Entity {
  ...
  name=TOP
  ...
}

file B:

ABC 111 222 333; 33 484 
DEF 333 444 555 xyz; 333
TOP 894 23 979;

As you can see I need 'TOP' in file A to be recognized as an ID, whereas in file B TOP denotes the beginning of a data line.

If I understand aright, the problem is that the lexer consumes 'TOP' before it applies the 'ID' rule.

Is there a nice way to deal with such a situation, e.g. telling the lexer that it should delay the recognition of 'TOP' tokens until the ID rule was applied to the input?

Thank you in advance!

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219330 is a reply to message #1219328]

Tue, 03 December 2013 11:28

Uli Merkel

Messages: 248
Registered: June 2013

Senior Member

I think thats what you get from your grammar already.

Because the "Entity" denotes that you are inside the AFile rules which do not recognise a "TOP" keyword.
So IIRC there is no conflict at all.

In the standard XTEXT terminals, ther is a build-in escape for IDs implemented:
a "^" in front of the ID which separetes the ID from a keyword is removed automatically.
So entering name=^TOP will be turned into a simple TOP

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219333 is a reply to message #1219330]

Tue, 03 December 2013 11:43

Wayne Campbell

Messages: 11
Registered: June 2013

Junior Member

Quote:

Because the "Entity" denotes that you are inside the AFile rules which do not recognise a "TOP" keyword.
So IIRC there is no conflict at all.

Hm, that's what I thought at first, too. But if I open file A with my Runtime Eclipse, then all occurences of 'TOP' are recognized as a keyword, causing an error ("Mismatched input 'TOP', expecting RULE_ID").
My standalone test tool reports the same syntax error.

Quote:

In the standard XTEXT terminals, ther is a build-in escape for IDs implemented:
a "^" in front of the ID which separetes the ID from a keyword is removed automatically.
So entering name=^TOP will be turned into a simple TOP

Yes, I am aware of that. Unfortunately I cannot modify the source files. The whole file format is a lot more complex, so I really need to avoid messing around with file contents manually.

--- EDIT ---
I changed the 'TOP' rule to 'TOP ', and that did the job, as there must be a whitespace anyway.

This works for now, and it does not smell as badly as manually crawling the input files. But if anybody has a even better solution I would be very happy Smile

--- /EDIT ---

[Updated on: Tue, 03 December 2013 12:05]

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219337 is a reply to message #1219333]

Tue, 03 December 2013 12:13

Uli Merkel

Messages: 248
Registered: June 2013

Senior Member

Hi Wayne,

great idea !!!!

Will put it into my "bag of tricks"

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219353 is a reply to message #1219333]

Tue, 03 December 2013 13:48

Sven Efftinge

Messages: 1823
Registered: July 2009

Senior Member

Hi Wayne,

for that you should declare a non-terminal rule like this:

MyIdWithKeywords :
ID | 'keyword1' | 'keyword2';

And then use that rule at the appropriate places.

Sven

Am 12/3/13 12:43 PM, schrieb Wayne Campbell:
> Quote:
>> Because the "Entity" denotes that you are inside the AFile rules which
>> do not recognise a "TOP" keyword.
>> So IIRC there is no conflict at all.
>
> Hm, that's what I thought at first, too. But if I open file A with my
> Runtime Eclipse, then all occurences of 'TOP' are recognized as a
> keyword, causing an error ("Mismatched input 'TOP', expecting RULE_ID").
> My standalone test tool reports the same syntax error.
>
>
> In the standard XTEXT terminals, ther is a build-in escape for IDs
> implemented:
> a "^" in front of the ID which separetes the ID from a keyword is
> removed automatically.
> So entering name=^TOP will be turned into a simple TOP
>
> Yes, I am aware of that. Unfortunately I cannot modify the source files.
> The whole file format is a lot more complex, so I really need to avoid
> messing around with file contents manually.

--
Need professional support for Xtext or other Eclipse Modeling technologies?
Go to: http://xtext.itemis.com
Twitter : @svenefftinge
Blog : http://blog.efftinge.de

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219377 is a reply to message #1219333]

Tue, 03 December 2013 15:12

Christian Dietrich

Messages: 13488
Registered: July 2009

Senior Member

Hi,

you can introduce a datatype rule for that

XID : ID | "keyword";

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219509 is a reply to message #1219353]

Wed, 04 December 2013 10:53

Uli Merkel

Messages: 248
Registered: June 2013

Senior Member

just for my curiosity:

for legacy DSL files like the one here.

Instead of adding possible keywords in any rule,
it looks like a nice way to specify keywords with a trailing space.
So there is never a conflicht with an ID
and users have not to be trained to use the caret

Or have I missed something important which makes this a bad option
beside the fact the user has to enter a space after each keyword?

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219518 is a reply to message #1219509]

Wed, 04 December 2013 11:42

Ed Willink

Messages: 6765
Registered: July 2009

Senior Member

Hi

Expect to have really confused users continually asking why "(x +
keyword)" fails to parse.

Regards

Ed Willink

On 04/12/2013 10:53, Uli Merkel wrote:
> just for my curiosity:
>
> for legacy DSL files like the one here.
>
> Instead of adding possible keywords in any rule,
> it looks like a nice way to specify keywords with a trailing space.
> So there is never a conflicht with an ID and users have not to be
> trained to use the caret
>
> Or have I missed something important which makes this a bad option
> beside the fact the user has to enter a space after each keyword?

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219531 is a reply to message #1219518]

Wed, 04 December 2013 12:29

Uli Merkel

Messages: 248
Registered: June 2013

Senior Member

Hi Ed,

thanks for pointing out a context where this will cause trouble.

I had not encountered this situation in the legacy areas I inspected,
just a simple KEYWORD myname KEYWORD1= abc, def, xyz
where the second keyword is "KEYWORD1="

So as long as a situation like the one you mentioned can be excluded,
it seems to be an acceptable way to specify the first KEYWORD as "KEYWORD ".
This way, an existing DSL will still be useable if it reads:

KEYWORD KEYWORD KEYWORD1= KEYWORD, KEYWORD1

[Updated on: Wed, 04 December 2013 15:21]

Report message to a moderator

Previous Topic:	Clean build in xtext
Next Topic:	Problem between Xbase 2.4.0 and 2.4.3

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Tue Jul 07 06:45:39 GMT 2020

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter