Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well?
[Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219328] Tue, 03 December 2013 11:07 Go to next message
Wayne Campbell is currently offline Wayne CampbellFriend
Messages: 11
Registered: June 2013
Junior Member
Hi,

I am currently devloping a grammar for a distributed file format. Each of these files has its own grammar, therefore my model looks like this:
...
terminal PLUS: '+';
terminal MINUS: '-';
terminal DOT: '.';
terminal ID : ('^'|'$'|DOT)? ('a'..'z'|'A'..'Z'|'_') (('a'..'z'|'A'..'Z'|'_'|'0'..'9'|PLUS|MINUS)|DOT)*;
...

Model: 
  AFile | BFile;

AFile:
  'Entity' '{'
  ...
  'name' '=' name=ID
  ... ;

BFile: ABCRecord | DEFRecord | TOPRecord;

ABCRecord: ... ;
DEFRecord: ... ;
TOPRecord: 'TOP' valA=INT valB=INT valC=INT ';' ;



Everything worked well until I found out that the following construct is legal:

file A:
Entity {
  ...
  name=TOP
  ...
}


file B:
ABC 111 222 333; 33 484 
DEF 333 444 555 xyz; 333
TOP 894 23 979;


As you can see I need 'TOP' in file A to be recognized as an ID, whereas in file B TOP denotes the beginning of a data line.

If I understand aright, the problem is that the lexer consumes 'TOP' before it applies the 'ID' rule.

Is there a nice way to deal with such a situation, e.g. telling the lexer that it should delay the recognition of 'TOP' tokens until the ID rule was applied to the input?


Thank you in advance!
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219330 is a reply to message #1219328] Tue, 03 December 2013 11:28 Go to previous messageGo to next message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 135
Registered: June 2013
Senior Member
I think thats what you get from your grammar already.

Because the "Entity" denotes that you are inside the AFile rules which do not recognise a "TOP" keyword.
So IIRC there is no conflict at all.

In the standard XTEXT terminals, ther is a build-in escape for IDs implemented:
a "^" in front of the ID which separetes the ID from a keyword is removed automatically.
So entering name=^TOP will be turned into a simple TOP
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219333 is a reply to message #1219330] Tue, 03 December 2013 11:43 Go to previous messageGo to next message
Wayne Campbell is currently offline Wayne CampbellFriend
Messages: 11
Registered: June 2013
Junior Member
Quote:

Because the "Entity" denotes that you are inside the AFile rules which do not recognise a "TOP" keyword.
So IIRC there is no conflict at all.

Hm, that's what I thought at first, too. But if I open file A with my Runtime Eclipse, then all occurences of 'TOP' are recognized as a keyword, causing an error ("Mismatched input 'TOP', expecting RULE_ID").
My standalone test tool reports the same syntax error.

Quote:

In the standard XTEXT terminals, ther is a build-in escape for IDs implemented:
a "^" in front of the ID which separetes the ID from a keyword is removed automatically.
So entering name=^TOP will be turned into a simple TOP

Yes, I am aware of that. Unfortunately I cannot modify the source files. The whole file format is a lot more complex, so I really need to avoid messing around with file contents manually.



--- EDIT ---
I changed the 'TOP' rule to 'TOP ', and that did the job, as there must be a whitespace anyway.

This works for now, and it does not smell as badly as manually crawling the input files. But if anybody has a even better solution I would be very happy Smile
--- /EDIT ---

[Updated on: Tue, 03 December 2013 12:05]

Report message to a moderator

Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219337 is a reply to message #1219333] Tue, 03 December 2013 12:13 Go to previous messageGo to next message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 135
Registered: June 2013
Senior Member
Hi Wayne,

great idea !!!!

Will put it into my "bag of tricks"
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219353 is a reply to message #1219333] Tue, 03 December 2013 13:48 Go to previous messageGo to next message
Sven Efftinge is currently offline Sven EfftingeFriend
Messages: 1771
Registered: July 2009
Senior Member
Hi Wayne,

for that you should declare a non-terminal rule like this:

MyIdWithKeywords :
ID | 'keyword1' | 'keyword2';

And then use that rule at the appropriate places.

Sven

Am 12/3/13 12:43 PM, schrieb Wayne Campbell:
> Quote:
>> Because the "Entity" denotes that you are inside the AFile rules which
>> do not recognise a "TOP" keyword.
>> So IIRC there is no conflict at all.
>
> Hm, that's what I thought at first, too. But if I open file A with my
> Runtime Eclipse, then all occurences of 'TOP' are recognized as a
> keyword, causing an error ("Mismatched input 'TOP', expecting RULE_ID").
> My standalone test tool reports the same syntax error.
>
>
> In the standard XTEXT terminals, ther is a build-in escape for IDs
> implemented:
> a "^" in front of the ID which separetes the ID from a keyword is
> removed automatically.
> So entering name=^TOP will be turned into a simple TOP
>
> Yes, I am aware of that. Unfortunately I cannot modify the source files.
> The whole file format is a lot more complex, so I really need to avoid
> messing around with file contents manually.


--
Need professional support for Xtext or other Eclipse Modeling technologies?
Go to: http://xtext.itemis.com
Twitter : @svenefftinge
Blog : http://blog.efftinge.de


--
Need professional support on Xtext or Xtend?
Mail to: xtext (at) itemis.com
Twitter : @svenefftinge
Blog : blog.efftinge.de
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219377 is a reply to message #1219333] Tue, 03 December 2013 15:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
Hi,

you can introduce a datatype rule for that

XID : ID | "keyword";
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219509 is a reply to message #1219353] Wed, 04 December 2013 10:53 Go to previous messageGo to next message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 135
Registered: June 2013
Senior Member
just for my curiosity:

for legacy DSL files like the one here.

Instead of adding possible keywords in any rule,
it looks like a nice way to specify keywords with a trailing space.
So there is never a conflicht with an ID
and users have not to be trained to use the caret

Or have I missed something important which makes this a bad option
beside the fact the user has to enter a space after each keyword?
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219518 is a reply to message #1219509] Wed, 04 December 2013 11:42 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 4188
Registered: July 2009
Senior Member
Hi

Expect to have really confused users continually asking why "(x +
keyword)" fails to parse.

Regards

Ed Willink


On 04/12/2013 10:53, Uli Merkel wrote:
> just for my curiosity:
>
> for legacy DSL files like the one here.
>
> Instead of adding possible keywords in any rule,
> it looks like a nice way to specify keywords with a trailing space.
> So there is never a conflicht with an ID and users have not to be
> trained to use the caret
>
> Or have I missed something important which makes this a bad option
> beside the fact the user has to enter a space after each keyword?
Re: [Xtext 2.4.3] What to do if 'keyword' can occur as an ID as well? [message #1219531 is a reply to message #1219518] Wed, 04 December 2013 12:29 Go to previous message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 135
Registered: June 2013
Senior Member
Hi Ed,

thanks for pointing out a context where this will cause trouble.

I had not encountered this situation in the legacy areas I inspected,
just a simple KEYWORD myname KEYWORD1= abc, def, xyz
where the second keyword is "KEYWORD1="

So as long as a situation like the one you mentioned can be excluded,
it seems to be an acceptable way to specify the first KEYWORD as "KEYWORD ".
This way, an existing DSL will still be useable if it reads:

KEYWORD KEYWORD KEYWORD1= KEYWORD, KEYWORD1

[Updated on: Wed, 04 December 2013 15:21]

Report message to a moderator

Previous Topic:Clean build in xtext
Next Topic:Problem between Xbase 2.4.0 and 2.4.3
Goto Forum:
  


Current Time: Sun Dec 21 22:18:24 GMT 2014

Powered by FUDForum. Page generated in 0.02046 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software