Eclipse Community Forums: TMF (Xtext) » problem with lexing terminal rules

Help

Home

Home » Modeling » TMF (Xtext) » problem with lexing terminal rules

Show: Today's Messages :: Show Polls :: Message Navigator

problem with lexing terminal rules [message #720310]

Tue, 30 August 2011 10:42

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Hi,

I have a very simple grammar, where I do not know how to solve lexing
problems for:

Model:
entries+=Entry*;

Entry:
Difference | DATE;

Difference :
INT '-' INT;

terminal DATE:
('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9');

My model simply consists out of lines which are either dates (in format
dd-mm-yyyy) or differences between integers (x-y).

The result of the parser is now:
30-08-11 -> correctly parsed as a DATE
5-3 -> correctly parsed as a Difference
12-5 -> Error: mismatched character '<EOF>' expecting set '0'..'9'

How can I make the third case working?

Thanks in advance for any hints!
Kai

Report message to a moderator

Re: problem with lexing terminal rules [message #720314 is a reply to message #720310]

Tue, 30 August 2011 11:15

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Hi,

make Date a datatype rule as well. You then register a value converter for date in order to ensure the correct format.

Alex

Report message to a moderator

Re: problem with lexing terminal rules [message #720366 is a reply to message #720314]

Tue, 30 August 2011 12:33

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Thanks a lot, Alex, this seems to be a valid workaround (although I
would have expected it to work with terminals as well).

Is it correct that I have to register the value converter on the "Entry"
rule and do the distinction between the possible data rules myself? At
least the toValue() never seems to be called for value converters for
the rules Date and Difference.

Cheers,
Kai

Am 8/30/2011 1:15 PM, schrieb Alexander Nittka:
> Hi,
>
> make Date a datatype rule as well. You then register a value converter
> for date in order to ensure the correct format.
>
> Alex

Report message to a moderator

Re: problem with lexing terminal rules [message #720370 is a reply to message #720366]

Tue, 30 August 2011 13:25

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Hi,

this is not a workaround but best practice! Terminal rules and datatype rules are used at different stages. Terminals are used during lexing and are used without context, the document is chopped up into tokens no matter whether they would be valid or not at the given position. Datatype rules are applied with respect to "structural" rules of the grammar.
The value converters can be registered only for terminal and datatype rules, as only they "return" simple data types. The remaining parser rules return EObjects. The toValue-method would be invoked during serialisation.

Alex

Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext@itemis.de

Report message to a moderator

Re: problem with lexing terminal rules [message #720508 is a reply to message #720310]

Tue, 30 August 2011 18:05

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

Take a look at:
http://henrik-eclipse.blogspot.com/2010/05/implementing-date-support-with-quickfix.html

It shows:
- Using an ecore data type in the grammar
- A Date value converter
- Overriding the SyntaxErrorMessageProvider
- Providing a quick fix for a ValueConverterException

- henrik

On 8/30/11 12:42 PM, Kai Kreuzer wrote:
> Hi,
>
> I have a very simple grammar, where I do not know how to solve lexing
> problems for:
>
> Model:
> entries+=Entry*;
>
> Entry:
> Difference | DATE;
>
> Difference :
> INT '-' INT;
>
> terminal DATE:
> ('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9');
>
>
> My model simply consists out of lines which are either dates (in format
> dd-mm-yyyy) or differences between integers (x-y).
>
> The result of the parser is now:
> 30-08-11 -> correctly parsed as a DATE
> 5-3 -> correctly parsed as a Difference
> 12-5 -> Error: mismatched character '<EOF>' expecting set '0'..'9'
>
> How can I make the third case working?
>
> Thanks in advance for any hints!
> Kai

Report message to a moderator

Re: problem with lexing terminal rules [message #720510 is a reply to message #720310]

Tue, 30 August 2011 18:05

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

Report message to a moderator

Re: problem with lexing terminal rules [message #720933 is a reply to message #720370]

Wed, 31 August 2011 13:49

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Hi Alex,

> this is not a workaround but best practice!

Even better :-)

> The toValue-method would be invoked during serialisation.

Isn't the toString-method called during serialisation and the
toValue-method during parsing? I'm confused now...

Doing all as data type rules, I indeed was able to get my example
working. Unfortunately, my REAL grammar is much more complex and I
stumble across another problem with the data type solution. I have
extracted it as a very simple grammar:

------
grammar org.xtext.example.mydsl.MyDsl with
org.eclipse.xtext.common.Terminals
generate myDsl "http://www.xtext.org/example/mydsl/MyDsl"

Model:
entries+= Entry*;

Entry:
value=MyDataType ';';

MyDataType :
INT ' ' INT (' ' INT)?;

------

The problem comes through the fact that ' ' is used in the optional part
of this data rule as well as it is treated as a hidden terminal.

The result is
1 2; --> ok
1 2 3; --> ok
1 2 ; --> fails with "expecting RULE_INT"
1 2 3 ; --> fails with "expecting ';'"

If I use another delimiter in my MyDataType (e.g. '-' instead of '-'),
it works as expected.

I get it working by changing the MyDataType rule to

MyDataType :
INT ' ' INT ((' ' INT ' '*) | ' '+)?;

but this now really looks like an ugly workaround! Do you have any
advice how to solve this in a nicer way?

~Kai

Report message to a moderator

Re: problem with lexing terminal rules [message #720934 is a reply to message #720508]

Wed, 31 August 2011 13:49

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Hi Henrik,

Thanks for the hint, I actually had already read your post some when
last year :-)
In general, I am well aware of how to use data rules and value
converters. Unfortunately, my current problem is a bit trickier as it
deals with overridden and hidden terminals.

Imagine for example you wanted to write your timestamps not as a STRING,
but as something like 2011-08-31 15:50:30 and this without any quotes.
My idea was to define a terminal rule for this, which would end up as a
single "chunk" in the parse tree.

~Kai

Am 8/30/2011 8:05 PM, schrieb Henrik Lindberg:
> Take a look at:
> http://henrik-eclipse.blogspot.com/2010/05/implementing-date-support-with-quickfix.html
>
>
> It shows:
> - Using an ecore data type in the grammar
> - A Date value converter
> - Overriding the SyntaxErrorMessageProvider
> - Providing a quick fix for a ValueConverterException
>
> - henrik
>
> On 8/30/11 12:42 PM, Kai Kreuzer wrote:
>> Hi,
>>
>> I have a very simple grammar, where I do not know how to solve lexing
>> problems for:
>>
>> Model:
>> entries+=Entry*;
>>
>> Entry:
>> Difference | DATE;
>>
>> Difference :
>> INT '-' INT;
>>
>> terminal DATE:
>> ('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9')'-'('0'..'9')('0'..'9');
>>
>>
>> My model simply consists out of lines which are either dates (in format
>> dd-mm-yyyy) or differences between integers (x-y).
>>
>> The result of the parser is now:
>> 30-08-11 -> correctly parsed as a DATE
>> 5-3 -> correctly parsed as a Difference
>> 12-5 -> Error: mismatched character '<EOF>' expecting set '0'..'9'
>>
>> How can I make the third case working?
>>
>> Thanks in advance for any hints!
>> Kai
>

Report message to a moderator

Re: problem with lexing terminal rules [message #720943 is a reply to message #720934]

Wed, 31 August 2011 14:15

Sven Efftinge

Messages: 1823
Registered: July 2009

Senior Member

I'd use a datatype rule and constraint the syntax regarding whitespace in a validation rule.

Sven

Report message to a moderator

Re: problem with lexing terminal rules [message #721258 is a reply to message #720943]

Thu, 01 September 2011 09:30

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Hi Sven,

> I'd use a datatype rule and constraint the syntax regarding whitespace
> in a validation rule.

Trying to use a data type rule brings me to the problems I mentioned to
Alex (see news://news.eclipse.org:119/j3ldg2$g67$1@news.eclipse.org).

Do I get it right, that if my data type rule contains white spaces (' ')
as part of its definition, I cannot "surround" such a data type with
hidden white spaces anymore? See my example "1 2;" vs. "1 2 ;", where
the latter does not work.

~Kai

Report message to a moderator

Re: problem with lexing terminal rules [message #721283 is a reply to message #721258]

Thu, 01 September 2011 10:30

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Hi,

have you tried
MyDatatypeRule: INT INT INT?;

Alex

Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext@itemis.de

Report message to a moderator

Re: problem with lexing terminal rules [message #721285 is a reply to message #721283]

Thu, 01 September 2011 10:41

Kai Kreuzer

Messages: 42
Registered: July 2009

Member

Not a bad idea, it makes at least my example cases work.
What I would nonetheless like is to make sure that there is only a
single white space between the parts, i.e. "1 2;" should NOT be
valid... I cannot do this check in the value converter as I already gets
the stripped down version "1 2" as an input for toValue().

~Kai

Report message to a moderator