Eclipse Community Forums: TMF (Xtext) » Help with expressions, pretty please?

Home

Home » Modeling » TMF (Xtext) » Help with expressions, pretty please?

Help with expressions, pretty please? [message #755884]

Wed, 09 November 2011 20:33

Eclipse User

Greetings,

I apologize in advance for the length of this question Smile

I've struggling for a couple of days trying to figure out how to add expressions to the protobuf-dt grammar. I read Sven's post on expressions and it was great! I think I understood it. The problem is that I don't know how to apply what I learned.

What I'm trying to accomplish is something called "message notation" in protocol buffer grammar. For example:

option (abc) = 3;
option (spanner.database_options) = {
acls: {
owner: "chubby!*"
admin: "chubby!*"
field_acls: {
reader: "chubby!*"
writer: "chubby!*"
}
}
};

whose grammar can be defined as (this doesn't work in Xtext, but it is the simplest way to describe it)

ValueRef:
MessageNotation | SimpleRef;

// { foo: 1, bar: 2 }
MessageNotation:
'{'
fields+=FieldNotation ((',')? fields+=FieldNotation)*
'}'
;

FieldNotation:
name=Name ':' value=SimpleRef;

SimpleRef:
LiteralRef | BooleanRef | NumberRef | StringRef;

The value of an option can be a simple value (SimpleRef) or message notation (MessageNotation). MessageNotation has fields, and the value of each field can be a SimpleRef or another MessageNotation.

I have tried everything I could image, and nothing. Whatever I try fails when I generate Java code from the grammar.

Any help will be greatly, greatly, greatly appreciated! Smile

Many thanks in advance,
-Alex

Re: Help with expressions, pretty please? [message #755885 is a reply to message #755884]

Wed, 09 November 2011 20:34

Eclipse User

Oops! mistake:

FieldNotation:
name=Name ':' value=ValueRef;

Very Happy

Re: Help with expressions, pretty please? [message #756084 is a reply to message #755885]

Thu, 10 November 2011 14:45

Eclipse User

Hey Alex,

you may want to write sth like:

FieldNotation:
name=Name ':' (simpleRef=SimpleRef|msgNotation=MessageNotation);

FieldNotation:
  name=Name ':' Property;

Property:
  Attribute | MessageNotation;

Attribute:
  type=SimpleRef;

...

Always be aware of what the generator makes of it. As far as I know you cannot extend a Java class from an enum so that you have to use a container class.

regards,
Max

[Updated on: Thu, 10 November 2011 14:46] by Moderator

Re: Help with expressions, pretty please? [message #756095 is a reply to message #755884]

Thu, 10 November 2011 15:40

Eclipse User

On 10.11.11 02:33, Alex Ruiz wrote:
> Whatever I try fails when I generate Java code from the grammar.

Hi Alex,

please provide more information about the failure and about the concrete
attempt that lead to the error.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Re: Help with expressions, pretty please? [message #756284 is a reply to message #756095]

Fri, 11 November 2011 14:09

Eclipse User

Hi Max and Sebastian. Thank you so much for your help. It turns out that enabling backtrack in my .mwe2 file solved the problem. I Googled it to learn more about it but couldn't find anything useful. Even though this is something Antlr-related, it would be great to have the Xtext documentation explain what this does (just a suggestion.)

BTW, I had to set 'backtrack = true' in both parser.antlr.XtextAntlrGeneratorFragment and parser.antlr.XtextAntlrUiGeneratorFragment to make it work. Also, a lot of warnings I had before (when generating Java code) went away when enabling backtrack.

Cheers,
-Alex

Re: Help with expressions, pretty please? [message #756311 is a reply to message #756284]

Fri, 11 November 2011 16:56

Eclipse User

Please note that turning on backtracking without knowing why may lead to
surprises - you get a parse result, but may not be what you intended.

The ordering of rules become very important (longest sequence/highest
precedence first).

A : B | C | D;

Without backtracking - select the one that matches.
With backtracking - select the first that matches.

(I was bitten by this)

- henrik

On 11/11/11 8:09 PM, Alex Ruiz wrote:
> Hi Max and Sebastian. Thank you so much for your help. It turns out that
> enabling backtrack in my .mwe2 file solved the problem. I Googled it to
> learn more about it but couldn't find anything useful. Even though this
> is something Antlr-related, it would be great to have the Xtext
> documentation explain what this does (just a suggestion.)
>
> BTW, I had to set 'backtrack = true' in both
> parser.antlr.XtextAntlrGeneratorFragment and
> parser.antlr.XtextAntlrUiGeneratorFragment to make it work. Also, a lot
> of warnings I had before (when generating Java code) went away when
> enabling backtrack.
>
> Cheers,
> -Alex

Re: Help with expressions, pretty please? [message #756317 is a reply to message #756311]

Fri, 11 November 2011 17:20

Eclipse User

Thanks so much Henrik for the tip. Luckily I have a decent test suite...and so far everything is working as usual Smile

By any chance you have a link handy that explains "backtrack"?

Cheers,
-Alex

Re: Help with expressions, pretty please? [message #756355 is a reply to message #756317]

Sat, 12 November 2011 06:04

Eclipse User

On 11/11/11 11:20 PM, Alex Ruiz wrote:
> Thanks so much Henrik for the tip. Luckily I have a decent test
> suite...and so far everything is working as usual :)
>
> By any chance you have a link handy that explains "backtrack"?
>
If you really want to get to the details, I recommend the ANTLR book:
http://pragprog.com/book/tpantlr/the-definitive-antlr-reference

In general: http://en.wikipedia.org/wiki/Backtracking

- henrik

Re: Help with expressions, pretty please? [message #756361 is a reply to message #756355]

Sat, 12 November 2011 06:26

Eclipse User

On the other hand: the situation looks simple enough to be able to do it without turning on backtracking, by an appropriate left-factorization. As Henrik points out, backtracking has its own problems and may even negatively influence the performance of the generated parser - especially on mall-formed input.

Re: Help with expressions, pretty please? [message #756975 is a reply to message #755884]

Tue, 15 November 2011 15:03

Eclipse User

Thanks Henrik and Meinte,

I've been reading about left factoring, and the examples I read were pretty easy to understand. My problem is that I don't know how to apply it to my grammar Sad

Here is the offending piece:

ComplexValue:
  '{'
  fields+=ValueField ((',')? fields+=ValueField)*
  '}';

ValueField:
  SimpleValueField | ComplexValueField;

SimpleValueField:
  name=FieldName ':' value=SimpleValueLink;

ComplexValueField:
  name=FieldName ':'? values=ComplexValue;

Any hints on how to left-factor this will be greatly appreciated Smile

Thanks,
-Alex

Re: Help with expressions, pretty please? [message #756979 is a reply to message #756975]

Tue, 15 November 2011 15:19

Eclipse User

I'm missing the SimpleValueLink and ComplexValue rules, which we'd need for left-factorization. The rules shown are not left-recursive per sé, but the parser might have a hard time distinguishing between SimpleValueLink and ComplexValue after having consumed the tokens for the FieldName rule and the semi-colon.

Re: Help with expressions, pretty please? [message #757000 is a reply to message #756979]

Tue, 15 November 2011 17:06

Eclipse User

Thanks so much Meinte for your help! I truly appreciate it.

I didn't know if I should specify the whole chain for rules, since it is pretty long. But at the same time, any of the involved rules might be responsible for the left-recursion. I tried to remove as much noise as possible.

SimpleValueLink:

SimpleValueLink:
  BooleanLink | NumberLink | StringLink;

BooleanLink:
  target=BOOL;

enum BOOL:
  true
  | false;

NumberLink:
  LongLink | DoubleLink;

LongLink:
  target=LONG;

terminal LONG returns ecore::ELong:
  ('-')? (NUMBER)+;

DoubleLink:
  target=DOUBLE;

terminal DOUBLE returns ecore::EDouble:
  (('-')? (NUMBER)* ('.' (NUMBER)+)? (('e'|'E')('-')? (NUMBER)+)?) | 'nan' | 'inf' | '-inf';

terminal NUMBER:
  '0'..'9';

StringLink:
  target=STRING;

terminal STRING:
  SL_STRING (SL_STRING)*;

terminal SL_STRING:
  '"' ('\\' ('"' | "'" | '\\' | !('\\' | '"')) | !('\\' | '"'))* '"' (WS)* |
  "'" ('\\' ('"' | "'" | '\\' | !('\\' | "'")) | !('\\' | "'"))* "'" (WS)*;

ComplexValue:

ComplexValue:
  '{'
  fields+=ValueField ((',')? fields+=ValueField)*
  '}';

I'm so sorry for abusing your kindness, but I was thinking that probably looking at the whole grammar would be better. If you think that is the case, the grammar can be found here: http://code.google.com/p/protobuf-dt/source/browse/com.google.eclipse.protobuf/src/com/google/eclipse/protobuf/Protobuf.xtext

Many, many thanks!

-Alex

Re: Help with expressions, pretty please? [message #757003 is a reply to message #757000]

Tue, 15 November 2011 17:43

Eclipse User

I got it! The problem was not there. The issue is that the protobuf language allows keywords to be identifiers. So I had this:

ame:
  ID | 'package' | 'import' | 'public' | 'option' | 'extend' | 'message' | 'optional' | 'required' | 'repeated' |
  'group' | 'enum' | 'service' | 'rpc' | 'returns' | 'true' | 'false' | 'default' | 'extensions' | 'to' | 'max' |
  'double' | 'float' | 'int32' | 'int64' | 'uint32' | 'uint64' | 'sint32' | 'sint64' | 'fixed32' | 'fixed64' |
  'sfixed32' | 'sfixed64' | 'bool' | 'string' | 'bytes';

Once I made Name to be just ID, the grammar compiles fine. Now I have to figure out how to have keywords as identifiers Smile

Cheers!

Re: Help with expressions, pretty please? [message #757051 is a reply to message #757003]

Wed, 16 November 2011 04:20

Eclipse User

You could absolve your guilty conscience by hiring me: http://www.dslconsultancy.com/ Wink

Anyway, you pattern for having keywords as identifiers is essentially correct, but it would cause left-recursion in cases where any of those keywords can also have a non-ID meaning (inside an expression). You can deal with this in a number of ways: (1) write your own, custom lexer (not recommended: a lot of work and brittle), (2) use different datatype rules for the various cases so you can weed out the keywords which have contention, (3) trust ANTLR and a whole bunch of unit tests to choose the right alternative - this only works if you only have ambiguity and not true non-LL(*) behavior/left-recursion.

Previous Topic:	Problem with name provide class
Next Topic:	Xtext generate is not deterministic!!

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 04:50:36 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter