Eclipse Community Forums: TMF (Xtext) » [Xtext] Grammar - Enhancing the infered model for language expressions

Help

Home

Home » Modeling » TMF (Xtext) » [Xtext] Grammar - Enhancing the infered model for language expressions

Show: Today's Messages :: Show Polls :: Message Navigator

[Xtext] Grammar - Enhancing the infered model for language expressions [message #660925]

Tue, 22 March 2011 11:38

Didier Villevalois

Messages: 86
Registered: July 2009

Member

Hi,

I create this new thread to resolve another problem I have on my
grammar. I have expressions in my language. The EBNF-like Xtext format
forces me to define my expression syntax in a non-left-recursive way.
Also I have operator precedence and this forces me to have different
rules for the different operators.

However from a generation point of view I would be glad to common
supertypes for UnaryOperationExpressions and BinaryExpressions.

Here is the ideal type hierarchy I would like:

BinaryExpression (left:Expression, right:Expression)
- ConditionalExpression (operator: enum ConditionalOperator)
- EqualityExpression (operator: enum EqualityOperator)
- RelationalExpression (operator: enum RelationalOperator)
- ArithmeticExpression (operator: enum ConditionalOperator)

Here is my actual grammar fragment for expressions:

<code>
Expression returns Expression:
{If} 'if' condition=ConditionalOrExpression 'then' then=Expression
'else' else=Expression |
{Let} 'let' definitions+=VariableDefinition ("," definitions
+=VariableDefinition)* 'in' result=Expression |
{Map} 'map' value=AccessExpression ('->' to=[ecore::EClassifier|
QualifiedName])? |
ConditionalOrExpression;

VariableDefinition returns VariableDefinition:
name=ID ':=' value=Expression;

ConditionalOrExpression returns Expression:
ConditionalAndExpression ({ConditionalExpression.left=current}
operator='or' right=ConditionalAndExpression)*;

ConditionalAndExpression returns Expression:
EqualityExpression ({ConditionalExpression.left=current} operator='and'
right=EqualityExpression)*;

enum ConditionalOperator:
Or='or' | And='and';

EqualityExpression returns Expression:
RelationalExpression ({EqualityExpression.left=current}
operator=EqualityOperator right=RelationalExpression)?;

enum EqualityOperator:
Equal='=' | Different='!=';

RelationalExpression returns Expression:
AdditiveExpression ({RelationalExpression.left=current}
operator=RelationalOperator right=AdditiveExpression)?;

enum RelationalOperator:
Less='<' | Greater='>' | LessOrEqual='<=' | GreaterOrEqual='=>';

AdditiveExpression returns Expression:
MultiplicativeExpression ({ArithmeticExpression.left=current}
operator=('+' | '-') right=MultiplicativeExpression)*;

MultiplicativeExpression returns Expression:
UnaryExpression ({ArithmeticExpression.left=current} operator=('*' |
'/') right=UnaryExpression)*;

enum ArithmeticOperator:
Add='+' | Substract='-' | Multiply='*' | Divide='/';

UnaryExpression returns Expression:
{UnaryExpression} operator=UnaryOperator operand=AccessExpression |
AccessExpression;

enum UnaryOperator:
Negation='!' | Minus='-';

AccessExpression returns Expression:
PrimaryExpression ({FeatureAccess.object=current} '.'
feature=[ecore::EStructuralFeature]
({Invocation.access=current} '(' (parameters+=Expression (","
parameters+=Expression)*)? ')')?)*;

PrimaryExpression returns Expression:
'(' Expression ')' | ObjectExpression | Literal |
{FunctionCall} name=ID '(' (parameters+=Expression ("," parameters
+=Expression)*)? ')';

ObjectExpression returns Expression:
{Source} 'source' |
{Extent} 'extent' |
{VariableUse} variable=[VariableDefinition|ID];

Literal returns Expression:
{BooleanLiteral} value=BooleanLiteralValue |
{IntegerLiteral} value=INT |
{FloatLiteral} value=FLOAT |
{StringLiteral} value=STRING;

enum BooleanLiteralValue:
False='false' | True='true';

FLOAT hidden():
INT '.' INT;
</code>

Here are my problems:

1) I can't use the ConditionalOperator enum as my ConditionalExpressions
must be expressed as two separate rules to ensure operator precedence.

I guess Xtext misses something like access a particular enum value.
Something like MyEnum.literalName. That would enable something like:

ConditionalOrExpression returns Expression:
ConditionalAndExpression ({ConditionalExpression.left=current}
operator=ConditionalOperator.Or right=ConditionalAndExpression)*;

Is there an other way to do the same thing ?

2) I can't have a common ConditionalExpression supertype. I tried to add
an 'artificial' rule (as said in the Xtext user guide) but this creates
a warning in ANTLR as this obviously creates a left-recusion even if the
rule is never called from anywhere.

ConditionalExpression:
ConditionalOrExpression | ConditionalAndExpression;

3) I can't pull up left and right feature although there are everywhere
of type Expression. I also tried to add an 'artificial' rule but this
doesn't pull up the left and right features as expected.

BinaryExpression:
ConditionalOrExpression | ConditionalAndExpression | EqualityExpression
| RelationalExpression | AdditiveExpression | MultiplicativeExpression;

The user guide says that it should pull up the common features (end of
section 3.3.2) or am I misunderstanding something ??

4) I will need to enhance my type checking (I use xtext-typesystem which
by the way would be good to integrate in Xtext) I will need to further
differentiate Multiplication from Division (as a Division needs to be of
the type of the left operand). But from a generation point of view I
really need to have my model to have common supertypes.

Infered model enhancement through Xtend looks attractive but I have no
hints in how modifying the type hierarchy (feature pull-ups and feature
types string->enum modification) would break what Xtext has generated.

Thanks for your help. Didier.

Report message to a moderator

Re: [Xtext] Grammar - Enhancing the infered model for language expressions [message #660954 is a reply to message #660925]

Tue, 22 March 2011 13:45

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

At some point you most likely want to shift to an external model instead
of generating it from the grammar. Using an imported model gives you
full control.

- henrik

On 3/22/11 12:38 PM, Didier Villevalois wrote:
> Hi,
>
> I create this new thread to resolve another problem I have on my
> grammar. I have expressions in my language. The EBNF-like Xtext format
> forces me to define my expression syntax in a non-left-recursive way.
> Also I have operator precedence and this forces me to have different
> rules for the different operators.
>
> However from a generation point of view I would be glad to common
> supertypes for UnaryOperationExpressions and BinaryExpressions.
>
> Here is the ideal type hierarchy I would like:
>
> BinaryExpression (left:Expression, right:Expression)
> - ConditionalExpression (operator: enum ConditionalOperator)
> - EqualityExpression (operator: enum EqualityOperator)
> - RelationalExpression (operator: enum RelationalOperator)
> - ArithmeticExpression (operator: enum ConditionalOperator)
>
> Here is my actual grammar fragment for expressions:
>
> <code>
> Expression returns Expression:
> {If} 'if' condition=ConditionalOrExpression 'then' then=Expression
> 'else' else=Expression |
> {Let} 'let' definitions+=VariableDefinition ("," definitions
> +=VariableDefinition)* 'in' result=Expression |
> {Map} 'map' value=AccessExpression ('->' to=[ecore::EClassifier|
> QualifiedName])? |
> ConditionalOrExpression;
>
> VariableDefinition returns VariableDefinition:
> name=ID ':=' value=Expression;
>
> ConditionalOrExpression returns Expression:
> ConditionalAndExpression ({ConditionalExpression.left=current}
> operator='or' right=ConditionalAndExpression)*;
>
> ConditionalAndExpression returns Expression:
> EqualityExpression ({ConditionalExpression.left=current} operator='and'
> right=EqualityExpression)*;
>
> enum ConditionalOperator:
> Or='or' | And='and';
>
> EqualityExpression returns Expression:
> RelationalExpression ({EqualityExpression.left=current}
> operator=EqualityOperator right=RelationalExpression)?;
>
> enum EqualityOperator:
> Equal='=' | Different='!=';
>
> RelationalExpression returns Expression:
> AdditiveExpression ({RelationalExpression.left=current}
> operator=RelationalOperator right=AdditiveExpression)?;
>
> enum RelationalOperator:
> Less='<' | Greater='>' | LessOrEqual='<=' | GreaterOrEqual='=>';
>
> AdditiveExpression returns Expression:
> MultiplicativeExpression ({ArithmeticExpression.left=current}
> operator=('+' | '-') right=MultiplicativeExpression)*;
>
> MultiplicativeExpression returns Expression:
> UnaryExpression ({ArithmeticExpression.left=current} operator=('*' |
> '/') right=UnaryExpression)*;
>
> enum ArithmeticOperator:
> Add='+' | Substract='-' | Multiply='*' | Divide='/';
>
> UnaryExpression returns Expression:
> {UnaryExpression} operator=UnaryOperator operand=AccessExpression |
> AccessExpression;
>
> enum UnaryOperator:
> Negation='!' | Minus='-';
>
> AccessExpression returns Expression:
> PrimaryExpression ({FeatureAccess.object=current} '.'
> feature=[ecore::EStructuralFeature]
> ({Invocation.access=current} '(' (parameters+=Expression (","
> parameters+=Expression)*)? ')')?)*;
>
> PrimaryExpression returns Expression:
> '(' Expression ')' | ObjectExpression | Literal |
> {FunctionCall} name=ID '(' (parameters+=Expression ("," parameters
> +=Expression)*)? ')';
>
> ObjectExpression returns Expression:
> {Source} 'source' |
> {Extent} 'extent' |
> {VariableUse} variable=[VariableDefinition|ID];
>
> Literal returns Expression:
> {BooleanLiteral} value=BooleanLiteralValue |
> {IntegerLiteral} value=INT |
> {FloatLiteral} value=FLOAT |
> {StringLiteral} value=STRING;
>
> enum BooleanLiteralValue:
> False='false' | True='true';
>
> FLOAT hidden():
> INT '.' INT;
> </code>
>
>
> Here are my problems:
>
> 1) I can't use the ConditionalOperator enum as my ConditionalExpressions
> must be expressed as two separate rules to ensure operator precedence.
>
> I guess Xtext misses something like access a particular enum value.
> Something like MyEnum.literalName. That would enable something like:
>
> ConditionalOrExpression returns Expression:
> ConditionalAndExpression ({ConditionalExpression.left=current}
> operator=ConditionalOperator.Or right=ConditionalAndExpression)*;
>
> Is there an other way to do the same thing ?
>
> 2) I can't have a common ConditionalExpression supertype. I tried to add
> an 'artificial' rule (as said in the Xtext user guide) but this creates
> a warning in ANTLR as this obviously creates a left-recusion even if the
> rule is never called from anywhere.
>
> ConditionalExpression:
> ConditionalOrExpression | ConditionalAndExpression;
>
> 3) I can't pull up left and right feature although there are everywhere
> of type Expression. I also tried to add an 'artificial' rule but this
> doesn't pull up the left and right features as expected.
>
> BinaryExpression:
> ConditionalOrExpression | ConditionalAndExpression | EqualityExpression
> | RelationalExpression | AdditiveExpression | MultiplicativeExpression;
>
> The user guide says that it should pull up the common features (end of
> section 3.3.2) or am I misunderstanding something ??
>
> 4) I will need to enhance my type checking (I use xtext-typesystem which
> by the way would be good to integrate in Xtext) I will need to further
> differentiate Multiplication from Division (as a Division needs to be of
> the type of the left operand). But from a generation point of view I
> really need to have my model to have common supertypes.
>
>
> Infered model enhancement through Xtend looks attractive but I have no
> hints in how modifying the type hierarchy (feature pull-ups and feature
> types string->enum modification) would break what Xtext has generated.
>
> Thanks for your help. Didier.
>

Report message to a moderator

Re: [Xtext] Grammar - Enhancing the infered model for language expressions [message #661015 is a reply to message #660954]

Tue, 22 March 2011 17:44

Didier Villevalois

Messages: 1
Registered: March 2011

Junior Member

Hi Henrik,

Henrik Lindberg wrote on Tue, 22 March 2011 09:45

At some point you most likely want to shift to an external model instead
of generating it from the grammar. Using an imported model gives you
full control.

OK I understand that maintaining my own model would enable me to have the needed type hierarchy and have features at the needed level in this hierarchy.

However, I think I still can't use values from the same enum in different parser rules. Am I wrong ?

> 1) I can't use the ConditionalOperator enum as my ConditionalExpressions
> must be expressed as two separate rules to ensure operator precedence.
>
> I guess Xtext misses something like access a particular enum value.
> Something like MyEnum.literalName. That would enable something like:
>
> ConditionalOrExpression returns Expression:
> ConditionalAndExpression ({ConditionalExpression.left=current}
> operator=ConditionalOperator.Or right=ConditionalAndExpression)*;
>
> Is there an other way to do the same thing ?

Report message to a moderator

Re: [Xtext] Grammar - Enhancing the infered model for language expressions [message #661023 is a reply to message #661015]

Tue, 22 March 2011 18:14

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

On 3/22/11 6:44 PM, Didier Villevalois wrote:
> Henrik Lindberg wrote on Tue, 22 March 2011 09:45
>> At some point you most likely want to shift to an external model
>> instead of generating it from the grammar. Using an imported model
>> gives you full control.
>
>
> OK I understand that maintaining my own model would enable me to have
> the needed type hierarchy and have features at the needed level in this
> hierarchy.
>
> However, I think I still can't use values from the same enum in
> different parser rules. Am I wrong ?
>

Yes, but you need two different rules - both returning the same Enum
type. Something like:

Add returns MyModel::Operator : op = '+' ;
Mul returns MyModel::Operator : op = '*' ;

or somesuch... But I don't know how well that works with the Xtext
grammar, if it needs to understand that it is an Enum or not. I ran into
some issues when using Enums in Eclipse b3, and ended up writing some
helpers (enum data type). For operators I used strings (and validation).
I know that others use references/links for the same thing.

- henrik

Report message to a moderator

Re: [Xtext] Grammar - Enhancing the infered model for language expressions [message #661294 is a reply to message #661023]

Thu, 24 March 2011 01:03

Sebastian Zarnekow

Messages: 3116
Registered: July 2009

Senior Member

Hi,

use the following and a value converter instead:

Add returns MyModel::Operator : '+' ;
Mul returns MyModel::Operator : '*' ;

See the docs for details on the IValueConverter.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 22.03.11 19:14, schrieb Henrik Lindberg:
> On 3/22/11 6:44 PM, Didier Villevalois wrote:
>> Henrik Lindberg wrote on Tue, 22 March 2011 09:45
>>> At some point you most likely want to shift to an external model
>>> instead of generating it from the grammar. Using an imported model
>>> gives you full control.
>>
>>
>> OK I understand that maintaining my own model would enable me to have
>> the needed type hierarchy and have features at the needed level in this
>> hierarchy.
>>
>> However, I think I still can't use values from the same enum in
>> different parser rules. Am I wrong ?
>>
>
> Yes, but you need two different rules - both returning the same Enum
> type. Something like:
>
> Add returns MyModel::Operator : op = '+' ;
> Mul returns MyModel::Operator : op = '*' ;
>
> or somesuch... But I don't know how well that works with the Xtext
> grammar, if it needs to understand that it is an Enum or not. I ran into
> some issues when using Enums in Eclipse b3, and ended up writing some
> helpers (enum data type). For operators I used strings (and validation).
> I know that others use references/links for the same thing.
>
> - henrik

Report message to a moderator

Re: [Xtext] Grammar - Enhancing the infered model for language expressions [message #1847761 is a reply to message #661294]

Sat, 06 November 2021 05:05

Mirko Raner

Messages: 124
Registered: July 2009
Location: New York City, NY

Senior Member

Has anyone found a solution to OP's question #3 in the meantime? (i.e., pulling up common features in an expression grammar)

I struggle with a very similar issue right now. My grammar contains an expression grammar like this:

Expression: Partition;

Partition returns Expression:
  Qualification ({Partition.dispatchObject=current} '??' => cases += ClosureWithParameters+)*;

Qualification returns Expression:
  Disjunction ({Qualification.left=current} operator=QualificationOperator right=Qualification)?;

Disjunction returns Expression:
  Conjunction ({Disjunction.left=current} operator=DisjunctionOperator right=Conjunction)*;

// ... about 10 to 15 additional productions with the same left/operator/right structure...

Exponentiation returns Expression:
  Prefixation ({Exponentiation.left=current} operator=ExponentiationOperator right=Prefixation)*;

Prefixation returns Expression:
  {Prefixation}
  operator=(PrefixOperator|PlusAdditionOperator|MinusAdditionOperator|PlusMinusAdditionOperator|MinusPlusAdditionOperator)
  operand=Prefixation | Invocation;

Invocation returns Expression:
	Primary
	(=>({Navigation.object=current} '.'? member=[MethodDeclaration])
	|=>({Invocation.target=current} arguments=Arguments))*;

In other words, most, but not all, of the expression grammar productions have a left/operator/right structure. I tried to extract common features into its own type by adding a production like this:

BinaryOperation: Qualification | Disjunction | ... | Exponentation;

This produces the desired hierarchy structure but does not pull up the left/right or operator features (BinaryOperation has no features of its own at all).

As a result, I have a significant amount of code duplication in the scoping, validation, and code generation areas of my code. For example, even though the code generation is identical for all binary operators, I have 15 or 20 similar methods that extract left/operator/right and pass it to a common method. I'd really like to get rid of these redundant methods as well as similar sets of methods in several other areas.

Is there a way in Xtext today to extract common features of types that appear in an expression grammar?

Report message to a moderator

Previous Topic:	log4j.properties not updating
Next Topic:	Simple SuperType Problem in Grammar - Cannot find compatible feature

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Jun 10 15:00:05 GMT 2023

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter