Eclipse Community Forums: TMF (Xtext) » [Xtext] Embedding ANTLR code into XText?

Home » Modeling » TMF (Xtext) » [Xtext] Embedding ANTLR code into XText?(Is it possible to include directives for the underlying parser technology in XText?)

[Xtext] Embedding ANTLR code into XText? [message #1706123]

Sun, 23 August 2015 17:49

Eclipse User

Dear all, I am currently working on an importer for transforming C code into EMF models (no validation required, I just want to have models for playing around). I have found that XText does not support semantic predicates (yet) which are required to, e.g., distinguish between typedefs types and variable names:
typedef int number;
number x;
int y;

It is possible to write a valid ANTLR grammar which is able to parse the example above correctly by using @header , @members, the semantic predicate for the rule type_id:
{isTypeName(input.LT(1).getText())}? IDENTIFIER
and other directives in ANTLR.

My question now is: As semantic predicates are not (yet) supported by XText, is it somehow possible to embed these ANTLR parser directives into XText?

Re: [Xtext] Embedding ANTLR code into XText? [message #1706430 is a reply to message #1706123]

Wed, 26 August 2015 15:17

Eclipse User

Okay, here is what I have found out so far:

In the work-flow file (.mwe2 extension in the same folder as the xtext grammar) the following snippet is found:

// The antlr parser generator fragment.
fragment = parser.antlr.XtextAntlrGeneratorFragment auto-inject

this injects XtextAntlrGeneratorFragment.java from https://github.com/eclipse/xtext/blob/master/plugins/org.eclipse.xtext.generator/src/org/eclipse/xtext/generator/parser/antlr/XtextAntlrGeneratorFragment.java

This class then generates the InternalC.g file in the src-gen folder based on the XtextAntlrGeneratorFragment.xpt template (found in the same folder as the XtextAntlrGeneratorFragment.java class.

Through expanding, the template AntlrGrammar.xpt gets processed and the @parser::header and @parser::members { directives get included into the ANTLR grammar file.

Re: [Xtext] Embedding ANTLR code into XText? [message #1706432 is a reply to message #1706430]

Wed, 26 August 2015 15:27

Eclipse User

So by now, I have added the necessary import ... statements and also the necessary addTypeName and isTypeName() Java-methods manually into the InternalC.g file. This could be later automated by using the information described above. Also adding a directive like if (this.isTypedef()) { this.addTypedef(input.get(0).getText()); } is no problem to be inserted manually and could be later automated. Where I still struggle is with the semantic predicates. Because when I do the following (manually at this moment):

// Rule typedefName
ruletypedefName returns [AntlrDatatypeRuleToken current=new AntlrDatatypeRuleToken()] 
    @init { enterRule(); 
    }
    @after { leaveRule(); }:
    
    {isTypeName(input.LT(1).getText())}?=>
    
    this_IDENTIFIER_0=RULE_IDENTIFIER    {
		$current.merge(this_IDENTIFIER_0);
		System.out.println("ID='" + $RULE_IDENTIFIER + "'");
    }

    { 
    newLeafNode(this_IDENTIFIER_0, grammarAccess.getTypedefNameAccess().getIDENTIFIERTerminalRuleCall()); 
    }

    ;

I get the following output message:

warning(200): InternalC.g:382:1: Decision can match input such as "{RULE_TYPEDEF..RULE_IDENTIFIER, '__m128'..'__m128i'}" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
Semantic predicates were present but were hidden by actions.

So the semantic predicate is ignored by ANTLR because of the structure of the generated grammar.

I have not found any documentation about XText's generated ANTLR grammar (found in InternalC.g).

Re: [Xtext] Embedding ANTLR code into XText? [message #1706433 is a reply to message #1706432]

Wed, 26 August 2015 15:48

Eclipse User

Additional note: This website http://meri-stuff.blogspot.co.at/2012/12/antlr-semantic-predicates.html#BasicsTerminologyGatedSemanticPredicate explains in detail what semantic predicates are.

Re: [Xtext] Embedding ANTLR code into XText? [message #1706451 is a reply to message #1706123]

Wed, 26 August 2015 21:31

Eclipse User

I have attached two ANTLR grammar files - both simplified versions of the C-grammar: the first C.g is a plain ANTLR-file created by myself where the semantic predicate is working without problems, the second InternalC.g is generated by XText where I have made changes explained above manually.

If I open them in AntlrWorks, both present the same syntax tree. Nevertheless, C.g is processed without problems by Antlr and InternalC.g shows the error messages mentioned above. Does anybody know why or how this could be improved? Anyone?

Attachment: C.g
(Size: 9.00KB, Downloaded 146 times)
Attachment: InternalC.g
(Size: 29.68KB, Downloaded 130 times)

Re: [Xtext] Embedding ANTLR code into XText? [message #1706452 is a reply to message #1706451]

Wed, 26 August 2015 23:29

Eclipse User

Oke, I finally managed to find the source of the problem: ANTLR can not do hoisting if there are any actions in front of the alternatives which call rule typedefName (from rule typeSpecifier up to rule declarationSpecifiers). See https://github.com/timeraider4u/xtext-predicates/blob/master/xtext/InternalC.g.v2.patch for the changes that have been required to compile without any multiple alternatives warning.

I do not know how this could be fixed in a "good way" Sad

Re: [Xtext] Embedding ANTLR code into XText? [message #1706587 is a reply to message #1706452]

Thu, 27 August 2015 18:05

Eclipse User

According to http://www.antlr.org/papers/LL-star/predicate-hoisting.pdf it is not possible to hoist semantic predicates if there are actions on the left side of the production: Quote:

We cannot hoist predicates over actions because they might be a function of that action

. XText is generating actions on the left side of the production to create the corresponding model. They are not influencing the predicates in any way.

I have now opened a question in the corresponding ANTLR Google discussion group under https://groups.google.com/forum/#!topic/antlr-discussion/UrZEt-uaMVA
to ask if it is possible to enforce the hoisting over actions under these circumstances.

Re: [Xtext] Embedding ANTLR code into XText? [message #1708033 is a reply to message #1706123]

Fri, 11 September 2015 11:03

Eclipse User

I have been successful with adding another grammar option to ANTLR to enforce hoisting of semantic predicates over actions.
My changes on the ANTLRv3 source code made so far can be viewed under https://github.com/antlr/antlr3/pull/177

Previous Topic:	Adding to the import list in the generated code
Next Topic:	new formatting API and Xbase inherited grammar

Goto Forum:

-=] Back to Top [=-

Current Time: Sat Jul 05 18:21:23 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter