Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » [Xtext] Embedding ANTLR code into XText?(Is it possible to include directives for the underlying parser technology in XText?)
[Xtext] Embedding ANTLR code into XText? [message #1706123] Sun, 23 August 2015 21:49 Go to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Dear all, I am currently working on an importer for transforming C code into EMF models (no validation required, I just want to have models for playing around). I have found that XText does not support semantic predicates (yet) which are required to, e.g., distinguish between typedefs types and variable names:
typedef int number;
number x;
int y;

It is possible to write a valid ANTLR grammar which is able to parse the example above correctly by using @header , @members, the semantic predicate for the rule type_id:
{isTypeName(input.LT(1).getText())}? IDENTIFIER
and other directives in ANTLR.

My question now is: As semantic predicates are not (yet) supported by XText, is it somehow possible to embed these ANTLR parser directives into XText?
Re: [Xtext] Embedding ANTLR code into XText? [message #1706430 is a reply to message #1706123] Wed, 26 August 2015 19:17 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Okay, here is what I have found out so far:

In the work-flow file (.mwe2 extension in the same folder as the xtext grammar) the following snippet is found:
// The antlr parser generator fragment.
fragment = parser.antlr.XtextAntlrGeneratorFragment auto-inject 


this injects XtextAntlrGeneratorFragment.java from https://github.com/eclipse/xtext/blob/master/plugins/org.eclipse.xtext.generator/src/org/eclipse/xtext/generator/parser/antlr/XtextAntlrGeneratorFragment.java

This class then generates the InternalC.g file in the src-gen folder based on the XtextAntlrGeneratorFragment.xpt template (found in the same folder as the XtextAntlrGeneratorFragment.java class.

Through expanding, the template AntlrGrammar.xpt gets processed and the @parser::header and @parser::members { directives get included into the ANTLR grammar file.
Re: [Xtext] Embedding ANTLR code into XText? [message #1706432 is a reply to message #1706430] Wed, 26 August 2015 19:27 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
So by now, I have added the necessary import ... statements and also the necessary addTypeName and isTypeName() Java-methods manually into the InternalC.g file. This could be later automated by using the information described above. Also adding a directive like if (this.isTypedef()) { this.addTypedef(input.get(0).getText()); } is no problem to be inserted manually and could be later automated. Where I still struggle is with the semantic predicates. Because when I do the following (manually at this moment):
// Rule typedefName
ruletypedefName returns [AntlrDatatypeRuleToken current=new AntlrDatatypeRuleToken()] 
    @init { enterRule(); 
    }
    @after { leaveRule(); }:
    
    {isTypeName(input.LT(1).getText())}?=>
    
    this_IDENTIFIER_0=RULE_IDENTIFIER    {
		$current.merge(this_IDENTIFIER_0);
		System.out.println("ID='" + $RULE_IDENTIFIER + "'");
    }

    { 
    newLeafNode(this_IDENTIFIER_0, grammarAccess.getTypedefNameAccess().getIDENTIFIERTerminalRuleCall()); 
    }

    ;

I get the following output message:
warning(200): InternalC.g:382:1: Decision can match input such as "{RULE_TYPEDEF..RULE_IDENTIFIER, '__m128'..'__m128i'}" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
Semantic predicates were present but were hidden by actions.

So the semantic predicate is ignored by ANTLR because of the structure of the generated grammar.

I have not found any documentation about XText's generated ANTLR grammar (found in InternalC.g).
Re: [Xtext] Embedding ANTLR code into XText? [message #1706433 is a reply to message #1706432] Wed, 26 August 2015 19:48 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Additional note: This website http://meri-stuff.blogspot.co.at/2012/12/antlr-semantic-predicates.html#BasicsTerminologyGatedSemanticPredicate explains in detail what semantic predicates are.
icon9.gif  Re: [Xtext] Embedding ANTLR code into XText? [message #1706451 is a reply to message #1706123] Thu, 27 August 2015 01:31 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
I have attached two ANTLR grammar files - both simplified versions of the C-grammar: the first C.g is a plain ANTLR-file created by myself where the semantic predicate is working without problems, the second InternalC.g is generated by XText where I have made changes explained above manually.

If I open them in AntlrWorks, both present the same syntax tree. Nevertheless, C.g is processed without problems by Antlr and InternalC.g shows the error messages mentioned above. Does anybody know why or how this could be improved? Anyone?
  • Attachment: C.g
    (Size: 9.00KB, Downloaded 113 times)
  • Attachment: InternalC.g
    (Size: 29.68KB, Downloaded 99 times)
Re: [Xtext] Embedding ANTLR code into XText? [message #1706452 is a reply to message #1706451] Thu, 27 August 2015 03:29 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Oke, I finally managed to find the source of the problem: ANTLR can not do hoisting if there are any actions in front of the alternatives which call rule typedefName (from rule typeSpecifier up to rule declarationSpecifiers). See https://github.com/timeraider4u/xtext-predicates/blob/master/xtext/InternalC.g.v2.patch for the changes that have been required to compile without any multiple alternatives warning.

I do not know how this could be fixed in a "good way" Sad .
Re: [Xtext] Embedding ANTLR code into XText? [message #1706587 is a reply to message #1706452] Thu, 27 August 2015 22:05 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
According to http://www.antlr.org/papers/LL-star/predicate-hoisting.pdf it is not possible to hoist semantic predicates if there are actions on the left side of the production: Quote:
We cannot hoist predicates over actions because they might be a function of that action
. XText is generating actions on the left side of the production to create the corresponding model. They are not influencing the predicates in any way.

I have now opened a question in the corresponding ANTLR Google discussion group under https://groups.google.com/forum/#!topic/antlr-discussion/UrZEt-uaMVA
to ask if it is possible to enforce the hoisting over actions under these circumstances.
Re: [Xtext] Embedding ANTLR code into XText? [message #1708033 is a reply to message #1706123] Fri, 11 September 2015 15:03 Go to previous message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
I have been successful with adding another grammar option to ANTLR to enforce hoisting of semantic predicates over actions.
My changes on the ANTLRv3 source code made so far can be viewed under https://github.com/antlr/antlr3/pull/177
Previous Topic:Adding to the import list in the generated code
Next Topic:new formatting API and Xbase inherited grammar
Goto Forum:
  


Current Time: Tue Mar 19 10:48:18 GMT 2024

Powered by FUDForum. Page generated in 0.02300 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top