Eclipse Community Forums: TMF (Xtext) » dynamically accessing the list of authorized keywords

Home

Home » Modeling » TMF (Xtext) » dynamically accessing the list of authorized keywords

dynamically accessing the list of authorized keywords [message #533457]

Thu, 13 May 2010 22:58

Eclipse User

Hi,
I'm currently building a DSL with Xtext for a simulation platform
( http://code.google.com/p/gama-platform/source/browse/#svn/br anches )

The problem, in the language used in the platform, is that the lists of keywords (for example control commands or operators) are dynamically defined (basically, and without going into further details, the keywords available depend on the loading of specific libraries by the user).

Is there a way, like in a regular ANTLR grammar, to define the grammar without hard-coding the list of keywords ? (for instance, within the grammar, to know if a keyword belongs to the set of authorized operators, I'd like to call a function in Java). And without having to regenerate the whole XText project every time a new keyword is added ?

Thank you in advance !

Cheers

Re: dynamically accessing the list of authorized keywords [message #533514 is a reply to message #533457]

Fri, 14 May 2010 07:21

Eclipse User

Hi,

depending on what the rest of the grammar looks like, you can solve the problem, but not on the grammar level.

Rule: keyword=ID...;

Then do a semantic check whether the keyword entered is valid. Code completion and highlighting will have to be customised as well, of course.

Alex

Re: dynamically accessing the list of authorized keywords [message #533543 is a reply to message #533514]

Fri, 14 May 2010 09:52

Eclipse User

I have a similar issue in b3, and I did exactly what Alex suggests.

In my case I can get by with a fairly simple ("dynamic") grammar of:
ID '{' (ID '=' Expression)* '}'
This is all my extensions are allowed to specify, and this is easy to
handle in terms of validation, coloring, and code completion.

However, it gets complicated when there is a requirement to dynamically
add support for complex grammar fragments - i.e. could be as complex as
implementing support for the original grammar!
It would be really neat if it was possible to inject grammar/parser
fragments dynamically.

I started toying with the idea to do something like:

CustomRule : key=ID '{' extended += (ExtRules | ANY)* '}'
ExtKeywords : ID | STRING | INT | "keyword1" | "keyword2" | ... ;

And use a second parser on the result of the "extended", but I never
tried since my grammar is quite large, and my list of keywords would be
quite large, and for my (current) simple needs this was total overkill.
I realized that this was a poor man's "modal lexer", so with a bit of
googling...

I found that ANTLR has "lexer states" support by using a multiplexing
token stream
http://sds.sourceforge.net/src/antlr/doc/streams.html#lexers tates which
looks really interesting for this sort of thing. It requires being able
to switch lexer streams from within the Xtext grammar though. I did not
explore how that could be done, but I know it is possible to customize
the lexer used for an XText grammar (search earlier posts in this
newsgroup).

I hope that helps with some starting points for figuring out how to
handle this

Regards
- henrik

On 5/14/10 1:21 PM, Alexander Nittka wrote:
> Hi,
>
> depending on what the rest of the grammar looks like, you can solve the
> problem, but not on the grammar level.
>
> Rule: keyword=ID...;
>
> Then do a semantic check whether the keyword entered is valid. Code
> completion and highlighting will have to be customised as well, of course.
>
> Alex

Re: dynamically accessing the list of authorized keywords [message #533809 is a reply to message #533457]

Mon, 17 May 2010 03:54

Eclipse User

Well you could also use an intermediate solution and write your grammar normally but pre-process the grammar with a M2M transformation so that it gets transformed to what you want and then that transformed grammar is the one actually used in the xtext classical generation.

I did that successfully in some cases.

Re: dynamically accessing the list of authorized keywords [message #535528 is a reply to message #533457]

Mon, 24 May 2010 05:30

Eclipse User

actually what I'd like to do is something close to ANTLR:

unary : {isUnary (input.LT(1).getText())}? IDENTIFIER | type ; 
binary: {isBinary(input.LT(1).getText())}? IDENTIFIER ; 
type  : INT 
        | FLOAT 
        | POINT 
        | COLOR
        | STRING
        | BOOL
        | {isType(input.LT(1).getText())}? IDENTIFIER;

and something like:

syntax: binExp ';';
binExp: l=unarExp (op=binary r+=binExp)*;
unarExp: v=IDENTIFIER | op=unary u=unarExp;

is there any way to put some dynamic predicate (Java) in the Xtext grammar ?
or any other way to make this kind of simple grammar works ?

Re: dynamically accessing the list of authorized keywords [message #535646 is a reply to message #535528]

Tue, 25 May 2010 02:36

Eclipse User

Hi,

there is no way to express that in the grammar. The above suggestions
for customising validation, highlighting and code completion still
apply. Of course, you are free to replace almost any component in the
framework with your own implementation. But there is no out-of-the-box
support for what you want.

If it really is only about checking the validity of a keyword, the
easiest way is first accepting any ID and then doing the rest on the
semantic level rather than trying to solve that on the grammar level.
Reason: going that way, almost everything you have to do is already
documented somewhere and you don't have to dive too deep into the
internals of lexing, model instantiation etc.

Alex

Re: dynamically accessing the list of authorized keywords [message #536132 is a reply to message #533457]

Thu, 27 May 2010 04:23

Eclipse User

Hi Alex,

There is no problem for validation, highlighting and completion, where I check for keywords from a list.

But I would like to be able to differentiate

a b c

  b
 / \
a   c

from

a
 \
  b
   \
    c

because at the end I will have:

variables : ID;
unary     : ID;
binary    : ID;

and I need to instantiate the list of unary and binary keyword at the start of my application.

I was looking around, for instance in the Mwe2.xtext grammar

import "platform:/resource/org.eclipse.xtext.common.types/model/JavaVMTypes.ecore" as types

DeclaredProperty:
  'var' (type=[types::JvmType|FQN])? name=FQN ('=' default=Value)?;

actually, I'm quite interested in:
(type=[types::JvmType|FQN])

Is there a way to define some types like in org.eclipse.xtext.common.types associating a list of keyword that would be set before initializing the plugin, but at the startup of the final application ?

I'm looking for documentation, and read lots of code, hopping to figure out the best way.

Thank you for answers.
Pierrick

Re: dynamically accessing the list of authorized keywords [message #538904 is a reply to message #533457]

Tue, 08 June 2010 23:56

Eclipse User

reply to myself (and googlers),
cross-reference cannot be used as predicate cause for the parser it is seen as a FQN (ID) rule.
and cross-ref defined in the grammar are then managed by the linker later on. ( http://www.eclipse.org/Xtext/documentation/latest/xtext.html #syntax -> Cross References )

Previous Topic:	How to override Activator?
Next Topic:	Xtext 0.8.0 with Eclipse 3.5

Goto Forum:

-=] Back to Top [=-

Current Time: Sat Aug 16 05:50:19 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter