Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Making an editor with xtext for an existing DSL (Robot Framework)
Making an editor with xtext for an existing DSL (Robot Framework) [message #837273] Thu, 05 April 2012 09:15 Go to next message
Mikko Korpela is currently offline Mikko Korpela
Messages: 9
Registered: April 2012
Junior Member
Hi,

(I'm a newbie in eclipse development and xtext)
I've been experimenting with xtext for a couple of days now. My goal has been to investigate if xtext could be the tech for our (Robot Framework Core) teams new editor. I have to say that my progress so far has been too good to be true - xtext is a very impressive tool.

Some backgroud about Robot Framework:
* Testing language (or Framework)
* Space sensitive syntax (separator between things is two or more spaces)
* In most places things are space and case insensitive "This is valid" == "t HISi Sv AliD"
* Can have direct links to Python and Java methods

So I have some questions related to our teams goals and also about ugly implementation details related to my prototype (hope someone can answer at least some of them):

1) Does xtext allow linking to definitions in other languages (Java and Python)? and is it easy to do?

2) Robot Framework has similar comments as in python ( line comment starting with "#") but the "#" can be escaped for other usages with "\". How should this kind of situations be handled? I assume I can not have a non matching look behind for "\" in my COMMENT terminal?

3) Where can I find more information about namespaces and scopes and some actual examples? (For example I need to solve the case and space insensitivity and also local variables etc.)

4) Is there a way to give preference to a match over some other? My problem is that in some cases I have a situations where there could be multiple matches and the grammar definition doesn't let me define these situations.

For example:

FooOrOtherWiseBar:
Foo | Bar;

Foo:
ID;

Bar:
ID;

--- Fails - I would like to have some way to say that TRY first match to Foo and if it fails try to match to Bar

Ok.. These are my questions so far. Hope this is not tl;dr post.

Cheers,
Mikko Korpela
Re: Making an editor with xtext for an existing DSL (Robot Framework) [message #837412 is a reply to message #837273] Thu, 05 April 2012 12:25 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
On 2012-05-04 15:15, Mikko Korpela wrote:
> Hi,
>
> (I'm a newbie in eclipse development and xtext)
> I've been experimenting with xtext for a couple of days now. My goal has
> been to investigate if xtext could be the tech for our (Robot Framework
> Core) teams new editor. I have to say that my progress so far has been
> too good to be true - xtext is a very impressive tool.
>
> Some backgroud about Robot Framework:
> * Testing language (or Framework)
> * Space sensitive syntax (separator between things is two or more spaces)
> * In most places things are space and case insensitive "This is valid"
> == "t HISi Sv AliD"
> * Can have direct links to Python and Java methods
>
> So I have some questions related to our teams goals and also about ugly
> implementation details related to my prototype (hope someone can answer
> at least some of them):
>
> 1) Does xtext allow linking to definitions in other languages (Java and
> Python)? and is it easy to do?
>
Xtend is written with Xtext and integrates with Java in a very nice way.
You can link to anything that is backed by an EMF model.

> 2) Robot Framework has similar comments as in python ( line comment
> starting with "#") but the "#" can be escaped for other usages with "\".
> How should this kind of situations be handled? I assume I can not have a
> non matching look behind for "\" in my COMMENT terminal?
>
You can not have COMMENT both be hidden and visible at the same time,
but you can write a rule like:

EscapedComment hidden(WS) : '\' COMMENT ;

> 3) Where can I find more information about namespaces and scopes and
> some actual examples? (For example I need to solve the case and space
> insensitivity and also local variables etc.)
>
The documentation is a good starting point.

> 4) Is there a way to give preference to a match over some other? My
> problem is that in some cases I have a situations where there could be
> multiple matches and the grammar definition doesn't let me define these
> situations.
>
Either use left factoring, or use semantic predicates. Look at how
general Expressions are expressed - that should give you an idea.

> For example:
>
> FooOrOtherWiseBar:
> Foo | Bar;
>
> Foo:
> ID;
>
> Bar:
> ID;
>
As your example is constructed it is impossible to pick one or the other
- it is truly ambiguous and picking one means that the other one would
never be picked.

> --- Fails - I would like to have some way to say that TRY first match to
> Foo and if it fails try to match to Bar
>
In order to get that behavior you need to turn on backtracking - which
should be your last resort (as you then must fully understand your
constructs and why you turn on backtracking - it is easy to get the
grammar to pass, but it may parse in ways that you find surprising and
not giving you the result you want at all. If you however turn on
backtracking, the first path where parsing succeeds gets picked.

Contrast this with semantic predicate, where the semantic predicate
means, pick the marked path if the marked tokens are in the look-ahead.

Hope that helps.
- henrik
Re: Making an editor with xtext for an existing DSL (Robot Framework) [message #837500 is a reply to message #837412] Thu, 05 April 2012 14:59 Go to previous messageGo to next message
Mikko Korpela is currently offline Mikko Korpela
Messages: 9
Registered: April 2012
Junior Member
Hi,

Thanks for the response!

Still a bit overwhelmed from jumping in to the land of xtext after spending last 1.5 years with pythons and robots Smile

About my question 4:
I abstracted too much out of the example. So this little language definition hopefully points out what I'm aiming at.

Model:
(foos+=Foo)+
'{' (bars+=Bar ';')+ '}';

Foo:
bar=[Bar] | some=ID;

Bar:
name=ID;


So I have some text that can be a reference to another location or not.. and I'm unable to identify it before I have the actual elements (Bars). Are the situation and solutions still similar as the ones pointed out in the original response?

Kind Regards,
Mikko Korpela

[Updated on: Thu, 05 April 2012 15:01]

Report message to a moderator

Re: Making an editor with xtext for an existing DSL (Robot Framework) [message #837676 is a reply to message #837500] Thu, 05 April 2012 20:45 Go to previous message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2492
Registered: July 2009
Senior Member
On 2012-05-04 20:59, Mikko Korpela wrote:

> About my question 4:
> I abstracted too much out of the example. So this little language
> definition hopefully points out what I'm aiming at.
>
> grammar org.xtext.example.mydsl.MyDsl with
> org.eclipse.xtext.common.Terminals
>
> generate myDsl "HERE WAS A LINK BUT I CAN NOT USE IT BECAUSE I HAVE LESS
> THAN 25 MESSAGES POSTED TO THiS NEWSGROUP"
>
Hint, post "Forum rule that 25 posts must have been made in order to
post a link sucks!" 25 times in a suitable forum... (one that the
webmasters read ;)

> Model:
> (foos+=Foo)+
> '{' (bars+=Bar ';') '}';
>
> Foo:
> bar=[Bar] | some=ID;
>
> Bar:
> name=ID;
>
>
> So I have some text that can be a reference to another location or not..
> and I'm unable to identify it before I have the actual elements (Bars).
> Are the situation and solutions still similar as the ones pointed out in
> the original response?

ok, so that does not work - the parser is doing its work long before the
linker kicks in - the parser simply can not know what to do when it sees
an ID - should it use it as a reference to a Bar, or as "some" ID that
is not a reference to something that exists elsewhere. There is no
simple way to specify this in the grammar.

There are a couple of different solutions.

- Does ID always reference something that exists (ie. in both the Bar
and "some" cases)? If so, make it a reference to the supertype of all
the things that it may reference.

- Use some syntactic marker (but guessing that is out of the question if
the grammar for the language already exists) (e.g. ('+' bar = [Bar]) |
some = ID ;

- If what follows ID when used as a bar reference makes it
distinguishable from a "some" then you can use a predicate. i.e.

Foo: =>(bar = [Bar] SOMETHING_NOT_FOLLOWING_SOME) | some = ID ;

or vice versa after "some":

Foo: bar = [Bar] | =>(some = ID SOMETHING_NOT_FOLLOWING_BAR_REF) ;

Which means, if you at the point of the => see an ID, and then
SOMETHING_NOT_FOLLOWING_xxx then take this path. This disambiguates the
ID. The SOMETHING_NOT... is some token/rule that makes the combination
unique.

(Remember [x] is shorthand for [x | ID])

- Do your own linking - i.e. assign the ID to a feature, and then check
if it is a valid reference in the hook "afterModelLinked" in the linker.
(There are a few more things to do though regarding keeping track of
references (unresolved and resolved) to get the builds compute the
correct build order / what to rebuild when something changes. (I use
this approach in cloudsmith/geppetto @ github).

- If there is something in the ID that makes it possible to determine if
it is a Bar reference or not then you could try to tackle the problem
that way.

- You can write your own lexer that makes use of the global index of
exported elements. It looks up the ID, and if there is an exported Bar
in the index it produces a BarReference token instead of an ID token.

This is more complicated if there are references to a Bar defined in the
same resource. You basically have to parse the file with a second parser
to find all the bars, and then also include these in your lookup.

Your grammar then becomes:

Foo : bar = [Bar | BarReference] | some = ID ;
Bar : name = ID;

Hope this helps explaining how it works, and some of your options.

Regards
- henrik
Previous Topic:Model Access in Xtext
Next Topic:How to enable mark occurrences in Xtext 2.3M6?
Goto Forum:
  


Current Time: Sun Apr 20 18:10:13 EDT 2014

Powered by FUDForum. Page generated in 0.01691 seconds