Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » OSS project needs help
icon4.gif  OSS project needs help [message #1240683] Thu, 06 February 2014 23:45 Go to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
Hi, I'm here to get one or more Xtext developers onboard an Open Source project which has got stalled due to our lack of knowledge with Xtext.

The contribution we are looking for is relative to some parser/lexer customization needed to improve grammar support to the Gherkin language.

I have identified a couple of viable solutions for the issues we are experiencing, but I'm struggling with having those implemented with Xtext: I'm sure somebody with a greater experience will have an easy time on that while collecting his own share of glory for helping out.

The community behind the project itself is bigger than it looks and I'm sure it will get much bigger once those issues will be resolved.

It would be nice if you jump in and give a look at what we have: if you are willing to lend an hand I'll be gratefull.

The project source code is available at https://github.com/rlogiacco/Natural/

[Updated on: Thu, 06 February 2014 23:46]

Report message to a moderator

Re: OSS project needs help [message #1241935 is a reply to message #1240683] Sat, 08 February 2014 19:46 Go to previous messageGo to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
Wow, almost 18k visualizations and not even a contact? Come on folks...
Re: OSS project needs help [message #1242091 is a reply to message #1240683] Sun, 09 February 2014 01:50 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2014-07-02 24:45, rlogiacco@xxxxxxxx Lo Giacco wrote:
> Hi, I'm here to get an Xtext developer onboard an Open Source project
> wich has been stalled due to my lack of knowledge with Xtext.
>
> The contribution we are looking for is relative to some parser/lexer
> customization needed to improve grammar support to the Gherkin language.
>
> I have identified a couple of viable solutions for the issues we are
> experiencing, but I'm struggling with having those implemented with
> Xtext: I'm sure somebody with a greater experience will have an easy
> time on that while collecting his own share of glory for helping out.
>
> The community behind the project itself is bigger than it looks and I'm
> sure it will get much bigger once those issues will be resolved.
>
> It would be nice if you jump in and give a look at what we have: if you
> are willing to lend an hand I'll be gratefull.
>
> The project source code is available at
> https://github.com/rlogiacco/Natural/

Love to help you, but I am afraid that jumping in and reading up on
everything you have done to date trying to find the problems you are
experiencing just takes too much time.

Will be happy to help you with answering concrete questions.

I have some immediate observations though by looking at the Xtext
grammar for JBehave.xtext and Ccumber.xtext that I found here:
https://github.com/rlogiacco/Natural/blob/master/org.agileware.natural.jbehave/src/org/agileware/natural/jbehave/JBehave.xtext
and here
https://github.com/rlogiacco/Natural/blob/master/org.agileware.natural.cucumber/src/org/agileware/natural/cucumber/Cucumber.xtext

I see you are using terminals for all sorts of things. That will simply
not work well. Strongly recommend using the standard terminals, possibly
with removal of the ^ escape that makes keywords into identifiers. Then
try to use Data Rules, and use hidden(...) if you
want your grammar rules to see whitespace.

I have not made a detail study of your terminals - but:

* it looks like they may be overlapping (will cause you lots of problems)
* you have spaces in what appears to be keywords (not good)
* terminals does not behave as keywords in editors etc.

As an example:

terminal I_WANT_TO: 'I want to';

Should be written as a rule:

i_want_to
: 'I' 'want' 'to'
;

i.e. no terminals, just keywords. If it is illegal to have more than a
single space between the words, then this should be validated. Also, if
you want to be able to support a user when user writes "i want to" and
help them upcase the 'i', you need to accept also a lower case 'i' and
then validate it as being in error. etc. etc.

In this terminal, you can experience surprises since it matches
all unicode characters until newline (including control characters, and
all kinds of punctuation).

terminal WORD: ('a'..'z' | 'A'..'Z') !(SPACE | '\n' | '\r')*;

This is problematic:

terminal EOL: '\r'? '\n'?;

Since that says that "nothing" is an EOL, and you use it both as
something required, and something optional in different rules.

In general:

* Add one language feature at a time

* Starts small - say, a grammar that accepts terminals in any order
create a parser and check that you get all tokens as you expect. If
you have errors here, it will drive you nuts trying to figure out
why the rest of the grammar does not work.

* Make the grammar as lenient as possible, do not try to capture every
semantic requirement with grammar rules (you are both likely to fail,
and it will result in a grammar that is poor at giving good
information to the user). Instead, deffer as many checks as possible
to validation.

* Once you are past the exploratory phase - switch to an external model
(copy the generated model, and maintain it by hand). If you don't then
you risk changing the API of the model substantially (you rather want
to be slapped by the grammar generation than having things
change automatically. (It is a pain to maintain the model manually
while exploring though).


OTOH - maybe the design "sort of works", you mentioned you have problems
with it - if so, what are the problems? Concrete questions are far
easier to answer by the community.

I hope that helps.

Best Regards
- henrik
Re: OSS project needs help [message #1242980 is a reply to message #1242091] Mon, 10 February 2014 10:45 Go to previous messageGo to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
Hi Henrik,
Nice to hear you are willing to help.

I believe you are not very familiar with the Gherkin language, but you got right away many of the issues we have.

One of the solutions I was envisioning for the project, probably the one providing the maximum level of compatibility, was to replace the Antlr lexer generated by Xtext with the one provided by the Gherkin library. My difficulty in doing it myself was to understand how the editor classes interact with the generated lexer/parser.

Would you advice to follow this route? If you do, we should be considering the xtext grammar files as the structure to define the language semantic context only, leveraging the existing parser for lexing and parsing the files.

On the other hand, if we don't go for the external parser, the grammar file will have to address:

- within some contexts the spaces are meaningless, while in others they are meaningful; as an example, spaces between two step definitions or between two scenarios are meaningless, while spaces within a step definition are meaningful
- once a step keyword is encountered, every character up to the end of line, including spaces, tabs or anything else, is part of the step definition
- multi language support (the Given, Then, And, When and But keywords should be localizable depending on a comment on the very first line of the file), not a very high priority, but definitely on the radar
- the EOL rule is definitely messy, but I didn't find any other way to describe that both an EOL or an EOF can be used to terminate a table, a pystring or a step


As you can imagine the Gherkin language (and the JBehave one is not much different) are not the usual formal languages. With regards to Gherkin I usually point my friends and colleagues to this page for reference: it's not 100% complete, but it provides a decent introduction to the language.

I'm very happy to have you onboard, I look forward to have you joining the github project!
Re: OSS project needs help [message #1243194 is a reply to message #1242980] Mon, 10 February 2014 16:50 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2014-10-02 11:45, rlogiacco@xxxxxxxx Lo Giacco wrote:
> Hi Henrik,
> Nice to hear you are willing to help.
>
> I believe you are not very familiar with the Gherkin language, but you
> got right away many of the issues we have.
>
> One of the solutions I was envisioning for the project, probably the one
> providing the maximum level of compatibility, was to replace the Antlr
> lexer generated by Xtext with the one provided by the
> https://github.com/cucumber/gherkin/tree/master/java. My difficulty in
> doing it myself was to understand how the editor classes interact with
> the generated lexer/parser.
>
I have an external lexer in the Geppetto project. There are some hoops
to jump through as there is more than one lexer involved. (Geppetto is
found at github puppetlabs geppetto.

> Would you advice to follow this route? If you do, we should be
> considering the xtext grammar files as the structure to define the
> language semantic context only, leveraging the existing parser for
> lexing and parsing the files.
>
You can use an external lexer, but not an external parser (or at least
not with a reasonable amount of work). Suggest copying, and possibly
adapting the existing lexer.

A chore is to sync the tokens between the Xtext grammar and the external
lexer - the token names/numbers must match up. I developed some
utilities to help with that (I think I checked them into the geppetto
respository).

> On the other hand, if we don't go for the external parser, the grammar
> file will have to address:
>
> - within some contexts the spaces are meaningless, while in others they
> are meaningful; as an example, spaces between two step definitions or
> between two scenarios are meaningless, while spaces within a step
> definition are meaningful

You can control this in an Xtext grammar by using the specification
hidden, which specifies the tokens that are not given to the rule. By
default this is WS, SL_COMMENT, ML_COMMENT. You can specify this per
rule. It is sometimes tricky to get this right in nested rule calls.
Again, you can check the geppetto grammar for how I use this.

> - once a step keyword is encountered, every character up to the end of
> line, including spaces, tabs or anything else, is part of the step
> definition
That is best handled in the lexer.

You may later also have to adjust the support for "Regions" which is
important for certain editing features.

> - multi language support (the Given, Then, And, When and But keywords
> should be localizable depending on a comment on the very first line of
> the file), not a very high priority, but definitely on the radar

Again, something that is best solved in the lexer, but that you may also
have to deal with in the higher functions (suggestions, etc.)

> - the EOL rule is definitely messy, but I didn't find any other way to
> describe that both an EOL or an EOF can be used to terminate a table, a
> pystring or a step
>
Also best done in the lexer.
>
> As you can imagine the Gherkin language (and the JBehave one is not much
> different) are not the usual formal languages. With regards to Gherkin I
> usually point my friends and colleagues to
> http://docs.behat.org/guides/1.gherkin.html for reference: it's not 100%
> complete, but it provides a decent introduction to the language.
>
> I'm very happy to have you onboard, I look forward to have you joining
> the github project!

Not sure I will do much but help answer questions...

- henrik
Re: OSS project needs help [message #1244407 is a reply to message #1243194] Wed, 12 February 2014 09:39 Go to previous messageGo to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
Henrik Lindberg wrote on Mon, 10 February 2014 11:50


Not sure I will do much but help answer questions...



Arghh... While I do really appreciate all the time you are dedicating answering my questions and resolving my doubts, what the project currently needs is active contribution: I've spent too much time myself trying to understand or find a way out of those problems to be able to fix them by myself. I have to admit I don't have enough background knowledge to be productive and learning everything necessary is going to take too long to be effective.

Please, trust me when I say I do appreciate your contribution and help, but we need somebody who steps into and codes away those issues.

I can understand you are too busy or not interested enough to be the one we are looking for, in which case I will bump my S.O.S.: anybody who is willing to dedicate some time to this? Please, it's an OSS project and I believe it's very much interesting one.
Re: OSS project needs help [message #1245832 is a reply to message #1244407] Fri, 14 February 2014 10:26 Go to previous message
Axel Guckelsberger is currently offline Axel GuckelsbergerFriend
Messages: 347
Registered: July 2009
Senior Member
Hi Roberto,

slightly off-topic, but still worth mentioning: do you know Jnario?

http://jnario.org/
https://github.com/sebastianbenz/Jnario/

Maybe it would be reasonable to verify whether synergies could be generated
between this and your project.

Regards,
Axel
Previous Topic:JvmField expression initializer generates function application
Next Topic:Use code generator from Eclipse and CLI
Goto Forum:
  


Current Time: Sat Nov 28 12:02:57 GMT 2020

Powered by FUDForum. Page generated in 0.02316 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top