Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » whitespaces not allowed in grammar
whitespaces not allowed in grammar [message #870725] Wed, 09 May 2012 13:21 Go to next message
Tim Student is currently offline Tim StudentFriend
Messages: 12
Registered: May 2012
Junior Member
I'm trying to build a Xtext Grammar Rule that accepts the following Syntax:
system    check_hardware
system    check_hardware
system    check_hardware -all
system    check_hardware -all -i
system    check_hardware -all -i "HDD"
system -i check_hardware
system -i check_hardware
system -i check_hardware -all
system -i check_hardware -all -i
system -i check_hardware -all -i "HDD"

You can immagine, the keyword system announces: a systemcommand will follow.
Optional the keyword system is able to handle one option -i.
After that (instead of my example check_hardware) each linux command resectively programm can follow with its options, for example 'echo "Hello World"' or maybe 'grep -i test *'

My approach for that is the following rule:
system:	'system' '-i'? prog=ID ('-'ID STRING?)* ; 

So we have the keyword system, an optional option -i the progam-/commandname (which has usually the Syntax of an ID) an finally the options for the command...
Here are some results, from this Rule:
	system echo				//OK
	system -i echo				//OK
	system -i echo -n "Hello World!"	//OK
	system -i echo -i "Hello World!"	//NOT OK -> "no viable alternative at input '-i'" //(every other character as an option except -i would be ok)

Can somebody tell me, what is wrong with my grammar?

Other open issue is regarding whitespaces...
How can I prevent xtext to accept:
	system -i echo -   n "Hello World!"
	system-i  echo -n    "Hello World!"
	system-iecho-n"Hello World!"

The last three inputs will be recognized as good. But they should be bad...

Great Thanks in advance.
Re: whitespaces not allowed in grammar [message #870748 is a reply to message #870725] Wed, 09 May 2012 14:12 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-09-05 15:21, Tim Student wrote:
> I'm trying to build a Xtext Grammar Rule that accepts the following Syntax:
>
> system check_hardware
> system check_hardware
> system check_hardware -all
> system check_hardware -all -i
> system check_hardware -all -i "HDD"
> system -i check_hardware
> system -i check_hardware
> system -i check_hardware -all
> system -i check_hardware -all -i
> system -i check_hardware -all -i "HDD"
>
> You can immagine, the keyword system announces: a systemcommand will
> follow. Optional the keyword system is able to handle one option -i.
> After that (instead of my example check_hardware) each linux command
> resectively programm can follow with its options, for example 'echo
> "Hello World"' or maybe 'grep -i test *'
>
> My approach for that is the following rule:
>
> system: 'system' '-i'? prog=ID ('-'ID STRING?)* ;
> So we have the keyword system, an optional option -i the
> progam-/commandname (which has usually the Syntax of an ID) an finally
> the options for the command...
> Here are some results, from this Rule:
>
> system echo //OK
> system -i echo //OK
> system -i echo -n "Hello World!" //OK
> system -i echo -i "Hello World!" //NOT OK -> "no viable alternative at
> input '-i'" //(every other character as an option except -i would be ok)
>
> Can somebody tell me, what is wrong with my grammar?
>
'-i' is a keyword, and will never be matched by '-' ID.

> Other open issue is regarding whitespaces...
> How can I prevent xtext to accept:
>
Use hidden(WS) and specify where WS is allowed (and how many).
SomeRule hidden(WS) : 'something' WS+ 'somethingElse' ;

> system -i echo - n "Hello World!"
> system-i echo -n "Hello World!"
> system-iecho-n"Hello World!"
>
> The last three inputs will be recognized as good. But they should be bad...
>
> Great Thanks in advance.

I would simplify the grammar to just accept a series of arguments, and
then use validation to handle all the issues (space/no space). This way
you can offer much better errors messages and do things like offer a
quick fix for fixing "illegal/non-existing spacing".

- henrik
Re: whitespaces not allowed in grammar [message #870778 is a reply to message #870748] Wed, 09 May 2012 15:38 Go to previous messageGo to next message
Tim Student is currently offline Tim StudentFriend
Messages: 12
Registered: May 2012
Junior Member
Thanks the approach is clear so far, but can you tell it to me some more detailed?

My Problem is, that I cannot handle whitespace characters. For example I would like to realise something like a variable assignment, that is restricted to:
//only this should be valid:
var="anystring";

//and not
var= "anystring";
//and not
var ="anystring";
//and not
var = "anystring";


When my grammar says:
variable:	name=ID '=' varcontent=STRING;

... all the three above mentioned entries will be valid, becaus Xtext allows an arbitrary number of whitespaces between the single elements.

What can I do there?
Re: whitespaces not allowed in grammar [message #870799 is a reply to message #870778] Wed, 09 May 2012 18:03 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-09-05 17:39, Tim Student wrote:
> Thanks the approach is clear so far, but can you tell it to me some more
> detailed?
>
> My Problem is, that I cannot handle whitespace characters. For example I
> would like to realise something like a variable assignment, that is
> restricted to:
>
> //only this should be valid:
> var="anystring";
>
> //and not
> var= "anystring";
> //and not
> var ="anystring";
> //and not
> var = "anystring";
>
>
> When my grammar says:
>
> variable: name=ID '=' varcontent=STRING;
>
> ... all the three above mentioned entries will be valid, becaus Xtext
> allows an arbitrary number of whitespaces between the single elements.
>
> What can I do there?
>
What I meant, but did not say - I wrote hidden(WS) - it should be the
other way around - you specify what should be hidden i.e.
hidden(SL_COMMENT, ML_COMMENT, ...) if you want to allow comments, or
hidden() if nothing should be hidden - you then specify WS where it is
allowed.

Rule hidden() :
variable: name=ID '=' varcontent=STRING;

should work.
- henrik
Re: whitespaces not allowed in grammar [message #871983 is a reply to message #870799] Tue, 15 May 2012 08:50 Go to previous messageGo to next message
Tim Student is currently offline Tim StudentFriend
Messages: 12
Registered: May 2012
Junior Member
Thanks Henrik, that was the right hint, now this part wors fine!
Re: whitespaces not allowed in grammar [message #875369 is a reply to message #870725] Tue, 22 May 2012 15:59 Go to previous message
Felix Feisst is currently offline Felix FeisstFriend
Messages: 20
Registered: February 2012
Location: Germany
Junior Member
Tim Student wrote on Wed, 09 May 2012 15:21
I'm trying to build a Xtext
	system echo				//OK
	system -i echo				//OK
	system -i echo -n "Hello World!"	//OK
	system -i echo -i "Hello World!"	//NOT OK -> "no viable alternative at input '-i'" //(every other character as an option except -i would be ok)

Can somebody tell me, what is wrong with my grammar?


The problem is your lexer tokens. Let me explain the problem to you.
Before the actual parser parses your input, the input is preprocessed by the so-called lexer. The lexer converts the input which is just a stream of characters to a so called token stream. Lets identify the tokens in your grammar:
'system'
'-i'
'-'
ID
STRING

These tokens follow directly from your grammar where ID and STRING are predefined tokens (also called terminals).

What the lexer now does, is to convert the stream of characters to a stream of tokens. Lets look at two of your examples:
	system -i echo -n "Hello World!" ==> 'system' '-i' ID '-' ID STRING
	system -i echo -i "Hello World!" ==> 'system' '-i' ID '-i' STRING

Note in the second example the "-i" is converted to '-i' instead of '-' ID. This is because the lexer prefers token '-i' over ID.

What your parser expects is the following stream of tokens:
'system' '-i'? ID ('-' ID STRING?)*

In the second example the parser cannot regognize the input because it sees the '-i' token but expects the '-' token.

I hope the problem is clear to you now.

There are several ways to fix this problem:
One solution would be to allow '-i' as an alternative to '-'ID:
system:	'system' '-i'? prog=ID (('-'ID | '-i') STRING?)* ; 

This solution feels a little dirty for me.

Another solution which I would prefer is to allow any ID as flag after system and use the validator to ensure that only "-i" is used. I would also define an own token (terminal) for the flags:
system: 'system' FLAG? prog=ID (FLAG STRING?)*

terminal FLAG: '-' ('a'..'z'|'A'..'Z'|'0'..'9')+;

The own FLAG token also solves your problem with the spaces between the '-' and the flag characters.
Don't forget to implement the validation that ensures that the first FLAG just is "-i".

I hope that was helpful,
Felix

[Updated on: Tue, 22 May 2012 16:00]

Report message to a moderator

Previous Topic:grammar file not copied to bin
Next Topic:[Xtext 2.3] Using the new debugging features
Goto Forum:
  


Current Time: Sat Dec 05 01:06:49 GMT 2020

Powered by FUDForum. Page generated in 0.02188 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top