Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » EOF not matching RULE_EOL
EOF not matching RULE_EOL [message #865840] Mon, 30 April 2012 15:53 Go to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
I'm having difficulties in understanding why I need an additional line at the end of my files with the attached grammar.

The message is "mismatched input <EOF> expecting RULE_EOL" but my EOL rule should accept both a new line or the EOF token...
Re: EOF not matching RULE_EOL [message #865948 is a reply to message #865840] Mon, 30 April 2012 17:00 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13661
Registered: July 2009
Senior Member
Hi,

did you see the bunch of warning you get with your dsl:

Decision can match input such as "RULE_EOL" using multiple alternatives


if your EOL is allowed everywhere why dont you make it generally hidden
(and maybe make it visible explicitely where it is allowed only at certain places)


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: EOF not matching RULE_EOL [message #866608 is a reply to message #865948] Tue, 01 May 2012 00:06 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-30-04 19:00, Christian Dietrich wrote:
> Hi,
>
> did you see the bunch of warning you get with your dsl:
>
>
> Decision can match input such as "RULE_EOL" using multiple alternatives
>
>
> if your EOL is allowed everywhere why dont you make it generally hidden
> (and maybe make it visible explicitely where it is allowed only at
> certain places)
>
+1 to that...

....or even allow it everywhere using hidden, and validate the few places
where it is not allowed as you can then have an error message that makes
sense "Newline not allowed here" and provide a quickfix to remove it.
The alternative is that users that enter a new line where they are not
supposed to will get syntax errors like "got EOL, expected xxx" which a
lot of the time is just gobbledygook to a user that is unaware of the
actual grammar rule/terminal names.

(it also makes your grammar cleaner and removes ambiguities).

- henrik
Re: EOF not matching RULE_EOL [message #867389 is a reply to message #866608] Tue, 01 May 2012 09:12 Go to previous messageGo to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
I had it hidden but then I was having trouble in describing that an element is ended by a line, can you help me with it? I'm experiencing other problems as well, like I can't use Given, When, Then, And or But in any part of the file.
My current grammar and an example file are attached.

I will appreciate if you can help me with this.
Re: EOF not matching RULE_EOL [message #867685 is a reply to message #867389] Tue, 01 May 2012 12:22 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-01-05 11:12, rlogiacco@xxxxxxxx Lo Giacco wrote:
> I had it hidden but then I was having trouble in describing that an element is ended by a line, can you help me with it? I'm experiencing other problems as well, like I can't use Given, When, Then, And or But in any part of the file.
> My current grammar and an example file are attached.
>
> I will appreciate if you can help me with this.

Without fully digging into the grammar and your sample input (I don't
have the time for that), here are a few things that I did notice...

You have:

terminal STEP_KEYWORD:
('Given' | 'When' | 'Then' | 'And' | 'But');

change that to just:

STEP_KEYWORD:
('Given' | 'When' | 'Then' | 'And' | 'But');

as you are using the quoted words as keywords in one place, but then you
squash all of the different words into the same token "STEP_KEYWORD" in
the terminal - so this:

type=('Given' | 'When' | 'Then' | 'And' | 'But') EOL*

will never work as the lexer would return the token STEP_KEYWORD.

If you really want it as a terminal, you could instead change it to:

type = STEP_KEYWORD EOL*

Looks like you may have an issue with single quoted string in WORD as it
clashes with STRING - which makes a'a' produce WORD and not WORD STRING
(but that may be what you want).

Regarding the EOL / EOF issues - I take back what I said earlier, now
that I seen the sample input - you really do need to have EOL as
something significant and can't in most cases hide it.

It seems the problems are mostly lexical, and trying to write good
lexing rules with the grammar can be difficult. If you continue to have
trouble it may be best to switch to an external lexer where you can do
more advanced things.

Hope that helps.

Regards
- henrik
Re: EOF not matching RULE_EOL [message #868041 is a reply to message #867685] Tue, 01 May 2012 16:05 Go to previous messageGo to next message
Roberto Lo Giacco is currently offline Roberto Lo GiaccoFriend
Messages: 17
Registered: April 2012
Junior Member
Hi Henrik,
it does help, but I'm sure I'm still missing something.
As you can see, in my grammar the keyword And has a specific meaning only after a Scenario: or Scenario Outline: token... I don't know why the lexer is recognizing it as a Step while it is expecting a Line.

Correct me if I'm wrong but the terminals should be recognized first and then used into the grammar.
This should mean that if I have a terminal for And and I use it somewhere else, the terminal get parsed first. In my grammar:

Feature:
	'Feature:' title=FeatureDescription EOL+
	description+=FeatureDescription* EOL*
	scenarios+=Scenario EOL*;
Scenario:
	'Scenario:' title=Line EOL*
	description+=Line* EOL*
	conditions+=Step+ EOL*;
Step:
	type=STEP_KEYWORD EOL* 
	description=Line EOL*
Line:
	content=CharSequence EOL;
CharSequence:
	(STRING | WORD | INT | ANY_OTHER)+;
FeatureDescription:
	(CharSequence | STEP_KEYWORD)+;	
terminal STEP_KEYWORD:
	('Given' | 'When' | 'Then' | 'And' | 'But');	
terminal WORD:
	'^'? ('a'..'z' | 'A'..'Z' | '_' ) ('a'..'z' | 'A'..'Z' | '_' | '0'..'9' | "'" | "#" |
		('\\' ( 'u' | '"' | "'" | '\\'))
	)*;
terminal INT returns ecore::EInt:
	('0'..'9')+;
terminal STRING:
	'"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | '"'))* '"' |
	"'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') | !('\\' | "'"))* "'";
terminal SL_COMMENT:
	'#' !('\n' | '\r')* EOL;
terminal SPACE:
	(' ' | '\t')+;
terminal EOL:
	(('\r'? '\n') | EOF);
terminal ANY_OTHER:
	.;


Whenever I encounter And a STEP_KEYWORD token should be identified and if I'm parsing a Feature then it should be allowed as part of the feature description. In my case it get's identified as a step type, but at this point I can't have a step type as a Step can appear only inside a Scenario....

About the EOF, whenever an EOF is encountered it should get parsed into an EOL token and, as you can see, I've an EOL* at the end of a step and an EOL at the end of Line... If my logic makes sense I shouldn't need any CR/LF at the end of a Step description as the EOF should get captured by the Line EOL fulfilling the grammar requirement.

I don't understand why it seems requiring a CR/LF at the end of the file, may be this is related to the multiple (10 times) warnings I'm getting when compiling my grammar:
Decision can match input such as "RULE_EOL" using multiple alternatives: 1, 2


Sorry to bother you guys, but it seems I'm lost here.
Re: EOF not matching RULE_EOL [message #868150 is a reply to message #868041] Tue, 01 May 2012 17:11 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-01-05 18:05, rlogiacco@xxxxxxxx Lo Giacco wrote:
> Hi Henrik,
> it does help, but I'm sure I'm still missing something.
> As you can see, in my grammar the keyword And has a specific meaning
> only after a Scenario: or Scenario Outline: token... I don't know why
> the lexer is recognizing it as a Step while it is expecting a Line.
> Correct me if I'm wrong but the terminals should be recognized first and
> then used into the grammar. This should mean that if I have a terminal
> for And and I use it somewhere else, the terminal get parsed first. In
> my grammar:
>
In general I have found that you will run into trouble when keywords are
expressed as terminals.

The lexer parses a terminal, then checks if the parsed content is a
keyword, and if so, delivers the token for the keyword instead of the
parsed terminal. Hence my suggestion to change the "STEP_KEYWORD" rule
to be a parser rule instead of a terminal.

>
> Feature:
> 'Feature:' title=FeatureDescription EOL+
> description+=FeatureDescription* EOL*
> scenarios+=Scenario EOL*;
> Scenario:
> 'Scenario:' title=Line EOL*
> description+=Line* EOL*
> conditions+=Step+ EOL*;
> Step:
> type=STEP_KEYWORD EOL* description=Line EOL*
> Line:
> content=CharSequence EOL;
> CharSequence:
> (STRING | WORD | INT | ANY_OTHER)+;
> FeatureDescription:
> (CharSequence | STEP_KEYWORD)+;
> terminal STEP_KEYWORD:
> ('Given' | 'When' | 'Then' | 'And' | 'But');
> terminal WORD:
> '^'? ('a'..'z' | 'A'..'Z' | '_' ) ('a'..'z' | 'A'..'Z' | '_' | '0'..'9'
> | "'" | "#" |
> ('\\' ( 'u' | '"' | "'" | '\\'))
> )*;
> terminal INT returns ecore::EInt:
> ('0'..'9')+;
> terminal STRING:
> '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') |
> !('\\' | '"'))* '"' |
> "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') |
> !('\\' | "'"))* "'";
> terminal SL_COMMENT:
> '#' !('\n' | '\r')* EOL;
> terminal SPACE:
> (' ' | '\t')+;
> terminal EOL:
> (('\r'? '\n') | EOF);
> terminal ANY_OTHER:
> .;
>
>
> Whenever I encounter And a STEP_KEYWORD token should be identified and
> if I'm parsing a Feature then it should be allowed as part of the
> feature description. In my case it get's identified as a step type, but
> at this point I can't have a step type as a Step can appear only inside
> a Scenario....
>
Basically have two different rules; one that allows STEP_KEYWORDS and
treats them as non special, and one that does not include them as text).

> About the EOF, whenever an EOF is encountered it should get parsed into
> an EOL token and, as you can see, I've an EOL* at the end of a step and
> an EOL at the end of Line... If my logic makes sense I shouldn't need
> any CR/LF at the end of a Step description as the EOF should get
> captured by the Line EOL fulfilling the grammar requirement.
>
I am not sure the EOF surfaces in the grammar the way you expect.
I would try to write a minimal grammar with a single terminal and see
what happens - if EOF is actually delivered and recognized by the
parser. If you do this as a minimal grammar, you should also be able to
debug the lexer and see exactly what happens.

> I don't understand why it seems requiring a CR/LF at the end of the
> file, may be this is related to the multiple (10 times) warnings I'm
> getting when compiling my grammar:
> Decision can match input such as "RULE_EOL" using multiple alternatives:
> 1, 2
>
You have to fix those...

> Sorry to bother you guys, but it seems I'm lost here.
Re: EOF not matching RULE_EOL [message #868406 is a reply to message #868150] Tue, 01 May 2012 22:33 Go to previous message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Btw, have you looked at Jnario ?
Seems to be a parser for the same type of language.

- henrik

On 2012-01-05 19:11, Henrik Lindberg wrote:
> On 2012-01-05 18:05, rlogiacco@xxxxxxxx Lo Giacco wrote:
>> Hi Henrik,
>> it does help, but I'm sure I'm still missing something.
>> As you can see, in my grammar the keyword And has a specific meaning
>> only after a Scenario: or Scenario Outline: token... I don't know why
>> the lexer is recognizing it as a Step while it is expecting a Line.
>> Correct me if I'm wrong but the terminals should be recognized first and
>> then used into the grammar. This should mean that if I have a terminal
>> for And and I use it somewhere else, the terminal get parsed first. In
>> my grammar:
>>
> In general I have found that you will run into trouble when keywords are
> expressed as terminals.
>
> The lexer parses a terminal, then checks if the parsed content is a
> keyword, and if so, delivers the token for the keyword instead of the
> parsed terminal. Hence my suggestion to change the "STEP_KEYWORD" rule
> to be a parser rule instead of a terminal.
>
>>
>> Feature:
>> 'Feature:' title=FeatureDescription EOL+
>> description+=FeatureDescription* EOL*
>> scenarios+=Scenario EOL*;
>> Scenario:
>> 'Scenario:' title=Line EOL*
>> description+=Line* EOL*
>> conditions+=Step+ EOL*;
>> Step:
>> type=STEP_KEYWORD EOL* description=Line EOL*
>> Line:
>> content=CharSequence EOL;
>> CharSequence:
>> (STRING | WORD | INT | ANY_OTHER)+;
>> FeatureDescription:
>> (CharSequence | STEP_KEYWORD)+;
>> terminal STEP_KEYWORD:
>> ('Given' | 'When' | 'Then' | 'And' | 'But');
>> terminal WORD:
>> '^'? ('a'..'z' | 'A'..'Z' | '_' ) ('a'..'z' | 'A'..'Z' | '_' | '0'..'9'
>> | "'" | "#" |
>> ('\\' ( 'u' | '"' | "'" | '\\'))
>> )*;
>> terminal INT returns ecore::EInt:
>> ('0'..'9')+;
>> terminal STRING:
>> '"' ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') |
>> !('\\' | '"'))* '"' |
>> "'" ('\\' ('b' | 't' | 'n' | 'f' | 'r' | 'u' | '"' | "'" | '\\') |
>> !('\\' | "'"))* "'";
>> terminal SL_COMMENT:
>> '#' !('\n' | '\r')* EOL;
>> terminal SPACE:
>> (' ' | '\t')+;
>> terminal EOL:
>> (('\r'? '\n') | EOF);
>> terminal ANY_OTHER:
>> .;
>>
>>
>> Whenever I encounter And a STEP_KEYWORD token should be identified and
>> if I'm parsing a Feature then it should be allowed as part of the
>> feature description. In my case it get's identified as a step type, but
>> at this point I can't have a step type as a Step can appear only inside
>> a Scenario....
>>
> Basically have two different rules; one that allows STEP_KEYWORDS and
> treats them as non special, and one that does not include them as text).
>
>> About the EOF, whenever an EOF is encountered it should get parsed into
>> an EOL token and, as you can see, I've an EOL* at the end of a step and
>> an EOL at the end of Line... If my logic makes sense I shouldn't need
>> any CR/LF at the end of a Step description as the EOF should get
>> captured by the Line EOL fulfilling the grammar requirement.
>>
> I am not sure the EOF surfaces in the grammar the way you expect.
> I would try to write a minimal grammar with a single terminal and see
> what happens - if EOF is actually delivered and recognized by the
> parser. If you do this as a minimal grammar, you should also be able to
> debug the lexer and see exactly what happens.
>
>> I don't understand why it seems requiring a CR/LF at the end of the
>> file, may be this is related to the multiple (10 times) warnings I'm
>> getting when compiling my grammar:
>> Decision can match input such as "RULE_EOL" using multiple alternatives:
>> 1, 2
>>
> You have to fix those...
>
>> Sorry to bother you guys, but it seems I'm lost here.
>
Previous Topic:Null pointer when serializing cross references
Next Topic:Confused about the map function
Goto Forum:
  


Current Time: Fri Nov 27 18:43:20 GMT 2020

Powered by FUDForum. Page generated in 0.02270 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top