Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » grammar ignore until next keyword
grammar ignore until next keyword [message #1060592] Mon, 27 May 2013 08:23 Go to next message
Stella Levin is currently offline Stella LevinFriend
Messages: 89
Registered: July 2009
Member
Hi, I try to build the grammar for the language (very simplified)
START_MODEL
VAR X ignore everything until the next keyword
still ignore the line
VAR Y again ignore until the next keyword
END_MODEL

I defined it as:

grammar org.mlnx.xtext.RBV hidden(WS, ML_COMMENT, SL_COMMENT)
import "http://www.eclipse.org/emf/2002/Ecore" as ecore
generate rBV "http://www.mlnx.org/xtext/RBV"
Model:
	START_KWD (name=ID)?
	(items += Item)+
	END_KWD 
	;
Item: 
	VarDeclaration
;	
VarDeclaration:
	VAR_KWD (name=SIMPLE_NAME) NOT_KWD
;
/***************
 * Terminals
 * The sequence of terminals is important
 ***************/
terminal START_KWD:		'START_MODEL';
terminal END_KWD:             	'END_MODEL';
terminal VAR_KWD:		'VAR';						
terminal SIMPLE_NAME:		('A'..'Z') ('A'..'Z'|'_'|'0'..'9')*;
//*********** grammar org.eclipse.xtext.common.Terminals
terminal ID  		: '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
terminal INT returns ecore::EInt: ('0'..'9')+;
terminal STRING	: 
			'"' ( '\\' ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') | !('\\'|'"') )* '"' |
			"'" ( '\\' ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') | !('\\'|"'") )* "'"; 
terminal ML_COMMENT	: '/*' -> '*/';
terminal SL_COMMENT 	: '//' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS		: (' '|'\t'|'\r'|'\n')+;
//******************************************************
terminal NOT_KWD        :  !(START_KWD|END_KWD|VAR_KWD)+;

Then I get error:
Quote:
mismatched input 'START_MODEL mymodel\n...' expecting RULE_START_KWD


If I change the following lines in the grammar (move '+' sigh from NOT_KWD to VarDeclaration)
VarDeclaration:
	VAR_KWD (name=SIMPLE_NAME) NOT_KWD+
;
terminal NOT_KWD: 	!(START_KWD|END_KWD|VAR_KWD);


The I get error on "VAR X ignore..." line:
Quote:
required (...)+ loop did not match anything at input 'ignore'

Is there a way to define it in xtext?
I need parsing rules in order to define outline window and get keywords coloring.
Thanks, Stella
Re: grammar ignore until next keyword [message #1060677 is a reply to message #1060592] Mon, 27 May 2013 16:49 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
This line does not work as you expect

terminal NOT_KWD: !(START_KWD|END_KWD|VAR_KWD);

Suggest you instead create a ANY_KWD rule, and list
all the keywords you want to include.

Also, it is usually a very bad idea to state keywords as terminals.
Your SIMPLE_NAME is an overlapping terminal - you will not get any ID
tokens.

Hope that helps
- henrik

On 2013-27-05 1:23, Stella Mising name wrote:
> Hi, I try to build the grammar for the language (very simplified)
> START_MODEL
> VAR X ignore everything until the next keyword
> still ignore the line
> VAR Y again ignore until the next keyword
> END_MODEL
>
> I defined it as:
>
> grammar org.mlnx.xtext.RBV hidden(WS, ML_COMMENT, SL_COMMENT)
> import "http://www.eclipse.org/emf/2002/Ecore" as ecore
> generate rBV "http://www.mlnx.org/xtext/RBV"
> Model:
> START_KWD (name=ID)?
> (items += Item)+
> END_KWD ;
> Item: VarDeclaration
> ;
> VarDeclaration:
> VAR_KWD (name=SIMPLE_NAME) NOT_KWD
> ;
> /***************
> * Terminals
> * The sequence of terminals is important
> ***************/
> terminal START_KWD: 'START_MODEL';
> terminal END_KWD: 'END_MODEL';
> terminal VAR_KWD: 'VAR';
> terminal SIMPLE_NAME: ('A'..'Z') ('A'..'Z'|'_'|'0'..'9')*;
> //*********** grammar org.eclipse.xtext.common.Terminals
> terminal ID : '^'?('a'..'z'|'A'..'Z'|'_')
> ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*;
> terminal INT returns ecore::EInt: ('0'..'9')+;
> terminal STRING : '"' ( '\\'
> ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') | !('\\'|'"') )* '"' |
> "'" ( '\\' ('b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\') |
> !('\\'|"'") )* "'"; terminal ML_COMMENT : '/*' -> '*/';
> terminal SL_COMMENT : '//' !('\n'|'\r')* ('\r'? '\n')?;
>
> terminal WS : (' '|'\t'|'\r'|'\n')+;
> //******************************************************
> terminal NOT_KWD : !(START_KWD|END_KWD|VAR_KWD)+;
>
> Then I get error: Quote:
>> mismatched input 'START_MODEL mymodel\n...' expecting RULE_START_KWD
>
>
> If I change the following lines in the grammar (move '+' sigh from
> NOT_KWD to VarDeclaration)
> VarDeclaration:
> VAR_KWD (name=SIMPLE_NAME) NOT_KWD+
> ;
> terminal NOT_KWD: !(START_KWD|END_KWD|VAR_KWD);
>
> The I get error on "VAR X ignore..." line:
> Quote:
>> required (...)+ loop did not match anything at input 'ignore'
>
> Is there a way to define it in xtext?
> I need parsing rules in order to define outline window and get keywords
> coloring.
> Thanks, Stella
Re: grammar ignore until next keyword [message #1060684 is a reply to message #1060677] Mon, 27 May 2013 18:37 Go to previous messageGo to next message
Stella Levin is currently offline Stella LevinFriend
Messages: 89
Registered: July 2009
Member
Hi Henrik, great thanks for reply. The problem is that the full grammar is VERY complicated, the statements are multi-line. I just want to create simplified editor, with basic keyword coloring and outline view. Why is it usually a very bad idea to state keywords as terminals? Why "terminal NOT_KWD" doesn't work as expected? Thanks again. Stella
Re: grammar ignore until next keyword [message #1060827 is a reply to message #1060684] Tue, 28 May 2013 14:37 Go to previous message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
On 2013-27-05 20:37, Stella Levin wrote:
> Hi Henrik, great thanks for reply. The problem is that the full grammar
> is VERY complicated, the statements are multi-line. I just want to
> create simplified editor, with basic keyword coloring and outline view.
> Why is it usually a very bad idea to state keywords as terminals? Why
> "terminal NOT_KWD" doesn't work as expected? Thanks again. Stella

You have overlapping terminals. You will probably never trigger NOT_KWD
terminal rule, because there are other terminal rules with higher
precedence.

Keywords work differently than terminals in the lexer; for rules such as
ID, if it matches a keyword the keyword token will be emitted. That will
not happen if you have the keyword as a terminal on its own. There are
also functionality tied to being a keyword as opposed to other types of
punctuation tokens. I.e. rule of thumb "do not use terminals for keywords".

If you want to be able to create a model (to show in the outline) you
naturally need to parse enough of the syntax. One way this can be done
is to write an external lexer that simplifies the task in the grammar.

Hope that helps.

Regards
- henrik
Previous Topic:New Target Language
Next Topic:xtext project creation programmatically
Goto Forum:
  


Current Time: Thu Dec 18 10:37:03 GMT 2014

Powered by FUDForum. Page generated in 0.01688 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software