Eclipse Community Forums: TMF (Xtext) » Unquoted string collide with keyword

Help

Home

Home » Modeling » TMF (Xtext) » Unquoted string collide with keyword(Unquoted string collide with keyword)

Show: Today's Messages :: Show Polls :: Message Navigator

Unquoted string collide with keyword [message #1564599]

Wed, 14 January 2015 22:11

Luis De Bello

Messages: 95
Registered: January 2015

Member

Hi guys,

I am creating a grammar which allows me to use unquoted strings but now I want to define some URI's and they can contain ":" "/", "." but I am using the ":" as keyword and as the first letter for some keywords.

Grammar:
grammar org.Testing hidden(WS, SL_COMMENT)

import "http://www.eclipse.org/emf/2002/Ecore" as ecore

/*
* Document
*/
Document:
namespaces=Namespace?
content=Content;

Namespace:
"namespace" prefix=UNQUOTED_STRING namespace=NamespaceUri;

NamespaceUri:
UNQUOTED_STRING;

/*
* Content
*/
Content:
elements+=Object+ | elements+=SingleObject;

Object:
{Object} "{" (elements+=KeyValuePair ("," elements+=KeyValuePair)*)? "}";

SingleObject:
{SingleObject} elements+=KeyValuePair;

KeyValuePair:
key=UNQUOTED_STRING ":" value=UNQUOTED_STRING ("as" type=Types)?;

enum Types:
ARRAY = ":array" | BOOLEAN = ":boolean" | NUMBER = ":number";

/*
* Terminals
*/
terminal UNQUOTED_STRING:
('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '0'..'9')*;

terminal STRING:
'"' ('\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\' | '"'))* '"' |
"'" ('\\' . /* 'b'|'t'|'n'|'f'|'r'|'u'|'"'|"'"|'\\' */ | !('\\' | "'"))* "'";

// TODO (ldebello) Check 0 for INT
terminal INT returns ecore::EInt:
('0'..'9')+;

terminal ANY_DATE: '|' -> '|';

terminal SL_COMMENT: '#' !('\n'|'\r')* ('\r'? '\n')?;

terminal WS:
(' ' | '\t' | '\r' | '\n')+;

------------------------------
Using this grammar I can use the following input without any issue

-------
Input
-------
namespace test1 test2
{
data: Hello,
data2: true as :boolean
}

However when I try to replace test2 for urn:urn it is not possible because the ":" is not accepted for the terminal UNQUOTED_STRING but I do not want to include this ":" in the terminal because it is also a keyword, I want to keep the structure UNQUOTED_STRING ":" UNQUOTED_STRING

Does anyone any idea on how to face this issue? and keep the current structure in the body and accept any character in the namespace.

Thanks and Regards,
Luis

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565209 is a reply to message #1564599]

Thu, 15 January 2015 06:27

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi you can use datatype rules for that

Eg TERMINALORKEYWORD : "keyword" | TERMINAL;

And the lever is eager and eats up as much as it can
So what about splitting up the keywords

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565290 is a reply to message #1564599]

Thu, 15 January 2015 07:30

Sebastian Zarnekow

Messages: 3118
Registered: July 2009

Senior Member

Hi Luis,

I'd assume it'll help if you split your enum Types and remove the colon,
e.g.

... ('as' ':' type=Types)?;

enum Types:
ARRAY = 'array' | .. ;

Best,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565763 is a reply to message #1565290]

Thu, 15 January 2015 13:16

Luis De Bello

Messages: 95
Registered: January 2015

Member

Hi Sebastian,

I tried to split the colon and the keywords and use the advice of Christian but after doing that I got an error from antler to indicate "the decision cannot distinguish between alternative(s) 2,3...", also if I split the keyword from ":number" to ":" "number" I will not able to use "number" as a key in my object.

I was thinking that maybe I should customize the lexer generated by xtext but I am not sure if that is the best way to go or if I can solve my issue with xtext

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565782 is a reply to message #1565763]

Thu, 15 January 2015 13:32

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

i dont get that: the following works for me

KeyValuePair:
key=UNQUOTED_STRING ":" value=UNQUOTED_STRING ("as" ":" type=Types)?;

enum Types:
ARRAY = "array" | BOOLEAN = "boolean" | NUMBER = "number";

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565796 is a reply to message #1565782]

Thu, 15 January 2015 13:41

Luis De Bello

Messages: 95
Registered: January 2015

Member

Hi Christian,

That works ok but I need to include any kind of characters in the namespace

for instance

-------
Input
-------
namespace <Here I want to accept (a..z | A..Z)> <Here I want to accept (a..z | A..Z | : | / | .)>
{
data: Hello,
data2: true as :boolean
}

My Rule for namespace is :

Namespace:
"namespace" prefix=UNQUOTED_STRING namespace=NamespaceUri;

NamespaceUri:
(UNQUOTED_STRING | ":")*; // This give the error from antlr

Regards,
Luis

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565837 is a reply to message #1565796]

Thu, 15 January 2015 14:15

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

hi where does the namespace end and the SingleObject start?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565865 is a reply to message #1565837]

Thu, 15 January 2015 14:33

Luis De Bello

Messages: 95
Registered: January 2015

Member

The namespace should start after the prefix and ends in the first blank or line break.
example:
S--> Start and E --> End

S E
namespace prefix urn:urn
{
data: value
}

Thanks & Regards,
Luis

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565873 is a reply to message #1565865]

Thu, 15 January 2015 14:39

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

but your grammar does not respect that.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565892 is a reply to message #1565873]

Thu, 15 January 2015 14:50

Luis De Bello

Messages: 95
Registered: January 2015

Member

but if I have the following grammar

Document:
namespaces=Namespace?
content=Content;

Namespace:
"namespace" prefix=UNQUOTED_STRING namespace=NamespaceUri;

NamespaceUri:
UNQUOTED_STRING;

the document contains the namespace as optional and the namespace rule is "namespace" UNQUOTED_STRING NamespaceUri so the namespace is separated in three parts and then the start object is using "{"

Maybe I am wrong but I think that the structure is fine.

Regards,
Luis

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565927 is a reply to message #1565892]

Thu, 15 January 2015 15:15

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi

you can have a singleobject as well

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565957 is a reply to message #1565927]

Thu, 15 January 2015 15:39

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

hmm tried to reduce your problem.

thus i created a simplified grammar and told xtext to generate a debug grammar (DebugAntlrGeneratorFragment)
antlrworks did not complain about the grammar

since xtext create entry rule rules for each rule some additional stuff is created

so the error is a bug in xtext imho.
can you please file a ticket

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Unquoted string collide with keyword [message #1565964 is a reply to message #1565957]

Thu, 15 January 2015 15:45

Luis De Bello

Messages: 95
Registered: January 2015

Member

Hi Christian,

Thanks for the feedback I will create a ticket for this, in the meanwhile I am creating a custom lexer in order to fix this situation.

I will post my lexer.g as soon as I finish

Report message to a moderator

Re: Unquoted string collide with keyword [message #1566582 is a reply to message #1565964]

Thu, 15 January 2015 23:42

Luis De Bello

Messages: 95
Registered: January 2015

Member

Hi guys,

Today, I have created a custom lexer to fix my issue, I am not sure if this is the best way to go but I cannot change my grammar because this is an existing in house language so that is not an option. In order to create the lexer I followed the following tutorial "http://consoliii.blogspot.com.ar/2013/04/xtext-is-incredibly-powerful-framework.html" it is really good.

After having my lexer and parser in different files I added some logic to provide some context to use one or other terminal

------------
Lexer
------------
/*
* generated by Xtext
*/
lexer grammar InternalDSLLexer;

@header {
package org.data.testing.dsl.parser.antlr.lexer;

// Hack: Use our own Lexer superclass by means of import.
// Currently there is no other way to specify the superclass for the lexer.
import org.eclipse.xtext.parser.antlr.Lexer;
}

@members {
private boolean isNamespacePrefix = false;
private boolean isNamespaceUri = false;
}

Localdatetime : ':localdatetime';

Namespace : '%namespace' {isNamespacePrefix = true; isNamespaceUri = false;};

Localtime : ':localtime';

DistinctBy : 'distinctBy';

StartsWith : 'startsWith';

Datetime : ':datetime';

LeapYear : ':leapYear';

Timezone : ':timezone';

Boolean : ':boolean';

Decimal : ':decimal';

Integer : ':integer';

Contains : 'contains';

EndsWith : 'endsWith';

Dsl10 : '%dsl1.0';

Output : '%output';

Number : ':number';

Object : ':object';

Period : ':period';

String : ':string';

Default : 'default';

GroupBy : 'groupBy';

Matches : 'matches';

OrderBy : 'orderBy';

Const : '%const';

Array : ':array';

Blank : ':blank';

Empty : ':empty';

Range : ':range';

Reduce : 'reduce';

SizeOf : 'sizeOf';

Date : ':date';

Even : ':even';

Null_1 : ':null';

Time : ':time';

False : 'false';

Lower : 'lower';

Match : 'match';

Split : 'split';

Upper : 'upper';

Where : 'where';

Odd : ':odd';

Uri : ':uri';

Find : 'find';

Json : 'json';

Null : 'null';

Scan : 'scan';

Trim : 'trim';

True : 'true';

HyphenMinusHyphenMinusHyphenMinus : '---';

And : 'and';

Map : 'map';

Not : 'not';

Xml : 'xml';

ExclamationMarkEqualsSign : '!=';

DollarSignDollarSign : '$$';

PlusSignPlusSign : '++';

HyphenMinusHyphenMinus : '--';

HyphenMinusGreaterThanSign : '->';

FullStopFullStop : '..';

LessThanSignLessThanSign : '<<';

LessThanSignEqualsSign : '<=';

EqualsSignEqualsSign : '==';

GreaterThanSignEqualsSign : '>=';

GreaterThanSignGreaterThanSign : '>>';

As : 'as';

Is : 'is';

Or : 'or';

ExclamationMark : '!';

DollarSign : '$';

Ampersand : '&';

LeftParenthesis : '(';

RightParenthesis : ')';

Asterisk : '*';

PlusSign : '+';

Comma : ',';

HyphenMinus : '-';

FullStop : '.';

Solidus : '/';

Colon : ':';

LessThanSign : '<';

EqualsSign : '=';

GreaterThanSign : '>';

QuestionMark : '?';

LeftSquareBracket : '[';

RightSquareBracket : ']';

LeftCurlyBracket : '{';

RightCurlyBracket : '}';

RULE_UNQUOTED_STRING : ('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')* {if (isNamespacePrefix == true) {isNamespacePrefix = false; isNamespaceUri = true;}};

RULE_STRING : ('"' ('\\' .|~(('\\'|'"')))* '"'|'\'' ('\\' .|~(('\\'|'\'')))* '\'');

RULE_NAMESPACE_URI : {isNamespaceUri}?=>('a'..'z'|'A'..'Z') ('a'..'z'|'A'..'Z'|':'|'/'|'.'|'0'..'9')* {isNamespaceUri = false;};

RULE_INT : ('0'..'9')+;

RULE_ANY_DATE : '|' ( options {greedy=false;} : . )*'|';

RULE_SL_COMMENT : '#' ~(('\n'|'\r'))* ('\r'? '\n')?;

RULE_WS : (' '|'\t'|'\r'|'\n')+;

If have added two boolean variables to handle the context for the namespace, to be honest I am not totally happy with my solution because I need to update the lexer grammar each time that I modify the xtext file but I was not able to cover my case in other ways.

I hope this information will be useful to you

Regards,
Luis

Report message to a moderator

Previous Topic:	Parser for Column Based Programming Language
Next Topic:	Changing cross reference proxy URIs?

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 05:29:17 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter