Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Handling older files which contain unescaped keywords
Handling older files which contain unescaped keywords [message #1731477] Thu, 05 May 2016 22:46 Go to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I have an issue in our internal model with loading old models which use identifiers which are now illegal due to the introduction of new rules.

My grammar:
https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl/src/com/pellcorp/mydsl/MyDsl.xtext

My old xtext file:
https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl.tests/src/com/pellcorp/mydsl/tests/test.mydsl

The syntax error I get:
mismatched input 'percentage' expecting '}'

I was wondering if there is a way to handle this without changing the syntax. Perhaps with some refinements to the .xtext model or maybe overriding some parts of the parser process?

Re: Handling older files which contain unescaped keywords [message #1731489 is a reply to message #1731477] Fri, 06 May 2016 04:58 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
You can use a datatype rule instand of ID

MyID: ID | ... | "keywordiwanttouseasid"


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731574 is a reply to message #1731489] Sat, 07 May 2016 03:55 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Do you mean on the Field rule i should add all the keywords that i would like to support as field names?

Why cant xtext just recognise that when percentage appears inside an Entity it should not be treated as a keyword. The parsing should be contextual, rather than percentage being a global keyword.

This is only a sample grammar to demonstrate the issue. My actual grammar is proprietary and so i cant post it but the issue is a much larger issue in the other grammar. And its not just field names
Re: Handling older files which contain unescaped keywords [message #1731580 is a reply to message #1731574] Sat, 07 May 2016 05:10 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
Because Xtext uses Antr and Antr uses Lexer based parsing. And the Lexer is context free.
Thus use bla=MyId Inside Entity.

Of Course you Could think of switching to a custom (Maybe) stateful Lexer but that is Not trivial done


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731584 is a reply to message #1731580] Sat, 07 May 2016 07:51 Go to previous messageGo to next message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 246
Registered: June 2013
Senior Member
what about using the old grammar and give the generator an additional feature to generate a modified DSL-File prefixing with a "^" IDs which became a keyword in between ?
Re: Handling older files which contain unescaped keywords [message #1731651 is a reply to message #1731584] Mon, 09 May 2016 02:11 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Christian,
Can you provide any pointers about how I might go about implementing a stateful Lexer. Are you aware of any one doing this with xtext?
Re: Handling older files which contain unescaped keywords [message #1731652 is a reply to message #1731651] Mon, 09 May 2016 03:04 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
Yes i know Guys Doing this but I have no all in one link.

Googling gives you eg http://consoliii.blogspot.de/2013/04/xtext-is-incredibly-powerful-framework.html

But I still don't get why you don't use the MyID approach. It is way more easy


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731763 is a reply to message #1731652] Tue, 10 May 2016 05:00 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
If I understood correctly, in order to use MyID, I would have to define every single possible keyword. The sample grammar I provided is a fraction of the size of the real one. Having to define a MyID rule of:

MyID: ID | "percentage" | "response" | "error" | "prepare" | .... and on it goes for potentially many 10s of keywords that currently cause clashes.

Because you have mentioned it several times, I feel like I must be missing something. Is the above not what you meant?
Re: Handling older files which contain unescaped keywords [message #1731764 is a reply to message #1731763] Tue, 10 May 2016 05:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
yes this is what i meant. but i dont see the problem.

from your initial post it sounded like "a few" keywords anyway


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731766 is a reply to message #1731763] Tue, 10 May 2016 05:29 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Ok, so in my sample I did as you suggested. Definately works nicely and its potentially a solution for us, even if it does require a bit of work to identify all the keywords.

https://github.com/pellcorp/xtext/commit/0b07b79d3adcccff6af5152bca69fcdd78e5f028
Re: Handling older files which contain unescaped keywords [message #1731767 is a reply to message #1731766] Tue, 10 May 2016 05:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
what about letting the computer do that job for you

import org.eclipse.xtext.GrammarUtil;
import org.eclipse.xtext.IGrammarAccess;

import com.google.inject.Injector;

public class Main {
	
	public static void main(String[] args) {
		Injector injector = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration();
		IGrammarAccess ga = injector.getInstance(IGrammarAccess.class);
		System.out.println(GrammarUtil.getAllKeywords(ga.getGrammar()));
	}

}



Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731785 is a reply to message #1731767] Tue, 10 May 2016 07:35 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Hi Christian,

I had come to that same conclusion, thanks for confirming my thinking was not crazy Smile
Re: Handling older files which contain unescaped keywords [message #1731796 is a reply to message #1731785] Tue, 10 May 2016 09:03 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Any suggestions for handling old files which already have the ^ escaped keywords? Any way to get the parser to automatically remove the ^ and no re-saving with the ^ prefix.
Re: Handling older files which contain unescaped keywords [message #1731801 is a reply to message #1731796] Tue, 10 May 2016 09:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
Did you implement an ivalueconverter for MyID?

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731804 is a reply to message #1731801] Tue, 10 May 2016 09:38 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I had forgotten about that, have now

https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl/src/com/pellcorp/mydsl/services/MyDslValueConverter.java

The carets still get added when the model is serialized to text, but the in memory model does have the IDs minus the ^

If I really wanted to remove the carets from what is serialized is there another class I need to implement?

Re: Handling older files which contain unescaped keywords [message #1731813 is a reply to message #1731804] Tue, 10 May 2016 10:32 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
i dont know which "serialized" you are talking about but this is done by valueconverter.toString

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731820 is a reply to message #1731813] Tue, 10 May 2016 11:24 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Saving the memory model to text restores the carets. Can i prevent that?
Re: Handling older files which contain unescaped keywords [message #1731822 is a reply to message #1731820] Tue, 10 May 2016 11:32 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
was the model parsed before? or is it newly created?

at least you should implement the value converters to string.


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731828 is a reply to message #1731822] Tue, 10 May 2016 11:46 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I loaded the model and confirmed with debug the the value converter removed the caret. I resaved this loaded model to text and the carets were back.

Sorry was reading on phone missed your info about toString

Will take a look. I did put a break point in the toString method and the method was not executed.



[Updated on: Tue, 10 May 2016 11:49]

Report message to a moderator

Re: Handling older files which contain unescaped keywords [message #1731829 is a reply to message #1731828] Tue, 10 May 2016 11:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
this is done here: org.eclipse.xtext.serializer.tokens.ValueSerializer.serializeAssignedValue(EObject, RuleCall, Object, INode, Acceptor)
it wont use the value converted value by default.
feel free to customize that


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731832 is a reply to message #1731829] Tue, 10 May 2016 11:51 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Ok thanks. Thats great news.
Re: Handling older files which contain unescaped keywords [message #1731843 is a reply to message #1731832] Tue, 10 May 2016 12:51 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Am I right that the carets actually do remain in the Node model, but are removed from the emf one. So that is why the carets are re-serialized by default.

What would be the recommended interface to override in order to remove the caret from the node before serializing? I can see I have several choices.

I can implement a IValueSerializer, but when I extend org.eclipse.xtext.serializer.tokens.ValueSerializer, eclipse warns me that this is not a good idea, so should I re-implement the entirety of the IValueSerializer?

Or should I override the TokenUtil class which is used by the org.eclipse.xtext.serializer.tokens.ValueSerializer

Any others I have missed.

[Updated on: Tue, 10 May 2016 12:57]

Report message to a moderator

Re: Handling older files which contain unescaped keywords [message #1731846 is a reply to message #1731843] Tue, 10 May 2016 12:55 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
id subclass ValueSerializer and "ignore" the restriction warning and will live with potential broken api when doing a xtext update

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731848 is a reply to message #1731846] Tue, 10 May 2016 13:10 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I am not sure if this is a good idea or not, would appreciate your opinion.

@SuppressWarnings("restriction")
public class MyDslValueSerializer extends ValueSerializer {
	public String serializeAssignedValue(EObject context, RuleCall ruleCall, Object value, INode node, Acceptor errors) {
		String name = ruleCall.getRule().getName();
		if ("MyID".equals(name))  {
                        // passing 'null' node, to force use of value converter
			return super.serializeAssignedValue(context, ruleCall, value, null, errors);
		} else {
			return super.serializeAssignedValue(context, ruleCall, value, node, errors);
		}
	}
}
Re: Handling older files which contain unescaped keywords [message #1731855 is a reply to message #1731848] Tue, 10 May 2016 13:55 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13193
Registered: July 2009
Senior Member
sounds ok for me

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731919 is a reply to message #1731855] Wed, 11 May 2016 03:44 Go to previous message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I have integrated this stuff into my main model at work, and it works flawlessly

Christian, thanks so much for your guidance, very much appreciated.
Previous Topic:Generating DSL2 files from DSL1
Next Topic:Not allowing empty lines at beginning of the page
Goto Forum:
  


Current Time: Sat Nov 16 21:08:29 GMT 2019

Powered by FUDForum. Page generated in 0.04454 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top