Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Handling older files which contain unescaped keywords
Handling older files which contain unescaped keywords [message #1731477] Thu, 05 May 2016 22:46 Go to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I have an issue in our internal model with loading old models which use identifiers which are now illegal due to the introduction of new rules.

My grammar:
https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl/src/com/pellcorp/mydsl/MyDsl.xtext

My old xtext file:
https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl.tests/src/com/pellcorp/mydsl/tests/test.mydsl

The syntax error I get:
mismatched input 'percentage' expecting '}'

I was wondering if there is a way to handle this without changing the syntax. Perhaps with some refinements to the .xtext model or maybe overriding some parts of the parser process?

Re: Handling older files which contain unescaped keywords [message #1731489 is a reply to message #1731477] Fri, 06 May 2016 04:58 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
You can use a datatype rule instand of ID

MyID: ID | ... | "keywordiwanttouseasid"


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731574 is a reply to message #1731489] Sat, 07 May 2016 03:55 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Do you mean on the Field rule i should add all the keywords that i would like to support as field names?

Why cant xtext just recognise that when percentage appears inside an Entity it should not be treated as a keyword. The parsing should be contextual, rather than percentage being a global keyword.

This is only a sample grammar to demonstrate the issue. My actual grammar is proprietary and so i cant post it but the issue is a much larger issue in the other grammar. And its not just field names
Re: Handling older files which contain unescaped keywords [message #1731580 is a reply to message #1731574] Sat, 07 May 2016 05:10 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
Because Xtext uses Antr and Antr uses Lexer based parsing. And the Lexer is context free.
Thus use bla=MyId Inside Entity.

Of Course you Could think of switching to a custom (Maybe) stateful Lexer but that is Not trivial done


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731584 is a reply to message #1731580] Sat, 07 May 2016 07:51 Go to previous messageGo to next message
Uli Merkel is currently offline Uli MerkelFriend
Messages: 250
Registered: June 2013
Senior Member
what about using the old grammar and give the generator an additional feature to generate a modified DSL-File prefixing with a "^" IDs which became a keyword in between ?
Re: Handling older files which contain unescaped keywords [message #1731651 is a reply to message #1731584] Mon, 09 May 2016 02:11 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Christian,
Can you provide any pointers about how I might go about implementing a stateful Lexer. Are you aware of any one doing this with xtext?
Re: Handling older files which contain unescaped keywords [message #1731652 is a reply to message #1731651] Mon, 09 May 2016 03:04 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
Yes i know Guys Doing this but I have no all in one link.

Googling gives you eg http://consoliii.blogspot.de/2013/04/xtext-is-incredibly-powerful-framework.html

But I still don't get why you don't use the MyID approach. It is way more easy


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731763 is a reply to message #1731652] Tue, 10 May 2016 05:00 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
If I understood correctly, in order to use MyID, I would have to define every single possible keyword. The sample grammar I provided is a fraction of the size of the real one. Having to define a MyID rule of:

MyID: ID | "percentage" | "response" | "error" | "prepare" | .... and on it goes for potentially many 10s of keywords that currently cause clashes.

Because you have mentioned it several times, I feel like I must be missing something. Is the above not what you meant?
Re: Handling older files which contain unescaped keywords [message #1731764 is a reply to message #1731763] Tue, 10 May 2016 05:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
yes this is what i meant. but i dont see the problem.

from your initial post it sounded like "a few" keywords anyway


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731766 is a reply to message #1731763] Tue, 10 May 2016 05:29 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Ok, so in my sample I did as you suggested. Definately works nicely and its potentially a solution for us, even if it does require a bit of work to identify all the keywords.

https://github.com/pellcorp/xtext/commit/0b07b79d3adcccff6af5152bca69fcdd78e5f028
Re: Handling older files which contain unescaped keywords [message #1731767 is a reply to message #1731766] Tue, 10 May 2016 05:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
what about letting the computer do that job for you

import org.eclipse.xtext.GrammarUtil;
import org.eclipse.xtext.IGrammarAccess;

import com.google.inject.Injector;

public class Main {
	
	public static void main(String[] args) {
		Injector injector = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration();
		IGrammarAccess ga = injector.getInstance(IGrammarAccess.class);
		System.out.println(GrammarUtil.getAllKeywords(ga.getGrammar()));
	}

}



Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731785 is a reply to message #1731767] Tue, 10 May 2016 07:35 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Hi Christian,

I had come to that same conclusion, thanks for confirming my thinking was not crazy Smile
Re: Handling older files which contain unescaped keywords [message #1731796 is a reply to message #1731785] Tue, 10 May 2016 09:03 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Any suggestions for handling old files which already have the ^ escaped keywords? Any way to get the parser to automatically remove the ^ and no re-saving with the ^ prefix.
Re: Handling older files which contain unescaped keywords [message #1731801 is a reply to message #1731796] Tue, 10 May 2016 09:12 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
Did you implement an ivalueconverter for MyID?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731804 is a reply to message #1731801] Tue, 10 May 2016 09:38 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I had forgotten about that, have now

https://github.com/pellcorp/xtext/blob/master/com.pellcorp.mydsl/src/com/pellcorp/mydsl/services/MyDslValueConverter.java

The carets still get added when the model is serialized to text, but the in memory model does have the IDs minus the ^

If I really wanted to remove the carets from what is serialized is there another class I need to implement?

Re: Handling older files which contain unescaped keywords [message #1731813 is a reply to message #1731804] Tue, 10 May 2016 10:32 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
i dont know which "serialized" you are talking about but this is done by valueconverter.toString

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731820 is a reply to message #1731813] Tue, 10 May 2016 11:24 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Saving the memory model to text restores the carets. Can i prevent that?
Re: Handling older files which contain unescaped keywords [message #1731822 is a reply to message #1731820] Tue, 10 May 2016 11:32 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
was the model parsed before? or is it newly created?

at least you should implement the value converters to string.


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731828 is a reply to message #1731822] Tue, 10 May 2016 11:46 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I loaded the model and confirmed with debug the the value converter removed the caret. I resaved this loaded model to text and the carets were back.

Sorry was reading on phone missed your info about toString

Will take a look. I did put a break point in the toString method and the method was not executed.



[Updated on: Tue, 10 May 2016 11:49]

Report message to a moderator

Re: Handling older files which contain unescaped keywords [message #1731829 is a reply to message #1731828] Tue, 10 May 2016 11:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
this is done here: org.eclipse.xtext.serializer.tokens.ValueSerializer.serializeAssignedValue(EObject, RuleCall, Object, INode, Acceptor)
it wont use the value converted value by default.
feel free to customize that


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731832 is a reply to message #1731829] Tue, 10 May 2016 11:51 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Ok thanks. Thats great news.
Re: Handling older files which contain unescaped keywords [message #1731843 is a reply to message #1731832] Tue, 10 May 2016 12:51 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
Am I right that the carets actually do remain in the Node model, but are removed from the emf one. So that is why the carets are re-serialized by default.

What would be the recommended interface to override in order to remove the caret from the node before serializing? I can see I have several choices.

I can implement a IValueSerializer, but when I extend org.eclipse.xtext.serializer.tokens.ValueSerializer, eclipse warns me that this is not a good idea, so should I re-implement the entirety of the IValueSerializer?

Or should I override the TokenUtil class which is used by the org.eclipse.xtext.serializer.tokens.ValueSerializer

Any others I have missed.

[Updated on: Tue, 10 May 2016 12:57]

Report message to a moderator

Re: Handling older files which contain unescaped keywords [message #1731846 is a reply to message #1731843] Tue, 10 May 2016 12:55 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
id subclass ValueSerializer and "ignore" the restriction warning and will live with potential broken api when doing a xtext update

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731848 is a reply to message #1731846] Tue, 10 May 2016 13:10 Go to previous messageGo to next message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I am not sure if this is a good idea or not, would appreciate your opinion.

@SuppressWarnings("restriction")
public class MyDslValueSerializer extends ValueSerializer {
	public String serializeAssignedValue(EObject context, RuleCall ruleCall, Object value, INode node, Acceptor errors) {
		String name = ruleCall.getRule().getName();
		if ("MyID".equals(name))  {
                        // passing 'null' node, to force use of value converter
			return super.serializeAssignedValue(context, ruleCall, value, null, errors);
		} else {
			return super.serializeAssignedValue(context, ruleCall, value, node, errors);
		}
	}
}
Re: Handling older files which contain unescaped keywords [message #1731855 is a reply to message #1731848] Tue, 10 May 2016 13:55 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
sounds ok for me

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Handling older files which contain unescaped keywords [message #1731919 is a reply to message #1731855] Wed, 11 May 2016 03:44 Go to previous message
Jason Pell is currently offline Jason PellFriend
Messages: 55
Registered: February 2011
Member
I have integrated this stuff into my main model at work, and it works flawlessly

Christian, thanks so much for your guidance, very much appreciated.
Previous Topic:Generating DSL2 files from DSL1
Next Topic:Not allowing empty lines at beginning of the page
Goto Forum:
  


Current Time: Fri Mar 29 08:35:19 GMT 2024

Powered by FUDForum. Page generated in 0.03554 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top