Home » Modeling » TMF (Xtext) » Serialization - order of unordered features
| Serialization - order of unordered features [message #869939] |
Fri, 04 May 2012 18:44  |
 Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
Hi,
Say I have grammar rules as such, where "out" and "in" are optional and the
order does not matter:
InputOutput:
'(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
(';'out=Output ')' | ')');
Output:
'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
Input:
'input' inputs+=MyInput (',' inputs+=MyInput)*;
When I serialize InputOutput that has both "out" and "in", I have noticed
that the "out" and "in" does always get serialized in the same order.
Sometimes, "out" gets written out before "in" and sometimes the other way
around. I want to get a consistent order such that if both "out" and "in"
exist, then "in" always gets written out first.
Is the ordering not consistent because of the way my grammar rule is
written? Will it help if I unravel it and list all possible options as such:
InputOutput:
'(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
out=Output ')' | '(' out=Output ';' in=Input ')'
Or would I have to customize the sequencer?
Mary
|
|
|
| Re: Serialization - order of unordered features [message #869946 is a reply to message #869939] |
Fri, 04 May 2012 21:25  |
Henrik Lindberg
Messages: 2463
Registered: July 2009 |
Senior Member |
|
|
Which version are you using? Do you have backtracking turned on?
Which serializer are you using (the old or the new)?
I think Moritz is working on a fix for the new searializer to make order
consistent (I have also found differences in order under certain
circumstances, and I don't think it really helps to modify the grammar.
- henrik
On 2012-05-05 24:44, Mary Komor wrote:
> Hi,
>
> Say I have grammar rules as such, where "out" and "in" are optional and
> the order does not matter:
>
> InputOutput:
> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
> (';'out=Output ')' | ')');
>
> Output:
> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>
> Input:
> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>
>
> When I serialize InputOutput that has both "out" and "in", I have
> noticed that the "out" and "in" does always get serialized in the same
> order. Sometimes, "out" gets written out before "in" and sometimes the
> other way around. I want to get a consistent order such that if both
> "out" and "in" exist, then "in" always gets written out first.
>
> Is the ordering not consistent because of the way my grammar rule is
> written? Will it help if I unravel it and list all possible options as
> such:
>
> InputOutput:
> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
> out=Output ')' | '(' out=Output ';' in=Input ')'
>
> Or would I have to customize the sequencer?
>
>
> Mary
|
|
|
| Re: Serialization - order of unordered features [message #869955 is a reply to message #869946] |
Sat, 05 May 2012 01:47 |
Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
I'm using Xtext 2.0.1 and the new Serializer:
fragment = serializer.SerializerFragment {}
Has this problem been addressed in the latest version? Just wondering if I
need to look into customizing the sequencer or not.
Mary
"Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
news:jo1vi8$9go$1@xxxxxxxxe.org...
> Which version are you using? Do you have backtracking turned on?
> Which serializer are you using (the old or the new)?
>
> I think Moritz is working on a fix for the new searializer to make order
> consistent (I have also found differences in order under certain
> circumstances, and I don't think it really helps to modify the grammar.
>
> - henrik
>
> On 2012-05-05 24:44, Mary Komor wrote:
>> Hi,
>>
>> Say I have grammar rules as such, where "out" and "in" are optional and
>> the order does not matter:
>>
>> InputOutput:
>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>> (';'out=Output ')' | ')');
>>
>> Output:
>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>
>> Input:
>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>
>>
>> When I serialize InputOutput that has both "out" and "in", I have
>> noticed that the "out" and "in" does always get serialized in the same
>> order. Sometimes, "out" gets written out before "in" and sometimes the
>> other way around. I want to get a consistent order such that if both
>> "out" and "in" exist, then "in" always gets written out first.
>>
>> Is the ordering not consistent because of the way my grammar rule is
>> written? Will it help if I unravel it and list all possible options as
>> such:
>>
>> InputOutput:
>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>
>> Or would I have to customize the sequencer?
>>
>>
>> Mary
>
|
|
|
| Re: Serialization - order of unordered features [message #870604 is a reply to message #869955] |
Tue, 08 May 2012 19:33 |
Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
Any advice as to how I should proceed with "fixing" this problem?
The order for the unordered elements seems to be quite random and seems to
even out at 50% either way. The randomness is seriously impeding our
progress in writing automated test cases. Some pointers on how to proceed
will be greatly appreciated.
If I do have to customize the Sequencer, are there any pointers on where to
start or where best to intervene? It will save me from digging through
layers of Xtext code.
Thanks in advance,
Mary
"Mary Komor" <eclipse.bugzilla.post@xxxxxxxx> wrote in message
news:jo2euf$5ll$1@xxxxxxxxe.org...
> I'm using Xtext 2.0.1 and the new Serializer:
>
> fragment = serializer.SerializerFragment {}
>
> Has this problem been addressed in the latest version? Just wondering if I
> need to look into customizing the sequencer or not.
>
> Mary
>
> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
> news:jo1vi8$9go$1@xxxxxxxxe.org...
>> Which version are you using? Do you have backtracking turned on?
>> Which serializer are you using (the old or the new)?
>>
>> I think Moritz is working on a fix for the new searializer to make order
>> consistent (I have also found differences in order under certain
>> circumstances, and I don't think it really helps to modify the grammar.
>>
>> - henrik
>>
>> On 2012-05-05 24:44, Mary Komor wrote:
>>> Hi,
>>>
>>> Say I have grammar rules as such, where "out" and "in" are optional and
>>> the order does not matter:
>>>
>>> InputOutput:
>>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>>> (';'out=Output ')' | ')');
>>>
>>> Output:
>>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>>
>>> Input:
>>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>>
>>>
>>> When I serialize InputOutput that has both "out" and "in", I have
>>> noticed that the "out" and "in" does always get serialized in the same
>>> order. Sometimes, "out" gets written out before "in" and sometimes the
>>> other way around. I want to get a consistent order such that if both
>>> "out" and "in" exist, then "in" always gets written out first.
>>>
>>> Is the ordering not consistent because of the way my grammar rule is
>>> written? Will it help if I unravel it and list all possible options as
>>> such:
>>>
>>> InputOutput:
>>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>>
>>> Or would I have to customize the sequencer?
>>>
>>>
>>> Mary
>>
|
|
|
| Re: Serialization - order of unordered features [message #870607 is a reply to message #870604] |
Tue, 08 May 2012 20:23 |
Henrik Lindberg
Messages: 2463
Registered: July 2009 |
Senior Member |
|
|
On 2012-09-05 1:33, Mary Komor wrote:
> Any advice as to how I should proceed with "fixing" this problem?
>
> The order for the unordered elements seems to be quite random and seems
> to even out at 50% either way. The randomness is seriously impeding our
> progress in writing automated test cases. Some pointers on how to
> proceed will be greatly appreciated.
>
> If I do have to customize the Sequencer, are there any pointers on where
> to start or where best to intervene? It will save me from digging
> through layers of Xtext code.
>
> Thanks in advance,
>
I have been working for some time on a new "semantic formatter" for
xtext to use in cloudsmith / geppetto @ github. It is near completion,
and I am basically waiting for the "random order" and "sequencer skips
trailing stuff" problems to be fixed (Moritz working on both) - and I
have one remaining feature to add regarding comment reconciliation when
formatter/sequencer injects/modifies syntax with associated comments.
When working on this, there were a handful of constructs in the geppetto
grammar that requires manual intervention in the sequencing logic.
You can look at my code in the branch called
https://github.com/cloudsmith/geppetto/tree/feature-formatter - I
override things to ensure that serialization produces correct code where
the grammar is too lenient. The same approach should be possible to used
to overcome random selection of serialized elements. I did however not
explicitly address the random serialization order in my grammar though.
You can look at:
/org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSemanticSequencer.java
and
/org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSyntacticSequencer.java
for some examples.
(To complete the picture if you start poking around in the
serialization/formatting code, the packages org.cloudsmith.xtext.*
contains the new formatter, and overridden classes from xtext - esp.
classes in org.cloudsmith.xtext.serializer.acceptor that override the
serializer/sequencer acceptors).
Having said all that....
I know Moritz made this change:
http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=986c7821ef45da8a50bad35fd5979e776d6472fe
Don't know if that is yet in the latest 2.3 - or if it will make it into
2.2.x stream.
Hope that helps you.
Regards
- henrik
> Mary
>
> "Mary Komor" <eclipse.bugzilla.post@xxxxxxxx> wrote in message
> news:jo2euf$5ll$1@xxxxxxxxe.org...
>> I'm using Xtext 2.0.1 and the new Serializer:
>>
>> fragment = serializer.SerializerFragment {}
>>
>> Has this problem been addressed in the latest version? Just wondering
>> if I need to look into customizing the sequencer or not.
>>
>> Mary
>>
>> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
>> news:jo1vi8$9go$1@xxxxxxxxe.org...
>>> Which version are you using? Do you have backtracking turned on?
>>> Which serializer are you using (the old or the new)?
>>>
>>> I think Moritz is working on a fix for the new searializer to make
>>> order consistent (I have also found differences in order under
>>> certain circumstances, and I don't think it really helps to modify
>>> the grammar.
>>>
>>> - henrik
>>>
>>> On 2012-05-05 24:44, Mary Komor wrote:
>>>> Hi,
>>>>
>>>> Say I have grammar rules as such, where "out" and "in" are optional and
>>>> the order does not matter:
>>>>
>>>> InputOutput:
>>>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>>>> (';'out=Output ')' | ')');
>>>>
>>>> Output:
>>>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>>>
>>>> Input:
>>>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>>>
>>>>
>>>> When I serialize InputOutput that has both "out" and "in", I have
>>>> noticed that the "out" and "in" does always get serialized in the same
>>>> order. Sometimes, "out" gets written out before "in" and sometimes the
>>>> other way around. I want to get a consistent order such that if both
>>>> "out" and "in" exist, then "in" always gets written out first.
>>>>
>>>> Is the ordering not consistent because of the way my grammar rule is
>>>> written? Will it help if I unravel it and list all possible options as
>>>> such:
>>>>
>>>> InputOutput:
>>>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>>>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>>>
>>>> Or would I have to customize the sequencer?
>>>>
>>>>
>>>> Mary
>>>
|
|
|
| Re: Serialization - order of unordered features [message #871129 is a reply to message #870607] |
Fri, 11 May 2012 00:47 |
Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
Hi Henrik,
Thanks very much for all the information and pointers.
I spent some time stepping through Xtext's Serializer/Sequencer code, trying
to understand it and trying to figure out where I would need to "intervene".
Unfortunately, I do not have the luxury of time to figure out a clean or
elegant solution right now. I finally resorted to a hack where I bypass the
Serializer. I pull the parsed text from the root node, use the
ContentFormatter to format the text and then write out the formatted text to
the file. Seems to be working well so far (and the added bonus is that it is
much, much faster and is very noticeable when serializing a huge file than
using Xtext's backtracking Serializer).
Mary
"Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
news:jocdf7$qr0$1@xxxxxxxxe.org...
> On 2012-09-05 1:33, Mary Komor wrote:
>> Any advice as to how I should proceed with "fixing" this problem?
>>
>> The order for the unordered elements seems to be quite random and seems
>> to even out at 50% either way. The randomness is seriously impeding our
>> progress in writing automated test cases. Some pointers on how to
>> proceed will be greatly appreciated.
>>
>> If I do have to customize the Sequencer, are there any pointers on where
>> to start or where best to intervene? It will save me from digging
>> through layers of Xtext code.
>>
>> Thanks in advance,
>>
> I have been working for some time on a new "semantic formatter" for xtext
> to use in cloudsmith / geppetto @ github. It is near completion, and I am
> basically waiting for the "random order" and "sequencer skips trailing
> stuff" problems to be fixed (Moritz working on both) - and I have one
> remaining feature to add regarding comment reconciliation when
> formatter/sequencer injects/modifies syntax with associated comments.
>
> When working on this, there were a handful of constructs in the geppetto
> grammar that requires manual intervention in the sequencing logic.
>
> You can look at my code in the branch called
> https://github.com/cloudsmith/geppetto/tree/feature-formatter - I override
> things to ensure that serialization produces correct code where the
> grammar is too lenient. The same approach should be possible to used to
> overcome random selection of serialized elements. I did however not
> explicitly address the random serialization order in my grammar though.
>
> You can look at:
> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSemanticSequencer.java
>
> and
>
> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSyntacticSequencer.java
>
> for some examples.
>
> (To complete the picture if you start poking around in the
> serialization/formatting code, the packages org.cloudsmith.xtext.*
> contains the new formatter, and overridden classes from xtext - esp.
> classes in org.cloudsmith.xtext.serializer.acceptor that override the
> serializer/sequencer acceptors).
>
> Having said all that....
>
> I know Moritz made this change:
> http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=986c7821ef45da8a50bad35fd5979e776d6472fe
>
> Don't know if that is yet in the latest 2.3 - or if it will make it into
> 2.2.x stream.
>
> Hope that helps you.
>
> Regards
> - henrik
>
>> Mary
>>
>> "Mary Komor" <eclipse.bugzilla.post@xxxxxxxx> wrote in message
>> news:jo2euf$5ll$1@xxxxxxxxe.org...
>>> I'm using Xtext 2.0.1 and the new Serializer:
>>>
>>> fragment = serializer.SerializerFragment {}
>>>
>>> Has this problem been addressed in the latest version? Just wondering
>>> if I need to look into customizing the sequencer or not.
>>>
>>> Mary
>>>
>>> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
>>> news:jo1vi8$9go$1@xxxxxxxxe.org...
>>>> Which version are you using? Do you have backtracking turned on?
>>>> Which serializer are you using (the old or the new)?
>>>>
>>>> I think Moritz is working on a fix for the new searializer to make
>>>> order consistent (I have also found differences in order under
>>>> certain circumstances, and I don't think it really helps to modify
>>>> the grammar.
>>>>
>>>> - henrik
>>>>
>>>> On 2012-05-05 24:44, Mary Komor wrote:
>>>>> Hi,
>>>>>
>>>>> Say I have grammar rules as such, where "out" and "in" are optional
>>>>> and
>>>>> the order does not matter:
>>>>>
>>>>> InputOutput:
>>>>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>>>>> (';'out=Output ')' | ')');
>>>>>
>>>>> Output:
>>>>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>>>>
>>>>> Input:
>>>>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>>>>
>>>>>
>>>>> When I serialize InputOutput that has both "out" and "in", I have
>>>>> noticed that the "out" and "in" does always get serialized in the same
>>>>> order. Sometimes, "out" gets written out before "in" and sometimes the
>>>>> other way around. I want to get a consistent order such that if both
>>>>> "out" and "in" exist, then "in" always gets written out first.
>>>>>
>>>>> Is the ordering not consistent because of the way my grammar rule is
>>>>> written? Will it help if I unravel it and list all possible options as
>>>>> such:
>>>>>
>>>>> InputOutput:
>>>>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>>>>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>>>>
>>>>> Or would I have to customize the sequencer?
>>>>>
>>>>>
>>>>> Mary
>>>>
>
|
|
|
| Re: Serialization - order of unordered features [message #871184 is a reply to message #871129] |
Fri, 11 May 2012 06:28 |
Henrik Lindberg
Messages: 2463
Registered: July 2009 |
Senior Member |
|
|
The serializer is required though when there is no node model, or where
parts of the node model is missing (when doing semantic modifications
etc.).
Regards
- henrik
On 2012-11-05 6:47, Mary Komor wrote:
> Hi Henrik,
>
> Thanks very much for all the information and pointers.
>
> I spent some time stepping through Xtext's Serializer/Sequencer code,
> trying to understand it and trying to figure out where I would need to
> "intervene". Unfortunately, I do not have the luxury of time to figure
> out a clean or elegant solution right now. I finally resorted to a hack
> where I bypass the Serializer. I pull the parsed text from the root
> node, use the ContentFormatter to format the text and then write out the
> formatted text to the file. Seems to be working well so far (and the
> added bonus is that it is much, much faster and is very noticeable when
> serializing a huge file than using Xtext's backtracking Serializer).
>
> Mary
>
> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
> news:jocdf7$qr0$1@xxxxxxxxe.org...
>> On 2012-09-05 1:33, Mary Komor wrote:
>>> Any advice as to how I should proceed with "fixing" this problem?
>>>
>>> The order for the unordered elements seems to be quite random and seems
>>> to even out at 50% either way. The randomness is seriously impeding our
>>> progress in writing automated test cases. Some pointers on how to
>>> proceed will be greatly appreciated.
>>>
>>> If I do have to customize the Sequencer, are there any pointers on where
>>> to start or where best to intervene? It will save me from digging
>>> through layers of Xtext code.
>>>
>>> Thanks in advance,
>>>
>> I have been working for some time on a new "semantic formatter" for
>> xtext to use in cloudsmith / geppetto @ github. It is near completion,
>> and I am basically waiting for the "random order" and "sequencer skips
>> trailing stuff" problems to be fixed (Moritz working on both) - and I
>> have one remaining feature to add regarding comment reconciliation
>> when formatter/sequencer injects/modifies syntax with associated
>> comments.
>>
>> When working on this, there were a handful of constructs in the
>> geppetto grammar that requires manual intervention in the sequencing
>> logic.
>>
>> You can look at my code in the branch called
>> https://github.com/cloudsmith/geppetto/tree/feature-formatter - I
>> override things to ensure that serialization produces correct code
>> where the grammar is too lenient. The same approach should be possible
>> to used to overcome random selection of serialized elements. I did
>> however not explicitly address the random serialization order in my
>> grammar though.
>>
>> You can look at:
>> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSemanticSequencer.java
>>
>>
>> and
>>
>> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSyntacticSequencer.java
>>
>>
>> for some examples.
>>
>> (To complete the picture if you start poking around in the
>> serialization/formatting code, the packages org.cloudsmith.xtext.*
>> contains the new formatter, and overridden classes from xtext - esp.
>> classes in org.cloudsmith.xtext.serializer.acceptor that override the
>> serializer/sequencer acceptors).
>>
>> Having said all that....
>>
>> I know Moritz made this change:
>> http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=986c7821ef45da8a50bad35fd5979e776d6472fe
>>
>>
>> Don't know if that is yet in the latest 2.3 - or if it will make it
>> into 2.2.x stream.
>>
>> Hope that helps you.
>>
>> Regards
>> - henrik
>>
>>> Mary
>>>
>>> "Mary Komor" <eclipse.bugzilla.post@xxxxxxxx> wrote in message
>>> news:jo2euf$5ll$1@xxxxxxxxe.org...
>>>> I'm using Xtext 2.0.1 and the new Serializer:
>>>>
>>>> fragment = serializer.SerializerFragment {}
>>>>
>>>> Has this problem been addressed in the latest version? Just wondering
>>>> if I need to look into customizing the sequencer or not.
>>>>
>>>> Mary
>>>>
>>>> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
>>>> news:jo1vi8$9go$1@xxxxxxxxe.org...
>>>>> Which version are you using? Do you have backtracking turned on?
>>>>> Which serializer are you using (the old or the new)?
>>>>>
>>>>> I think Moritz is working on a fix for the new searializer to make
>>>>> order consistent (I have also found differences in order under
>>>>> certain circumstances, and I don't think it really helps to modify
>>>>> the grammar.
>>>>>
>>>>> - henrik
>>>>>
>>>>> On 2012-05-05 24:44, Mary Komor wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Say I have grammar rules as such, where "out" and "in" are
>>>>>> optional and
>>>>>> the order does not matter:
>>>>>>
>>>>>> InputOutput:
>>>>>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>>>>>> (';'out=Output ')' | ')');
>>>>>>
>>>>>> Output:
>>>>>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>>>>>
>>>>>> Input:
>>>>>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>>>>>
>>>>>>
>>>>>> When I serialize InputOutput that has both "out" and "in", I have
>>>>>> noticed that the "out" and "in" does always get serialized in the
>>>>>> same
>>>>>> order. Sometimes, "out" gets written out before "in" and sometimes
>>>>>> the
>>>>>> other way around. I want to get a consistent order such that if both
>>>>>> "out" and "in" exist, then "in" always gets written out first.
>>>>>>
>>>>>> Is the ordering not consistent because of the way my grammar rule is
>>>>>> written? Will it help if I unravel it and list all possible
>>>>>> options as
>>>>>> such:
>>>>>>
>>>>>> InputOutput:
>>>>>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>>>>>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>>>>>
>>>>>> Or would I have to customize the sequencer?
>>>>>>
>>>>>>
>>>>>> Mary
>>>>>
>>
|
|
|
| Re: Serialization - order of unordered features [message #871320 is a reply to message #871184] |
Fri, 11 May 2012 18:57 |
Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
In my case, I always have a node model. But I can see your point.
"Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
news:joipjn$9ri$1@xxxxxxxxe.org...
> The serializer is required though when there is no node model, or where
> parts of the node model is missing (when doing semantic modifications
> etc.).
>
> Regards
> - henrik
>
> On 2012-11-05 6:47, Mary Komor wrote:
>> Hi Henrik,
>>
>> Thanks very much for all the information and pointers.
>>
>> I spent some time stepping through Xtext's Serializer/Sequencer code,
>> trying to understand it and trying to figure out where I would need to
>> "intervene". Unfortunately, I do not have the luxury of time to figure
>> out a clean or elegant solution right now. I finally resorted to a hack
>> where I bypass the Serializer. I pull the parsed text from the root
>> node, use the ContentFormatter to format the text and then write out the
>> formatted text to the file. Seems to be working well so far (and the
>> added bonus is that it is much, much faster and is very noticeable when
>> serializing a huge file than using Xtext's backtracking Serializer).
>>
>> Mary
>>
>> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
>> news:jocdf7$qr0$1@xxxxxxxxe.org...
>>> On 2012-09-05 1:33, Mary Komor wrote:
>>>> Any advice as to how I should proceed with "fixing" this problem?
>>>>
>>>> The order for the unordered elements seems to be quite random and seems
>>>> to even out at 50% either way. The randomness is seriously impeding our
>>>> progress in writing automated test cases. Some pointers on how to
>>>> proceed will be greatly appreciated.
>>>>
>>>> If I do have to customize the Sequencer, are there any pointers on
>>>> where
>>>> to start or where best to intervene? It will save me from digging
>>>> through layers of Xtext code.
>>>>
>>>> Thanks in advance,
>>>>
>>> I have been working for some time on a new "semantic formatter" for
>>> xtext to use in cloudsmith / geppetto @ github. It is near completion,
>>> and I am basically waiting for the "random order" and "sequencer skips
>>> trailing stuff" problems to be fixed (Moritz working on both) - and I
>>> have one remaining feature to add regarding comment reconciliation
>>> when formatter/sequencer injects/modifies syntax with associated
>>> comments.
>>>
>>> When working on this, there were a handful of constructs in the
>>> geppetto grammar that requires manual intervention in the sequencing
>>> logic.
>>>
>>> You can look at my code in the branch called
>>> https://github.com/cloudsmith/geppetto/tree/feature-formatter - I
>>> override things to ensure that serialization produces correct code
>>> where the grammar is too lenient. The same approach should be possible
>>> to used to overcome random selection of serialized elements. I did
>>> however not explicitly address the random serialization order in my
>>> grammar though.
>>>
>>> You can look at:
>>> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSemanticSequencer.java
>>>
>>>
>>> and
>>>
>>> /org.cloudsmith.geppetto.pp.dsl/src/org/cloudsmith/geppetto/pp/dsl/serializer/PPSyntacticSequencer.java
>>>
>>>
>>> for some examples.
>>>
>>> (To complete the picture if you start poking around in the
>>> serialization/formatting code, the packages org.cloudsmith.xtext.*
>>> contains the new formatter, and overridden classes from xtext - esp.
>>> classes in org.cloudsmith.xtext.serializer.acceptor that override the
>>> serializer/sequencer acceptors).
>>>
>>> Having said all that....
>>>
>>> I know Moritz made this change:
>>> http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=986c7821ef45da8a50bad35fd5979e776d6472fe
>>>
>>>
>>> Don't know if that is yet in the latest 2.3 - or if it will make it
>>> into 2.2.x stream.
>>>
>>> Hope that helps you.
>>>
>>> Regards
>>> - henrik
>>>
>>>> Mary
>>>>
>>>> "Mary Komor" <eclipse.bugzilla.post@xxxxxxxx> wrote in message
>>>> news:jo2euf$5ll$1@xxxxxxxxe.org...
>>>>> I'm using Xtext 2.0.1 and the new Serializer:
>>>>>
>>>>> fragment = serializer.SerializerFragment {}
>>>>>
>>>>> Has this problem been addressed in the latest version? Just wondering
>>>>> if I need to look into customizing the sequencer or not.
>>>>>
>>>>> Mary
>>>>>
>>>>> "Henrik Lindberg" <henrik.lindberg@xxxxxxxx> wrote in message
>>>>> news:jo1vi8$9go$1@xxxxxxxxe.org...
>>>>>> Which version are you using? Do you have backtracking turned on?
>>>>>> Which serializer are you using (the old or the new)?
>>>>>>
>>>>>> I think Moritz is working on a fix for the new searializer to make
>>>>>> order consistent (I have also found differences in order under
>>>>>> certain circumstances, and I don't think it really helps to modify
>>>>>> the grammar.
>>>>>>
>>>>>> - henrik
>>>>>>
>>>>>> On 2012-05-05 24:44, Mary Komor wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Say I have grammar rules as such, where "out" and "in" are
>>>>>>> optional and
>>>>>>> the order does not matter:
>>>>>>>
>>>>>>> InputOutput:
>>>>>>> '(' ')' | '(' out=Output (';' in=Input ')' | ')') | '(' in=Input
>>>>>>> (';'out=Output ')' | ')');
>>>>>>>
>>>>>>> Output:
>>>>>>> 'output' outputs+=MyOutput (',' outputs+=MyOutput)*;
>>>>>>>
>>>>>>> Input:
>>>>>>> 'input' inputs+=MyInput (',' inputs+=MyInput)*;
>>>>>>>
>>>>>>>
>>>>>>> When I serialize InputOutput that has both "out" and "in", I have
>>>>>>> noticed that the "out" and "in" does always get serialized in the
>>>>>>> same
>>>>>>> order. Sometimes, "out" gets written out before "in" and sometimes
>>>>>>> the
>>>>>>> other way around. I want to get a consistent order such that if both
>>>>>>> "out" and "in" exist, then "in" always gets written out first.
>>>>>>>
>>>>>>> Is the ordering not consistent because of the way my grammar rule is
>>>>>>> written? Will it help if I unravel it and list all possible
>>>>>>> options as
>>>>>>> such:
>>>>>>>
>>>>>>> InputOutput:
>>>>>>> '(' ')' | '(' in=Input ')' | '(' out=Output ')' | '(' in=Input ';'
>>>>>>> out=Output ')' | '(' out=Output ';' in=Input ')'
>>>>>>>
>>>>>>> Or would I have to customize the sequencer?
>>>>>>>
>>>>>>>
>>>>>>> Mary
>>>>>>
>>>
>
|
|
|
| Re: Serialization - order of unordered features [message #881881 is a reply to message #871320] |
Tue, 05 June 2012 08:17 |
Moritz Eysholdt
Messages: 138
Registered: July 2009
Location: Kiel, Germany |
Senior Member |
|
|
Hi Mary,
yes, there was some nondeterminism in the serializer about how to oder tokens during serialization when the grammar allows multiple orders. This, however, (and as Henrik already mentioned) has been fixed.... now the order in the serializers output is always expected to be the same as in the grammar. If you want to look at the code, org.eclipse.xtext.serializer.analysis.ISemanticSequencerNfaProvider.ISemState.getOrderID() is a good starting point.
So, if you reorder the elements in your grammar, you can customize the oder of the output during serialization.
hth,
Moritz
|
|
|
| Re: Serialization - order of unordered features [message #891809 is a reply to message #881881] |
Tue, 26 June 2012 01:55 |
Mary Komor
Messages: 48
Registered: July 2009 |
Member |
|
|
Hi Moritz,
Which version did the fix go into?
Mary
"Moritz Eysholdt" <forums-noreply@xxxxxxxx> wrote in message
news:jqktcm$hi$1@xxxxxxxxe.org...
> Hi Mary,
>
> yes, there was some nondeterminism in the serializer about how to oder
> tokens during serialization when the grammar allows multiple orders. This,
> however, (and as Henrik already mentioned) has been fixed.... now the
> order in the serializers output is always expected to be the same as in
> the grammar. If you want to look at the code,
> org.eclipse.xtext.serializer.analysis.ISemanticSequencerNfaProvider.ISemState.getOrderID()
> is a good starting point.
> So, if you reorder the elements in your grammar, you can customize the
> oder of the output during serialization.
>
> hth,
> Moritz
|
|
| |
Goto Forum:
Current Time: Tue Oct 08 08:05:00 EDT 2013
Powered by FUDForum. Page generated in 0.14193 seconds
|
] 
Search
Help
Register
Login
Home
