Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Debugging non-deterministic serialization failure
Debugging non-deterministic serialization failure [message #1746530] Mon, 31 October 2016 18:06 Go to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
Using Xtext 2.8.4 I have a test case for a quick fix which non-deterministically alternates between passing and "Could not serialize EObject via backtracking."

Does anyone have some hints on how to debug this? What could possibly cause such non-determinism?

thanks,
Stephan
Re: Debugging non-deterministic serialization failure [message #1746531 is a reply to message #1746530] Mon, 31 October 2016 18:09 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
did you have a look what contrained makes this fail? (SequencerDiagnosticProvider)
what is the cause there?

Re: Debugging non-deterministic serialization failure [message #1746532 is a reply to message #1746530] Mon, 31 October 2016 18:24 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
Further observation: during one IDE session the quick fix either consistently works or consistently fails. Apparently s.t. goes havoc during initialization of my plugin? In both cases the error log is free of relevant entries, though.
Re: Debugging non-deterministic serialization failure [message #1746533 is a reply to message #1746532] Mon, 31 October 2016 18:28 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
looks very strange. thus it would be interesting which contraint is affected. do you have grammar inhertiance etc involved?
Re: Debugging non-deterministic serialization failure [message #1746534 is a reply to message #1746533] Mon, 31 October 2016 18:38 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
No grammar inheritance, only cross references to elements from other DSLs.

Here are the details of the exception, not sure if it helps any without the (somewhat complicated) grammar:
Caused by: java.lang.RuntimeException: Could not serialize EObject via backtracking.
Constraint: null ((annotations+=DslAnnotation* incomplete?='incomplete'? | incomplete?='incomplete'? usingDeprecatedAtSyntax?='@') name=ID prototype=[InstanceDefinition|FQN]? entity=[Entity|FQN]? attributeDefinitions+=AttributeDefinition* | (annotations+=DslAnnotation* inherit='inherited' incomplete?='incomplete'? prototype=[InstanceDefinition|ID] | inherit='specialized' prototype=[InstanceDefinition|ID] (attributeDefinitions+=AttributeDefinitionAbbrev | attributeDefinitions+=AttributeDefinitionPlain)) | annotations+=DslAnnotation* inherit='specialized' incomplete?='incomplete'? prototype=[InstanceDefinition|ID] entity=[Entity|FQN]? attributeDefinitions+=AttributeDefinition*) null
Values: inherit(1), prototype(1), incomplete(0-1), resolving(0-1), usingDeprecatedAtSyntax(0-1)
Semantic Object: DomainInstanceModel.bundleDefinitionList[1]->BundleDefinition'b1'.instanceList[0]->InstanceDefinition
Context: SubInstanceDefinitionInBundle


Does this provide any clue where to continue searching?
Stephan
Re: Debugging non-deterministic serialization failure [message #1746535 is a reply to message #1746534] Mon, 31 October 2016 18:41 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
where is the grammar for the "resolving" thing? which attributes/references are actually set in the object you try to serialize?

[Updated on: Mon, 31 October 2016 18:46]

Report message to a moderator

Re: Debugging non-deterministic serialization failure [message #1746537 is a reply to message #1746535] Mon, 31 October 2016 19:11 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
"resolving" is marked as transient in EMF, so this shouldn't be looked at at all.

"inherit" is set to the string "inherited" (thus uniquely identifying the alternative)

"prototype" is set as a valid cross reference (from another resource).

Everything else is unset.

Re: Debugging non-deterministic serialization failure [message #1746538 is a reply to message #1746537] Mon, 31 October 2016 19:17 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
Can you Double Check the transient value Service picks up the Emf Value correct
Re: Debugging non-deterministic serialization failure [message #1746539 is a reply to message #1746538] Mon, 31 October 2016 19:39 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
the other point would be the cross ref serializer,
does the name of the prototype be serializable as iD?

unfortunately the error message is not that good in the alternatives case if not.
Re: Debugging non-deterministic serialization failure [message #1746540 is a reply to message #1746538] Mon, 31 October 2016 19:40 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
We actually have our own transient value service (motivated by a tweak concerning another attribute).
That service implements both org.eclipse.xtext.serializer.sequencer.ITransientValueService and org.eclipse.xtext.parsetree.reconstr.ITransientValueService.
Via the former API we answer (as per the super implementation) ValueTransient.PREFERABLY (because the attribute is not of type String and thus answers true from defaultValueIsSerializeable()).
Via the latter API this is reported as 'true' (is transient).

I experimentally added explicit treatment to let this guy consistently answer ValueTransient.YES, but test keeps failing sometimes.

At least that experiment makes "resolving" disappear from the error message.
Re: Debugging non-deterministic serialization failure [message #1746541 is a reply to message #1746540] Mon, 31 October 2016 19:46 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
ok then the second thing is about the scoping for prototype.
your naming sheme seems to differ between branches
Re: Debugging non-deterministic serialization failure [message #1746542 is a reply to message #1746540] Mon, 31 October 2016 19:50 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
We should be looking at this alternative:
annotations+=DslAnnotation* inherit='inherited' incomplete?='incomplete'? prototype=[InstanceDefinition|ID]


The target element for the "prototype" reference has a normal "name" attribute and simple name resolution is granted by the scope provider.

That scope provider is a tad complex, in this particular context it has to travel a few references within the model (same resource).
Re: Debugging non-deterministic serialization failure [message #1746543 is a reply to message #1746542] Mon, 31 October 2016 19:52 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
id debug org.eclipse.xtext.serializer.tokens.CrossReferenceSerializer.getCrossReferenceNameFromScope(EObject, CrossReference, EObject, IScope, Acceptor) and look for valueConverter exceptions
Re: Debugging non-deterministic serialization failure [message #1746544 is a reply to message #1746543] Mon, 31 October 2016 19:54 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
p.s.

for testing purposes i used this grammar


Root:
	prototype=Prototype
	model=Model
;

Prototype:
	name=FQN
;
Model:
"a" 

(
	(
		annotations+=DslAnnotation* incomplete?='incomplete'? 
		| incomplete?='incomplete'? usingDeprecatedAtSyntax?='@'
	) 
	name=ID 
	prototype=[Prototype|FQN]?
	entity=FQN? 
	attributeDefinitions+=AttributeDefinition* 

	| 

	(
		annotations+=DslAnnotation* 
		inherit='inherited' 
		incomplete?='incomplete'? 
		prototype=[Prototype|ID]

		| 

		inherit='specialized' 
		prototype=[Prototype|ID]
		(
		attributeDefinitions+=AttributeDefinitionAbbrev 
		| attributeDefinitions+=AttributeDefinitionPlain
		)
	) 

	| 

	annotations+=DslAnnotation* 
	inherit='specialized' 
	incomplete?='incomplete'? 
	prototype=[Prototype|ID]
	entity=FQN? "#"
	attributeDefinitions+=AttributeDefinition*
) 


"b";

DslAnnotation:
	"@" name=ID "("")"
;

FQN: ID ("."ID)*;

AttributeDefinition:
	"attr" name=ID
;

AttributeDefinitionAbbrev:
	"a" name=ID
;

AttributeDefinitionPlain:
	"ax" name=ID
;


and test


class Main {
	
	def static void main(String[] args) {
		val Injector i = new MyDslStandaloneSetup().createInjectorAndDoEMFRegistration
		val ISerializer s = i.getInstance(ISerializer)
		val p = MyDslFactory.eINSTANCE.createPrototype => [
			name = "a.b.c"
		]
		val m = MyDslFactory.eINSTANCE.createRoot => [
				prototype = p
			model = MyDslFactory.eINSTANCE.createModel => [
				inherit = "inherited"
				prototype = p
				incomplete = false
			]
			
			]
		val rs = i.getInstance(ResourceSet)
		val r = rs.createResource(URI.createURI("dummy.mydsl1"))
		r.contents += m
		println(s.serialize(
			m
		))
	}
	
	
}
Re: Debugging non-deterministic serialization failure [message #1746545 is a reply to message #1746544] Mon, 31 October 2016 20:07 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
No ValueConverterException from CrossReferenceSerializer
(only a bunch from AssignementQuantityAllocator.allowTransient(..) -> .. IDValueConverter.assertValidValue(), but these seem to be part of normal control flow)
Re: Debugging non-deterministic serialization failure [message #1746546 is a reply to message #1746545] Mon, 31 October 2016 20:10 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
hmmmmm.

can you, for test, eliminate the other branches.
this would give you a better error message
Re: Debugging non-deterministic serialization failure [message #1746547 is a reply to message #1746546] Mon, 31 October 2016 20:16 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
> can you, for test, eliminate the other branches.

will do.

Meanwhile I put a print point into the final loop in BacktrackingSemanticSequencer.createSequence(EObject, EObject) to see all 'ti'. In the bad case the output is *much* shorter, much fewer elements in the 'trace'...
Re: Debugging non-deterministic serialization failure [message #1746548 is a reply to message #1746547] Mon, 31 October 2016 20:23 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
removed 3 out of 5 alternatives and couldn't reproduce the serialization bug in 10 test runs.
Re: Debugging non-deterministic serialization failure [message #1746549 is a reply to message #1746548] Mon, 31 October 2016 20:34 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
hmmmm i am kind a out of ideas. the question is: why is the "right path" considered bad.
are you sure there are no relevant errors added to org.eclipse.xtext.serializer.sequencer.BacktrackingSemanticSequencer.SerializableObject.isValueValid(ISemState, int, Object)
Re: Debugging non-deterministic serialization failure [message #1746550 is a reply to message #1746549] Mon, 31 October 2016 20:38 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
I couldn't ever reproduce in any combination of alternatives other than the original full set of 5. Each seems to be necessary to trigger the bug.
Re: Debugging non-deterministic serialization failure [message #1746551 is a reply to message #1746550] Mon, 31 October 2016 20:41 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
btw how do you solve the ambiguity in the first one inside your grammar? is there keywords to distinguish prototype and enitity
Re: Debugging non-deterministic serialization failure [message #1746552 is a reply to message #1746550] Mon, 31 October 2016 20:45 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
Bingo: in the bad case there *is* a 'false' inside isValueValid().
It's the element *before* the one created by the prefix.
state is "inherit='specialized'" (which you can see in one of the other alternatives)

still have to figure out what the values mean hear ...
Re: Debugging non-deterministic serialization failure [message #1746553 is a reply to message #1746552] Mon, 31 October 2016 20:58 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
This looks suspecious (in AssignmentFinder):
	protected Iterable<AbstractElement> findValidValueAssignments(EObject semanticObj,
			Iterable<AbstractElement> assignedElements, Object value) {
		// keywords have precedence over everything else
		for (AbstractElement ass : assignedElements)
			if (ass instanceof Keyword && keywordSerializer.isValid(semanticObj, (Keyword) ass, value, null))
				return Collections.singletonList(ass);

		// now check the remaining assignments
...


In assignedElements there are two keywords with the same value, but one triggers invalid value above. It seems, the loop randomly picks one from the two possible (and necessary) answers.
Re: Debugging non-deterministic serialization failure [message #1746554 is a reply to message #1746553] Mon, 31 October 2016 21:04 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
What are the two keywords, compared with the constants in mydslgrammaraccess.
And where does it take the keywords from?
Re: Debugging non-deterministic serialization failure [message #1746555 is a reply to message #1746553] Mon, 31 October 2016 21:07 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
And here's the fix:
	protected Iterable<AbstractElement> findValidValueAssignments(EObject semanticObj,
			Iterable<AbstractElement> assignedElements, Object value) {
		List<AbstractElement> result = Lists.newArrayList();
		for (AbstractElement ass : assignedElements)
			if (ass instanceof Keyword && keywordSerializer.isValid(semanticObj, (Keyword) ass, value, null))
				result.add(ass);
		// keywords have precedence over everything else
		if (!result.isEmpty())
			return result;

		// now check the remaining assignments
...


Thanks Christian, for asking the right questions to guide me to the culprit!
Stephan
Re: Debugging non-deterministic serialization failure [message #1746557 is a reply to message #1746555] Mon, 31 October 2016 21:11 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 10427
Registered: July 2009
Senior Member
youre welcome. maybe you can backport the current 2.10 code

protected Set<AbstractElement> findValidValueAssignments(EObject semanticObj,
			Iterable<AbstractElement> assignedElements, Object value) {
		Set<AbstractElement> result = Sets.newLinkedHashSet();
		for (AbstractElement ass : assignedElements) {
			if (ass instanceof Keyword && keywordSerializer.isValid(semanticObj, (Keyword) ass, value, null))
				result.add(ass);
			else if (ass instanceof RuleCall) {
				RuleCall rc = (RuleCall) ass;
				if (rc.getRule() instanceof EnumRule) {
					if (enumLiteralSerializer.isValid(semanticObj, rc, value, null))
						result.add(ass);
				} else if (valueSerializer.isValid(semanticObj, rc, value, null))
					result.add(ass);
			}
		}
		return result;
	}
Re: Debugging non-deterministic serialization failure [message #1746558 is a reply to message #1746555] Mon, 31 October 2016 21:18 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
Quote:

What are the two keywords, compared with the constants in mydslgrammaraccess.
And where does it take the keywords from?

Two of them are the same value, but they can be assigned via different rules which makes two occurrences of
  inherit='specialized'

produce two distinct "Keyword"s (you actually have these two occurrences in you're extracted grammar).

good night! Smile
Stephan
Re: Debugging non-deterministic serialization failure [message #1746650 is a reply to message #1746557] Wed, 02 November 2016 13:39 Go to previous message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1464
Registered: July 2009
Senior Member
Christian Dietrich wrote on Mon, 31 October 2016 22:11
maybe you can backport the current 2.10 code ...


Thanks, that version works as well.

I was briefly worried if more locations in Xtext might work under a wrong assumption: apparently the same keyword can be represented by different Keyword instances, but the old code in findValidValueAssignments() assumed finding one Keyword per value is sufficient.

Anyway, one more indication that migrating to a newer version would have helped / will help.

Stephan
Previous Topic:Serialization: Indent missing at start of semantic modification
Next Topic:XText Get original String from AST
Goto Forum:
  


Current Time: Tue Apr 25 17:03:55 GMT 2017

Powered by FUDForum. Page generated in 0.03743 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software