Skip to main content


Eclipse Community Forums

      Home
Home » Eclipse Projects » Eclipse Titan » XML Decoder order problem
XML Decoder order problem [message #1764719] Thu, 01 June 2017 12:02 Go to next message
Eclipse UserFriend
Hello everyone,

Today I was testing the XML codec for oneM2M and noted that it enforces strict order even if there is no specific order in the record declaration (or just something is simply wrong). Let me explain:

In the reference guide of Titan, Chapter 4.25.2 XML Encoder and Decoder, Attributes, there is a section for the attribute "useOrder" that explains that :

Quote:
The encoding instruction designates that the encoder shall encode the values of the fields corresponding to the children elements of the all constructor according to the order identified by the elements of the order field. At decoding, the received values of the XML elements shall be placed in their corresponding record fields and a new record of element shall be inserted into the order field for each XML element processed (the final order of the record of elements shall reflect the order of the XML elements in the encoded XML document).


This is also confirmed by the TTCN-3 spec (ETSI ES 201 873-9, Annex B).

But there is no "record of (...) order" and "useOrder" inside the oneM2M Types module, and yet Titan doesn't decode certain elements (leaving them with "omit"). Let me show an example:

HTTP/1.1 201 Created\r
Content-Location: /in-cse/CAE178153928\r
X-M2M-RI: TC_CSE_DMR_CRE_BV_002_01-m_createAe661389\r
X-M2M-Origin: /in-cse\r
X-M2M-RSC: 2001\r
Content-Type: application/xml;charset=ISO-8859-1\r
Content-Length: 372\r
Server: Jetty(8.1.16.v20140903)\r
\r
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<m2m:ae xmlns:m2m=\"http://www.onem2m.org/xml/protocols\" rn=\"MyAe\">
   <ty>2</ty>
   <ri>/in-cse/CAE178153928</ri>
   <pi>/in-cse</pi>
   <ct>20170601T160956</ct>
   <lt>20170601T160956</lt>
   <acpi>/in-cse/acp-347340622</acpi>
   <et>20180601T160956</et>
   <api>myAppId</api>
   <aei>CAE178153928</aei>
   <rr>true</rr>
</m2m:ae>


will be converted to:

OneM2M_Types.MsgIn : {
    primitive := {
        responsePrimitive := {
            responseStatusCode := int2001 (2001),
            requestIdentifier := "TC_CSE_DMR_CRE_BV_002_01-m_createAe661389",
            primitiveContent := {
                any_1 := {
                    {
                        AE_optional := {
                            resourceName := "MyAe",
                            resourceType := int2 (2),
                            resourceID := "/in-cse/CAE178153928",
                            parentID := "/in-cse",
                            creationTime := "20170601T160956",
                            lastModifiedTime := "20170601T160956",
                            labels := omit,
                            accessControlPolicyIDs := {
                                "/in-cse/acp-347340622"
                            },
                            expirationTime := "20180601T160956",
                            dynamicAuthorizationConsultationIDs := omit,
                            announceTo := omit,
                            announcedAttribute := omit,
                            appName := omit,
                            app_ID := "myAppId",
                            aE_ID := "CAE178153928",
                            pointOfAccess := omit,
                            ontologyRef := omit,
                            nodeLink := omit,
                            requestReachability := true,
                            contentSerialization := omit,
                            e2eSecInfo := omit,
                            choice := omit
                        }
                    }
                }
            },
            to_ := omit,
            from_ := "/in-cse",
            originatingTimestamp := omit,
            resultExpirationTimestamp := omit,
            eventCategory := omit,
            contentStatus := omit,
            contentOffset := omit,
            assignedTokenIdentifiers := omit,
            tokenRequestInformation := omit
        }
    }
}


but this one:

HTTP/1.1 201 Created\r
X-Powered-By: Express\r
Content-Location: /mobius-yt/MyAe\r
X-M2M-RI: TC_CSE_DMR_CRE_BV_002_01-m_createAe838176\r
X-M2M-RSC: 2001\r
Content-Type: application/xml\r
Date: Thu, 01 Jun 2017 14:15:06 GMT\r
Connection: keep-alive\r
Content-Length: 373\r
\r
<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>
<m2m:ae xmlns:m2m=\"http://www.onem2m.org/xml/protocols\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" rn=\"MyAe\">
   <ty>2</ty>
   <pi>rJ1BZsp--</pi>
   <ri>r1lQOZop-b</ri>
   <ct>20170601T141506</ct>
   <et>20190601T141506</et>
   <lt>20170601T141506</lt>
   <api>re.kr.keti.testSystem.Id</api>
   <rr>true</rr>
   <aei>Ctsae001-ae</aei>
</m2m:ae>


is decoded to this:

OneM2M_Types.MsgIn : {
    primitive := {
        responsePrimitive := {
            responseStatusCode := int2001 (2001),
            requestIdentifier := "TC_CSE_DMR_CRE_BV_002_01-m_createAe838176",
            primitiveContent := {
                any_1 := {
                    {
                        AE_optional := {
                            resourceName := "MyAe",
                            resourceType := int2 (2),
                            resourceID := omit,
                            parentID := "rJ1BZsp--",
                            creationTime := omit,
                            lastModifiedTime := omit,
                            labels := omit,
                            accessControlPolicyIDs := omit,
                            expirationTime := omit,
                            dynamicAuthorizationConsultationIDs := omit,
                            announceTo := omit,
                            announcedAttribute := omit,
                            appName := omit,
                            app_ID := omit,
                            aE_ID := omit,
                            pointOfAccess := omit,
                            ontologyRef := omit,
                            nodeLink := omit,
                            requestReachability := omit,
                            contentSerialization := omit,
                            e2eSecInfo := omit,
                            choice := omit
                        }
                    }
                }
            },
            to_ := omit,
            from_ := omit,
            originatingTimestamp := omit,
            resultExpirationTimestamp := omit,
            eventCategory := omit,
            contentStatus := omit,
            contentOffset := omit,
            assignedTokenIdentifiers := omit,
            tokenRequestInformation := omit
        }
    }
}


with the following warnings:
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `resourceID'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `creationTime'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `expirationTime'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `lastModifiedTime'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `App-ID'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `requestReachability'
Warning: While XER-decoding type '@OneM2M_Types.anytype': Alternative 'AE_optional': Unprocessed XML tag `AE-ID'


Is this a correct behaviour? And is there a possibility (a flag maybe) that can be set in order to not check the order of the elements? Because I don't see any other reason that will convert the first one, and not the second message.

Thanks in advance.

Best regards,
Naum
Re: XML Decoder order problem [message #1764724 is a reply to message #1764719] Thu, 01 June 2017 12:24 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,

can you pls attach the type declaration (plus encoding instructions of course)?

BR

Elemer
Re: XML Decoder order problem [message #1764763 is a reply to message #1764724] Fri, 02 June 2017 03:22 Go to previous messageGo to next message
Eclipse UserFriend
Hello Elemer,

The type in question is this (line 6544 in the OneM2M_Types module): 
type record AE_optional {
		XSD.NCName resourceName optional,
		ResourceType resourceType optional,
		XSD.ID resourceID optional,
		NhURI parentID optional,
		Timestamp creationTime optional,
		Timestamp lastModifiedTime optional,
		Labels labels optional,
		AcpType accessControlPolicyIDs optional,
		Timestamp expirationTime optional,
		ListOfURIs dynamicAuthorizationConsultationIDs optional,
		ListOfURIs announceTo optional,
		record length(1 .. infinity) of XSD.NCName announcedAttribute optional,
		XSD.String appName optional,
		XSD.String app_ID optional,
		XSD.ID aE_ID optional,
		PoaList pointOfAccess optional,
		XSD.AnyURI ontologyRef optional,
		XSD.AnyURI nodeLink optional,
		XSD.Boolean requestReachability optional,
		Serializations contentSerialization optional,
		E2eSecInfo e2eSecInfo optional,
		union {
			record length(1 .. infinity) of ChildResourceRef childResource_list,
			record length(1 .. infinity) of union {
				Container container,
				Group group_,
				AccessControlPolicy accessControlPolicy,
				Subscription subscription,
				PollingChannel pollingChannel,
				Schedule schedule,
				SemanticDescriptor semanticDescriptor,
				TimeSeries timeSeries,
				TrafficPattern trafficPattern,
				Sg_flexContainerResource_group sg_flexContainerResource
			} choice_list
		} choice optional
	}
	with {
	  variant "element";
	  variant (resourceName) "attribute";
	  variant (announcedAttribute) "list";
	  variant (app_ID) "name as 'App-ID'";
	  variant (aE_ID) "name as 'AE-ID'";
	  //variant (requestReachability) "text 'true' as '1'";
	  //variant (requestReachability) "text 'false' as '0'";
	  variant (choice) "untagged";
	  variant (choice.childResource_list) "untagged";
	  variant (choice.childResource_list[-]) "name as 'childResource'";
	  variant (choice.choice_list) "untagged";
	  variant (choice.choice_list[-]) "untagged";
	  variant (choice.choice_list[-].group_) "name as 'group'";
	  
	};


I'm not interested in the outer data (request/responsePrimitive) because that is done manually (the HTTP part), only this. The decoder is called on the body element of the HTTP message (which, in this case is AE_optional). If you want, I can provide you the whole messages in octetstring, so you can verify on your side.

Best regards,
Naum

[Updated on: Fri, 02 June 2017 03:24] by Moderator

Re: XML Decoder order problem [message #1764799 is a reply to message #1764763] Fri, 02 June 2017 06:13 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,

AE_optional is declared as a record which assumes strict order of elements;


the first sample 
<m2m:ae xmlns:m2m=\"http://www.onem2m.org/xml/protocols\" rn=\"MyAe\">
   <ty>2</ty>
   <ri>/in-cse/CAE178153928</ri>
   <pi>/in-cse</pi>
   <ct>20170601T160956</ct>
   <lt>20170601T160956</lt>
   <acpi>/in-cse/acp-347340622</acpi>
   <et>20180601T160956</et>
   <api>myAppId</api>
   <aei>CAE178153928</aei>
   <rr>true</rr>
</m2m:ae>


respects this order, but the second 

<m2m:ae xmlns:m2m=\"http://www.onem2m.org/xml/protocols\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" rn=\"MyAe\">
   <ty>2</ty>
   <pi>rJ1BZsp--</pi>
   <ri>r1lQOZop-b</ri>
   <ct>20170601T141506</ct>
   <et>20190601T141506</et>
   <lt>20170601T141506</lt>
   <api>re.kr.keti.testSystem.Id</api>
   <rr>true</rr>
   <aei>Ctsae001-ae</aei>
</m2m:ae>


is in violation of it; e.g. pi comes before ri, et comes before lt etc. so this structure is nor decodable by the rules expressed .

If you reorder the second structure as per the type it should become decodable.


Best regards
Elemer
Re: XML Decoder order problem [message #1764801 is a reply to message #1764799] Fri, 02 June 2017 06:18 Go to previous messageGo to next message
Eclipse UserFriend
I think I see where the confusion comes from: possibly you expect some XML encoding rule to enforce this order, but
this is not needed: the XML structural information is mapped to TTCN-3 structural information, and whenever needed, to encoding rules to complete that.
So the main rule is expressed in the TTCN-3 code, and the encoding rules only add to that if needed.
Re: XML Decoder order problem [message #1764854 is a reply to message #1764801] Fri, 02 June 2017 11:29 Go to previous messageGo to next message
Eclipse UserFriend
Then there are two possible solutions to this problem:

1. "Fail" the test because the information isn't decoded properly (but the information that the problem is actually in the order will be difficult to set in the verdict or in an actual error - because the order is verified implicitly)
2. Restructure the received XML as needed (if the order is not correct). This implies that the verdict isn't 100% correct, because there might be some other errors hidden behind this.

Thank you for the help.

Best regards,
Naum
Re: XML Decoder order problem [message #1764857 is a reply to message #1764854] Fri, 02 June 2017 11:42 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,

a question : can you copy your XML decoding function declaration here so I can take a look?


If the received XML looks like your second sample that is definitely a sign of a fault in SUT that should be fixed in SUT. I don't believe we should adapt to a faulty behavior by restructuring the XML.
To put things in perspective:
the TTCN-3 code is generated based on the XML schema; you can validate the XML directly against the schema (with xmlllint for instance ) and this will tell you the same thing: that the XML does not conform to the governing schema document. This has nothing to do with TTCN-3 or Titan btw.

Titan can detect the fault on two levels: in the decoder by throwing errors/warnings or when matching the incoming variable against the template.
This is more or less the decision of the user.


BR

Elemer
Re: XML Decoder order problem [message #1764964 is a reply to message #1764857] Mon, 05 June 2017 04:05 Go to previous messageGo to next message
Eclipse UserFriend
Hello Elemer,

I'm using the decvalue() function to decode the XML, like this:
decvalue(bitBodyString, pl_out.primitive.requestPrimitive.primitiveContent.any_1[0]);


I agree that the problem is on the SUT side. Because in this case we can't modify the TTCN-3 code (at least not without the approval of ETSI STF 531/oneM2M TF 001), the only thing that we can do, as you said, is to throw a warning/error in the decoder, but as I can see, the message shown in the console is that some elements are missing (not the correct message in this case). Is there a possibility to alter the codec from C++?

Best regards,
Naum
Re: XML Decoder order problem [message #1764966 is a reply to message #1764964] Mon, 05 June 2017 04:27 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,

let me clarify: throwing a warning/error in codec is technically a possibility, but in my opinion the correct (at least from TTCN-3 perspective) way of handling such a fault is to let verdict be set to false when matching.

When using external function declarations, the default error handling is set to ERROR, ad that can be altered to WARNING by setting the errorbehavior flag .
I will have to check with the situation with encvalue/decvalue ; as here there's no room to alter error handling, possibly it's left at WARNING.
We can change that to ERROR and let codec declare that it cannot decode the structure along the Schema, but throwing an error would probably be too disruptive as it does not permit the test case to reach a verdict.
Warning messages are a guess at best and a sign that the codec is struggling to decode the structure; such a fault will require some analysis anyhow, so don'be thrown off of them.

I'm not sure exactly what codec alteration you have in mind; In my opinion there's not much the codec could do beyond the above.

In summary, my suggestion is that you just let the test case fail and write a trouble report against the SUT.
Don't forget that the scope of the exercise is not to have all test cases pass and look good on paper but to detect faults.

Best regards
Elemer
Re: XML Decoder order problem [message #1764991 is a reply to message #1764966] Mon, 05 June 2017 11:15 Go to previous messageGo to next message
Eclipse UserFriend
Yes, I understand that the test case serves to detect faults and incompatibilities with the specification, but in this case there isn't much space for manoeuvring (changing the verdict and the reason); I'll discuss in the STF about this problem and hope that we will find a proper solution.

Otherwise, I detected (I think) an error in the XML codec: the variant "name as uncapitalized" doesn't work. The oneM2M_Types.ttcn file is always the same.
The type in question is "ServiceSubscribedAppRule_optional" with the following structure:

type record ServiceSubscribedAppRule_optional
	{
		XSD.NCName resourceName optional,
		ResourceType resourceType optional,
		XSD.ID resourceID optional,
		NhURI parentID optional,
		Timestamp creationTime optional,
		Timestamp lastModifiedTime optional,
		Labels labels optional,
		AcpType accessControlPolicyIDs optional,
		Timestamp expirationTime optional,
		ListOfURIs dynamicAuthorizationConsultationIDs optional,
		ListOfM2MID applicableCredIDs optional,
		ListOfM2MID allowedApp_IDs optional,
		ListOfM2MID allowedAEs optional,
		union {
			record length(1 .. infinity) of ChildResourceRef childResource_list,
			record length(1 .. infinity) of Subscription subscription_list
		} choice optional
	}
	with {
	  variant "name as uncapitalized";
	  variant "element";
	  variant (resourceName) "attribute";
	  variant (allowedApp_IDs) "name as 'allowedApp-IDs'";
	  variant (choice) "untagged";
	  variant (choice.childResource_list) "untagged";
	  variant (choice.childResource_list[-]) "name as 'childResource'";
	  variant (choice.subscription_list) "untagged";
	  variant (choice.subscription_list[-]) "name as 'subscription'";
	};


Here is an example of TTCN-3 data:
ServiceSubscribedAppRule_optional := {
                            resourceName := omit,
                            resourceType := omit,
                            resourceID := omit,
                            parentID := omit,
                            creationTime := omit,
                            lastModifiedTime := omit,
                            labels := omit,
                            accessControlPolicyIDs := omit,
                            expirationTime := omit,
                            dynamicAuthorizationConsultationIDs := omit,
                            applicableCredIDs := {
                                "None"
                            },
                            allowedApp_IDs := {
                                "myAppId"
                            },
                            allowedAEs := {
                                "C-AE-ID-STEM"
                            },
                            choice := omit
                        }
                    }


The codec doesn't take into account the first variant and translates it into:
<m2m:ServiceSubscribedAppRule_optional xmlns:m2m="http://www.onem2m.org/xml/protocols">
    <applicableCredIDs>None</applicableCredIDs>
    <allowedApp-IDs>myAppId</allowedApp-IDs>
    <allowedAEs>C-AE-ID-STEM</allowedAEs>
</m2m:ServiceSubscribedAppRule_optional>


instead of "serviceSubscribedAppRule_optional".

Should I report this formally as a bug?

Best regards,
Naum
Re: XML Decoder order problem [message #1764995 is a reply to message #1764991] Mon, 05 June 2017 11:49 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,

I will need the complete type declaration, including encoding instructions, to look into this.
If you have a reduced example that reproduces the fault pls. upload it here.

Best regards

Elemer
Re: XML Decoder order problem [message #1765106 is a reply to message #1764995] Tue, 06 June 2017 12:05 Go to previous messageGo to next message
Eclipse UserFriend
Hello Elemer,

Here is an example. I took the same type and data as above (ServiceSubscribedAppRule_optional). Normally, the XML should be "serviceSubscribedAppRule_optional", but it is "ServiceSubscribedAppRule_optional".

The test case file is test_codec.tc_codecTest().

And the logs are here:
MTC@egm-VirtualBox: Test case tc_codecTest started.
MTC@egm-VirtualBox: Initializing variables, timers and ports of component type OneM2M_TestSystem.CseTester inside testcase tc_codecTest.
MTC@egm-VirtualBox: Port acPort was started.
MTC@egm-VirtualBox: Port mcaPort was started.
MTC@egm-VirtualBox: Port mccPort was started.
MTC@egm-VirtualBox: Component type OneM2M_TestSystem.CseTester was initialized.
MTC@egm-VirtualBox: Function rnd() returned 0.685228.
MTC@egm-VirtualBox: { operation := int1 (1), to_ := "NotInitialized", from_ := "NotInitialized", requestIdentifier := "tc_codecTest-m_createServiceSubscribedAppRule685229", resourceType := int19 (19), primitiveContent := { any_1 := { { ServiceSubscribedAppRule_optional := { resourceName := omit, resourceType := omit, resourceID := omit, parentID := omit, creationTime := omit, lastModifiedTime := omit, labels := omit, accessControlPolicyIDs := omit, expirationTime := omit, dynamicAuthorizationConsultationIDs := omit, applicableCredIDs := { "None" }, allowedApp_IDs := { "myAppId" }, allowedAEs := { "C-AE-ID-STEM" }, choice := omit } } } }, roleIDs := omit, originatingTimestamp := omit, requestExpirationTimestamp := omit, resultExpirationTimestamp := omit, operationExecutionTime := omit, responseType := omit, resultPersistence := omit, resultContent := omit, eventCategory := omit, deliveryAggregation := omit, groupRequestIdentifier := omit, filterCriteria := omit, discoveryResultType := omit, tokens := omit, tokenIDs := omit, localTokenIDs := omit, tokenRequestIndicator := omit }
MTC@egm-VirtualBox: '3C6D326D3A536572766963655375627363726962656441707052756C655F6F7074696F6E616C20786D6C6E733A6D326D3D27687474703A2F2F7777772E6F6E656D326D2E6F72672F786D6C2F70726F746F636F6C73273E0A093C6170706C696361626C65437265644944733E4E6F6E653C2F6170706C696361626C65437265644944733E0A093C616C6C6F7765644170702D4944733E6D7941707049643C2F616C6C6F7765644170702D4944733E0A093C616C6C6F7765644145733E432D41452D49442D5354454D3C2F616C6C6F7765644145733E0A3C2F6D326D3A536572766963655375627363726962656441707052756C655F6F7074696F6E616C3E0A0A'O ("<m2m:ServiceSubscribedAppRule_optional xmlns:m2m='http://www.onem2m.org/xml/protocols'>\n\t<applicableCredIDs>None</applicableCredIDs>\n\t<allowedApp-IDs>myAppId</allowedApp-IDs>\n\t<allowedAEs>C-AE-ID-STEM</allowedAEs>\n</m2m:ServiceSubscribedAppRule_optional>\n\n")
MTC@egm-VirtualBox: Terminating component type OneM2M_TestSystem.CseTester.
MTC@egm-VirtualBox: Port acPort was stopped.
MTC@egm-VirtualBox: Port mcaPort was stopped.
MTC@egm-VirtualBox: Port mccPort was stopped.
MTC@egm-VirtualBox: Component type OneM2M_TestSystem.CseTester was shut down inside testcase tc_codecTest.
MTC@egm-VirtualBox: Waiting for PTCs to finish.
MTC@egm-VirtualBox: Setting final verdict of the test case.
MTC@egm-VirtualBox: Local verdict of MTC: none
MTC@egm-VirtualBox: No PTCs were created.
MTC@egm-VirtualBox: Test case tc_codecTest finished. Verdict: none


Best regards,
Naum
Re: XML Decoder order problem [message #1765342 is a reply to message #1765106] Thu, 08 June 2017 10:52 Go to previous messageGo to next message
Eclipse UserFriend
Hi Naum,


this is the bug tracker created:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=518005

We are workng on it.

BR

Elemer
Re: XML Decoder order problem [message #1765526 is a reply to message #1765342] Mon, 12 June 2017 01:40 Go to previous message
Eclipse UserFriend
Hi Naum,

this issue has been solved; it is related to use of "anytype";

please test it at your convenience.

Best regards
Elemer
Previous Topic:Using Titan as a fuzzing engine part 2
Next Topic:ttcn-3 xmpp example
Goto Forum:
  


Current Time: Wed Jul 23 21:12:46 EDT 2025

Powered by FUDForum. Page generated in 0.25847 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top

Sign up to our Newsletter

A fresh new issue delivered monthly