Eclipse Community Forums: Eclipse Titan » Generating invalid UTF-8

Help

Home

Home » Eclipse Projects » Eclipse Titan » Generating invalid UTF-8

Show: Today's Messages :: Show Polls :: Message Navigator

Generating invalid UTF-8 [message #1797533]

Thu, 01 November 2018 10:23

Alexander Kaiser

Messages: 28
Registered: May 2017

Junior Member

Hello,

while implementing conformance test for MQTT, I discovered the need of generating invalid UTF-8 characters/strings.
This need originates from the MQTT specification. There, the conformance statements [MQTT-1.5.3-1] and [MQTT-1.5.3-2] state that invalid UTF-8 characters must be handled by the SUT by closing the network connection.

For this purpose, we have already a few defined and implemented test cases (e.g. f_TC_MQTT_BROKER_CONNECT_020 in MQTT_Testcase_Functions.ttcn). However, these TCs started to fail with a Dynamic test case error after updating Titan.
After an investigation on this issue, I figured out that the error is caused by the @update feature. The attached UTF8.ttcn file illustrates the error with a minimal example.

However, with regard to Titan's release notes and TTCN-3 Core Language specification the @update feature seems to be unofficial anyway.
Unfortunately, I don't see any "standard compliant" solution to this issue. Anyone out there with a proposal or handy tips?

Best regards,
Alexander

Attachment: UTF8.ttcn
(Size: 2.44KB, Downloaded 32 times)

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797546 is a reply to message #1797533]

Thu, 01 November 2018 16:00

Alexander Kaiser

Messages: 28
Registered: May 2017

Junior Member

Hello again,

after diving into the TTCN-3 Core Language specification I stumbled across oct2unichar which converts octetstrings to universal charstrings in UTF-8 (I'm still astonished why I didn't use this one from the early beginning).

I replaced the @update instructions with oct2unichar and Titan seems now to digest the encoding of the strings without any complaint. The following snippet from the log demonstrates a user_name string with an appended oct2unichar('EDA080'O) (which sould represent U+D800)

user_name := { stringLength := 36, stringItem := "4bb415c8-2c5b-48af-9a0e-46c34f7672e4" & char(0, 0, 216, 0) }

However, I'm still struggling with the verification whether the produced string is really invalid according to the MQTT specification. At least Mosquitto and most of the other Brokers I checked so far seem to accept those strings.

BR,
Alexander

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797695 is a reply to message #1797546]

Mon, 05 November 2018 11:19

Botond Baranyi

Messages: 34
Registered: February 2016

Member

Hi Alexander,

I've not managed to reproduce the @update issue with my own resources.
Could you upload the module 'MQTT_v3_1_1_Types' so I can try your example out?

Best regards,
Botond Baranyi

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797698 is a reply to message #1797695]

Mon, 05 November 2018 11:37

Elemer Lelik

Messages: 1063
Registered: January 2015

Senior Member

Hi Alex,

nothwithstanding this concrete example, let me tell you that the idea of @erronous, @update has been accepted by ETSI and it will be part of the standard
probably from the next release; the syntax might be slightly changed though.

So whatever you do it will be future proof to an extent.

Best regards

Elemer

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797718 is a reply to message #1797533]

Mon, 05 November 2018 15:31

Alexander Kaiser

Messages: 28
Registered: May 2017

Junior Member

Hi Botond and Elemer,

thank you for your responses.

@Botond
I attached the 'MQTT_v3_1_1_Types' file I used. However, this should be identical to the latest one from Titan's Git repository.

I'm using the following version of ttcn3_compiler

ttcn3_compiler -v
TTCN-3 and ASN.1 Compiler for the TTCN-3 Test Executor
Product number: CRL 113 200/6 R4A
Build date: Jun 12 2018 08:36:32
Compiled with: GCC 7.3.0
Using OpenSSL 1.1.0g  2 Nov 2017

Copyright (c) 2000-2018 Ericsson Telecom AB

@Elemer
These are great news :)
While I was able to send some invalid (according to the MQTT specification) UTF-8 characters like U+D800 and U+DFFF, completely invalid UTF-8 characters (e.g. 0xC0AE as stated in RFC3629 Section 10) are not possible with oct2unichar. For now our Test Cases (TC_MQTT_BROKER_CONNECT_[020..025]) should work again as intentended.
But once we figured out the issue with @update I will try to define some additional test purposes with some "evil UTF-8 characters" specifically for security considerations.

Best regards,
Alexander

Attachment: MQTT_v3_1_1_Types.ttcn
(Size: 11.08KB, Downloaded 19 times)

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797808 is a reply to message #1797718]

Tue, 06 November 2018 16:29

Botond Baranyi

Messages: 34
Registered: February 2016

Member

Hi Alexander,

I've looked into the mentioned dynamic test case error. It's caused by a limitation in the RAW encoder, and has nothing to do with the '@update' feature. RAW coding instructions (such as 'LENGTHTO' or 'TAG') don't work if the affected fields have been altered by 'erroneous' attributes.

In the given example field msg.connect_msg.payload.client_identifier.stringItem is set to a different value through the '@update' command, but its value would be needed to calculate field stringLength in the same record (because of the 'LENGTHTO' attribute). This causes a conflict in the encoder, and an error is displayed.

If you set this type of encoding error to warning (or disable it), then the value will be encoded, but field stringLength will be omitted.

This can be done by adding the 'errorbehavior' extension to the RAW encoder function declaration.

For example:

external function f_MQTT_enc_negtest(in MQTT_v3_1_1_Message msg) return octetstring
  with { extension "prototype(convert) encode(RAW) errorbehavior(NEGTEST_CONFL:WARNING)" };

If you need the length field to be encoded too, then you should add an 'erroneous' attribute for it. too.

For example:

@update(t_connect)
with {
  erroneous (msg.connect_msg.payload.client_identifier.stringItem) "value(raw) := c_oCID";
  erroneous (msg.connect_msg.payload.client_identifier.stringLength) "value := 3";
}

Best regards,
Botond Baranyi

Report message to a moderator

Re: Generating invalid UTF-8 [message #1797842 is a reply to message #1797808]

Wed, 07 November 2018 10:42

Alexander Kaiser

Messages: 28
Registered: May 2017

Junior Member

Hi Botond,

thank you for your investigation and explanation.

Disabling the error solved the issue. As I was setting the stringLength also via the '@update' feature as described in your example, ignoring this error should be an appropriate solution.
Now the encoded octetstrings look good, thanks again.

Best regards,
Alex

Report message to a moderator

Previous Topic:	L2 and L3 Network support TTCN
Next Topic:	DTE: Decoding of integer failed when component stops

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Jul 10 20:19:03 GMT 2020

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter