| Re: [iottestware-dev] Open CoAP test suite for negative testing |
|
Hello Sascha, Thank you for the fast response ! That’s great. I’m also preparing the tests for the IoT Week in ETSI in two weeks using our online TAAS platform to run them. The tests use specific URIs to test the actual result (for example “Simple resource”, Any resource”, etc). I know that this is limitation of the implementations but in order to assure
that we have real conformance tests we need to make some abstraction layers. For example, we can specify some pixits (module parameters) for different URIs (resources that exist/not exist, read-only resources, etc). Another way to do this is to make a preamble
that will create the resource (if the implementation supports it) and then send the actual test message. The postamble in the end of the testcase serves to revert the SUT in the initial condition (for example a test that will create a resource inside the SUT,
so the postamble will delete it to assure that the next testcase will be executed in the same conditions). I wanted to make some fast tests to see if Californium allows to create new resources (with custom URIs), but my virtual machine is crashing right now, so I can’t provide you with a
concrete answer how we can do that. I will fix it ASAP and come back with the findings. I also have some questions regarding the AddressList pixit (the multiple endpoints/SUTs, etc), but we can discuss that in a separate thread or maybe at the meeting in Sophia, because
it isn’t urgent. Best regards, Naum De : iottestware-dev-bounces@xxxxxxxxxxx [mailto:iottestware-dev-bounces@xxxxxxxxxxx]
De la part de Kretzschmann, Sascha thank you very much for your interest for the IoT-Testware. We really appreciate your feedback. As you may know, we are doing some basic improvements at the moment, including easily installing the project locally for interested parties and be more generic with our test suite. The latter is indeed the part, that causes the error. Just to let you know and to make it clear. When we started with the test suites, we were looking for a SUT (especially for CoAP). So we decided to implement our own local CoAP server (quick-and-dirty).
This is the SUT included when you clone the CoAP-Testware. “Also, the tests mustn't rely on specific resources / specific implementations.” Could you tell me more about this point? As you know, the challenge with CoAP is (in contrast to MQTT) that you need to know about the resources the CoAP-Server provides (including the methods they implement). In other words, if you change
the SUT, you need to change the resources. Otherwise the majority of the test cases will be failing. This is my point of view. Unfortunately, we still have a manual process of changing the resources. I’m working on that with the help of module parameter. Preamble/Postamble: Yes indeed. If you have concrete ideas what needs to be done beforehand and afterwards, feel free to let us know or push it directly. Thank you and have a nice day. Sascha -- Sascha Kretzschmann (M.Sc.)
Fraunhofer Gesellschaft e.V. Tel.: +49 (0)30 3463 7255 Von:
iottestware-dev-bounces@xxxxxxxxxxx [mailto:iottestware-dev-bounces@xxxxxxxxxxx]
Im Auftrag von Naum SPASESKI Hello Sascha, everyone, I was analysing the CoAP testcases, along with the TP document that Axel shared with me last May/June. After I run them on OM2M (an Eclipse implementation of oneM2M) and found the following : TC_COAP_SERVER_001 PASS TC_COAP_SERVER_GET_001 FAIL: resource doesn't exist, status 4.04 instead of 2.05.
CoAP specs 5.9.1.5.: "This Response Code is like HTTP 200 "OK" but only used in response to GET requests. CoAP specs 5.9.2.5.: "This Response Code is like HTTP 404 "Not Found"." As the resource doesn't exist, I think that the correct reponse is 4.04, not 2.05. Or we have to provide a preamble that will create the resource. TC_COAP_SERVER_GET_002 FAIL: Malformed URI, status 4.00 instead of 2.05 CoAP specs 5.9.1.5.: "This Response Code is like HTTP 200 "OK" but only used in response to GET requests. CoAP specs 5.9.2.1.: "This Response Code is Like HTTP 400 "Bad Request"." Because oneM2M uses specific url structures, the test fails. But as in the previous, I think that it should return at least 4.04, because there are no resources behind. Or we have to provide
a preamble that will create the resource. TC_COAP_SERVER_GET_003 FAIL: resource doesn't exists, status 4.04 instead of 4.05. I can't see the logic behind this test. Is there an option to disable the GET operation for certain resources? I checked this with Californium. In the TP the uri is 'Simple_resource' but in the testcase it is 'Calculator_Resource', because if it is the former uri, the testcase will be duplicate of
001. Californium, stangely, with Calculator_Resource responds with 4.05, and with Simple_resource responds with 4.04. TC_COAP_SERVER_GET_005 FAIL: Malformed URI, status 4.00 instead of 4.04 Because of oneM2M (which supports only certain types of addresses) the test fails. But, again as the test 001, the actual uri is Any_resource, instead of Simple_resource. If I change to Simple_resource, it will be a duplicate of 001 and it will return 2.05. TC_COAP_SERVER_POST_001 FAIL: status 4.04 instead of 2.01. The client isn't registered to the oneM2M server, as per oneM2M specs.
CoAP specs 5.8.2: "The actual function performed by the POST method is determined by the origin server and dependent on the target resource. It usually results in a new resource being created
or the target resource being updated." I think that we should think of something more precise than this test, because there can be situations where Payload "New1/New2" will not result with new Location-Paths "New1" and "New2". TC_COAP_SERVER_POST_002 FAIL: status 4.04 instead of 2.04. The client isn't registered to the oneM2M server, as per oneM2M specs.
In the TTCN-3, this test is a duplicate of 001 (and in the TP it is described with 3 uri paths and payload). But the test passes because the resource is created in the previous test. TC_COAP_SERVER_POST_003 FAIL: status 4.04 instead of 2.04. The client isn't registered to the oneM2M server, as per oneM2M specs.
The TTCN-3 doesn't correspond to the TP, there is misalignment in Location-Path New2, which, according to TP, shall be New2a. In both cases it fails with Californium too. TC_COAP_SERVER_POST_005 FAIL: status 4.04 instead of 4.05. The client isn't registered to the oneM2M server, as per oneM2M specs. I can't see the logic behind this test, as in GET_003. TC_COAP_SERVER_PUT_001 FAIL: status 4.04 instead of 2.04. The client isn't registered to the oneM2M server, as per oneM2M specs. TC_COAP_SERVER_PUT_002 FAIL: status 4.04 instead of 2.04. The client isn't registered to the oneM2M server, as per oneM2M specs. The test in TTCN is a duplicate of 001. TC_COAP_SERVER_DELETE_001 FAIL: status 4.04 instead of 2.02. The client isn't registered to the oneM2M server, as per oneM2M specs. The test should contain a preamble to create the resource. TC_COAP_SERVER_DELETE_001 FAIL: status 4.04 instead of 2.02. The client isn't registered to the oneM2M server, as per oneM2M specs. The test uses 001 as a preamble to delete the resource. As a general remark, I have to say that we need preamble and postamble for every test (you can see an example in oneM2M:
https://git.onem2m.org/TST/ATS) in order for the tests to be independent from each other. Also, the tests mustn't rely on specific resources / specific implementations. I will continue with the CoAP negative tests and also with the analysis of the MQTT suite and get back with some comments. Best regards, Naum De :
iottestware-dev-bounces@xxxxxxxxxxx [mailto:iottestware-dev-bounces@xxxxxxxxxxx]
De la part de Kretzschmann, Sascha Hello project partners, we recently published [1] a restructered CoAP test suite. The reason for the changes is a more detailed codec [2] provided by Ericsson. Now we can also run more exhausting (negative) tests. If you have any feedback, please let us know. It would be appreciated. Best regards Sascha [1]
https://github.com/eclipse/iottestware.coap/commit/fcc165132291ea8ad482861547465dc093fe7b4e -- Sascha Kretzschmann (M.Sc.) Fraunhofer Gesellschaft e.V. Tel.: +49 (0)30 3463 7255 |
