Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [escet-dev] How to test 3D digital twins?
  • From: "Beek, Bert van" <D.A.v.Beek@xxxxxx>
  • Date: Wed, 7 Jul 2021 15:04:37 +0000
  • Accept-language: en-GB, en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=tue.nl; dmarc=pass action=none header.from=tue.nl; dkim=pass header.d=tue.nl; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c8LPfCh92XxBp8/TN6YdoYXJtSxuEip+AHWEFo0X0r4=; b=gUd4t9qh/h58ocTgZJIReOnuqfxt3NTC1TkMIGEl/iY0LY4d4JqQW4G3MkxdpsxDEUcNkvoM3WY0+PEhgHbKSvRb2ZOUUxAWKI5LHOiN9mQVB3euaQC7JJqI4DIOzu7TWeY+mn1m2p9tUFEHuuYx8S1ylvXLbufJDEn0rdHsxP0zUmAjyAW5PWBJWSoNZweJxAiI8CvzngX9Iz5OIqImxr3GFVOfd6V46sKl97wvLJMEMcKHErJBqKSVy1L+SBwTd9EJCkOSs6ldmkTakfRekxnQoQnpHKvxOWPifC9BRStoyDX59eOad6yZ8tAE5Jnsq+gcNlyxeaxGdwjH0lFk0g==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ATUBgoWM/QG5XnVZkvL/rzZ2JSMXS5g01udDrvUJKRR7lTHNysV+GUW0hS7Zn9WkYMGVbAXSjr6BFbIKzccA2SL2MMvXBFSTtbM3K8E6eCA4vAQYpk30KSVB+HA6A208DHSXMO1nQPZLWj6s+sO7TwNqG1FJsajAIPyMJG16cz70tgrNCU8yYLaC/w06DgerZ+V4QJ8OetxQG5+LElcnJlqlo1Cs4rLHx/a/6qGIC5xCfJREsE8U4qV/7D002JWLSu5nfdMqbMT9FlGHE6dOPOk1cshggW6/5vl+KN+GQ12j5rRAoCN3+WUO8mp1CDF5hN7uwuhFLt8dRrTXQ86jLA==
  • Delivered-to: escet-dev@xxxxxxxxxxx
  • List-archive: <https://dev.eclipse.org/mailman/private/escet-dev/>
  • List-help: <mailto:escet-dev-request@eclipse.org?subject=help>
  • List-subscribe: <https://dev.eclipse.org/mailman/listinfo/escet-dev>, <mailto:escet-dev-request@eclipse.org?subject=subscribe>
  • List-unsubscribe: <https://dev.eclipse.org/mailman/options/escet-dev>, <mailto:escet-dev-request@eclipse.org?subject=unsubscribe>
  • Thread-index: AQHXcXus0oQNYmquKkmc6Z7GXbnz86s0jW8AgAGeogCAADImgIABQZ2A
  • Thread-topic: [escet-dev] How to test 3D digital twins?

Hi Dennis,

Thanks for your answer. I am fully aware of the problem of the exploding state space and the coverage problem. 

I think I did not formulate the problem properly. I now understand your previous question “Is full automation a requirement”. That is exactly the problem. For ESCET you have defined a number of tests that run fully automated and that each have a perfectly defined required output result. For any new functionality always also new tests are defined, which are then automatically added to the total set of tests to be executed, if I understand correctly.

I am trying to see if a similar approach can be executed for the digital twins. Of course the coverage problem is similar to the test set used for ESCET. There are however, several additional problems:
  1. How do I define and measure the required outcome for a test on a digital twin?
  2. What kind of tests do I use:
    • complete controllers that have been submitted by student groups in the past?
    • or a number of well defined small tests that each test a small aspect of the digital twin?
  3. How could I automate such tests (in a way similar as the CIF test set in ESCET)?

Today, two TAs are doing a lot of manual work in the Festo lab: running controllers from previous final assignments on the actual Festo machines and running the same controllers on the digital twins and visually comparing the output. 

I was wondering if we could learn from the ESCET testing experience, but maybe the testing problem for a digital twin is so very much different from the testing problem in ESCET CIF development that it is difficult to learn from the ESCET experience?

Bert

On 6 Jul 2021, at 21:53, Dennis Hendriks <dh_tue@xxxxxxxxxxx> wrote:

Hi Bert,

Disclaimer: I'm not an expert on testing per se.

I assume you have to test the physical system manually? This is not automated? I assume you can test the digital twin fully automatically, if you provide the scenarios to execute?

One issue is how to get good coverage, I think. You could perform random simulations on the digital twin and observe the outcome (success, failure, states, or so). Then you could perform the same scenarios on the real system, and maybe observe the outcome as well. But to get good coverage, you'll need many scenarios. Some states may then be difficult to reach, as some states have higher chances of being reached than others. A classical problem is an automaton with as alphabet all 10 digits and which accepts only a specific password or phone number. The number of possible sequences is exponential in the sequence length and only one such sequence is accepted. It's highly unlikely you'll ever find it by black box testing. The same problem occurs in active automata learning, where conformance testing also plays a large role. There is quite some literature on this. This is not just theoretical, but a very practical problem as well.

But that assumes black box testing. If you know the model, you can use whitebox testing techniques. You could generate test scenarios from the model (digital twin model) and test them on the physical twin. You would need to generate both scenarios for behavior accepted by the digitial twin model, and for behavior not accepted by it. You would have to check that (non)acceptance against the (non)acceptance of the physical one. There are algorithms that can generate a 'complete' test set, under certain conditions. But still, the number of tests may be very large.

Another question is: what constitutes acceptance? It could be accepting states, if that is something you can define for your system. You could also define input and output subsets, and see for a sequence of inputs whether the sequence of outputs matches, similar to Mealy machines.

Does this help you in any way?

Dennis



Van: escet-dev <escet-dev-bounces@xxxxxxxxxxx> namens Beek, Bert van <D.A.v.Beek@xxxxxx>
Verzonden: dinsdag 6 juli 2021 18:54
Aan: escet developer discussions <escet-dev@xxxxxxxxxxx>
Onderwerp: Re: [escet-dev] How to test 3D digital twins?
 
Hi Dennis,

The goal of this exercise is to find errors in the digital twins. Last year, due to the Corona restrictions, testing was optional. There was only a single group who tested on the festo workstations. Their controller worked without errors on the digital twin, but it did not work on the Festo workstation. The reason for this was that the actuator setting to immediately stop the sliding manipulator was true/true (pressure on both sides), whereas in the digital twin, the actuator setting for this was false/false. False/false (absence of pressure) on the Festo workstation caused the sliding manipulator to slowly stop => stopping at an incorrect location.

“Behave in the same way” means that a controller that works correctly on the twin should work correctly on the Festo system, and that a controller that shows erroneous behavior on the Festo workstation should show the same erroneous behavior on the twin.

Ideally, we would like to consider the full state space, but obviously that is impossible. The controller is hybrid (timed) so has infinite state space.

In principle we only look at the observable behavior at the festo workstation and at the twin (it a product picked up properly and release at the required target location), but we could probably generate certain graphs of behavior in TwinCAT.

  • Is full automation a requirement?
I don’t understand this question.

Bert

On 5 Jul 2021, at 18:09, Dennis Hendriks <dh_tue@xxxxxxxxxxx> wrote:

Hi Bert,

Some random questions that immediately pop into my mind:
  • What is the goal of this exercise? What do you hope to gain?
  • What does 'behave in the same way' mean for you?
  • Do you consider the full state space of behavior, or only a finite set of scenarios?
  • Do you consider the protocol only (order of events), or also timing?
  • Is full automation a requirement?
Dennis


 
Van: escet-dev <escet-dev-bounces@xxxxxxxxxxx> namens Beek, Bert van <D.A.v.Beek@xxxxxx>
Verzonden: maandag 5 juli 2021 10:56
Aan: escet developer discussions <escet-dev@xxxxxxxxxxx>
Onderwerp: [escet-dev] How to test 3D digital twins?
 
How would you test the functionality of digital twins for the Festo workstations as shown in https://cstweb.wtb.tue.nl/4tc00/festo/digital_twin/index.html?

A teacher assistant aims to test all CIF controllers submitted last year as final assignment for the course 4TC00 on the digital twin and on the Festo workstation to see if they behave in the same way.

I could also imagine that we write some small controllers that test specific functionality of specific sensor actuator relations/behavior.

Any thoughts or tips?

Thanks in advance
Bert
_______________________________________________
escet-dev mailing list
escet-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/escet-dev

_______________________________________________
escet-dev mailing list
escet-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/escet-dev


Back to the top