|
Hello,
I see lots of the questions got already covered. Hioefully I can add a bit more depth to some parts.
Indeed the big question is what "behaves the same" means exactly. That is, you need a deterministic decision procedure that yields a true/false answer for every possible situation that may happen. (In the real system I suppose, since the twins are supposed
to match the real system and not the other way around.)
In our tests, we have unit tests that perform exhaustive testing on the elementary parts, but even there, there is a hierarchy.
For example, the tree tests for multi-value trees below:
It starts with exhastive tests on a single node in the tree. Both "equalityValueTest" and "equalityIndexTest" do complete verification that the single node "v3 = 5" (or 6 in the second case) delivers the correct node graph.
The "equalityChainTest" does the same, but for 2 nodes. Note this isn't quite complete, as I test only one sequence of building the graph, and more variations exist there (I use knowledge from the implementation in this case to reduce testing effort).
So after these tests, I am fairly sure a node or a sequence of nodes works as expected, and I can move on to conjunction.
I start with conjunctions over elementary false and true values in "conjunctFalseTrueTests". After checking that works, I can perform conjunction over single nodes, and then over sequences of nodes. and then over trees of nodes.
And so you build from small to more complicated operations. The reason for this is that you need the more elementary operations to work before you can even think about correctness of the more complicated operations. If a single node gives me the wrong graph,
conjunction over a tree of nodes has no meaning any more.
Fixing tests also works like this. If tests break, you start from the broken test with the most elementary operation, and then work your way up to restore functionality.
In your system the above roughly translates to "if a sensor isn't truthfully replicated, there isn't anything useful you can say about equality of a block at a conveyor belt being stopped by detecting the block with the sensor."
We also have system tests (cif.tests plugin is full of them), those work the other way around. Larger scale but still small, which is more aimed at system tests, wehere you exercise the system in more realistic way. Basically it checks how the various parts
respond to each other. I see them mostly as "smoke" tests. Check that the system doesn't go up in smoke when you try something simple.
It gives a yes/no answer (always easy to obtain with a computer-only test), but if it breaks, you don't get much information what exactly isn't working (ie the answer that you get is "the system as w hole fails"). It needs more work to pinpoint the precise
cause.
Of course we are doing just one way of testing, there are many more variations, depending on what you consider important to have confidence in (and how much cinfidence).
For real-system testing, you may want to consult the PhD work by Niels Braspenning and Roel Boumen(?) as inspiration.
Albert
From: escet-dev <escet-dev-bounces@xxxxxxxxxxx> on behalf of Beek, Bert van <D.A.v.Beek@xxxxxx>
Sent: Wednesday, July 7, 2021 23:51
To: escet developer discussions <escet-dev@xxxxxxxxxxx>
Subject: Re: [escet-dev] How to test 3D digital twins?
Thanks Dennis, that was very helpful.
Bert
On 7 Jul 2021, at 18:48, Dennis Hendriks <dh_tue@xxxxxxxxxxx> wrote:
Hi Bert,
For ESCET we use both unit and integration tests. The integration tests have a purpose similar to your goal: expected output (digital twin) and tool output (physical twin) should match. For ESCET we have quite good coverage I think for our integration tests.
For software tools that take files as input and produce output files, this is relatively straightforward. For your case it is non-trivial. Indeed digital and physical twins are quite
different from ESCET applications. I gave some hints on how to define/measure the outcome in my previous mail.
For ESCET we opt for relatively small test models, each with a specific purpose. Then figuring out what went wrong is easier. Also, it prevents testing the same thing (large
overlap) for many different test cases.
How to automate depends a lot on the tools and environments you use. So I don't think I can be of much help there.
Dennis
Van: escet-dev <escet-dev-bounces@xxxxxxxxxxx> namens Beek, Bert van <D.A.v.Beek@xxxxxx>
Verzonden: woensdag 7 juli 2021 17:04
Aan: escet developer discussions <escet-dev@xxxxxxxxxxx>
Onderwerp: Re: [escet-dev] How to test 3D digital twins?
Hi Dennis,
Thanks for your answer. I am fully aware of the problem of the exploding state space and the coverage problem.
I think I did not formulate the problem properly. I now understand your previous question “Is full automation a requirement”. That is exactly the problem. For ESCET you have defined a number of tests that run fully automated and that each have
a perfectly defined required output result. For any new functionality always also new tests are defined, which are then automatically added to the total set of tests to be executed, if I understand correctly.
I am trying to see if a similar approach can be executed for the digital twins. Of course the coverage problem is similar to the test set used for ESCET. There are however, several additional problems:
- How do I define and measure the required outcome for a test on a digital twin?
- What kind of tests do I use:
- complete controllers that have been submitted by student groups in the past?
- or a number of well defined small tests that each test a small aspect of the digital twin?
- How could I automate such tests (in a way similar as the CIF test set in ESCET)?
Today, two TAs are doing a lot of manual work in the Festo lab: running controllers from previous final assignments on the actual Festo machines and running the same controllers on the digital twins and visually comparing the output.
I was wondering if we could learn from the ESCET testing experience, but maybe the testing problem for a digital twin is so very much different from the testing problem in ESCET CIF development that it is difficult to learn from the ESCET experience?
Bert
Hi Bert,
Disclaimer: I'm not an expert on testing per se.
I assume you have to test the physical system manually? This is not automated? I assume you can test the digital twin fully automatically, if you provide the scenarios to execute?
One issue is how to get good coverage, I think. You could perform random simulations on the digital twin and observe the outcome (success, failure, states, or so). Then you could perform the same scenarios on the real system, and maybe observe the outcome as
well. But to get good coverage, you'll need many scenarios. Some states may then be difficult to reach, as some states have higher chances of being reached than others. A classical problem is an automaton with as alphabet all 10 digits and which accepts only
a specific password or phone number. The number of possible sequences is exponential in the sequence length and only one such sequence is accepted. It's highly unlikely you'll ever find it by black box testing. The same problem occurs in active automata learning,
where conformance testing also plays a large role. There is quite some literature on this. This is not just theoretical, but a very practical problem as well.
But that assumes black box testing. If you know the model, you can use whitebox testing techniques. You could generate test scenarios from the model (digital twin model) and test
them on the physical twin. You would need to generate both scenarios for behavior accepted by the digitial twin model, and for behavior not accepted by it. You would have to check that (non)acceptance against the (non)acceptance of the physical one. There
are algorithms that can generate a 'complete' test set, under certain conditions. But still, the number of tests may be very large.
Another question is: what constitutes acceptance? It could be accepting states, if that is something you can define for your system. You could also define input and output subsets,
and see for a sequence of inputs whether the sequence of outputs matches, similar to Mealy machines.
Does this help you in any way?
Dennis
Hi Dennis,
The goal of this exercise is to find errors in the digital twins. Last year, due to the Corona restrictions, testing was optional. There was only a single group who tested on the festo workstations. Their controller worked without errors on the
digital twin, but it did not work on the Festo workstation. The reason for this was that the actuator setting to immediately stop the sliding manipulator was true/true (pressure on both sides), whereas in the digital twin, the actuator setting for this was
false/false. False/false (absence of pressure) on the Festo workstation caused the sliding manipulator to slowly stop => stopping at an incorrect location.
“Behave in the same way” means that a controller that works correctly on the twin should work correctly on the Festo system, and that a controller that shows erroneous behavior on the Festo workstation should show the same erroneous behavior on
the twin.
Ideally, we would like to consider the full state space, but obviously that is impossible. The controller is hybrid (timed) so has infinite state space.
In principle we only look at the observable behavior at the festo workstation and at the twin (it a product picked up properly and release at the required target location), but we could probably generate certain graphs of behavior in TwinCAT.
- Is full automation a requirement?
I don’t understand this question.
Bert
Hi Bert,
Some random questions that immediately pop into my mind:
- What is the goal of this exercise? What do you hope to gain?
- What does 'behave in the same way' mean for you?
- Do you consider the full state space of behavior, or only a finite set of scenarios?
- Do you consider the protocol only (order of events), or also timing?
- Is full automation a requirement?
Dennis
How would you test the functionality of digital twins for the Festo workstations as shown in https://cstweb.wtb.tue.nl/4tc00/festo/digital_twin/index.html?
A teacher assistant aims to test all CIF controllers submitted last year as final assignment for the course 4TC00 on the digital twin and on the Festo workstation to see if they behave in the same way.
I could also imagine that we write some small controllers that test specific functionality of specific sensor actuator relations/behavior.
Any thoughts or tips?
Thanks in advance
Bert
_______________________________________________
escet-dev
mailing list
escet-dev@xxxxxxxxxxx
To
change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/escet-dev
_______________________________________________
escet-dev
mailing list
escet-dev@xxxxxxxxxxx
To
change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/escet-dev
_______________________________________________
escet-dev mailing list
escet-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/escet-dev
|
