Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Titan » Strategies for shutdding down complex component hierarchy
Strategies for shutdding down complex component hierarchy [message #1836986] Wed, 20 January 2021 15:19 Go to next message
Harald Welte is currently offline Harald WelteFriend
Messages: 128
Registered: July 2017
Location: Berlin, Germany
Senior Member

Dear TITAN community,

at Osmocom., we have one problem that hits us again and again over the years: How to properly shut down a complex hierarchy of components exchainging messages safely, without running into Dynamic test case errors during teardown.

In one of our typical test cases, we set up dozens to sometimes hundreds of components, most of them with internal ports connected between components. Maybe only half of those components are actual test cases, the others are just emulatiing some underlying protocol stack, or in some way facilitating the connection beween the IUT and the tets.

At some point, the actual test case components are terminating, and let's assume the test concluded successfully. All components up to now have either verdict "none" or "pass".

Then, the runtime starts stopping the various components, in whatever order. At this point it may happen that one of the components still is processing some message and sends it to another component that has alreay been shut down -> boom. Dynamic Test Error is reported, and the overal lverdict becomes "error", even though all of the tests had passed before.

Originally I had hoped that it is sufficient to simply make sure that all external ports like IPL4asp are clsoed first, so no external messages can be received anymore. That helps, but...

This can also be triggered by timeouts, as many protocol layers have internal timers for sending keep-alives. If such a timer fires while whetever is the next component has already been terminated -> boom.

We already tried to do an explicit "all component.stop()", it doesns't help either.

The most obvious solution to this problem would be to either have

* some way to "lock" the current verdict, i.e. whatever happens beyond this point can no longer affect the overall verdict. The test author could simply put that "lock" instruction once he knwos that anything relevant to his test has finished, and everything else happening thereafter is irrelevant for the overall verdict and must be ignored

or

* some way to reliably prevent all components from sending furrther messages through their ports

I'm somewhat surprised that there appears to be no obvious solution to the problem.

Sure, in "textbook TTCN3" you would probably normally have all of this complex non-testcase code as part of the "system simulator" which re resides outside of your TTCN3 runtime and hence on the "other" side of the test ports.

However, in TITAN with its capacity for "internal" ports between components, it is much more likely that there is a lot of code that just provides underlying logic/transport for the actual test casese. Even Ericsson has released such code like the SCCP_Emulation - so it seems to be an acceptable programming paradigm.

I also though it would be possible to designate entire component (types) as "not relevant to the final verdict", but that does not help. You still want to be able to catch violations of lower-lyer protocols inside those "emulation" components and do a setverdict(fail) if you encouter such a problem.

How do other people solve this ? How do you suggest to proceed?

Thanks,
Harald
Re: Strategies for shutdding down complex component hierarchy [message #1836990 is a reply to message #1836986] Wed, 20 January 2021 15:43 Go to previous messageGo to next message
Gábor Szalai is currently offline Gábor SzalaiFriend
Messages: 86
Registered: December 2015
Member
In our internal simulator, solved the issue using a controller component.
Every components connect to the controller component. The controller component sends the "shut down" order to every other component, waits for the confirmation. After the last component confirmed the shut down order, the components can be safely ended.
Every component stops sending outgoing messages after the "shut down" order, and ignores incoming messages.

Re: Strategies for shutdding down complex component hierarchy [message #1837003 is a reply to message #1836990] Wed, 20 January 2021 20:12 Go to previous messageGo to next message
Harald Welte is currently offline Harald WelteFriend
Messages: 128
Registered: July 2017
Location: Berlin, Germany
Senior Member

Gábor Szalai wrote on Wed, 20 January 2021 16:43
In our internal simulator, solved the issue using a controller component.


Yes, it is obviously possible to do this. But it is a *lot* of effort to implement such a port in every component, and repeat the related code over and over again. To me, that is a very expensive work-around for something that sounds relatively easy to do within the runtime.

We often have relatively complex/deep hierarchies, so it would be very difficult for one central component to even know all other components / component references. There is normally no need for that kind of knowledge. Yes, one could delegate the task of forwarding/flooding this "shutdown" request across the hierarchy. But really? Repeating the same code again and again?

Furthermore, not all components we use are developed by Osmocom. Take for example the SCCP_Emulation we use from the TITAN project. Adding such an "SHUTDOWN" interface would mean we'd have to fork every 3rd party component we use, maintain our own patchset on top, ...

If there weas some generic support by the TITAN runtime, one would reduce a lot of explicit extra code in every TITAN component out there and avoid all of the extra effort.

I still have a hard time believing that this has not been solved before in a "one size fits all" approach, for everyone. I hoped I simply had missed this feature. Given that you also seem to create elaborate work-arounds, it seems like it really doesn't exist :(

Do you think it's feasible to implement either of the approaches I suggested? I'm not familiar with the TITAN internals, but I might be able to have a look how verdicts are collected and try to implement the "verdict freeze'
Re: Strategies for shutdding down complex component hierarchy [message #1837288 is a reply to message #1836986] Wed, 27 January 2021 08:18 Go to previous messageGo to next message
Gábor Szalai is currently offline Gábor SzalaiFriend
Messages: 86
Registered: December 2015
Member
It seems to me that the "stop" component operation can be used to end the test.
See 21.3.3 of the standard. Stopping an alive-type component keeps the ports connected/mapped.
Re: Strategies for shutdding down complex component hierarchy [message #1837298 is a reply to message #1837288] Wed, 27 January 2021 09:21 Go to previous messageGo to next message
Harald Welte is currently offline Harald WelteFriend
Messages: 128
Registered: July 2017
Location: Berlin, Germany
Senior Member

Hi Gabor,

yes, "stop" can be used (and is used by us). However, if you stop let's say 50 parallel test components (e.g. with "all component.stop") then that "stop" is neither atomic, nor does it leave the overall system state in a way that prevents further DTE.

Particularly, incoming messages arriving on those ports meanwhile from other components still result in dynamic test case errors. Or maybe it's on te transmit side, if you try to send to a component reference of a stopped PTC - at least it still causes DTE.

So either

a) the 'all component.stop' would have to be somehow atomic/synchronized, to prevent any other component from sending further messages before all of them are stopped
b) the internal ports between the PTC somehow have to enter a new state that doesn't cause a DTE if a message is sent to a stopped PTC.

Please note the above reflects my "external" view. I have very limited knowledge of the TITAN internals / implementation, I just share my experience as somebody who is developing relatively complex test suites on it and observing the behavior.
Re: Strategies for shutdding down complex component hierarchy [message #1837302 is a reply to message #1837298] Wed, 27 January 2021 09:31 Go to previous message
Harald Welte is currently offline Harald WelteFriend
Messages: 128
Registered: July 2017
Location: Berlin, Germany
Senior Member

Also, another problem in this context is that "all component.stop" seems to only work when execute from the MTC. Not sure if that's a TTCN3 language limitation or one of the TITAN implementation?

That's rather inconvenient, as in complex component hierarchies, the decision if and when to stop (for example in case of "failure" cases) can be anywhere. And when you terminate, you don't want that "fail" verdict to occasionally turn into an "error" due to a DTE occurring due to the somehow race-ful shutdown of the component hierarchy.

So I'm still wondering what the "correct" approach here is to avoid all of the above, or where my misunderstanding lies.
Previous Topic:TTCN-3 specification version supported
Next Topic:Log FileMask / ConsoleMask filter by component type?
Goto Forum:
  


Current Time: Wed Jun 23 10:11:38 GMT 2021

Powered by FUDForum. Page generated in 0.02295 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top