Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Titan » test component sends messages to already disconnected component on shutdown(test component sends messages to already disconnected component on shutdown causing testcase error)
test component sends messages to already disconnected component on shutdown [message #1782716] Wed, 28 February 2018 10:09 Go to next message
Philipp Maier is currently offline Philipp MaierFriend
Messages: 2
Registered: February 2018
Junior Member
Hello folks,

I am Philipp Maier and I work for sysmocom as a software developer. We recently introduced TTCN3 to test our GSM software components.

Now we stumbled upon a problem that seems to be a race condition between the shutdown of the test components and the system under test that is still running and still sending messages into the test system while it is shut down.

The particular testsuit tests a paging situation on a BSC. The paging request is sent into the BSC via BSSMAP and the resulting paging commands on the RSL/Abis interface are monitored but we may exit the testcase while the BSC still keeps on paging.

(see also https_//git_osmocom_org/osmo-ttcn3-hacks/tree/bsc/BSC_Tests.ttcn)

While the testcase shuts down, it may happen that an RSL message is picked up by an underlaying component (IPA_Emulation.ttcnpp) and is forwarded to an upper component (RSL_Emulation.ttcn) which is already disconnected.

The result is then a testcase error:

19:34:31.554035 30 IPA_Emulation.ttcnpp:485 Dynamic test case error: Sending data on the connection of port IPA_RSL_PORT to 1:IPA_RSL[2] failed. (Broken pipe)
(see also attached result.log)

Our first idea was to freeze all communication by stopping all ports at once by calling

all port.stop;

before we perform setverdict(pass); and exit the testcase. But it seems not to solve the problem. We still have testcase errors.

(see also https_//gerrit_osmocom_org/#/c/6968/)

One other solution would be to stop the components manually in the correct order so that it is impossible that one component tries to send something to an already disconnected one. We already tried that and it seems to work fine:

private function f_shutdown_helper() runs on test_CT {
var integer i;

/* Make sure all IPA_Emulation instances are stopped */
for (i := 0; i < sizeof(bts); i := i + 1) {
bts[i].rsl.vc_IPA.stop;
}

/* Make sure the Osmocom_CTRL_Adapter is stopped */
vc_CTRL_IPA.stop;

setverdict(pass);
}

But this would require to implement a well thought out shutdown sequence for every single testcase. We wonder if this might be a common problem and if there is some kind of best practice to resolve this.

best regards,
Philipp Maier
  • Attachment: result.log
    (Size: 6.83MB, Downloaded 74 times)
Re: test component sends messages to already disconnected component on shutdown [message #1782765 is a reply to message #1782716] Thu, 01 March 2018 07:56 Go to previous messageGo to next message
Elemer Lelik is currently offline Elemer LelikFriend
Messages: 1120
Registered: January 2015
Senior Member
Hi Philipp,

yes, it's a known problem and there's no simple solution to it.

The spawned PTC's are acting asynchronously and one needs to implement a logic for controlled shutdown.
A common mistake is to rely on timers and delays , which builds a number of implicit assumptions into the code;
eventually the architecture will fall apart displaying mysterious symptoms.

If your shutdown function has no side effects, it's just as good as any.

I personally prefer a graceful shutdown implemented with internal synchronization ports;
these ports carry bidirectionally simple synchronization messages (strings or integers) and connect every PTC to the MTC , where the test case executes.

The functions started on the PTC's, when they are about to terminate , send a request to the MTC and wait to be acknowledged.
When acknowledged, the function terminates and the component becomes done.

something like this :

testcase tc_TS002_cancel_nonExisting_action() runs on MTC_CT
{
	ConfigurationBuild();

	v_RequestorMapping_CT.start(f_ScanClientPorts(tsp_AuthType_requestor,"titan0","titan0",tsp_localPortRequestor));
	
	alt { []Ctlrm_PCO.receive("ready")    {};   }

	v_TranslatorR_CT.start(f_ScanTranslatorPorts());
	
	v_Requestor_CT.start(f_TS002_cancel_nonExisting_action());

        Twait.start(60.0); 
	alt
	{
		[]Ctlr_PCO.receive("halt")    
		{
			Ctlrt_PCO.send("halt");
			Ctlrm_PCO.send("halt");
		};
		[]Twait.timeout {};		

	}//endalt

	all component.done;
	ConfigurationKill();
}


//-----------------------------------------------------------------------------
function f_ScanServerPorts()   runs on ServerMapping_CT 
//-----------------------------------------------------------------------------
{//startfunction
:
  vl_result :=f_IPL4_listen(    TCP_PCO,    "",    tsp_ServerListenerPort,    {tcp:={}},    {}  )

  if (ispresent(vl_result.connId))
  {
    Ctl_PCO.send("ready"); 
  }
:
//function body here
:
:

alt {
      [] Ctl_PCO.receive("halt") {


   }//endalt
 
}//endfunction

//************************************************************************* 
function f_ScanTranslatorPorts()  runs on Translator_CT
//*************************************************************************
{//startfunction f_ScanTranslatorPorts
    
	alt
    {//startalt

:
//function body here
:	

      [] Ctl_PCO.receive("halt") {
    }//endalt

	}//endfunction  ScanTranslatorPorts



where
ConfigurationByuild/ConfigurationKill contain all create/connect/map and
unmap/disconnect/kill commands respectively:

//-----------------------------------------------------------------------------
function ConfigurationBuild() runs on MTC_CT
//-----------------------------------------------------------------------------

{//startfunction

    v_ServerMapping_CT:=ServerMapping_CT.create;
    v_TranslatorS_CT:=Translator_CT.create;
    v_Server_CT:=Server_CT.create;

    connect(v_ServerMapping_CT:Ctl_PCO,self:Ctlsm_PCO);
    connect(v_Server_CT:Ctl_PCO,self:Ctls_PCO);
    connect(v_TranslatorS_CT:Ctl_PCO,self:Ctlst_PCO);

    connect(v_ServerMapping_CT:Comm_PCO,v_Server_CT:Comm_PCO);
    connect(v_ServerMapping_CT:CommTrans_PCO,v_TranslatorS_CT:CommTrans_PCO);
    connect(v_Server_CT:InComm_PCO,v_TranslatorS_CT:InComm_PCO);

    map(v_ServerMapping_CT:TCP_PCO, system:TCPs_PCO);
}//endfunction 

//-----------------------------------------------------------------------------
function ConfigurationKill() runs on MTC_CT
//-----------------------------------------------------------------------------

{//startfunction

    unmap(v_ServerMapping_CT:TCP_PCO, system:TCPs_PCO);

    disconnect(v_ServerMapping_CT:Comm_PCO,v_Server_CT:Comm_PCO);
    disconnect(v_ServerMapping_CT:CommTrans_PCO,v_TranslatorS_CT:CommTrans_PCO);
    disconnect(v_Server_CT:InComm_PCO,v_TranslatorS_CT:InComm_PCO);


    disconnect(v_Server_CT:Ctl_PCO,self:Ctls_PCO);
    disconnect(v_ServerMapping_CT:Ctl_PCO,self:Ctlsm_PCO);
    disconnect(v_TranslatorS_CT:Ctl_PCO,self:Ctlst_PCO);
   
   all component.kill;
   
}//endfunction 



So what happens is that the test case starts one component , waits until it announces itself to be ready ( the transport layer has to stabilize end-to-end, all alarms silent , whatnot) and only then starts the other components.


When the component of interest terminates , it announces itself finished to the MTC, which only then gives permission to the other components to leave; they might have terminated already and are likely waiting in the final alt for permission.

Finally , when all components are done, the configuration is dismantled.


Now I'm not saying that everyone should do the same; for simple configurations this might be overkill; it definitely adds an overhead;
However with this such a mechanism arbitrarily large architectures can be
started/terminated in a controlled manner.

Again, if you are happy with your shutdown, use it.


I hope this helps somewhat

Best regards
Elemer




[Updated on: Sat, 03 March 2018 08:44]

Report message to a moderator

Re: test component sends messages to already disconnected component on shutdown [message #1782778 is a reply to message #1782716] Thu, 01 March 2018 10:14 Go to previous messageGo to next message
roland gecse is currently offline roland gecseFriend
Messages: 20
Registered: December 2015
Junior Member
Yet another possible solution would be to set up an intermediary, which would transparently relay signaling between Tester and SUT. In could be realized as a daemon that runs independently from the Tester. The intermediary would include a management channel through which the Tester could control its behavior, i.e. enable/disable forwarding at testcase beginning/end, respectively.

Re: test component sends messages to already disconnected component on shutdown [message #1782877 is a reply to message #1782765] Fri, 02 March 2018 21:37 Go to previous message
Philipp Maier is currently offline Philipp MaierFriend
Messages: 2
Registered: February 2018
Junior Member
Hello Elemer,

Thank you for clarifying that it is indeed a more complex problem. That already helped me to understand the nature of the problem.

I see. If it is done right, than the components need to be synchronized properly and I think at least for future testcase implementations we need to keep an eye on it.

However. I think in the particular situation we could get off cheaply by just taking care that there is no more message flow when a test ends. When we take the paging tests as an example, we would just wait long enough until all paging has been end. That would be very similar to @roland's idea.

Thanks for your help.

Best regards.
Philipp
Previous Topic:Porting the ETSI IPv6 test suite to Titan part I
Next Topic:Graduation notice
Goto Forum:
  


Current Time: Tue Apr 23 06:18:24 GMT 2024

Powered by FUDForum. Page generated in 0.03274 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top