Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Titan » Catching dynamic testcase errors(Non-standard exception handling in Titan)
Catching dynamic testcase errors [message #1738213] Mon, 18 July 2016 08:19 Go to next message
Elemer Lelik is currently offline Elemer LelikFriend
Messages: 807
Registered: January 2015
Senior Member
Dear all,


Exception handling in TTCN-3 is pretty summary, and with a reason: when an exception is encountered runtime, an error is thrown (in Titan, named Dynamic Testcase Error-DTE), verdict is set to error, and execution
is terminated. To understand the unassuming nature of exception handling in TTCN-3 , we have to have in mind the original intention of the language , which is to create test environments to validate SUT behaviour. Once an exception occurs, the whole environment is considered compromised, so continuing execution would be unreasonable.
The user will have to investigate the origin of the exception, correct it and resume execution afterwards.

One has to add that runtime exception handling is only the tip of the iceberg here: during compilation , strict verifications are assumed, so a large part of potential exceptions are eliminated compile-time. Not all exceptions can be detected during compilation though: in Titan, besides programming error, illegal values can originate in the configuration file ( which is only verified run-time), messages coming from test ports etc.
As long as TTCN-3 and Titan is used for the intended purpose, this (apparently) self-restrained exception handling mechanism should be sufficient.

However , along the years we have discovered that TTCN-3 and Titan , due to Titans' robustness and reliability, constitute an excellent development platform for signalling applications outside the scope of conformance/functional etc. testing, such as signalling gateways, traffic generators and so on. And as such , we have been confronted with a new situation: writing test frameworks and writing signalling applications are two different disciplines, requiring two different mindsets and technologies. For instance a test framework is better disseminated in source code, while often an application requires distributing binaries and so on. In all honesty, as the TTCN-3 language was not created for this, lacks support for application development is some areas.
But, even with the above deficiencies, we have found that TTCN-3 could be an alternative to Java or C/C++ , due to ease of formulating protocols, both in TTCN-3 and ASN.1, the matching mechanism, support for signalling concepts in general.

Exception handling though proved insufficient: for a traffic generator running say a 72 hours stability test is intolerable to have traffic stopped after two days because of a difficult-to-detect programming error or unbound variable in an incoming message.
This lead us to introducing the non-standard exception handling feature of Titan quite similar to the ones present in C++ or Java.
A note of warning here: although one may be tempted to use this in circumstances other than described above, this is counter-recommended: if you are using Titan in test applications, do yourself a favor and avoid try-catch; your scope should be to find errors , both in your code and in SUT as fast as possible. Using try-catch may push symptoms over other parts of the code, making debugging more difficult. Also, DTE handling is implemented using C++ exceptions; as a DTE exception is raised, the component can be left in an inconsistent/invalid state. The DTE caused by the unbound values, using the omitted field, overindexing, sending are safer to catch, but catching the DTE during the operations, which alter the internal state of the component such as connect, map, create are dangerous and should be avoided. So our recommendation is to use try-catch with moderation.

The mechanism of catching DTEs, as said above, is very similar to exception handling used in other languages: there is a try statement block which contains the guarded code and immediately after it there is a
catch statement block which contains the code that is executed if a DTE occurred in the guarded statement block. When a DTE occurs in a statement the rest of the statements in that block are skipped and control is transferred to the 1st statement of the catch block. Two TITAN specific keywords were introduced for this purpose: @try and @catch ,
these start with a "@" to avoid backward compatibility issues (new keywords clashing with identifiers used in existing code)
and to differentiate them from standard TTCN-3 keywords.



Syntax:

function MyFunc() {
  @try { // this is the guarded block, all DTEs are caught here
    <statements>
  }
  @catch(dte_str) { // dte_str will contain the error message of the DTE
    <statements>
  }
}


The identifier dte_str becomes an invisible variable definition in the @catch block, the code is semantically equivalent to:

@catch {
  var charstring dte_str := <DTE error message>;
  <statements>
}


This can be used as a normal charstring type variable whose scope is the @catch statement block.


The predefined str2int() function causes a DTE if the input is invalid;
this wrapper function makes it safe to use it on possibly invalid input strings

function safe_str2int(in charstring int_str, in integer defval) return integer {
  @try {
    return str2int(int_str);
  }
  @catch(err) {
    return defval;
  }
}



The predefined function string2ttcn does not have a return value;any error in the input string will cause
an exception that can be caught using @try - @catch blocks.

:
type record MyRecord { integer a, boolean b }
:
var template MyRecord my_rec

@try {
 	string2ttcn("complement ({1,?},{(1,2,3),false}) ifpresent", my_rec)
	log(my_rec)
}
@catch (err_str) {
 	log("string2ttcn() failed: ", err_str)
}






In the next example we check if the DTE was caused by the division by zero; if not then
the DTE with the same message is created again, so that any other DTE will
stop the execution of the test case with the original error message.
If it was a division by zero then the verdict is set to fail.


external function throw(in charstring msg);

testcase TC(in integer i) runs on MyComp {
  @try {
    i := 10 / i;
    somefunction(); // this function might cause other DTEs
    setverdict(pass); // this line is reached only if there was no DTE
  }
  @catch(err) {
    if (match(err, pattern "*division by zero*")) {
      log("division by zero detected");
      setverdict(fail); // the verdict is fail instead of error
    } else {
      throw(err); // external function used to re-throw the DTE
    }
  }
}




External functions can be used to re-throw the error in the catch block with a modified or original (as in the example above) error message.
The C++ implementation:

void throw_(const CHARSTRING& msg) {
  TTCN_error("%s", (const char*)msg);
}




@try-@catch blocks can be nested. In this example minor DTEs are ignored and the
for loop continues but in case of a major error the DTE is re-thrown an caught by
the outer catch which also terminates the test case with a fail verdict:

testcase TC() runs on MyComp {
  @try {
    for (var integer i:=0; i<100; i:=i+1) {
      @try {
        <statements that can cause DTEs>
      }
      @catch(dte_str) {
        if (match(err, <some pattern for minor errors>) {
          log("minor error ", dte_str, " ignored, continuing load test...");
        } else {
          throw(dte_str);
        }
      }
    }
    setverdict(pass);
  }
  @catch(dte_msg) {
    log("Something went very wrong: ", dte_msg);
    setverdict(fail);
  }
}





Best regards

Elemer

[Updated on: Mon, 18 July 2016 15:50]

Report message to a moderator

Re: Catching dynamic testcase errors [message #1738388 is a reply to message #1738213] Tue, 19 July 2016 11:57 Go to previous message
Gustavo Gonnet is currently offline Gustavo GonnetFriend
Messages: 34
Registered: October 2015
Location: Montreal, Quebec, Canada
Member
Hi,
In our case, the @try-@catch feature helped us make our suites more robust.

Some of our tests have a setup to do in the SUT before running the scenario and doing validations. This setup sometimes involves a change in configuration that potentially impacts other, subsequent tests.
Imagine that the simplest test looks like this:
1- SUT setup - configuration change
2- run the scenario (the goal of the test)
3- SUT cleanup - revert change

If something bad or unexpected happens when the scenario runs, or we made a mistake in our code in the test, the fact that the test ends abruptly (in step 2) without reverting those configuration changes (in step 3) may lead to subsequent tests (when running a suite of tests) failing in an unpleasant domino effect.

So this is how we fixed this particular challenge and made robust suites a reality for us:
1- SUT setup - configuration change
2- @try: run the scenario (the goal of the test)
3- @catch() {fail_the_test();}
4- SUT cleanup - revert change

In this way, we ensure that our tests leave the SUT in a nice and clean state.

Thanks,
Gustavo.
Previous Topic:Advanced TTCN-3 usage
Next Topic:Titan compiler crashing while compilation
Goto Forum:
  


Current Time: Fri Sep 21 02:52:45 GMT 2018

Powered by FUDForum. Page generated in 0.02144 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top