Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Titan » Support for HTTP/2 in Eclipse Titan- with a twist( @decoded and decmatch)
Support for HTTP/2 in Eclipse Titan- with a twist [message #1773897] Fri, 06 October 2017 09:54 Go to next message
Elemer Lelik is currently offline Elemer LelikFriend
Messages: 807
Registered: January 2015
Senior Member
Dear all,

I will continue with the HTTP2 example presented previously; however, this post is not focused on HTTP2 , but on some new language elements related to matching and decoding that I will try to introduce under the pretext of HTTP2.


First , let me summarize the problem we are facing: we are using a generic IPL4 port in TCP mode, which communicates with TCP ASPs (Abstract Service Primitives); let's just call them TCP structures for simplicity.
However as users we re not interested to see TCP structures, we'd rather deal with HTTP2 structures instead; but the problem is , these structures are encoded in the payload part of TCP structures
hence they are not directly readable to us, at least not without running them through decode first.

There are several possible answers to this dilemma:

-we could rewrite the generic TCP port and create an HTTP2 test port instead; although this seems convenient, in the long term it will prove a maintenance nightmare: protocol specific test port will star to proliferate uncontrollably and you will probably end up with several test ports for different protocols, test ports sharing the largest part of their code ; so any correction in the common part will have to be propagated through all akin code. We've been through this phase and I don't recommend this to anyone.
-we could use a dual faced /translation port ; this technical possibility has been thoroughly treated previously and I believe this is the best option
-we could decode the TCP structures after receiving them and match them using an if-else or similar structure ; this is what we have done in the previous HTTP2 example:

   alt 
  { 
    [] p.receive(ASP_RecvFrom:?) -> value v_ASP_RecvFrom { 
    
     f_HTTP2_decode_frame(v_ASP_RecvFrom.msg, v_HTTP2_Frame, v_err) 

     log("**********received frame:  ***********",v_HTTP2_Frame)
     
      if (ischosen(v_HTTP2_Frame.settings_frame))
      {
     if (match(v_HTTP2_Frame,t_HTTP2_init_settings_frame_ack ) ) 
     {  
     log("**********received init_settings_frame_ack   ") 
     repeat;
     } 
     else if  (match(v_HTTP2_Frame, t_HTTP2_init_settings_frame_r))
     {  
     log("**********received init_settings_frame    ") ;
     p.send(t_data2(v_cid,f_HTTP2_encode_frame(valueof(t_HTTP2_init_settings_frame_ack))));
     repeat;
     
     }
     
      }

    else  if (ischosen(v_HTTP2_Frame.header_frame))
      {
      log("**********received header frame    ") ;
        var HTTP2_header_block vl_HTTP2_header_block
    
    HTTP2_comp_context_decode(v_HTTP2_comp_context,vl_HTTP2_header_block, v_HTTP2_Frame.header_frame.header_block_fragment )
    
     log("************received header block:  ",vl_HTTP2_header_block)
     repeat;
      }

 else  if (ischosen(v_HTTP2_Frame.data_frame))
      {
           log("*************received data_frame    ") ;
      }

 else  if (ischosen(v_HTTP2_Frame.ping_frame))
      {
           log("*************received ping_frame    ") ;
      }

    };
    [] t.timeout{log("Bye")}

  }//endalt 
  



-finally we could use decmatch and @decoded, new language elements standardized recently, that try to provide an alternative to this dilemma

The usage of decmatch and @decoded can be exemplified as below:
 type MyType2 record {
 Header        header,
 octetstring   payload
 } 

 MyPort.receive(MyType2:{header := ?, payload := decmatch mw_myTemplate})  -> value (v_myVar := @decoded payload);
  // The encoded payload field of the received message is decoded and matched with
 // mw_myTemplate; if the matching is successful the decoded payload is stored in v_myVar
 



However , usage of decmatch and @decoded in Titan has a few caveats:

-first and foremost, this feature is only supported in the function ttst runtime RT2 , for reasons I hope are fairly obvious: although this mechanism is convenient, it's not the best approach for an architecture optimized for speed.

To introduce the second limitation, let's quickly recap how codecs can be invoked from within TTCN-3:
-the first option is to call the declared codec functions explicitly;
Say in the module HTTP_Types.ttcn (part of the protocol module HTTP2.0-which is an HTTP1.0/HTTP1.1 implementation !!!) the codecs are declared as:
  external function ef_HTTP_Encode(in HTTP_Message pl_pdu) return octetstring
  external function ef_HTTP_Decode(in octetstring pl_stream, in boolean pl_assemble_chunked:=true) return HTTP_Message
  

and invoked e.g. as:
  template  ASP_Send t_data1(in integer p_id ) :={
  connId:=p_id,
  proto:=omit,
  msg:=ef_HTTP_Encode(valueof(t_message("")))
}
 


-the second option is to call them implicitly, by using encvalue or decvalue; the TTCN-3 tool will infer and invoke the applicable codec ;
this assumes that the codec for the type in question is unique (else , for multiple encoding that has been standardized in 4.9.1,
the encvalue/decvalue will have to be extended with a parameter that points to the desired codec)
Note that encvalue and decvalue are using bitstring as output/input.
decmatch and @decoded use this latter logic, meaning that they invoke the codecs implicitly (and they also use bitstrings , not octetstrings)

Codec declarations are part of the specific protocol modules (and not directly part of Titan) and typically are using octetstrings as input/output.
This means that the existing protocol modules are to be complemented with additional code to make them encvalue/decvalue (and implicitly decmatch/@decoded) compatible.
Updating the already published dozens of protocol modules will take some time, but the example below will explain what needs to be done so if someone is in a hurry to deploy a protocol module with encvalue/decvalue can simply transpose the code.


Let's start with the runtime issue: as you might now, Titan contains two runtimes: one rich in features and one optimized for speed.
This practically means that during the first phase of compilation (TTCN-3 ---> C++) two different sets of code will be generated depending on the
presence/absence of the compiler flag -R. Load test runtime is the default , while function test runtime has to be triggered by using an appropriate Makefile.

So here are the steps to be taken when migrating a project from Runtime1 (load test runtime, default) to Runtime2(function test runtime):

1. add -DTITAN_RUNTIME_2 to CPPFLAGS:
# Flags for the C++ preprocessor (and makedepend as well):
CPPFLAGS = -D$(PLATFORM) -I$(TTCN3_DIR)/include -DIPL4_USE_SSL -I$(OPENSSL_DIR)/include   -DTITAN_RUNTIME_2
 

2. add -R to compiler flags:
# Flags for the TTCN-3 and ASN.1 compiler:
COMPILER_FLAGS = -L -R
 

3.change TTCN3_LIB to ttcn3-rt2 or ttcn3-rt2-parallel (from ttcn3 or ttcn3-parallel)
# Execution mode: (either ttcn3 or ttcn3-parallel)
TTCN3_LIB = ttcn3-rt2-parallel
  



Next, let's look into what's needed for encvalue/decvalue friendliness.
At this moment the HTTP2 code has not yet been updated for that in github; to make the codecs encvalue/decvalue friendly , the following additions have been made in the attached code:


in HTTP2_Types.ttcn:
//  encvalue/decvalue friendly codecs
//******************************************************************************

external function f_HTTP2_encode(in HTTP2_Frame pl_frame) return bitstring
with { extension "prototype(convert) encode(HTTP2)" }

external function f_HTTP2_decode(inout bitstring buff, out HTTP2_Frame pl_frame) return integer
with { extension "prototype(sliding) decode(HTTP2)" }
//******************************************************************************
  


and

// HTTP/2 Frame definition
// The length field handled automatically
type union HTTP2_Frame {
  HTTP2_Data_frame          data_frame,
  HTTP2_Header_frame        header_frame,
  HTTP2_Priority_frame      priority_frame,
  HTTP2_RST_Stream_frame    rst_frame,
  HTTP2_Settings_frame      settings_frame,
  HTTP2_Push_Promise_frame  push_promise_frame,
  HTTP2_Ping_frame          ping_frame,
  HTTP2_Goaway_frame        goaway_frame,
  HTTP2_Window_Update_frame window_update_frame,
  HTTP2_Continuation_frame  continuation_frame,
  HTTP2_Generic_frame       generic_frame
}
with {
 encode "HTTP2"
}
 



in HTTP2_EncDec.cc:

namespace HTTP2__Types {
// encvalue/decvalue friendly codecs

BITSTRING f__HTTP2__encode(const HTTP2__Frame& pl__frame)
{
	return oct2bit(f__HTTP2__encode__frame(pl__frame));
}



INTEGER f__HTTP2__decode(BITSTRING &buff, HTTP2__Frame &pl__frame)
{
	OCTETSTRING x(bit2oct(buff));
	HTTP2__decoder__error__descr dummy;
	INTEGER ret_val = f__HTTP2__decode__frame(x, pl__frame, dummy);
	buff = BITSTRING(0, NULL);
	return ret_val;
}

}
 



With these additions an encvalue/decvalue invocation referring to a value of type HTTP2_Frame will work properly; else the compiler will complain.
Besides the decmatch/@decoded inspired changes I have also extracted part of the code into a preamble/postamble.
So now the test case looks like this:



 //************************************************************************* 	
testcase TC_http2() runs on GeneralComp system SystemComp {
  //*************************************************************************


  f_preamble();


  //init compression context
  v_HTTP2_comp_context:=HTTP2_comp_context_init();
  // send connection preface "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"
  p.send(t_data2(v_cid,HTTP2_connection_preface));
  // send settings
  p.send(t_data2(v_cid,f_HTTP2_encode_frame(valueof(t_HTTP2_init_settings_frame))));
  // send window update
  p.send(t_data2(v_cid,f_HTTP2_encode_frame(valueof(t_HTTP2_window_update_frame))));

  if(not(tsp_start==upgrade_conn))
  {
    // send HEADER frame
    HTTP2_comp_context_encode(v_HTTP2_comp_context,valueof(t_HTTP2_header_block), v_data )
    //log("v_HTTP2_header_block",v_HTTP2_header_block)
    p.send(t_data2(v_cid,f_HTTP2_encode_frame(valueof(t_HTTP2_header_frame(v_data))))); 
  }

  t.start(1.0) 

  alt {

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_init_settings_frame_ack
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received init_settings_frame_ack   :",v_HTTP2_Frame)  ;repeat}

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_init_settings_frame_r
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received init_settings_frame_r   :",v_HTTP2_Frame)  
      log("**********received init_settings_frame r   ") ;
      p.send(t_data2(v_cid,f_HTTP2_encode_frame(valueof(t_HTTP2_init_settings_frame_ack))));
      repeat}

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_header_frame0
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received header_frame   :",v_HTTP2_Frame) 

      log("**********received header frame    ") ;
      var HTTP2_header_block vl_HTTP2_header_block

      HTTP2_comp_context_decode(v_HTTP2_comp_context,vl_HTTP2_header_block, v_HTTP2_Frame.header_frame.header_block_fragment )

      log("************received header block:  ",vl_HTTP2_header_block) ;
      repeat
    }

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_data_frame0
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received data_frame   :",v_HTTP2_Frame)  ;repeat}

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_ping_frame
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received ping_frame   :",v_HTTP2_Frame)  ;repeat}



    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= ?
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received something   :", v_HTTP2_Frame)  ;repeat}


    [] t.timeout{log("Bye")}

  }


  HTTP2_comp_context_free(v_HTTP2_comp_context);

  //----------------------------------------------------------------------------


  f_postamble();

  setverdict(pass);

}//endtestcase
 



major difference compared to the previosu HTTP2 example being the usage of decmatch/@decoded:
 :
 alt {

    [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_init_settings_frame_ack
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received init_settings_frame_ack   :",v_HTTP2_Frame)  ;repeat}
 
 :
 
  

The above snippet should be read like this:

"If the TCP ASP queued in the port contains a payload that when decode matches t_HTTP2_init_settings_frame_ack, then the ASP should be extracted from the queue, and the decoded payload deposited in
the variable v_HTTP2_Frame"


The same nghttp2.org has been used as before , with a request towards /httpbin/ip.
A browser request to this URL (https://nghttp2.org/httpbin/ip ) returns:
{
  "origin": "91.82.100.59"
}
 

The same as seen by Titan:

05:26:48.162027 HTTP2.ttcn:458 received something   :{
    data_frame := {
        stream_id := 1,
        end_stream_flag := true,
        data := '7B0A2020226F726967696E223A202239312E38322E3130302E3539220A7D0A'O ("{
  \"origin\": \"91.82.100.59\"
}
"),
        padding := omit
    }
} 
 


Code and logs attached.



I hope this will be useful if someone wishes to use decmatch/@decoded. However I'd personally prefer the translation port-based solution.


Best regards

Elemer

[Updated on: Mon, 09 October 2017 06:51]

Report message to a moderator

Re: Support for HTTP/2 in Eclipse Titan- with a twist [message #1774008 is a reply to message #1773897] Mon, 09 October 2017 07:01 Go to previous message
Elemer Lelik is currently offline Elemer LelikFriend
Messages: 807
Registered: January 2015
Senior Member
Hey guys,

I have just realized that I have made a mistake in this post , namely data frames are not received where they are supposed to be received , namely here:


   [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= decmatch t_HTTP2_data_frame0
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received data_frame   :",v_HTTP2_Frame)  ;repeat}



but here:
 [] p.receive(ASP_RecvFrom:{
        connId:=v_cid,
        remName:=tsp_hostname,
        remPort:=?,
        locName:=?,
        locPort:=?,
        proto:=?,
        userData:=?,
        msg:= ?
      }) -> value (v_HTTP2_Frame := @decoded msg) {     log("received something   :", v_HTTP2_Frame)  ;repeat}


which reduces from the exemplifying impact of the post.

By changing

template  HTTP2_Frame t_HTTP2_data_frame0:= {
  data_frame :={
    stream_id:=1,
    end_stream_flag :=false,
    data:=?,
    padding:=omit
  }
}



to
template  HTTP2_Frame t_HTTP2_data_frame0:= {
  data_frame :={
    stream_id:=1,
    end_stream_flag :=?,
    data:=?,
    padding:=omit
  }
}



things will fall into place and "received something"


05:26:48.162027 HTTP2.ttcn:458 received something   :{
    data_frame := {
        stream_id := 1,
        end_stream_flag := true,
        data := '7B0A2020226F726967696E223A202239312E38322E3130302E3539220A7D0A'O ("{
  \"origin\": \"91.82.100.59\"
}
"),
        padding := omit
    }
} 
 


will change to "received data_frame":

23:40:20.123542 HTTP2.ttcn:371 received data_frame   :{
    data_frame := {
        stream_id := 1,
        end_stream_flag := true,
        data := '7B0A2020226F726967696E223A202239312E38322E3130302E3539220A7D0A'O ("{
  \"origin\": \"91.82.100.59\"
}
"),
        padding := omit
    }
} 



I have updated the formatted log files accordingly, not the code though, so you will have to correct yourselves if interested.

Sorry and best regards
Elemer

[Updated on: Mon, 09 October 2017 07:04]

Report message to a moderator

Previous Topic:TTCN Titan Testport for socketCAN
Next Topic:Logging in Eclipse Titan part III: Custom logger plug-ins
Goto Forum:
  


Current Time: Sat Sep 22 11:53:50 GMT 2018

Powered by FUDForum. Page generated in 0.30621 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top