Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Trace Compass » Creating a custom text parser
Creating a custom text parser [message #1710325] Tue, 06 October 2015 06:42 Go to next message
Mikael Springer is currently offline Mikael SpringerFriend
Messages: 6
Registered: July 2012
Junior Member
Hi.

I'm trying to create a custom text parser to analyze text traces from our server software. I have managed to capture most of the content of the traces but there is one bit of information left that I cannot make the parser capture. Below is an example of one of our traces.

The information can be categorized in two categories; ordinary trace messages beginning with a time stamp and continuing to the end of the line, and messages specifying network communication beginning with a line containing "Sending message!" or "Received message!", spanning over several lines and ending when the next "ordinary" line begins. A network message has metadata in fields "Type", "Recipient", "Sender" and "Msg nr" and message content in lines beginning with the pattern "[A-Z]_FT:" where FT stands for Field Type.

By specifying ordinary trace messages containing a time stamp as Root Lines in the custom parser and using Child Lines for the metadata I can capture these ordinary trace lines and metadata fields in network messages. The one bit of information I cannot capture is the actual content of the network messages. These are lines following a "Sending message!" or "Received message!" and metadata, beginning with a pattern "[A-Z]_FT:".

I could capture these lines by using Child Lines in the same way that I'm capturing metadata but then I would have to specify one Child Line for every possible Field Type we have in the system and these are in the order of hundreds so this is not very practical. And I don't want to have each Field Type displayed in its own column since then I would end up with hundreds of columns in the trace table.

Is there a way to capture all lines beginning with "[A-Z]_FT:" and spanning several lines and place all of it in one column "Msg Content"?

Below the trace data I include an export of my custom parser.

10/05/15 09:00:01.33,  cc_main ItemStateQueued::finalizeItem(c_call* item 511851)
10/05/15 09:00:01.33,  cc_main c_call::changeItemState(ItemState* newState = 14ItemStateFinal), oldState = 15ItemStateQueued, callId = 511851
10/05/15 09:00:01.33,  cc_main c_call::dequeue_on_client(), call_id = 511851, is_dequeued_on_client = 1, is_counted_on_client = 1
10/05/15 09:00:01.33,  cc_main c_call::dequeue_on_client - already done
10/05/15 09:00:01.33,  cc_main c_client::end_call_in_goal_ratio, call_id=511851
10/05/15 09:00:01.33,  cc_main c_call::~call - call_id = 511851, removed item_id = INC511851
10/05/15 09:00:01.34,  cc_main Sending message!
Type :      CALL_END_MT
Sender :    -1
Recipient : -1
Msg nr :    10274-9038
CALL_ID_FT : 	511851
PBX_CALL_ID_FT : 	clearit_front-1444028343.106
REMOTE_ADDRESS_FT : 	0733906699
CALLED_ADDRESS_FT : 	0046774109093
10/05/15 09:00:01.34,  cc_main c_messenger::send()! MessageType = 501
10/05/15 09:00:01.34,  cc_main c_call::set_ivr_state(), call_id = 511851, current_state = state_no_ivr, new_state = state_final
10/05/15 09:00:01.34,  cc_main Node_Controller::passive_queue_is_activated(), _passive_queue_is_activated = false
10/05/15 09:00:01.34,  cc_main c_call::check_node_switch - the call has not swithed nodes
10/05/15 09:00:01.34,  cc_main c_call::removeItem - call id = 511851, _itemRepository.size = 2
10/05/15 09:00:01.34,  cc_main c_call::cleanupObservers(), call_id = 511851, _subscribers.size = 3
10/05/15 09:00:01.34,  cc_main c_VRU::set_state(e_state new_state), extension = 7382, pre_state = state_queue, state = state_call, new_state = state_ready
10/05/15 09:00:01.34,  cc_main Sending message!
Type :      DISCONNECT_EVENT_MT
Sender :    -1
Recipient : -1
Msg nr :    10274-9042
LOCAL_ADDRESS_FT : 	7382
10/05/15 09:00:01.34,  cc_main c_messenger::send()! MessageType = 47
10/05/15 09:00:04.36,  cc_main c_LCC_handler::receive_message, client_type = 1, msg_type = 6
10/05/15 09:00:04.36,  cc_main Received message!
Type :      LINE_AVAILABLE_MT
Sender :    2
Recipient : 1
Msg nr :    10519-815
LOCAL_ADDRESS_FT : 	7382
10/05/15 09:00:04.36,  cc_main c_VRU::handle_line_available, phone=7382


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<CustomTxtTraceDefinitionList>
	<Definition category="CI Trace" name="CPP Trace">
		<TimeStampOutputFormat>MM/dd/yy HH:mm:ss.SSS</TimeStampOutputFormat>
		<InputLine>
			<Cardinality max="2147483647" min="0" />
			<RegEx>(\d\d/\d\d/\d\d \d\d:\d\d:\d\d\.\d+),\s+(\w+)\s(.*)</RegEx>
			<InputData action="0" format="MM/dd/yy HH:mm:ss.SSS" name="Time Stamp" />
			<InputData action="0" format="" name="Process" />
			<InputData action="0" format="" name="Description" />
			<InputLine>
				<Cardinality max="2147483647" min="0" />
				<RegEx>Type\s:\s+(\w+_MT)</RegEx>
				<InputData action="0" format="" name="Msg Type" />
			</InputLine>
			<InputLine>
				<Cardinality max="2147483647" min="0" />
				<RegEx>Sender\s:\s+(.*)</RegEx>
				<InputData action="0" format="" name="Msg Sender" />
			</InputLine>
			<InputLine>
				<Cardinality max="2147483647" min="0" />
				<RegEx>Recipient\s:\s+(.*)</RegEx>
				<InputData action="0" format="" name="Msg Recipient" />
			</InputLine>
			<InputLine>
				<Cardinality max="1" min="0" />
				<RegEx>Msg nr\s:\s+(.*)</RegEx>
				<InputData action="0" format="" name="Msg Number" />
			</InputLine>
		</InputLine>
		<OutputColumn name="Time Stamp" />
		<OutputColumn name="Process" />
		<OutputColumn name="Description" />
		<OutputColumn name="Msg Type" />
		<OutputColumn name="Msg Sender" />
		<OutputColumn name="Msg Recipient" />
		<OutputColumn name="Msg Number" />
	</Definition>
</CustomTxtTraceDefinitionList>


Re: Creating a custom text parser [message #1728829 is a reply to message #1710325] Thu, 07 April 2016 17:10 Go to previous message
Patrick Tasse is currently offline Patrick TasseFriend
Messages: 53
Registered: July 2009
Member
Hi Mikael,

I'm very sorry I only noticed your message now, I guess we don't monitor the Trace Compass forum very well. I hope this still helps you or anyone else using the custom parsers.

Yes, what you describe is possible using the custom parser wizard. You have to capture both the field name and value from the input log and keep appending to the same output. I also captured the ':' to include a separator between the name and value, and for the name use 'Append with |' to separate the different fields. The resulting definition is included below (only the last InputLine and OutputColumn is added):

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<CustomTxtTraceDefinitionList>
<Definition category="CI Trace" name="CPP Trace">
<TimeStampOutputFormat>MM/dd/yy HH:mm:ss.SSS</TimeStampOutputFormat>
<InputLine>
<Cardinality max="2147483647" min="0"/>
<RegEx>(\d\d/\d\d/\d\d \d\d:\d\d:\d\d\.\d+),\s+(\w+)\s(.*)</RegEx>
<InputData action="0" format="MM/dd/yy HH:mm:ss.SSS" name="Time Stamp"/>
<InputData action="0" format="" name="Process"/>
<InputData action="0" format="" name="Description"/>
<InputLine>
<Cardinality max="2147483647" min="0"/>
<RegEx>Type\s:\s+(\w+_MT)</RegEx>
<InputData action="0" format="" name="Msg Type"/>
</InputLine>
<InputLine>
<Cardinality max="2147483647" min="0"/>
<RegEx>Sender\s:\s+(.*)</RegEx>
<InputData action="0" format="" name="Msg Sender"/>
</InputLine>
<InputLine>
<Cardinality max="2147483647" min="0"/>
<RegEx>Recipient\s:\s+(.*)</RegEx>
<InputData action="0" format="" name="Msg Recipient"/>
</InputLine>
<InputLine>
<Cardinality max="1" min="0"/>
<RegEx>Msg nr\s:\s+(.*)</RegEx>
<InputData action="0" format="" name="Msg Number"/>
</InputLine>
<InputLine>
<Cardinality max="2147483647" min="0"/>
<RegEx>(\S.*_FT) (:) \s*(.*?)\s*</RegEx>
<InputData action="2" format="" name="Msg Content"/>
<InputData action="1" format="" name="Msg Content"/>
<InputData action="1" format="" name="Msg Content"/>
</InputLine>
</InputLine>
<OutputColumn name="Time Stamp"/>
<OutputColumn name="Process"/>
<OutputColumn name="Description"/>
<OutputColumn name="Msg Type"/>
<OutputColumn name="Msg Sender"/>
<OutputColumn name="Msg Recipient"/>
<OutputColumn name="Msg Number"/>
<OutputColumn name="Msg Content"/>
</Definition>
</CustomTxtTraceDefinitionList>


It creates the following output with your example trace message (tab-delimited):

Time Stamp	Process	Description	Msg Type	Msg Sender	Msg Recipient	Msg Number	Msg Content
10/05/15 09:00:01.340	cc_main	Sending message!	CALL_END_MT	-1	-1	10274-9038	CALL_ID_FT:511851 | PBX_CALL_ID_FT:clearit_front-1444028343.106 | REMOTE_ADDRESS_FT:0733906699 | CALLED_ADDRESS_FT:0046774109093


Best regards,
Patrick
Previous Topic:parse, check and count
Next Topic:Live mode in Trace Compass
Goto Forum:
  


Current Time: Wed Sep 19 22:05:45 GMT 2018

Powered by FUDForum. Page generated in 0.01554 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top