A Guide for Creating a Parser and a Correlation Engine

How to create a parser and a correlation engine using the plug-in interfaces

Table of Contents

        Section 1.0 - The Basics

        Section 2.0 - Creating a Parser

               - 2.0 The Log File of 'Simple Parser V1.0'

               - 2.1 Creating the Plug-in

               - 2.2 Writing the Manifest File

               - 2.3 Writing 'MyParser.java'

               - 2.4 Before Self Hosting

               - 2.5 Importing a ‘syslog.log’ File Remotely

               - 2.6 Log Parser Sample

        Section 3.0 - Creating a Correlation Engine

               - 3.0 Creating a Correlation Engine

               - 3.1 Getting Started

               - 3.2 Creating the Plug-in

               - 3.3 Writing the Manifest File

               - 3.4 Writing the Code

               - 3.5 Self Hosting

               - 3.6 Log Correlation Engine Sample

1.0 Creating a Parser - The Basics
(Skip this section if you are familiar with the Log and Trace Analyzer platform.)

The Log and Trace Analyzer (LTA) platform offers developers to define extension points for any development organization. (Much like implementing an interface in Java that may have been originally defined by other developers). Extensions implement extension points and a manifest file defines the extension(s) or extension point(s) of a given plug-in . The extension point that will be of concern to us is defined in "org.eclipse.hyades.logc" with the id "logParser". The diagram below abstractly summarizes the contribution that we will make to the LTA's platform.

Figure 1.0 shows that "custom.myparser" is the plug-in that must be created in order to be able to parse a
desirable log file. This plug-in will extend an extension point of org.eclipse.hyades.logc and extend a class
from org.eclipse.hyades.logging.parsers. The class that will be extended contains generalized methods and data
fields that most parsers require.

Once this plug-in is created, we will be able to import the type of log file for which we have defined a parser. (see figure 1.1)

Figure 1.1 - The import log file dialog box. The created parser is indicated by a red arrow

The role of the parser is to extract the info from a specific log format and map it to the common base event model. Mapping a log file to the CBE model will allow entries to be easily filtered and/or analyzed based on a set of defined symptoms.

2.0 The Log File of 'Simple Parser V1.0'

The format of the log file that we wish to parse is as follows:

- The lines that are to be parsed begin with '##'
- The first component of the record is the severity of the message (i.e. Information, Error, or Warning)
- The second component of the record is the time stamp. The time stamp is in the following format: month.day.year.hour.min.sec. (e.g. 02.23.03.13.11.12 which corresponds to Feb 23, 2003 13:11:12)
- The third component is the text message describing an event.
- The last component is the record ID.

Assume that 'syslog.log' is the filename of the log file. Here's an example of what the log file can look like:

Initialized on Jan 4, 2003
The following errors occurred while attempting to launch the x program...
## <Error> <02.22.03.10.53.22> Missing library file 00000000.000
## <Error> <02.22.03.10.53.22> Missing class 00000001.000
## <Error> <02.22.03.10.53.22> Missing Environmental Variable 00000002.000
## <Error> <02.22.03.10.53.22> Exception 00000003.000
## <Warning> <02.22.03.10.53.22> Connection without a firewall 00000004.000
System idle...
Host names connecting...
host1.domain.com
host2.domain.com
host3.domain.com
## <Information> <02.22.03.13.11.10> First time user: host3.domain.com 00000005.000
## <Information> <02.22.03.13.11.10> Performing security check host3.domain.com 00000005.001
Security check: host3.domain.com...
## <Warning> <02.22.03.13.11.11> host3.domain.com does not meet security requirements 00000005.002
Disconnecting host3.domain.com...
## <Information> <02.23.03.13.11.12> host3.domain.com has been disconnected 00000005.003
System idle...

Only the lines that begin with "##" will appear as a log record.

2.1 Creating the Plug-in

Follow these steps to create the plug-in that will be used to create the parser:

- Start the workbench. (Note: you can start the workbench using another directory for the workspace by running the following command line from the installation directory of 'Log and Trace Analyzer': eclipse -data myWorkspace)
- From the File menu select New > Project
- Choose Plug-in Development, Plug-in Project and click on Next
- Choose 'custom.myparser' as the Project name. Click on Next twice
- Choose the "Create a blank plug-in project" radio button and click on finish

- Right click on the newly created plug-in and choose properties
- Click on 'Java Build Path' and select the libraries tab.
- Click on 'Add External JARs…'
- Include the following library files:

  <x>\plugins\org.eclipse.hyades.logging.core\hlcore.jar
  <x>\plugins\org.eclipse.hyades.logging.core\hlevents.jar
  <x>\plugins\org.eclipse.hyades.logging.parsers\hparse.jar, where <x> is LTA's home directory. Click OK to close the 'properties' dialog box.

- Right click on the plug-in again and select 'Import'
- Select 'File system' and click on next
- Click on browse and select the following directory: <x>\plugins\org.eclipse.hyades.logging.parsers, where <x> is LTA's home directory
- Check mark the 'icons' directory under the 'org.eclipse.hyades.logging.parsers' directory
- Click on Finish.
The icons directory contains an icon that will be used when displaying the parser in the import log list.
Note that you can create your own icons and have them imported using similar means.

- Right click on the plug-in one more time
- Select New > File
- Type 'plugin.properties' for the file name and click on Finish
This file will be used to define variables that the manifest file will use (see next section)

- Use the menu to select 'Window' > 'Preferences'
- Select 'Plug-In Development' > 'Target Platform'
- Click on 'Not In Workspace'. Click on OK to close the preference page.

2.2 Writing the Manifest File

The manifest file defining the extension(s) and extension point(s) of a plug-in is actually a XML file called 'plugin.xml'. You can find this file under the newly created plug-in. Before writing the manifest file, we will first need to define its variables in 'plugin.properties'. This file makes it easier to change the value of an attribute that has been repeatedly used in the manifest file. You are by no means obligated to include this file.

You can always substitute the value of the variables in place of the variable names.

This file appears under the plug-in created. Open it by double clicking on the file.

Here's the content of 'plugin.properties':

# The following is a list of variables used for the 'custom.myparser' plugin

FILE_PATH = file_path # Value of this variable should not be changed

DEFAULT_FILE_NAME = syslog.log

ID_FOR_VERSION = PRODUCT_VERSION

ID_FOR_RELEASE = PRODUCT_RELEASE

We are now prepared to write the manifest file. Open the file and switch to the 'source' tab using the tabs at the bottom of the window that displays the file.
Replace the code with the following:

<?xml version="1.0" encoding="UTF-8"?>

<plugin

        id="custom.myparser"

        name="Simple Parser"

        version="1.2.0">

        <runtime>

        </runtime>

        <requires>

                <import plugin="org.eclipse.hyades.logging.parsers" export="true"/>

                <import plugin="org.eclipse.core.runtime" export="true"/>

        </requires>

        <!-- =========================================================================== -->

        <!-- Simple Parser: Extension Point org.eclipse.hyades.logging.parsers.logParser -->

        <!-- =========================================================================== -->

        <extension

                point="org.eclipse.hyades.logging.parsers.logParser">

                <parser

                       name="Simple Parser V1.0"

                       icon="./icons/full/obj16/apache_error.gif"

                       description="A simple parser"

                       class="MyParser"

                       ui_name="Simple Parser V1.0"

                       id="MyParser">

                        <field

                                useBrowse="true"

                                defaultValue="%DEFAULT_FILE_NAME"

                                name="Enter the absolute path of the log file:"

                                helpContextId=""

                                tooltip="TOOLTIP"

                                id="file_path"

                                browseType="*.log">

                        </field>

                        <field

                                useBrowse="false"

                                defaultValue=""

                                name="Enter the version of the software that was used to generate the log file:"

                                helpContextId=""

                                tooltip="TOOLTIP"

                                id="%ID_FOR_VERSION">

                        </field>

                        <field

                                useBrowse="false"

                                defaultValue=""

                                name="Enter the release of the software that was used to generate the log file:"

                                helpContextId=""

                                tooltip="TOOLTIP"

                                id="%ID_FOR_RELEASE">

                        </field>

                </parser>

        </extension>

</plugin>

Brief Description:
   - The '<require>' replace is used to import the dependencies.
   - One of the attributes of the '<parser>' replace is 'class', which defines the location of the class to be used when parsing the file.
   - The '<extension>' replace is used to define the extension to the point defined in org.eclipse.hyades.logging.parsers.logParser.
   - Note that each replace under the '<extension>' replace is assigned a name and an id.
     The id of the field replaces happens to play an important role. (see the next section)
   - The '<field>' replace is used to define the fields that will appear in the log import wizard (see figure 1.3). The attribute 'ui_type' can be used to specify the type of widgets to include on the dialog box shown in figure 1.3. Currently the following widgets are supported: text_field , combobox, checkbox, radio, and text_area. This optional attribute has text_field as its default value.

The following are some examples of using this attribute:

A check box field:

<field id="checkbox_id"

       name="Check Box Label Value"

       defaultValue="true"

       useBrowse="false"

       tooltip="Check Box ToolTip"

       helpContextId=""

       ui_type="checkbox">

</field>

A combo box with 3 selection items:

<field id="combobox_id"

       name="Combo Box Label Value"

       defaultValue="selection1,selection2,selection3"

       useBrowse="false"

       tooltip="Combo Box ToolTip"

       helpContextId=""

       ui_type="combobox">

</field>

A list of 3 radio buttons:

 <field id="radio_id"

        name="Radio Button List Label Value"

        defaultValue="selection1,selection2,selection3"

        useBrowse="false"

        tooltip="Radio Button List ToolTip"

        helpContextId=""

        ui_type="radio">

</field>

A text area field:

 <field id="text_area"

        name="Text Area Label Value"

        defaultValue="activity.log"

        useBrowse="false"

        tooltip="Text Area ToolTip"

        helpContextId="">

        ut_type="text_area"

</field>

Figure 1.3 The effect of the <field> replace

2.3 Writing 'MyParser.java'

At this point we are ready to write the code that will do the actual parsing of the log file:

- Right click on the 'src' directory under the newly created plug-in
- Select New > Class
- Choose 'MyParser' as the name of the class, and click on Finish
- Before going through the code here's an overview:

The 'MyParser' class will extend org.eclipse.hyades.logging.parsers.Parser, to inherit a few generalized methods.
Any class that extends org.eclipse.hyades.logging.parsers.Parser must implement the following two abstract methods: getName( ) and getVersion( ).

In addition to implementing the above two methods, 'MyParser' also includes the following two methods: parse (org.apache.commons.logging.Log) and setUserInput (Hashtable).

Immediately after a log file has been imported (using the import wizard), the setUserInput (Hashtable)
is invoked. The inputs from the wizard are passed as a hashtable to this method. For example to find what the user has inputted for any particular field one can use the hashtable and the field id as the key to find the value inputted in the field. (see the source)

After setUserInput (Hashtable) the parse (Log) is invoked. This is the method that is responsible for the actual parsing of the log file. The method writes messages (with each message representing a log record) to the Log object (passed as parameter) and everything that is written to Log will be loaded to the workbench.

Here's a diagram summarizing the method invocations:

Here's the documented code for MyParser.java:

import java.text.ParsePosition;

import java.text.SimpleDateFormat;

import java.util.Date;

import java.util.Hashtable;

import org.apache.commons.logging.Log;

import org.eclipse.hyades.logging.events.IComponentIdentification;

import org.eclipse.hyades.logging.events.IReportSituation;

import org.eclipse.hyades.logging.events.ISituation;

import org.eclipse.hyades.logging.parsers.LogParserException;

import org.eclipse.hyades.logging.parsers.Parser;

import org.eclipse.hyades.logging.parsers.ParserUtilities;

/**

* A simple parser created to illustrate how one can create their own

* parsers for a given log file.

public class MyParser extends Parser {

/* The version of the software used to generate the log file */

private String prodVersion = null;

/* The release (eg. Alpha, Beta, etc..) of the software used to generate the log file */

private String prodReal = null;

/* Keys for the product version & the product release */

private final static String VER_KEY = "PRODUCT_VERSION";

private final static String REL_KEY = "PRODUCT_RELEASE";

/* The source ID (which is simply the concatenation of the release and the version of the

software used to generate the log file) */

private String sourceID = null;

/* The creation time, unique ID, severity, and the text field of a log record */

// private StringBuffer creationTime = new StringBuffer("");

/* The following field holds the creationTime of the previous log record */

private String prevCreationTime = null;

/* Store the message count */

int messageCounter = 0;

/* The following method is invoked right after the user clicks on the 'finish' button in the

* import wizard page. This method sets the appropriate fields based on the user's input */

public void setUserInput(Hashtable table) throws LogParserException {

/* The following line of code sets file_path to the path that has been chosen by the user */

super.setUserInput(table);

/* Store the product version and the product release */

prodVersion = (String) table.get(VER_KEY);

prodReal = (String) table.get(REL_KEY);

/* Create the sourceID */

if (prodVersion != null) {

sourceID = prodReal.concat(prodVersion);

}

else {

sourceID = prodReal;

}

/* The following method is invoked right after setUserInput(Hashtable) in order to parse the log file */

public void parse(Log log) throws LogParserException {

/* The following call causes the initialization of a few generalized fields */

super.parse(log);

/* Attempting to parse the log file */

try {

/* Read the log file line by line */

while (curLine != null) {

/* Go to the next line if the current line does not begin with "##" */

if ((curLine.charAt(0) == '#') && (curLine.charAt(1) == '#')) {

/* Initialize the Common Base Event. */

messages[arrayIndex].init();

//Create a new instance of a Source Component:

IComponentIdentification sourceComponentId = eventFactory.createComponentIdentification();

sourceComponentId.setLocation(localHostId);

sourceComponentId.setLocationType(localHostIdFormat);

sourceComponentId.setExecutionEnvironment("Java");

sourceComponentId.setComponent(sourceID);

sourceComponentId.setSubComponent("MyParser)");

sourceComponentId.setComponentIdType("Application");

sourceComponentId.setComponentType("HyadesLoggingSamples");

//Create a new instance of a report situation:

IReportSituation reportSituation = eventFactory.createReportSituation();

reportSituation.setReasoningScope("INTERNAL");

reportSituation.setReportCategory("LOG");

//Create a new instance of a situation:

ISituation situation = eventFactory.createSituation();

situation.setCategoryName("ReportSituation");

situation.setSituationType(reportSituation);

messages[arrayIndex].setSituation(situation);

messages[arrayIndex].setSourceComponentId(sourceComponentId);

/* Assertion: The current line begins with '##' and needs to be parsed */

curLine = curLine.substring(3);

/* Find the severity of the log record */

String severity = curLine.substring(1, curLine.indexOf('>'));

if (severity.trim().equals("Information"))

messages[arrayIndex].setSeverity((short) 10);

else if (severity.trim().equals("Error"))

messages[arrayIndex].setSeverity((short) 50);

else if (severity.trim().equals("Warning"))

messages[arrayIndex].setSeverity((short) 30);

/* Find the text body of the log record */

messages[arrayIndex].setMsg(curLine.substring(curLine.indexOf('>') + 2));

/* Find the creation time of the log record */

String creationTime = findCreationTime();

messages[arrayIndex].setCreationTime(creationTime);

/* Find the message count and the unique ID of the log record */

if (prevCreationTime.equals(creationTime)) {

messageCounter++;

messages[arrayIndex].setSequenceNumber((short) messageCounter);

}

else {

messageCounter = 0;

prevCreationTime = creationTime;

}

/* Write 'messages' to the logger and read the next line */

log.trace(messages[arrayIndex]);

}

curLine = readLine();

}

catch (Throwable throwable) {

ParserUtilities.exceptionHandler(throwable, curLineNumber, curLine, "Error parsing error log.");

}

/* Store the creation time in the desirable format */

private String findCreationTime() {

SimpleDateFormat myFormat;

/* This is the format found in the log file */

myFormat = new SimpleDateFormat("MM.d.yy.kk.mm.ss");

ParsePosition parseLoc = new ParsePosition(0);

StringBuffer creationTime = new StringBuffer();

curLine = curLine.substring(curLine.indexOf('>') + 2);

creationTime.append(curLine.substring(1, curLine.indexOf('>')));

Date date = myFormat.parse(creationTime.toString(), parseLoc);

/* This is the desirable format that we would like creation time to be in

* (The creation time has to be in this format) */

myFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

creationTime = new StringBuffer(myFormat.format(date));

creationTime.replace(10, 11, "T");

creationTime.append(".000000");

return (creationTime.toString().trim());

}

/* Returns the name of this parser */

public String getName() {

return "Simple Parser";

}

/* Returns the version of this parser */

public String getVersion() {

return "V1.0";

}

2.4 Before Self Hosting

Before self-hosting to test the newly created plug-in, ensure that the settings under the preferences is correct: Click on Window > preferences > Plug-in Development > Target Platform, and click on 'Not In Workspace'. Click on OK to close the preferences dialog box.

Once the log file shown previously is imported, the 'Log View' will look similar to the one shown below:

Figure 1.5 - Log View

2.5 Importing a ‘syslog.log’ File Remotely

The Remote Agent Controller (RAC) can be used to import any type of log file remotely from any one of the platforms that the RAC supports. The supported platforms of the RAC include: Windows (NT, 2000, XP), Linux x86, Linux 390, zSeries, iSeries, Solaris, AIX, and HP-UX. This of course provides the convenience of analyzing a log file from a remote machine without having to transfer a copy of the file to a local machine. For example, by installing the RAC on a Linux machine users can use the workbench to import and analyze a log file that is continuously updated by a running server.

Follow the steps below to configure the RAC for importing ‘syslog.log’ remotely:

1. Install the RAC on the remote machine (Please refer to the installation documents included with the RAC).

2.  The parser created needs to be transferred to the remote machine.
     When importing log files remotely, the RAC will use the parser to parse the file and redirect the output to the workbench.
     Transfer ‘MyParser.class’ to the following directory: <x>/lib, where <x> is the RAC home directory.
     You can use a FTP program or an external storage media (e.g. floppy disk) to transfer the file.
     ‘MyParser.class’ can be found under the following directory: <y>\custom.myparser\bin, where <y> is the workspace directory.

3. The RAC is a daemon process that can be controlled using a configuration file.
Open ‘serviceconfig.xml’ file under the following directory: <x>/config, where <x> is the home directory of the RAC.

4.  Locate the following XML fragment:
     <Application configuration="default" executable="RemoteLogParserLoader" location="%JAVA_PATH%" path="%JAVA_PATH%/java">
     This is the application element that handles remote imports of log files.
5.  Add the following attribute to the ‘Application’ element above:
     <Variable name="CLASSPATH" position="prepend" value="%RASERVER_HOME%/lib"></Variable>
     The following variable will ensure that ‘MyParser.class’ is included in the classpath.

6. Save ‘serviceconfig.xml’

7.  We will now need to restart the RAC in order for the new changes to take effect.
      In windows:
         - Click on Start->Settings->Control Panel->Administrative Tools->Services.
         - Locate ‘IBM Agent Controller’
         - Stop the service using the ‘Stop Service’ tool bar button
         - Start the service using the ‘Start Service’ tool bar button

      In other platforms:
         - Switch to the directory: <x>/bin, using the following command: cd <x>/bin, where <x> is the home directory of the RAC
         - Stop the RAC by running the following command: ./RAStop.sh
         - Start the RAC by running the following command: ./RAStart.sh
           In case the RAC fails to start, give it a few minutes before running RAStart.sh again.

Follow the steps below to importing ‘syslog.log’ remotely:

1. Ensure the remote machine configured previously contains a ‘syslog.log’ file

2. Start the workbench on the local host

3. Click on File->Import

4. Select ‘Log File’ and click on Next

5. Click on ‘Simple Parser V1.0’ and click on Next

6. Enter the host name of the remote machine configured previously, and click on ‘Add’

7. Select the host added and click on ‘Test Connection’. Ensure the connection succeeds.

8. Select the host added and click on Next

9. Specify the path of ‘syslog.log’ on the remote machine (e.g. /home/user/syslog.log), an arbitrary version and release.

10. Click on ‘Finish’ for the log file to be imported

11. You may need to refresh the log-view for all the records to be displayed.

2.6 Log Parser Sample

This sample log parser can be found in Log and Trace Analyzer 1.1 and higher.

Follow the steps below to run the Log Parser Sample in the Workbench:

1. Open the New Example wizard (File > New > Example).

2. Select ‘Hyades Logging’ then ‘Log Parser Sample’. Click Next.

3. Specify a project for the Log Parser sample. Click Finish.

4. Click Yes if prompted to switch to the Java perspective.

5. A project is created in the Java perspective for the Log Parser sample. This project contains the necessary jar files and source files for the sample.

6. Before running the Log Parser sample, go to the Target Platform preferences page (Window > Preferences > Plug-In Development > Target Platform) and click on the Not In Workspace button. Click Apply then OK.

7. Now run the Log Parser sample (Run > Run As > Run-time Workbench).

8. Follow the instructions in the Working with the Log Parser Sample readme file to import a log file.

3.0 Creating a Correlation Engine

The following section discusses how a correlation engine can be build for the type of log files associated with 'Simple Parser V1.0'. A correlation engine is generally used for visually displaying correlation between log records of one or more log files using a number of factor(s). For instance, the "Default Time-Based Log Record Correlator" supports all eight default log files (apache error, apache access, and etc…) and uses the creation time as a factor to correlate log records.

The correlation engine that will be created will only support one type of log file (i.e. syslog - the log file associated with "Simple Parser V1.0") and it will use the record ID as a factor to correlate log records. A developer is not limited to the types of log files that their correlation engine can support. Any number of log files can be supported for as long as a parser has been created for that log file.

The record ID is a new field that will be added to the log file. Rather than using one log file, we will use two in order to demonstrate how one can associate multiple files with a correlation engine.

Here's the content of the two log files:

syslog.log:

syslog2.log:

Initialized on Jan 4, 2003
## <Error> <02.22.03.10.53.22> c:\winnt\endc.dll is missing 00000000.001
## <Error> <02.22.03.10.53.22> c:\x\class1.class is missing 00000001.001
## <Error> <02.22.03.10.53.22> WAS_HOME is not set 00000002.001
## <Error> <02.22.03.10.53.22> Null Pointer Exception 00000003.001
System idle...
Performing security checks...
## <Information> <02.22.03.13.11.10> Restoring back-up files 00000006.000
## <Information> <02.22.03.13.11.11> System passed test 00000006.001
System idle...

The records will be correlated based on the first 8 digits of the record ID. For instance the first error message of syslog.log (i.e. Missing library file) will be correlated with the first error message of syslog2.log (i.e. c:\winnt\endc.dll is missing) since both messages have the same 8 digit record ID (i.e. 00000000). The following diagram illustrates how the two log files are correlated using the record IDs:

3.1 Getting Started

The extension point that the correlation engine will implement is defined in the same plug-in as the extension point for the parser (i.e. org.eclipse.hyades.logc). The ID of the extension point is 'logInteractionView'. The class to be written must implement the interface 'IlogRecordCorrelationEngine'. This class will be used to correlate log records.
There also exists another interface called 'IlogRecordFilter' which is used to filter the unwanted log records when displaying the correlation sequence diagram. The method, public EList filter(EList) is used to specify the rules that define the required filtration process. The following are some examples of rules that can be used to filter a given set of log records:
1) Creation time belongs to [a,b]
(i.e. Only log records that have a creation time belonging to the interval of 'a' to 'b' is displayed)
2) Severity level = 1
(i.e. Only log records with a severity level of one is included)
3) Displaying log records that have a text field beginning with the string 'file missing'
A separate class implementing 'IlogRecordFilter' is needed for every type of log file that the correlation engine supports. It is important to note that filtering is optional, and the developer is not obligated to filter the log records before correlation.

3.2 Creating the Plug-in

Follow the steps below to create the plug-in that will be used to create the correlation engine:

- Start the workbench. (Note: you can start the workbench using another directory for the workspace by running the following command line from the installation directory of 'Log and Trace Analyzer': eclipse -data myWorkspace)
- From the File menu select New > Project
- Choose Plug-in Development, Plug-in Project and click on Next
- Choose 'custom.mycorengine' as the Project name. Click on Next twice
- Choose the "Create a blank plug-in project" radio button and click on finish

- Right click on the newly created plug-in and choose properties
- Click on 'Java Build Path' and select the libraries tab.
- Click on 'Add External JARs…'
- Include the following library files:
a) <x>\plugins\org.eclipse.emf.common\runtime\common.jar
b) <x>\plugins\org.eclipse.emf.ecore\runtime\ecore.jar
c) <x>\ plugins\org.eclipse.hyades.models.cbe\cbe-model.jar
d) <x>\plugins\org.eclipse.hyades.models.hierarchy\hmodel.jar
e) <x>\plugins\org.eclipse.hyades.logc\logc.jar

Where <x> is the home directory of 'Log and Trace Analyzer'.

- Right click on the plug-in
- Select New > File
- Type 'plugin.properties' for the file name and click on Finish

3.3 Writing the Manifest File

The following is the content of 'plugin.properties':

# The following is a list of variables used for the 'custom.mycorengine' plugin

CORR_NAME       = Simple Correlation Engine V1.0

CORR_DESC       = Correlates log records using the Log Correlator Sample.

# This variable holds a listing of the log file that this correlation engine support.

# Multiple log files must be separated by a comma.

LOG_TYPES       = Simple Parser V1.0

LOG_TYPE_SIMPLE = Simple Parser V1.0

Double click on 'plugin.xml' under custom.mycorengine and click on the 'source' tab at the bottom of the window. Replace the code with the following:

<?xml version="1.0" encoding="UTF-8"?>

<plugin

        name="Hyades Correlation Engine"

        id="custom.mycorengine"

        version="1.2.0"

        provider-name="Eclipse.org">

        <runtime>

                <library name="corengine.jar">

                </library>

        </runtime>

        <requires>

                <import plugin="org.eclipse.core.runtime"/>

                <import plugin="org.eclipse.emf.common"/>

                <import plugin="org.eclipse.hyades.models.hierarchy"/>

                <import plugin="org.eclipse.hyades.models.cbe"/>

                <import plugin="org.eclipse.hyades.logc"/>

        </requires>

        <!-- ===================================================== -->

        <!-- Contribute to the logInteractionView extension point. -->

        <!-- ===================================================== -->

        <extension

                point="org.eclipse.hyades.logc.logInteractionView">

                <view

                        log_types="%LOG_TYPES"

                        name="%CORR_NAME"

                        description="%CORR_DESC">

                        <LogRecordFilter

                                log_type="%LOG_TYPE_SIMPLE"

                                class="SimpleParserFilter">

                        </LogRecordFilter>

                        <LogRecordCorrelationEngine

                                class="MyCorEngine">

                        </LogRecordCorrelationEngine>

                </view>

        </extension>

</plugin>

Brief Description:
- The value of the 'name', 'description', and the 'log_types' attributes of the '<view>' replace corresponds to the text that appears in the dialog box shown in figure 1.7
- The class attribute of the '<LogRecordCorrelationEngine>' replace is used to identify the class that has implemented the 'IlogRecordCorrelationEngine' interface.
- Similarly the attributes of the '<LogRecordFilter>' replace is used to identify the log type and the associated class that is used to filter the supported log file. As previously mentioned, the class specified must implement the 'IlogRecordFilter' interface.

Figure 1.7 - The attribute contributions

3.4 Writing the Code

Follow the steps below to create the required classes:

- Right click on the 'src' directory under the newly created plug-in
- Select New > Class
- Choose 'MyCorEngine' as the name of the class, and click on Finish
- Repeat the steps again to create a class called: 'SimpleParserFilter'
- Here's an overview of the code for 'MyCorEngine:

In 'MyCorEngine.java' we are required to only implement one method (i.e. correlate (EList logFiles)). This method only has one parameter of type org.eclipse.emf.common.util.EList which is similar to java.util.List. This parameter is used to extract the log records within one or more log files. Figure 1.8 illustrates how this parameter can be used to correlate log records. (The idea behind the use of this parameter applies to all correlation engines)

Figure 1.8 - The generalized idea behind the parameter passed to the 'correlate' method

In short, each element of the 'logFiles' has a list of log records, and each log record has a list of correlation types (or engines), finally each element of the correlation type has a list of log records.

In this example:

- 'logFiles' will consist of two elements, namely syslog.log and syslog2.log.
- Each element of 'logFiles' will consist of a list that holds the log records corresponding to either syslog.log or syslog2.log
- Each of the log records will have a list containing only one element which holds a reference to the correlation type that will be build.
- The correlation type of each log record will consist of other log records that have the same first eight digits in their record ID.

Figure 1.9 and 2.0 further clarifies this idea.

Figure 1.9 - The idea behind the parameter passed to the 'correlate' method

Here's the documented code for MyCorEngine:

/**

* A simple correlation engine created to illustrate how one can make correlations

* between log records.

import org.eclipse.emf.common.util.BasicEList;

import org.eclipse.emf.common.util.EList;

import org.eclipse.hyades.logc.extensions.ILogRecordCorrelationEngine;

import org.eclipse.hyades.logs.correlators.RecordList;

import org.eclipse.hyades.models.cbe.CBECommonBaseEvent;

import org.eclipse.hyades.models.hierarchy.CorrelationContainer;

import org.eclipse.hyades.models.hierarchy.CorrelationContainerProxy;

import org.eclipse.hyades.models.hierarchy.CorrelationEngine;

public class MyCorEngine implements ILogRecordCorrelationEngine {

private CorrelationEngine correlationEngine = null;

private CorrelationContainer correlationContainer = null;

/* The name and type of the simple correlation engine */

private final String CORRELATION_NAME = "Simple Correlation Engine V1.0";

private final String CORRELATION_TYPE = "Correlated";

* Makes the necessary correlation between the log records based up on a number

* of factor(s). Currently the factor that is used to correlate log records is

* the ID of each record included in the log file associated with 'Simple Parser

* V1.0'

* @param list - A listing of the processes (i.e. the log files) for which

* correlation are to be made made.

public void correlate(CorrelationContainerProxy correlationContainerProxy, EList logFiles) {

correlationEngine = correlationContainerProxy.getCorrelationEngine();

correlationContainer = correlationContainerProxy.getCorrelationContainer();

if (correlationEngine == null) {

correlationEngine.setType(CORRELATION_TYPE);

correlationEngine.setName(CORRELATION_NAME);

correlationEngine.setId(CORRELATION_NAME);

}

/* Traverse through each of the log files that a correlation needs to be made */

for (int i = 0; i < logFiles.size(); i++) {

/* For each of the existing log file, traverse through its log records and make the necessary correlations */

if (logFiles.get(i) != null) {

/* Store the list corresponding to the log records of the i-th logFile */

EList recordList = ((RecordList) logFiles.get(i)).getList();

/* Make the necessary correlations */

makeCorrelations(recordList, logFiles, i);

}

} // End of for-loop

} // End of correlator (EList)

* A helper method to 'correlate(EList)' that is used to make the necessary

* correlations between the log records

* @param recordList - A list of log records for which correlations are made

* logFiles - 'logRecord' belongs to a log file listed under this parameter

* logFileIndex - The index of 'logFiles' identifying the log file that

* logRecord belongs to

private void makeCorrelations(EList recordList, EList logFiles, int logFileIndex) {

/* Traverse through each of the log records and make the necessary correlations */

for (int j = 0; j < recordList.size(); j++) {

/* Make the correlation for the j-th log record */

setPartners(recordList.get(j), logFiles, logFileIndex);

} // End of for-loop

} // End of makeCorrelations (EList)

* A helper method to 'makeCorrelations (EList, EList, int)' used to set the partners

* @param logRec - The log record for which partners are set

* logFiles - 'logRecord' belongs to a log file listed under this parameter

* logFileIndex - The index of 'logFiles' identifying the log file that logRecord belongs to

private void setPartners(Object logRec, EList logFiles, int logFileIndex) {

/* The log records are mapped to a Common Base Event */

CBECommonBaseEvent logRecord = (CBECommonBaseEvent) logRec;

/* The list of records that will be checked */

EList recordList = null;

/* The correlators list and the correlated records of 'logRecord' */

EList correlators = null;

/* Get the record ID of the passed log record */

String recordID = getRecordID(logRecord);

/* Traverse through the logFiles (starting from index 'logFileIndex') and make the proper correlations*/

for (int i = logFileIndex; i < logFiles.size(); i++) {

/* The record list of the i-th logFile */

recordList = ((RecordList) logFiles.get(i)).getList();

/* Traverse through the record list of the i-th log file and make the proper correlations */

if (recordList != null) {

for (int j = 0; j < recordList.size(); j++) {

if (recordList.get(j) != null && recordList.get(j) != logRec && getRecordID(recordList.get(j)).equals(recordID) && recordList.get(j) != logRec) {

addCorrelation((CBECommonBaseEvent) logRec, (CBECommonBaseEvent) recordList.get(j));

}

} // End of the j-th loop

}

} // End of the i-th loop

} // End of setPartners (Object, EList, int)

private EList addCorrelation(CBECommonBaseEvent artifact, CBECommonBaseEvent associtatedEvent) {

EList correlations = (EList) correlationContainer.getCorrelations().get(artifact);

if (correlations == null) {

correlations = new BasicEList();

correlations.add(associtatedEvent);

correlationContainer.getCorrelations().put(artifact, correlations);

}

else {

correlations.add(associtatedEvent);

}

return correlations;

}

/* Returns the ID associated with a log record from the 'syslog' log file (i.e. The log file that is associated

* with the 'Simple Parser V1.0' */

private String getRecordID(Object logRecord) {

String textField = ((CBECommonBaseEvent) logRecord).getMsg();

return textField.substring(textField.lastIndexOf(' '), textField.lastIndexOf('.'));

}

} // End of MyCorEngine class

Here's the content of 'SimpleParserFilter':

/**

* The following class is a filteration for the Simple Parser log file.

import org.eclipse.emf.common.util.EList;

public class SimpleParserFilter implements org.eclipse.hyades.logc.extensions.ILogRecordFilter

{

/* Does not perform any filteration. It returns the list 'as is'. */

public EList filter(EList list)

{ return list; }

}

The following describes the architecture of the sampled correlation engine.

The first method that is called is of course the correlate( ) method. The 'logFiles' parameter of this method is a list that has each element modeling a log file. The entries of this list can be modeled as a tree such as the following:

3.5 Self Hosting

Follow the steps below to test the correlation engine in a self-hosting environment:

- Go to the 'Profiling' perspective and view the 'Profiling Monitor' if it is not displayed. Using the pop-up menu in the 'Profiling Monitor' select 'Import'.
- Select 'Log File' and click on Next. Select 'Simple Parser V1.0' and click on Next. Select 'localhost' and click on Next. Specify the path of 'syslog.log', any arbitrary version, and release. Click on Finish
- Repeat the same steps to import 'syslog2.log'
- Show the 'Hosts' of the profiling monitor tree, by clicking on the menu (circled below) and clicking on Show > Hosts

- Select the host and using the pop-up menu click on Show View > Log interactions. Select "Simple Correlation Engine V1.0" and click on OK
- The sequence diagram should look similar to Figure 2.1. Use the text hover as shown in the figure to view the messages of the log records

Figure 2.1 - The log interaction sequence diagram

3.6 Log Correlation Engine Sample

This sample log correlation engine can be found in Log and Trace Analyzer 1.1 and higher.

Follow the steps below to run the Log Correlator Sample in the Workbench:

1. Open the New Example wizard (File > New > Example).

2. Select ‘Hyades Logging’ then ‘Log Correlator Sample’. Click Next.

3. Specify a project for the Log Correlator sample. Click Finish.

4. Click Yes if prompted to switch to the Java perspective.

5. A project is created in the Java perspective for the Log Correlator sample. This project contains the necessary jar files and source files for the sample.

6. Before running the Log Correlator sample, go to the Target Platform preferences page (Window > Preferences > Plug-In Development > Target Platform) and click on the Not In Workspace button. Click Apply then OK.

7. Now run the Log Correlator sample (Run > Run As > Run-time Workbench).

8. Follow the instructions in the Working with the Log Correlator Sample readme file to import a log file and correlate the log records.

Copyright © 2003 Hyades project.
All rights reserved. This program and the accompanying materials
are made available under the terms of the Common Public License v0.5
which accompanies this distribution, and is available at
http://www.eclipse.org/legal/cpl-v05.html

Written by: Ali Mehregani