Creating a log parser using the Generic Log Adapter

     1.0 Overview of the Generic Log Adapter
     2.0 Creating a log parser
          2.1 Setting up a log parser plug-in project
          2.2 Adapter Configuration File structure
          2.3 Creating a rules-based adapter
               Creating adapter configuration file
               Configuring the context instance
               Configuring the sensor component
               Configuring the extractor component
               Configuring the parser component
               Configuring the outputter component
               Substitution rule examples
          2.4 Creating a static adapter
               Configuring the adapter context for a static parser
               Writing a static parser class
               Configuring the sensor component for a static parser
               Configuring the outputter component for a static parser
          2.5 Deploying the log parser
          2.6 Testing the log parser in the Log and Trace Analyzer
     3.0 Configuring an adapter for live application monitoring
     Reference
          Common Base Event format
          Adapter Configuration Editor
          Limitations of the Adapter Configuration Editor
          Regular expression grammar

1.0 Overview of the Generic Log Adapter

The Generic Log Adapter allows you to process application log files and transform their contents into the Common Base Event format. The Common Base Event specification prescribes a common format for logging, management, problem determination, and autonomic computing. It provides a consistent format that facilitates intercommunication between tools that support these goals. Common Base Event objects allow you to develop a common prescriptive event in a consistent format so that tools can be developed to support these goals.

In order to take advantage of tools such as the Log and Trace Analyzer that support Common Base Event objects, your application log files need to conform to the Common Base Event format. To produce application log records that conform, you can do one of the following:

The first approach is impractical unless the application is currently being developed. This approach does not solve the problem of integrating existing applicatin logs with the Common Base Event format.

The second approach is more practical. However, it leaves you with the tedious task of writing the transform. Typically, this is accomplished by either writing some custom code in either a third-generation programming language or a script that can be used by established scripting tools such as Perl. Once the transform is created, you must test the transform and integrate the transform solution into the tools that use Common Base Event objects.

The Generic Log Adapter simplifies the adoption of the Common Base Event format. The Adapter Configuration Editor and the runtime are components provided by the Generic Log Adapter that will help you create and test transforms quickly. The editor allows you to write the transforms using Java or script fragments using regular expressions (such as Perl) to describe the mapping of log file content to Common Base Event attributes. You can also test your script fragments as you write the regular expression scripts in the editor. The runtime takes as inputs the rules that you have written and your log file and produces Common Base Event objects as outputs. The Generic Log Adapter allows you to directly integrate your transform with the Log and Trace Analyzer tool that is used to analyze Common Base Event records.

2.0 Creating a log parser

Enterprise information systems of all kinds generate various types of log files such as access logs, activity logs and error logs. These log files usually contain log records constructed in proprietary formats. The lack of a common convention for logging makes cross-system analysis such as event correlation very difficult.

With the aid of the Generic Log Adapter framework, programmers can contribute static parsers and parsers based on pre-defined rules to the Import wizard in the Log and Trace Analyzer tool. In a rule-based parser, an adapter configuration file containing parsing rules expressed in regular expression grammar is provided to the Generic Log Adapter engine. In a static parser, the adapter configuration file contains a reference to a Java class that performs the log file parsing.

To create a log parser:

  1. Set up a log parser plug-in project
  2. Create an adapter configuration file for one of the following parsers:
  3. Configure a plug-in manifest file
  4. Test the log parser

2.1 Setting up a log parser plug-in project

Before you can create a log parser, you must create the plug-in project and configure the project build path.

Creating the plug-in project

  1. In the Navigator view, right-click and select New > Project. The New Project wizard opens.
  2. Select Plug-in Project and click Next.
  3. On the New Plug-in Project page, type the project name and select the directory where the project contents will be stored and click Next.
  4. On the Plug-in Project Structure page, under Project Settings, select Create a Java project radio button. Specify the name you want to use for the different components of your project. Click Next. The Plug-in Content window opens.
  5. On the Plug-in Content page, Specify a value for Provider Name. Change the other default values as required. Click Finish. The plug-in project is created. (You will be asked to switch to the plug-in perspective.)

Setting up the Java build path for the plug-in project

In order to use the Generic Log Adapter framework, the following plug-ins need to be included in your plug-in project:

If you have the Hyades source code in your workspace, include the above plug-ins in the Java build path of your plug-in project.

  1. In the Package Explorer, select your plug-in project name, right-click and select Properties. (Alternately, use the ALT-ENTER keys.) The Properties wizard opens.
  2. Select the Projects tab.
  3. Under Required projects on the build path, select the projects listed above. Click Okay.

If you do not have the Hyades source code, include the runtime libraries of the above plug-ins in your plug-in project's Java build path.

  1. In the Package Explorer view, select your plug-in project name, right-click and select Properties. (Alternately, use the ALT-ENTER keys.) The Properties window opens.
  2. Select the Libraries tab.
  3. Select the Add External Jars button. Navigate to the directory where Hyades is installed and add the runtime libraries of the above plug-ins as required. Click OK.

The next step in creating a log parser is to create an adapter configuration file. There are two types of adapters that can be created; rules-based adapters and static parser adapters.

 

2.2 Adapter Configuration File structure

An adapter configuration file contains a collection of contexts that are to be run by a Generic Log Adapter runtime instance. Each context describes how to process a single log file. Because the adapter configuration file can to contain many of these contexts, the adapter can process many log files at the same time.

Each context associated with a particular log file is constituted of a series of components. Each component has a role to play in processing a log file.

Context

The context describes the ordered grouping of components as they are chained together for log file processing. Each log file has a separate context, and multiple contexts can be defined in a single configuration. The Generic Log Adapter provides one context:

A context has the following properties:

Property Type Behavior
Description string A text description for the context. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The context refers to this string to identify which component instance to use.
Is Continuous Operation boolean Whether or not this log file should be monitored for changes based upon the Maximum Idle Time and Pause Interval properties.
Maximum Idle Time integer (milliseconds) The maximum amount of time the context will wait for a log file to change before shutting itself down.
Pause Interval integer (milliseconds) The amount of time the context waits after reaching the end of the log file before trying to read again.

Sensor

The sensor provides the mechanism to read the content for processing. The Generic Log Adapter provides the following sensors:

The following table lists the attributes that are common among the sensor types.

Sensor attributes

Attribute name Type Behavior
Sensor Type String The sensor type to be used by the adapter.
Description (required) String A text description for the sensor. This value is used only by the Adapter Configuration Editor.
Unique ID (required) String A unique identifier for this component. The sensor refers to this string to identify which component instance to use.
Maximum Blocking (required) Integer The maximum number of lines the sensor reads before forwarding to the extractor for processing. For the StaticParserSensor type, this field indicates the maximum number of Common Base Event objects created by the static parser before forwarding to the outputter for processing.
Confidence Buffer Integer The last n bytes of a file to be saved and used for determining if an append has occured to the file since the last time it was read.
Footer size Integer The n bytes of a file that contains variable content. The footer portion of a file will not be used when determining if an append has occured since the last time the file was read.

The following table lists the properties that are specific for each type of sensor. The following properties are known by the component code and must be spelled exactly as indicated in the table for them to be recognized and used by the component.

Sensor type Properties Application
SingleFileSensor
directory (required)
The directory location of the file to be read by the SinlgeFileSensor.
fileName (required)
The name of the file to be read by the SingleFileSensor.
shell
The UNIX shell to run the converter command in. Used only on UNIX platforms
converter
A command line for a preprocessing instruction to run before starting the sensor.
Used when creating an adapter for parsing a log file with regular expression rules
StaticParserSensor
directory (required)
The directory location of the file to be read by the StaticParserSensor.
fileName (required)
The name of the file to be read by the StaticParserSensor.
parserClassName (required)
the name of the Java class that implements the static parser. The class must implement the org.eclipse.hyades.logging.parsers.IParser interface.
Used when creating an adapter for parsing a log file with a static parser Java class.

Extractor

The extractor takes a collection of input lines provided by the sensor and separates them into message boundaries.

The Generic Log Adapter provides the following extractors:

The following properties are provided as part of the configuration for the extractor:

Property Type Behavior
Description string A text description for the extractor. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The extractor refers to this string to identify which component instance to use.
Contains Line Breaks boolean Indicates whether the messages in the log file span multiple lines.
Replace Line Breaks boolean Replace the line breaks in this file by another string segment. The replacement is made before the extractor searches for Start Pattern or End Pattern.
Line Break Symbol string The string segment to replace line breaks with. This is used only if Replace Line Breaks is true.
Include Start Pattern boolean Whether to include the StartPattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
Start Pattern string or regular expression The string segment that identifies the start of a message. You can always use the RegularExpressionExtractor class. However, if your start pattern is a simple string, the SimpleExtractor class incurs less overhead.
Include End Pattern boolean Whether to include End Pattern as part of the message to be forwarded to the next component. If the value is false, the matching portion of the message is stripped.
End Pattern string or regular expression The string segment that identifies the end of a message. If End Pattern is not specified, then the message is delimited by the next Start Pattern.

Parser

The parser takes the messages that have been delimited by the extractor and builds a set of mappings of string values to a data structure. In the case of the Generic Log Adapter the data structure is Common Base Event 1.0.1

The Generic Log Adapter provides the following parser class:

The parser has two phases of execution:

  1. The global processing phase, where a set of global regular expressions are executed against the message provided by the extractor.
  2. The attribute processing phase, where specific sets of substitution rules are executed to determine the value to be assigned to each attribute in the data structure (Common Base Event 1.0.1).

The two phases of execution allow the message to be tokenized into a series of attribute values during the global processing phase. The attribute values can then be referred to by attribute name or index during the attribute processing phase.

The following global properties are specified as part of the parser:

Property Type Behavior
Description string A text description for the parser. This value is used only by the Adapter Configuration Editor.
Unique Id string A unique identifier for this component. The parser refers to this string to identify which component instance to use.
Separator Token string or regular expression A token, or a regular expression that describes a series of tokens, that breaks the message up into attribute-value pairs.

For example, if each new line of your log file contains an attribute and its value, you could specify a Separator Token of \n. The advantage of breaking the message up is that the string segments can be referred to in the attribute processing phase.

Designation Token string or regular expression A token, or a regular expression that describes a series of tokens, that breaks the string segments created by the Separator Token into attribute-value pairs. For example, if each attribute and value are separated by a = sign , and the value of Designation Token is = , the attribute and its value can be referred to in the attribute processing phase. If a separator token is specified but a designation token is not specified then the message is broken up into a list of attribute values which can be referred to in the substitution rules by index.

The individual attributes of the Common Base Event have the following properties:

Property Type Behavior
Use Previous Match Substitution As Default boolean If the current set of substitution rules for this attribute returns a positive match, then store the match value so that it can be used as a default to process the next record if there is no positive match.
Default value String If none of the substitution rules for this attribute provide a positive match or there are no substitution rules for this attribute then this attribute will be assigned the default value specified here. If the same value is desired for all records for a particular attribute, this property can be used instead of creating a default substitution rule for that attribute. Note, the Use Previous Match Substitution as Default will take precedence if there is a saved value available.

Substitution rules are run as part of the attribute processing phase. They are processed in order from top to bottom. Once a rule is a match, no further rules are run against this property. If no match is found, then no value is placed in the attribute unless the Use Previous Match Substitution as Default or Default Value properties of the attribute are specified.

Substitution rules contain the following properties:

Property Type Behavior
Match regular expression The regular expression that identifies the value to be applied to this attribute if a match occurs in the scope of the string segment being processed by this Substitution Rule. If you do not provide a regular expression, the rule is considered a match.
Positions name-value attribute pairs Positions let you refer to the name-value attribute pairs created during the global processing phase.
  • If both a separator token and a designation tokens are specified, the Positions string can refer to many attributes by name using the following grammar: $h('Attr1')@@$h('Attr2')...
    $h('Attrn') identifies a previously created attribute name.
    where @@ is the separator between the attributes.
  • If a separator token is specified but not a designation token, then the Positions string can refer to one or more attributes by index instead of name using the following grammar: n1@@n2..
    where n1 and n2 are integers representing the indicies. The index starts at 1. If the message string begins with a separator token, the attribute value at index 1 will be null. The first attribute value will begin at index 2.

In the Match property, you can add the separator to indicate which regular expressions should be applied to which attribute value. The Substitute property can refer to these attribute atoms using $1, $2, and so on..

Substitute string The value to place in the attribute. This can be a string or it can refer to the regular expression in the Match property. The string segments that are extracted by the Match property can be referred to using the groupings in the regular expression. $1 indicates the first grouping, $2 the second, and so on.
Use built in function check box Use built-in function when selected for an attribute, if none of the previous substitution rules, if any exist, are matched, will cause the Generic Log Adapter runtime to generate a value for that attribute using its own internal functions. This feature is only supported the followingn CBE attributes:
  • localInstanceId
  • globalInstanceId
  • componentIdentification.location
  • componentIdentification.locationType
Substitution Class string Specifies the name of a user created Java class that can perform custom parsing logic on the value matched by the Positions, Match and Substitute specifications of the rule. If the substitution rule matches data in the record, this class will be called and the result will be used to create the Common Base Event object.
Specifications:
  • This class must implement the interface class org.eclipse.hyades.logging.adapter.parsers.ISubstitutionExtension or the extension class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension in order for it to be executed by the Generic Log Adapter.
  • Use the extension class when you want to save state data between log file records. The class org.eclipse.hyades.logging.adapter.parsers.IStatefulSubstitutionExtension causes the Generic Log Adapter to only allocate one instance of this class to parse all records of the log file.
  • The user class must be included in the classpath of the Generic Log Adapter runtime.
  • If the Generic Log Adapter is running in an Eclipse environment, the class can be created as an eclipse plug-in extension of the Generic Log Adapter by using the substitutionExtension extension point of the org.eclipse.hyades.logging.adapter plugin.
Time Format string This field only applies to substitution rules for the creationTime element and is used to simplify date time parsing. For example, some log files contain timestamps that include the names of a month instead of its numerical representation. This makes it difficult to create substituion rules to generate the string value in the form of the XML Schema dateTime data type required by the CBE creationTime element. In this case the user can specify a java simple date format string as documented in the javadoc for java.text.SimpleDateFormat (java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html) in the Time Format field. The value matched by the Positions, Match and Substitute specifications of the rule will be parsed based on the Time Format string and the resulting value will be used to generate the CBE creationTime value.

Formatter

The formatter takes the mappings of attributes to their values provided by the parser and builds the correct Java object instance. The adapter runtime contains a single formatter that creates Common Base Event objects conforming to the Common Base Event version 1.0.1 specification.

The formatter contains the following properties:

Property Type Behavior
Unique Id string A unique identifier for this component. The formatter refers to this string to identify which component instance to use.
Description string A text description for the formatter. This value is used only by the Adapter Configuration Editor.

Outputter

The outputter externalizes the resulting Common Base Event records provided by the formatter. Outputters provide, or wrap, the mechanism for storing the final outputs of the context. The Hyades Generic Log Adapter provides the following outputters:

Properties of the outputter:

Property Type Behavior
Unique Id string A unique identifier for this component. The outputter refers to this string to identify which component instance to use.
Description string A text description for the outputter. This value is used only by the Adapter Configuration Editor.
Outputter Type enumeration The type of outputter this is.

Outputter types and required properties:

Outputter type Required properties
StandardOutOutputter None
SingleFileOutputter directory, fileName
LoggingAgentOutputter agentName - name given to the logging agent when it is created
waitUntilLoggingTime(optional)- the time the outputter will wait for the user to attach and start monitoring the agent before it starts writing to the agent. If this property is not specified, the outputter will not wait for the agent to be monitored before writing to it.
NotificationOutputter resourceURL - URL of the Notification Sink which will receive the events

 

2.3 Creating a rules-based adapter

Prerequisites:
Setting up a log parser plug-in project for a log parser

A rule-based adapter uses regular expression rules to parse log file information and formats this information into the Common Base Event format. This type of adapter can usually be used for application log files that have a fixed log record format. The following illustrates the steps required in creating a rules-based adapter.

  1. Create the adapter configuration file.
  2. Configuring the context instance.
  3. Configure the sensor component.
  4. Configure the extractor component.
  5. Configure the parser component.
  6. Configure the outputter component.

Once you have completed the above steps, you can proceed to configure your plug-in project's manifest file to use your rule-based adapter.

Creating an adapter configuration file

Prerequisites:
Set up a log parser plug-in project

The Generic Log Adapter uses adapter configuration files to parse and covert log files to the Common Base Event format. To create an adapter configuration file:

  1. Create a folder in your plug-in project that will contain the file.
  2. Select File > New > Other. The Select page of the New wizard opens.
  3. Expand the Generic Log Adapter folder and select Generic Log Adapter File. Click Next.
  4. On the Generic Log Adapter File page, select the folder to contain the adapter configuration file.
  5. Enter a name for the adapter configuration file in the File name field. The adapter configuration file will have the extension .adapter. Optionally, click the Advanced button to link to a file in your file system.
  6. Select the XML encoding to use for your adapter configuration file in the XML Encoding field. Although you will view the adapter configuration file in the Adapter Configuration Editor, the file is actually an XML file.)
  7. Click Next. to display the Template  Log  File page.
  8. On the Template Log File page, select a log file which will be used to help define the parser rules.
  9. Click Finish to create the adapter configuration file. The new adapter configuration file is listed in the Navigator view and the file is opened in the Adapter Configuration Editor.

Once the adapter configuration file is created, you need to configure the context instance.

Configuring the context instance for a rules-based adapter

Prerequisites:
Create an Adapter Configuration File

In the Adapter Configuration File, the context instance describes the ordered grouping of components as they are chained together for log file processing. Each log file has a separate context, and multiple contexts can be defined in a single configuration. To configure the context instance, follow these steps:

  1. In the left pane of the Adapter Configuration Editor, expand the node Configuration. Select Context Instance. The right pane displays the Context Instance information.
  2. Select Continuous operation if you want the log file to be monitored for changes continuously. If continuous operation is selected, specify the Maximum Idle Time and Pause Interval values to be used for monitoring. If continuous operation is not specified the Maximum Idle Time and Pause Interval values will be ignored.
  3. Type the maximum amount of time you want the context to wait for a log file to change before it is shut down in the Maximum Idle Time field.
  4. Type the amount of time the context should wait after reaching the end of a log file before trying to read again in the Pause Interval field.
  5. Save the adapter configuration file changes.

Changes in the adapter configuration file are saved in the plug-in project you created. For details of the adapter configuration file structure, please see the related reference topics.

 

Configuring the sensor component for a rules-based parser

Prerequisites:

Creating an adapter configuration file

The role of the sensor is to read the log content for processing. To configure the sensor for a rules-based parser, follow these steps:

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Sensor. The right pane displays the sensor information.
  2. In the Sensor type list, select the SingleFileSensor.
  3. Type a description of your sensor in the Description field.
  4. In the Maximum Blocking field, type the number of lines you want the sensor to process before passing the information to the extractor.
  5. Some log files append a footer to the end of the log file. This part of the log file is not considered log data, and needs to be excluded when monitoring a log file for changes. You can specify the log file footer size in bytes in the File Footer Size field. The default size is 90 bytes.
  6. If a log file is being monitored in continuous operation, the last n bytes of the log file is stored in memory. This stored data, called a confidence buffer, is compared to the last n bytes of the log file being monitored to check whether or not the log file has changed. You can specify the size of the confidence buffer in the Confidence Buffer Size field. The default size is 1024 bytes.
  7. The the directory and fileName properties were automatically generated when you created the adapter configuration file. These properties correspond to the log file directory and file name. To update these properties, follow these steps:
    1. Select Configuration > Context Instance > Sensor > Property directory.
    2. Edit the property value.
    3. Save the adapter configuration file to apply the changes.
    4. Repeat steps a to c for the file name property.
  8. Two optional properties that can be added are the converter and shell properties. To configure these properties:
    1. Select Configuration > Context Instance > Sensor.
    2. Right-click on the sensor to bring up the pop-up menu. Select Add > Property. The right pane refreshes with fields to enter the property name and value.
    3. Type the name and value of the property.
    4. Save the adapter configuration file to apply the changes.
  9. To test that your log file is being loaded by the sensor, click the Re-run adapter  Picture of the Rerun adapter icon icon to run the adapter. The results will be shown in the Sensor Result view.

Refer to the Sensor properties table for an explaination of the properties required for the sensor component of a rule-based parser.

Once the sensor is configured, you can continue on to configure the extractor component.

Configuring the extractor component for a rules-based parser

Prerequisites:

In a rules-based log parser, the role of the extractor is to take a collection of input lines provided by the sensor and separate these into message boundaries. To configure the extractor, you need to first determine the extractor class to use. Then, you can configure the Extractor rules based on the selected extractor class.

Configuring the Extractor class

The default extractor class used is a regular expression extractor. You can change the extractor class by modifying the Extractor component in the Contexts section of the Adapter Configuration file.

  1. In the Adapter Configuration Editor, expand the node Contexts > Contexts Basic Context Implementation. The Basic Context Implementation details are shown in the right pane.
  2. Select Component Regular Expression Extractor. The Extractor properties are displayed in the right pane.

    the extractor name, description executable class are listed in the properties pane

  3. Update the executable class to the class you want to use. For details, see the Configuration file structure reference topic for the extractor classes provided by the Generic Log Adapter.
  4. Save the adapter configuration file changes.

Configuring the Extractor rules

Once you determine which extractor class to use, you can write corresponding rules for the extractor in the context instance.

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Extractor. The right pane displays the extractor information.

    Sensor Configuration

  2. If the log file contains records that span multiple lines in the file, select the Contains line breaks check box. Do not select this check box if each line in the log file is a separate record or if there are no line breaks in the log file.
  3. If you want to replace the line breaks with an alternative symbol, select Replace line breaks, and type the symbol in the Line Break Symbol field.
  4. Write a script fragment in the Start Pattern field to identify the start pattern. The default regular expression is the caret ^ symbol which is a null token matching the beginning of a string or line.
  5. To include the start pattern in the result, check the Include Start Pattern check box.
  6. Write a script fragment in the End Pattern field to identify the end pattern. The default regular expression is the dollar $ symbol which is a null token matching the end of a string or line.
  7. To include the end pattern in the result, select the Include End Pattern check box.
    Hint: To help you with identifying the start and end patterns, click the Re-run  icon Picture of the Rerun adapter icon, then click the Show log Show log icon icon. The log content that has been read by the sensor is shown in the Sensor Result view.
  8. Save the adapter configuration file to apply changes.
  9. To test your rules, click the Re-run adapter  Picture of the Rerun adapter icon icon to run the adapter. The results will be shown in the Extractor Result view.
  10. Click the Show Next Event  Picture of the Show Next Event icon icon to see each of the log records that have been processed by the extractor rules you have written.

    Image of the extractor rules, and the output results in the Extractor Result view with start pattern omitted and the default end pattern used.

For a detailed description of extractor properties, see the Adapter Configuration File structure reference document.

The next step in creating a rule-based adapter is to configure the parser component.

Configuring the parser component for a rules-based parser

Prerequisites:

The role of the parser is to take the messages that have been delimited by the extractor and build a set of mappings of string values to the Common Base Event attributes.

To configure the parser component, you need to configure rules for the two levels of parser execution:

Configuring rules for global processing

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Parser. The right pane displays the parser information.

    Parser Configuration details displayed in right pane

  2. Type the description of the parser in the Description field.
  3. If log records contain separator tokens that delimit the elements within a record, type the token in Separator Token field. For example, for the myapp application log record, the separator token between the elements is a double space:
    <March 14, 2004 10:25:21 EST>  application:myapp  severity:3  message:WARNING-file style.css missing
    The separator token can be specified with the regular expression [ ]{2}.
  4. If the log records contain elements in a name-value pair pattern, you can use specify the token that associates the name with the value in the Designation Token field. Using the same example as above, in the myapp log record, the designation token is a colon : .

Configuring rules for attribute processing

The second phase of execution in the parser applies specific substitution rules to identify which values to assign to the Common Base Event attributes.

Adding and configuring Common Base Event attributes

The following Common Base Event attributes are required:

For a complete listing of Common Base Event attributes, please see Common Base Event Model.

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Parser > Common Base Event. The right pane displays the Common Base Event information.
  2. To add a new attribute, right-click on the Common Base Event node, and select the attribute you want to add from the pop-up menu. The attribute is created beneath the Common Base Event node. In the example below, the msg attribute is added.

    The msg attribute is shown in the right pane

  3. To configure the attribute you just added, click on the attribute. The attribute details will be displayed in the right pane.
  4. Using the above example, if the log records do not all contain the msg attribute, you can select the Use Previous Match Substitution As Default check box. This will assign the value of the last matched message to the msg attribute.
  5. If you want to set a default value for the msg, type the value in the Default Value field. This default value will be used only if no substitution rules exist or match and the Use Previous Match Substitution As Default is not checked or if it is checked, there is no previous match value saved.
  6. Continue to add all required attributes following the steps above.
Adding and configuring attribute substitution rules

To add a substitution rule to the Common Base Event attribute, select the Attribute > Add > Substitution Rule. The substitution rule details are displayed in the right pane of the editor.

  1. The Positions field can be used when the log record contains name value pairs delimited by a token. The value of a name value pair can be extracted using the atom $h('nameofNameValuePair').
    For example, using the same log record from myapp application:
    <March 14, 2004 10:25:21 EST>  application:myapp  severity:3  message:WARNING-file style.css missing
    with the separator token = []{2} and the designation token = : , the atom to extract the message would be $h('message').
    For other applications of the Positions field, please see the Positions field usage topic.
  2. The Match specifies a pattern to search for in the record. If the Positions field is specified, a part of the record will be searched, otherwise the whole record will be searched for the specified pattern. Within this pattern, certain data can be grouped together using parantheses which can be referred to in the Substitute field. If the pattern matches data in the record then the Common Base Event attribute this rule is for will be given a value based on the Substitute field. If it is left empty, the rule will be considered a match.
    In the myapp log record example, to match the value of the message extracted by the positions field, the following regular expression can be used:
    ^(\w+)-(.*)
    This match pattern contains two groups. The first group is a set of word characters before a dash. The second group is everything after the dash to the end of the message value as defined by the Positions field specification.
  3. The Substitute field is the value to be assigned to the Common Base Event attribute as a result of a match. The values can be extracted from the matched string using the literals $1, $2, etc that represent the groups defined in the match pattern.
    In the myapp log record example, if you just want to assign the value of the warning to the Common Base Event msg attribute, specify the substitution as $2.

    Example of log record that is parsed and the result shown in the formatter results view

  4. The Substitution Extension Class can be used if you want to run custom parsing logic on the value matched by the Positions, Match and Substitute fields. The user class must implement the Generic Log Adapter interface
    classorg.eclipse.hyades.logging.adapter.parsers.ISubstitutionExtension 
    in order for it to be executed by the Generic Log Adapter.

  5. The Time Format field can be used to create a substitution rule for the creationTime attribute. Specify a Java time pattern string in this field. The pattern entered will be used to parse the value matched by the Positions, Match and Substitute fields using the java.text.SimpleDateFormat class to generate the date as a long value. This value is then used to generate the string value of the date in the XML Schema dateTime format required by Common Base Event.
    In the myapp log record example, the Time Format rule will be as follows:

    Sample rule for the time format field using	the Java simple date format class

  6. Apply the changes made by saving the adapter configuration file.
  7. To test your rules, click the Rerun Adapter Picture of the Rerun adapter icon to run the adapter. The results will be shown in the Formatter Result view.
  8. Click the Show Next Event Picture of the Show Next Event icon icon to view the Common Base Event attributes generated using the rules you have written.

Note: For more information on the Java simple date format, see the javadoc java.sun.com/j2se/1.4.2/docs/api/java/text/SimpleDateFormat.html.

Once the parser rules are completed, you can configure the outputter component of the adapter configuration file.

Configuring the outputter component for a rules-based parser

Prerequisites:

The outputter writes Common Base Event records provided by the formatter to an external source. The formatter is the component that takes the output provided by the parser and builds the correct Java object instance. The adapter runtime contains only a single pre-configured formatter that creates Common Base Event objects. To configure the outputter:

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Outputter. The right pane displays the outputter information.
  2. Select a type from the Outputter type list. In the example below, LoggingAgentOutputter is selected. This outputter writes the resulting Common Base Event records to a logging agent that can be monitored in the Profiling and Logging perspective to view the log data.

    Outputter Configuration

  3. To add a new property, follow these steps:
    1. Select Configuration > Context Instance > Outputter.
    2. Right-click to bring up the pop-up menu.Select Add > Property. The right pane refreshes with fields to enter the property name and value.
    3. Type the name and value of the property.
    4. Save the adapter configuration file to apply the changes.
    Refer to Outputter types and required properties to view the required properties for each outputter type.
  4. If the Outputter type is changed in the Configuration Context instance, the Outputter component in the Contexts node needs to be updated as well.  Select Contexts > Context Basic Context Implementation > Component Hyades Logging Agent Outputter . The details of the outputter component are shown in the right pane.

    Outputter component of the context is opened in the right pane

  5. Update the Name, Description and Executable Class for the Outputter.
    Refer to the Configuration file structure reference to obtain the correct name of outputter classes to use.
  6. Save the adapter configuration file changes.

Once the outputter is configured, you need to configure the plug-in manifest file for the log parser.

Substitution rule examples

Using the Positions, Match and Substitution fields

The following examples illustrate how to use the Positions, Match and Substitution fields of a substitution rule.

  1. Substitution rule to retreive msg attribute value with separator token and designation token specified.

    Log record:
    <March 14, 2004 10:25:21 EST> application:myapp severity:3 instance:01 message:WARNING-file style.css missing

    Rule specifications:
    Separator Token= []{2}
    Designation Token= :
    Postions=$h('message')
    Match=(.*)
    Substitution=$1


    Resulting value assigned to the msg atribute:
    WARNING-file style.css missing

  2. Substitution rule to retreive msg attribute value with only separator token specified.

    Log record:
    <March 14, 2004 10:25:21 EST>;myapp;3;01;WARNING-file style.css missing

    Rule specifications:
    Separator Token=;
    Designation Token=
    Postions= 5
    Match= (.*)
    Substitution= $1

    Resulting value assigned to the msg atribute:
    WARNING-file style.css missing

  3. Substitution rule to retreive msg attribute value with only separator token specified.

    Log record:
    <March 14, 2004 10:25:21 EST>;myapp;3;01;WARNING;file style.css missing

    Rule specifications:
    Separator Token=;
    Designation Token=
    Postions= 5@@6
    Match= (.*)@@(.*)
    Substitution= $1-$2


    Resulting value assigned to the msg atribute:
    WARNING-file style.css missing

Using the Time Format field

The following examples illustrate how to use the Time Format field of a substitution rule.

  1. Substitution rule to retreive creationTime attribute value with no separator or designation token specified.
    Log record:
    <March 14, 2004 10:25:21 EST> myapp 3 01 WARNING file style.css missing

    Rule specifications:
    Separator Token=
    Designation Token=
    Postions=
    Match= ^<(.*)>.*
    Substitution= $1
    Time Format= MMMM dd, yyyy hh:mm:ss z


    Resulting value assigned to the creationTime atribute:
    2004-03-14T15:25:21.000Z

2.4 Creating a static adapter

Prerequisites:
Set up a plug-in project for a log parser

A static adapter uses a Java class to parse log file information and format it into the Common Base Event format. Its configuration file can consist of a sensor, and an outputter component. To create a static adapter:

  1. Create the adapter configuration file.
  2. Configure the adapter context and configuration.
  3. Write the Java parser class.
  4. Configure the sensor component.
  5. Configure the outputter component.

Once you have completed the above steps, you can proceed to configure your plug-in project's manifest file to use your static adapter.

Configuring the adapter Contexts and Configuration for a static adapter

Prerequisites:
Create an Adapter Configuration File

The static parser uses a Java class to parse a log file. The Extractor, Parser and Formatter components in the Configuration Context Instance are replaced by custom Java code. The corresponding components in the Contexts section need to removed. To configure the adapter configuration file for a static adapter:

  1. Expand the node Contexts > Contexts Basic Context Implementation. The Basic Context Implementation details are shown in the right pane.
  2. Select Component Regular Expression Extractor,Component Generic Parser and Component CBE Formatter.
  3. Right-click and select Delete. The three components are removed from the Context.
  4. Expand the node Configuration > Context Instance.
  5. Select the Extractor, Parser and Formatter components.
  6. Right-click and select Delete. The three components are removed from the configuration.

    The extractor, parser and formatter components are removed from the context and the configuration

The next step in creating a static log parser is to write the Java class for the static parser.

Writng a Java class for the static adapter

In creating a static log parser, the extractor, parser and formatter components of the adapter configuration file are replaced with a Java class. The static parser class must implement the interface org.eclipse.hyades.logging.parser.IParser. This can be done by extending the abstract class org.eclipse.hyades.logging.parser.Parser. The following is the standard format required by the Java class.

Once you have written and tested your Java parser class, you can configure the sensor component of the adapter configuration file to refer to this class. .

Configuring the sensor component for a static adapter

Prerequisites:

The role of the sensor is to provide the means to read the content for processing. In a static parser, a property is added to the sensor to indicate that following the sensor processing, it should call a custom Java parser class. To configure the sensor:

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Sensor. The right pane displays the sensor information.
  2. Select StaticParserSensor from the Sensor type list.
  3. In the details pane, type a description of your sensor in the Description field.
  4. In the Maximum Blocking field, type the number of Common Base Event objects that are created by the static parser sensor before they are passed to the outputter. This value indicates how many Common Base Event objects the parseNext() method of the static parser class should return. This value is passed to the static parser class as the array_size parameter by the setConfiguration() method of the static parser class.
  5. For the static adapter, the fields Confidence Buffer Size and File Footer Size do not apply to the StaticParserSensor type.
  6. The directory and fileName properties are automatically generated when the adapter configuration file is created. These correspond to the log file directory and file name. To update these properties:
    1. Select Configuration > Context Instance > Sensor > Property fileName.
    2. Edit the property value.
    3. Save the adapter configuration file to apply the changes.
    4. Repeat the same steps for the directory property.
  7. Add the parserClassName property.
    1. Select Configuration > Context Instance > Sensor.
    2. Right-click to bring up the pop-up menu.Select Add > Property. The right pane refreshes with fields to enter the property name and value.
    3. Type parserClassName as the property name and the name of your Java parser class as the property value.
    4. Save the adapter configuration file to apply the changes.
  8. You can add other properties that contain configuration information that can be passed to the static parser Java class. For example, the log file version.
  9. Since you changed the Sensor Type, you need to update the corresponding sensor component under Contexts.
    1. Select Contexts > Context Basic Context Implementation > Component OS File Sensor and select the sensor component.
    2. Update the Name and Description. Type org.eclipse.hyades.logging.adapter.config.sensors.StaticParserSensor as the Executable class.

      The sensor	component in the context is opened and the details are shown in the details pane.

    3. Save the adapter configuration file to apply the changes.

Refer to the Configuration file structure for a summary of the properties required for the sensor component of a static parser.

Once the sensor is configured, you can procced to configure the outputter component.

Configuring the outputter component for a static adapter

Prerequisites:

The outputter writes the resulting Common Base Event records provided by the formatter to an external source. The formatter is the component that receives the output provided by the parser and builds the correct Java object instances. The adapter runtime contains only a single pre-configured formatter that creates Common Base Event objects. The outputter can be configured as follows for a static adapter:

  1. In the left pane of the Adapter Configuration Editor, select Configuration > Context Instance > Outputter. The right pane displays the outputter information.
  2. Select undeclared from the Outputter type list. There is no defined type for a static parser outputter. Update the description field to indicate it is a static parser outputter

    Outputter Configuration

  3. The static parser outputter does not require any additional properties. To remove any existing outputter properties, follow these steps:
    1. Select Configuration > Context Instance > Outputter.
    2. Select a property and right-click to bring up the pop-up menu. Select Delete. The property is removed from the Outputter.
    3. Save the adapter configuration file to apply the changes.
  4. Select Contexts > Context Basic Context Implementation > Component Hyades Logging Agent Outputter . The details of the outputter component are shown in the right pane.

    Outputter component of the context is opened in the right pane

  5. Update the Name, and Description. Update the Executable Class to org.eclipse.hyades.logging.adapter.config.outputters.StaticParserOutputter.
    Note: Please refer to the Configuration file structure reference to obtain the correct name of outputter classes to use.
  6. Save the adapter configuration file changes.

Once the outputter is configured, you need to configure the plug-in manifest file for the log parser.

2.5 Deploying the log parser plug-in project

Prerequisites

After you create a plug-in project to contain the artifacts for a log parser, you need to configure the plug-in so that it can be used by the Log and Trace Analyzer. Follow these steps:

  1. Add plug-in dependencies to the plug-in's manifest file
  2. Add an extension point for the log parser to the plug-in's manifest file
  3. Create a static wrapper parser class
  4. Verify the log parser plug-in settings
  5. Packaging your log parser plug-in for deployment

Add plug-in dependencies

The following plug-in dependencies must be included in the plug-in manifest file. Open the plugin.xml file in a text editor and add the following lines after the runtime element:

   
<requires>
 <import plugin="org.eclipse.ui"/>
 <import plugin="org.eclipse.core.runtime.compatibility"/>
 <import plugin="org.eclipse.hyades.logging.adapter"/>
 <import plugin="org.eclipse.hyades.logging.parsers"/>
 <import plugin="org.eclipse.hyades.logging.adapter.config"/>
</requires>

Add a log parser extension point

You need to define an extension point for each type of application log file that you want to parse and import into the Log and Trace Analyer. Below is a sample extension point for the MyApp application with both a rules adapter and static adapter defined:

 <extension
         point="org.eclipse.hyades.logging.parsers.logParser">
	      <parser
            name="Sample Rules Adapter for MyApp"
            icon=""
            description="%STR_MYAPP_PARSER_DESCRIPTION"
            class="RulesParser.StaticParserExtension"
            ui_name="MyApp myapp.log file"
            id="org.eclipse.hyades.logging.parsers.MyAppLogParser">  
         <field
               useBrowse="true"
               defaultValue="d:\temp\sample.log"
               name="Directory"
               helpContextId=""
               tooltip="%STR_MYAPP_TOOLTIP1"
               id="file_path"
               browseType="*.log">
         </field>         
         <field
               useBrowse="false"
               defaultValue="MyApp 1.0(rules), MyApp 1.0(static)"
               name="Supported versions"
               helpContextId=""
               tooltip="%STR_MYAPP_TOOLTIP2"
               ui_type="combobox"
               id="version">
         </field>
         <parserParameter
               name="MyApp 1.0(rules)"
               value="./MyAdapter/myadapter.adapter">
    </parserParameter>
	<parserParameter
               name="MyApp 1.0(static)"
               value="./MyAdapter/mystaticadapter.adapter">
         </parserParameter>
      </parser>
   </extension>

To customize the extension point for your log parser, the updates need to be made:

  1. Replace RulesParser in this extension point with the name of the plugin you created for the log parser plugin project.
  2. Replace MyAdapter in this extension point with the folder name in your plugin that contains your adapter files.
  3. The <parser> tag specifies high-level information for the parser and it is a required element. Replace the following attributes:
  4. The first <field> tag defines the UI field for selecting the location of the log file. It is a required element. Replace the following attributes:
  5. The second <field> tag defines the UI Combo box for selecting the version of the log file to be imported. It is a required element. Replace the following attributes:
  6. The <parserParameter> tag defines which Adapter Configuration file to use for each log file version supported. Replace the following attributes:
Hint: A Default should be defined in case a match for a version cannot be found. If you only have one adapter configuration file then you only need to have the following Default parserParameter:
<parserParameter
   name="Default"
   value="./MyAdapter/myadapter.adapter">

The fields specified above will create an option for MyApp myapp.log in the Import Log wizard as shown below:

Import Log wizard showing the myapp myapp.log as an option to import

You can use a plugin.properties file to define certain properties in the plugin.xml file that may need to have different versions. For example, if there are text strings that will be included on the Import Log File wizard that need to be translated to different languages, they can be defined in the plugin.properties file and you can include different properties files for the languages you want to support. In the plugin.properties file you define substitution variables that can be used in the plugin.xml file. The plugin.properties file for the above plugin.xml would look like this:

# Properties for RulesParser Plugin
pluginName = RulesParser
providerName = MyCompany
# logParser extension point message(s):
STR_MYAPP_PARSER_DESCRIPTION = MyApp rules parser v1.0
STR_MYAPP_TOOLTIP1           = Enter the location of the log file
STR_MYAPP_TOOLTIP2           = Select the version of the log file to import

Create a wrapper class for the parser

Create a static wrapper class that extends the org.eclipse.hyades.logging.adapter.config.StaticParserWrapper. This class is used for both static and rules-based parsers. Create the class in the log parser plug-in project. You can use the example class below by replacing RulesParser with the name of your log parser plug-in project.

/*
 * Created on Apr 12, 2004
 * StaticParserExtension class created to be used in RulesParser Plug-in
 */
package RulesParser;
import org.eclipse.hyades.logging.adapter.config.StaticParserWrapper;
/**
 * @author developer
 * StaticParserExtension class
 */
public class StaticParserExtension extends StaticParserWrapper {
	public StaticParserExtension(){
		super();
		currentPlugin="RulesParser"; 
	}
}

Verify the log parser plug-in settings

To verify that you have correctly configured the plug-in manifest file, you can run your plug-in project in a new run-time workbench. Follow these steps:

  1. Switch to the Plug-in perspective by selecting Window > Open perspective > Other > Plug-in Development.
  2. From the Plug-in Development perspective, select your plug-in project.
  3. From the toolbar, select Run > Run As > Run-time Workbench.
  4. In the new workbench, from the menu, select File > Import.
  5. In the Import wizard, select Log File and click Next.
  6. On the Log Files page, click Add to add a log file. In the Add Log File window, verify that your new log file type is included in the Selected log file list.
  7. Select your log file type. Verify that all of the fields and text on the Details tab is correct.
  8. Specify the value for the log file location.
  9. Click OK.
  10. Click Finish to import the log file of your new log file type. Verify that the log file records are displayed in the Log View.

Packaging your log parser plug-in for deployment

To deploy your log parser plug-in to an eclipse workbench you need to package the plug-in files by exporting them to a zip file. Follow these steps:

  1. Open the plugin.xml file in a text editor.
  2. Verify that it includes a runtime library jar file for your plug-in class files. For example:
    	<runtime>
    		<library name="parsers.jar">
    			<export name="*"/>
    		</library>
    	</runtime>
    	
  3. To deploy your log parser to a remote host so that you can import log files from that remote host, you must create an Agent Controller plug-in configuration file for your log parser. Follow these steps:
    1. Create a folder in your log parser plug-in called config.
    2. Create a file called pluginconfig.xml in the config folder. For example:
      <?xml version="1.0" encoding="UTF-8"?>
      <PluginConfiguration>
      <Application configuration="default" 
          executable="RemoteLogParserLoader" 
          extends="default" 
          location="%RASERVER_HOME%\plugins\RulesParser_1.0.0" 
          path="%JAVA_PATH%">
      <Variable name="CLASSPATH" 
          position="prepend" 
          value="%RASERVER_HOME%\plugins\RulesParser_1.0.0\parsers.jar"/>
      </Application>
      <Option name="RulesParser" type="version" value="1.0.0"/>
      </PluginConfiguration>
      
    3. Replace RulesParser with the name of your log parser plug-in project.
  4. Open the build.properties file in a text editor.
  5. Add the folder where your adapter configuration files are stored and the plugin.properties file to the Binary Build list. Also add the config folder if you will be deploying on a remote host. For example here is an example build.properties file:
    bin.includes = MyAdapters/,\
    config/,\
    plugin.xml,\
    plugin.properties,\
    parsers.jar
    source.parsers.jar = src/
    output.parsers.jar = bin/
    
  6. Export the plug-in files by selecting File > Export.
  7. In the Export wizard, select > Deployable plugins and fragments.
  8. Click Next.
  9. On the Export Plugins and Fragments page, select your plug-in in the Available Plugins and Fragments list.
  10. Select deploy as a single deployable ZIP file in the Export Options field.
  11. Specify the destination File name for the zip file.
  12. Click Finish.

You can deploy your log parser plug-in by unzipping the zip file you just created in the plugin directory of an eclipse installation. You are now ready to test your newly created log parser plug-in using the Log and Trace Analyzer.

 

2.6 Testing the log parser in the Log and Trace Analyzer

Prerequisite

To test your log parser, open the Log and Trace Analyzer tool.

  1. Select File > Import.... The Import wizard opens to the Select page.
  2. From the list of import source types, select Log File. Click Next. The Import Log File page opens.
  3. Add log files to be imported by clicking Add.
  4. Select the type of log from the Selected log files list. The value of the ui_name variable you specified in the plugin.xml file for your log adapter should appear here.
  5. Click the Host tab.
  6. Select either the local host or a remote host from the Default Hosts list. Click the Details tab.
  7. Type the path of the file being imported.
  8. Select the version details of the server that created the log. Click the Destination tab.
  9. Input the destination project in the Project field and the name for the grouping logical unit in the Monitor field.
  10. Specify whether the destination is an existing log file.  If it replaces an existing file or appends to one, follow these steps:
    1. Select the check box Merge the log file with an existing log file in the workspace.
    2. Select an existing log file for the merge. Click OK.
  11. If the log you are importing already exists in the workspace, a pop-up window will appear with the options to append to the log file or to replace the log file. Choose Replace or Append. Click OK. The imported file is then opened in the Log view.

By default, all the details for the imported log files will be saved in a Default Log Set, which allows you to easily re-import the files. You can create new log sets, or open and edit existing log sets.

Note:  Ensure the version of the log you are importing matches the application version selected in the Details tab. A mismatched version number will cause the wrong parser to be used when the log is imported. The Log Navigator view will list the log, but the Log view will be empty.

3.0 Configuring an adapter for live application monitoring

The Generic Log Adapter provides an outputter class that allows the Common Base Event objects generated by your adapter to be sent to a logging agent. The Log and Trace Analyzer can be used to attach to the logging agent, to monitor and to analyze the application log in real time.

Prerequisites:

Please follow these steps to configure your adapter for monitoring.

Update the Outputter component

Note: When a new adapter configuration file is created, it is configured to use the CBELogOutputter class by default. If you have not changed this setting, you can skip this procedure.

The outputter component in the adapter configuration file needs to be updated to use the CBELogOutputter class which writes the Common Base Event objects to a logging agent. You can either update the existing Outputter Component or add a new Outputter component.

To add a new outputter component:

  1. To edit the adapter configuration file, select the adapter configuration file in the Navigator pane. Right-click and select Open with > Adapter Configuration Editor. The file is opened in the editor.
  2. In the Adapter Configuration Editor window, expand the Adapter > Configuration > Context Instance node.
  3. Right-click on the Context Instance and select Add > Outputter. A new outputter is created in the Context Instance.
  4. In the Outputter Type field, select LoggingAgentOutputter.
  5. Click on the agentName property. The property details are shown in the right pane.
  6. Update the property value with the desired logging agent name. This is the name that you will see in the Log and Trace Analyzer as the Logging Agent to monitor.
  7. If you want the outputter to wait before logging information to the CBELogOutputter class, add a property called waitUntilLoggingTime and set the value of the property to the time in milliseconds. To add the property, right-click on the newly created Outputter and select Add > Property. A new property is created under the new Outputter node.
  8. To enable the CBELogOutputter class, you need to create the associated Outputter component in the Contexts Basic Context Implementation. Open Contexts > Context Basic Implementation Context.
  9. Right-click on Context Basic Implementation Context and select Add > Component. A new component is added to the context.
  10. Configure the new component with the following values:
    Name: Outputter name in the context
    Description: Description of what this outputter does
    Executable class: org.eclipse.hyades.logging.adapter.outputters.CBELogOutputter
    Logging level:The level of internal logging the outputter component should use.
    A value of 99 means the outputter will not log any messages.
    A value of 0 means the outputter will log messages of all severities.
    The recommended value is 50 which means that only Critical or Fatal internal outputter errors messages are logged.
    Role: Outputter
    UniqueID: Must point to the outputter defined in the Context Instance in step 2. Use the Browse button to select the correct ID.
  11. Save the adapter configuration file changes.

Update the context instance

If you want the log parser to be monitor the log file continuously, you need to update the context instance properties of the adapter configuration file for continuous operation.

  1. To edit the adapter configuration file, select the adapter configuration file in the Navigator perspective. Right-click and select Open with > Adapter Configuration Editor. The file is opened in the editor.
  2. In the editor window, select Adapter > Configuration > Context Instance . The Context Instance details are displayed in the right pane.
  3. Select the Continuous Operation check box.
  4. Configure the Maximum Idle Time and Pause Interval fields as required.
  5. Save the adapter configuration file changes.

Launch the Java process to run the adapter

The log adapter you created can be run externally from a batch file in the Windows environment or a shell script in non-Windows environments. Below are sample files that can be used to launch the Java process that will run your log adapter.
Note: To run the Generic Log Adapter on a system, Hyades must be installed on that system.

Windows batch file:

@rem USAGE:  runregex
@rem Change the following lines to point to the directory where you unzipped the runtime
@rem and the directory where your log adapter resides 
SET HYADES_HOME=d:\hyadesDC
SET YOUR_ADAPTER_DIRECTORY=MyAdapterDirectory
SET CONFIG_FILE=%HYADES_HOME%\%YOUR_ADAPTER_DIRECTORY%\myadapter.adapter
PUSHD %HYADES_HOME%\%YOUR_ADAPTER_DIRECTORY%
java -classpath "%HYADES_HOME%\lib\glaconfigs.jar;%HYADES_HOME%\lib\hexr.jar;%HYADES_HOME%\lib\hgla.jar;%HYADES_HOME%\lib\hglaconfig.jar;%HYADES_HOME%\lib\hlcore.jar;%HYADES_HOME%\lib\hlevents.jar;%HYADES_HOME%\lib\hparse.jar;;%HYADES_HOME%\lib\logparsers.jar;%HYADES_HOME%\lib\xalan.jar;" org.eclipse.hyades.logging.adapter.Adapter -ac "%CONFIG_FILE%" -cc "%CONFIG_FILE%"
POPD
non-Windows batch file:
#!/bin/sh
# USAGE:  runregex_example.sh
# Change the following line to point to the directory where you unzipped the runtime
HYADES_HOME=/home/hyadesDC
export HYADES_HOME
YOUR_DIRECTORY=/yourdirectory
export YOUR_DIRECTORY
CONFIG_FILE=$HYADES_HOME/$YOUR_DIRECTORY/myadapter.adapter
export CONFIG_FILE
CUR_DIR=`pwd`
export CUR_DIR
cd $HYADES_HOME/$YOUR_DIRECTORY
java -classpath "$HYADES_HOME\lib\glaconfigs.jar;$HYADES_HOME\lib\hexr.jar;$HYADES_HOME\lib\hgla.jar;$HYADES_HOME\lib\hglaconfig.jar;$HYADES_HOME\lib\hlcore.jar;$HYADES_HOME\lib\hlevents.jar;$HYADES_HOME\lib\hparse.jar;;$HYADES_HOME\lib\logparsers.jar;$HYADES_HOME\lib\xalan.jar;" org.eclipse.hyades.logging.adapter.Adapter -ac "$CONFIG_FILE" -cc "$CONFIG_FILE"
cd $CUR_DIR
Configure the file as required to launch the Java process that will run your adapter.

Monitoring the live application log

You are now ready to monitor the live application log.

  1. Ensure that logging is enabled in the preferences.
    1. From the main toolbar, select Window > Preferences . The Preferences wizard opens.
    2. Select Profiling and Logging from the left pane. The Profiling and Logging preferences are loaded in the right pane.
    3. Select Enable logging under Profiling and Logging options.

      Enable logging check box is selected in the preferences pane for Profiling and Logging

    4. Select OK to apply the changes.
  2. In the Profiling and Logging perspective, click the Profiling Icon of profiling tool icon in the toolbar and select Profile. The Profile wizard opens.
  3. Select Attach - Java Process and click the New button at the bottom of the list. A new Java process configuration is created.
  4. Click the Hosts tab and select the host on which the application log resides.
    Note: To monitor a live application log, the Hyades Data Collection Engine must be running on the same system as the application.
  5. Click the Agents tab and use the arrow keys to select the logging agent that you configured.

    In the profiling preferences wizard, the logging agent the user created is on the left pane

  6. Click Apply.
  7. Click Run to attach to the Logging agent.
  8. In the Profiling and Logging perspective in the Profiling Monitor view, select the logging agent.
  9. Right-click on the agent and select Start monitoring.
  10. Right-click on the agent and select Open with > Log View. The log events from the application log are loaded into the Log view.

 

Configuring an adapter for live application monitoring

The Generic Log Adapter provides an outputter class that allows the Common Base Event objects generated by your adapter to be sent to a logging agent. The Log and Trace Analyzer can be used to attach to the logging agent, to monitor and to analyze the application log in real time.

Prerequisites:

Please follow these steps to configure your adapter for monitoring.

Update the Outputter component

Note: When a new adapter configuration file is created, it is configured to use the CBELogOutputter class by default. If you have not changed this setting, you can skip this procedure.

The outputter component in the adapter configuration file needs to be updated to use the CBELogOutputter class which writes the Common Base Event objects to a logging agent. You can either update the existing Outputter Component or add a new Outputter component.

To add a new outputter component:

  1. To edit the adapter configuration file, select the adapter configuration file in the Navigator pane. Right-click and select Open with > Adapter Configuration Editor. The file is opened in the editor.
  2. In the Adapter Configuration Editor window, expand the Adapter > Configuration > Context Instance node.
  3. Right-click on the Context Instance and select Add > Outputter. A new outputter is created in the Context Instance.
  4. In the Outputter Type field, select LoggingAgentOutputter.
  5. Click on the agentName property. The property details are shown in the right pane.
  6. Update the property value with the desired logging agent name. This is the name that you will see in the Log and Trace Analyzer as the Logging Agent to monitor.
  7. If you want the outputter to wait before logging information to the CBELogOutputter class, add a property called waitUntilLoggingTime and set the value of the property to the time in milliseconds. To add the property, right-click on the newly created Outputter and select Add > Property. A new property is created under the new Outputter node.
  8. To enable the CBELogOutputter class, you need to create the associated Outputter component in the Contexts Basic Context Implementation. Open Contexts > Context Basic Implementation Context.
  9. Right-click on Context Basic Implementation Context and select Add > Component. A new component is added to the context.
  10. Configure the new component with the following values:
    Name: Outputter name in the context
    Description: Description of what this outputter does
    Executable class: org.eclipse.hyades.logging.adapter.outputters.CBELogOutputter
    Logging level:The level of internal logging the outputter component should use.
    A value of 99 means the outputter will not log any messages.
    A value of 0 means the outputter will log messages of all severities.
    The recommended value is 50 which means that only Critical or Fatal internal outputter errors messages are logged.
    Role: Outputter
    UniqueID: Must point to the outputter defined in the Context Instance in step 2. Use the Browse button to select the correct ID.
  11. Save the adapter configuration file changes.

Update the context instance

If you want the log parser to be monitor the log file continuously, you need to update the context instance properties of the adapter configuration file for continuous operation.

  1. To edit the adapter configuration file, select the adapter configuration file in the Navigator perspective. Right-click and select Open with > Adapter Configuration Editor. The file is opened in the editor.
  2. In the editor window, select Adapter > Configuration > Context Instance . The Context Instance details are displayed in the right pane.
  3. Select the Continuous Operation check box.
  4. Configure the Maximum Idle Time and Pause Interval fields as required.
  5. Save the adapter configuration file changes.

Launch the Java process to run the adapter

The log adapter you created can be run externally from a batch file in the Windows environment or a shell script in non-Windows environments. Below are sample files that can be used to launch the Java process that will run your log adapter.
Note: To run the Generic Log Adapter on a system, Hyades must be installed on that system.

Windows batch file:

@rem USAGE:  runregex
@rem Change the following lines to point to the directory where you unzipped the runtime
@rem and the directory where your log adapter resides 
SET HYADES_HOME=d:\hyadesDC
SET YOUR_ADAPTER_DIRECTORY=MyAdapterDirectory
SET CONFIG_FILE=%HYADES_HOME%\%YOUR_ADAPTER_DIRECTORY%\myadapter.adapter
PUSHD %HYADES_HOME%\%YOUR_ADAPTER_DIRECTORY%
java -classpath "%HYADES_HOME%\lib\glaconfigs.jar;%HYADES_HOME%\lib\hexr.jar;%HYADES_HOME%\lib\hgla.jar;%HYADES_HOME%\lib\hglaconfig.jar;%HYADES_HOME%\lib\hlcore.jar;%HYADES_HOME%\lib\hlevents.jar;%HYADES_HOME%\lib\hparse.jar;;%HYADES_HOME%\lib\logparsers.jar;%HYADES_HOME%\lib\xalan.jar;" org.eclipse.hyades.logging.adapter.Adapter -ac "%CONFIG_FILE%" -cc "%CONFIG_FILE%"
POPD
non-Windows batch file:
#!/bin/sh
# USAGE:  runregex_example.sh
# Change the following line to point to the directory where you unzipped the runtime
HYADES_HOME=/home/hyadesDC
export HYADES_HOME
YOUR_DIRECTORY=/yourdirectory
export YOUR_DIRECTORY
CONFIG_FILE=$HYADES_HOME/$YOUR_DIRECTORY/myadapter.adapter
export CONFIG_FILE
CUR_DIR=`pwd`
export CUR_DIR
cd $HYADES_HOME/$YOUR_DIRECTORY
java -classpath "$HYADES_HOME\lib\glaconfigs.jar;$HYADES_HOME\lib\hexr.jar;$HYADES_HOME\lib\hgla.jar;$HYADES_HOME\lib\hglaconfig.jar;$HYADES_HOME\lib\hlcore.jar;$HYADES_HOME\lib\hlevents.jar;$HYADES_HOME\lib\hparse.jar;;$HYADES_HOME\lib\logparsers.jar;$HYADES_HOME\lib\xalan.jar;" org.eclipse.hyades.logging.adapter.Adapter -ac "$CONFIG_FILE" -cc "$CONFIG_FILE"
cd $CUR_DIR
Configure the file as required to launch the Java process that will run your adapter.

Monitoring the live application log

You are now ready to monitor the live application log.

  1. Ensure that logging is enabled in the preferences.
    1. From the main toolbar, select Window > Preferences . The Preferences wizard opens.
    2. Select Profiling and Logging from the left pane. The Profiling and Logging preferences are loaded in the right pane.
    3. Select Enable logging under Profiling and Logging options.

      Enable logging check box is selected in the preferences pane for Profiling and Logging

    4. Select OK to apply the changes.
  2. In the Profiling and Logging perspective, click the Profiling Icon of profiling tool icon in the toolbar and select Profile. The Profile wizard opens.
  3. Select Attach - Java Process and click the New button at the bottom of the list. A new Java process configuration is created.
  4. Click the Hosts tab and select the host on which the application log resides.
    Note: To monitor a live application log, the Hyades Data Collection Engine must be running on the same system as the application.
  5. Click the Agents tab and use the arrow keys to select the logging agent that you configured.

    In the profiling preferences wizard, the logging agent the user created is on the left pane

  6. Click Apply.
  7. Click Run to attach to the Logging agent.
  8. In the Profiling and Logging perspective in the Profiling Monitor view, select the logging agent.
  9. Right-click on the agent and select Start monitoring.
  10. Right-click on the agent and select Open with > Log View. The log events from the application log are loaded into the Log view.

 

Common Base Event format specification

For details on the Common Base Event format specification, refer to the latest version of Canonical Situation Data Format: The Common Base Event v1.0.1 at www.eclipse.org/hyades.

Adapter Configuration Editor

The Adapter Configuration Editor enables you to write rules for mapping your log file content to the Common Base Event format.

The following figure shows an adapter configuration file in the editor. The editor provides two panes:

Picture of the Adapter Configuration Editor

Editing icons

The following table lists icons in the Adapter Configuration Editor:

Icon Description
Picture of the Adapter icon Adapter
Picture of the Context icon Context
Picture of the Contexts icon Contexts
Picture of the Component icon Component
PIcture of the Configuration icon Configuration
Picture of the Context instance icon Context instance
Picture of the Sensor icon Sensor
Picture of the Sensor outputter stream icon Sensor/outputter stream
Picture of the Extractor icon Extractor
Picture of the Parser icon Parser
Picture of the Common Base Event elements icon Common Base Event elements
PIcture of the Nestings icon Nestings
Picture of the Attributes icon Attributes
PIcture of the Process unit icon Process unit
Picture of the Formatter icon Formatter
Picture of the Outputter icon Outputter

Limitations of the Adapter Configuration Editor

Maximum of ten events are shown in the Extractor Result view

The Extractor Result view in the Generic Log Adapter perspective will only show a maximum of ten events at any one time. If you select the Show last event icon, the tenth event of the current set of 10 events loaded in memory will be displayed, not the last event of the log file. If you select the Show first event icon, the first event of the current set of 10 events loaded in memory will be displayed, not the first event of the log file.

There is no workaround for this limitation.

 

Regular expression grammar

The Hyades Adapter Configuration Editor allows you to use regular expressions to describe how log files should be transformed into Common Base Event records. The following tables are a guideline to regular expression usage.

General rules

Regular expression matching

Expression Matches
{n,m} at least n but not more than m times
{n,} at least n times
{n} exactly n times
* 0 or more times
+ 1 or more times
? 0 or 1 times
. everything except \n in a regular expression within parentheses
^ a null token matching the beginning of a string or line (i.e., the position right after a newline or right before the beginning of a string) in a regular expression within parentheses
$ a null token matching the end of a string or line (that is, the position right before a newline or right after the end of a string) in a regular expression within parentheses
\b backspace inside a character class ([abcd])
\b null token matching a word boundary (\w on one side and \W on the other)
\B null token matching a boundary that isn't a word boundary
\A only at beginning of string
\Z only at end of string (or before newline at the end)
\ newline
\r carriage return
\t tab
\f formfeed
\d digit [0-9]
\D non-digit [^0-9]
\w word character [0-9a-z_A-Z]
\W non-word character [^0-9a-z_A-Z]
\s a whitespace character [ \t\n\r\f]
\S a non-whitespace character [^ \t\n\r\f]
\xnn the hexadecimal representation of character nn
\cD the corresponding control character
\nn or \nnn the octal representation of character nn unless a backreference.
\1, \2, \3 ... whatever the first, second, third, and so on, parenthesized group matched. This is called a backreference. If there is no corresponding group, the number is interpreted as an octal representation of a character.
\0 the null character. Any other backslashed character matches itself .
*? 0 or more times
+? 1 or more times
?? 0 or 1 times
{n}? exactly n times
{n,}? at least n times
{n,m}? at least n but not more than m times

Grouping and extracting matches

To group parts of an expression, use the metacharacters ( ). This allows the regular expression in the parantheses to be treated as a single unit. For example, the regular expression

severity:(1|2)
matches the pattern severity:1 or severity:2.

To extract parts of a string that have been matched using the grouping metacharacters, use the special variables $1, $2, etc.

# Extract the name and URL from $pattern = <a href="secure_logon.html">Logon form</a>
$pattern =~ <a href=\"(.*)\">(.*)</a> ; # match using grouping
$url = $1;                # $1 equals secure_logon.html
$pagename = $2;           # $2 equals Logon form

Perl 5 extended regular expressions

Expression Matches
(?#text) An embedded comment causing text to be ignored.
(?:regexp) Groups things like "()" but doesn't cause the group match to be saved.
(?=regexp) A zero-width positive lookahead assertion. For example, \w+(?=\s) matches a word followed by whitespace, without including whitespace in the MatchResult
(?!regexp) A zero-width negative lookahead assertion. For example foo(?!bar) matches any occurrence of foo that isn't followed by bar. This is a zero-width assertion, which means that a(?!b)d matches ad because a is followed by a character that is not b (the d) and d follows the zero-width assertion.
(?imsx) One or more embedded pattern-match modifiers:
i enables case insensitivity
m
enables multiline treatment of the input
s
enables single-line treatment of the input
x
enables extended whitespace comments