High Level Design (HLD) Specification
Data Collection & Communication Component
Eclipse TPTP Core Platform Project
Revision 0.4.0
November 16, 2004
Table of Contents
Agent Controller Subcomponents
2.5.1 Connection Management & Secured Connection Management
2.5.6 Event Listeners & Handlers
2.5.8 Configuration Management – NEW?
2.5.9 File Transfer Management
2.6 External Interfaces of Client Library
2.6.1 Client Library external interfaces
2.6.2 Client library Java packages
2.6.9 Example of Client Interaction with Client Library
2.8 Agent-to-Agent Communication
3.1 Typical Client/Agent Communication Sequence
3.2 Client connection sequence
3.6 Client view for starting connection
3.7 Interaction for setting up data collection/connection..
3.9 Client view for sending command
3.10 Client configuration of an agent/Agent Controller
4.1.5 Message Buffer Memory Management
4.2 Data Transfer and Transport Layer
4.7 Remote File Browsing Agent
4.8.1 Command extractor XML Structure
4.8.2 Command extractor XML Parser
4.9 Agent Controller configuration
4.9.1 Agent Controller initialization
4.9.2 Agent Controller configuration for PayLoad Normalizer and Command Extractor
4.9.4 Client-Agent Controller interaction
4.13.1 Session concepts and usage
4.13.3 Session management rules
4.13.4 An example of “stateless” default session info processing
4.13.5 An example of explicit session info processing
4.14 Dynamic Agent Controller configuration
Agent Controller runtime configuration (serviceconfig.xml)
4.14.2 Agent runtime configuration (pluginconfig.xml)
4.15.2 Error Handling Mechanism
4.15.3 Synchronous and Asynchronous Errors
4.15.6 Common Base Event Model
4.15.7 Error Codes and Scenarios
5.1.5 Connection Sequence Diagram
5.3.1 Command Handlers for Events
5.7 Data Reception Sequence Diagram
5.8 Event Listeners & Handlers
5.8.1 ConnectionListener interface
5.8.2 ProcessListener interface
5.13 Asynchronous Event Handling
5.13.8 EventCommandHandler Interface
6.1 Client-Client Library Interface
6.1.1 Existing Hyades Client - Full Packages List
6.1.2 Java Packages that need Conversion to C++
6.1.3 List of Java Classes (Needs porting to C++)
6.4 Transport Layer Interface (Changes TBD)
6.4.2 Functions Provided By Transport Library
6.4.3 Functions Provided By Agent Controller
6.5 Payload Normalizer Interface
6.5.1 Agent Controller configuration for PayLoad Normalizer and Command Extractor
6.5.2 Functions Provided by Agent Controller
6.5.3 Functions Provided by PayLoad Normalizer
6.6 Command Extractor Interface
6.6.1 Functions Provided By Command Extractor
6.6.2 Functions Provided by the Agent Controller to interface with the Command Extractor
6.7 Socket Transport Interface
6.8 Named Pipe Transport Interface
6.8.1 Named Pipe Message Envelope
6.9.1 Sample HTTP Request Message Envelope
6.9.2 Sample HTTP Response Message Envelope
6.9.3 HTTP Connection Handshake
6.10 Agent Base Classes and Templates
6.10.1 General Class Definitions
6.10.2 Base Agent Interface – IBaseAgent and BaseAgentImpl Classes
6.10.3 Variable interface – IVariable and VariableImpl classes
6.10.4 Event Interfaces – IEventProvider and EventProviderImpl Classes
6.10.5 Data Provider Interface – IDataProvider and DataProviderImpl classes
6.10.6 Attachable interface – IAttachable and AttachableImpl Classes
6.10.7 Base Collector Template – BaseCollectorImpl Class
6.10.8 Example of an Agent class definition
7.1.1 Destination and Source IDs
7.1.3 Command Element and Interface
7.6 Process Controller Agent Interface
7.7 Environment Info Agent Interface
7.7.2 Message Buffer Memory Management
1 Introduction
This is the high-level technical design document for the new Agent Controller and associated development.
1.1 Purpose
This document specifies a high level design, including inter-component interfaces and dependencies sufficient to guide low-level design and implementation that will meet the needs described in the requirements documentation.
1.2 Intended Audience
This document is expected to be used by a technical audience of:
· Software developers tasked to implement the design.
· Technical staff involved with testing, maintenance, and support.
1.3 Organization of the Document
2 System Architecture
2.1 Basic Structure
At the most basic level, the Eclipse TPTP system consists of a client, an Agent Controller and one or more agents.
The client may be an Eclipse workbench with TPTP plug-ins installed, or it can be a standalone application built using a C++ implementation of the TPTP client library. The client can be running on the same machine as the Agent Controller or it can be on a different system. In either case, the client communicates with the Agent Controller using an interprocess communication transport that is provided with TPTP.
The agents can include both system agents provided as part of TPTP and third party data collection agents. The agents always run on the say machine as the Agent Controller. The agents communicate with the Agent Controller using an interprocess transport that is provided by TPTP.
The scope of this project includes development of the Agent Controller, a C++ client library, base classes and templates for C++ agent development and several system agents. The relationship of these components is shown in Figure 2-1.
Figure 2-1 – Basic System Structure
The components shown in gray above are within the scope of this project and will be described in greater detail in this document. The other components represent products that will be developed using TPTP. They are outside the scope of this document.
The actual system structure can be more complex than shown above. Each client can be attached to multiple Agent Controllers, and each Agent Controller can be in communication with multiple clients. Further details are provided throughout this document.
2.2 Expanded Architecture
The previous section described a very basic view of the system architecture. This section will further refine the description above and indicate the components involved. For the purposes of this discussion, only the native/C++ implementation will be considered. The existing Eclipse-based Java client will continue to be a primary platform, but its architecture will not change.
The Agent Controller will communicate with clients and agents through a pluggable message pipeline. In addition, common transport services needed by both the client library and the controller’s transport layer will be moved into a shared library. The required components are shown in Figure 2-2 below.
Figure 2-2 – Component Diagram
The diagram above shows the major components that have been identified for this project. Some other components, including an enumeration of agents and security-related components, are described later in this document but their position in the architecture should be clear from the above document. The arrows in this diagram do not indicate program flow. The arrows are intended to indicate a dependency relationship between components. The arrow points to the component that defines the interface between the two.
The Client component in the figure above represents the client application that is using TPTP. It does not correspond to a component which needs to be developed for this project. The Client Library is a native shared library which is designed to be used by the client application. The interface between the Client and the Client Library, including the hierarchy of classes exposed by the client library is detailed below.
The Socket Library component is a shared library that is used by both the Client Library and the Socket Transport Layer component. This library provides basic socket services. Similarly, the HTTP Library provides basic services related to HTTP communication and is shared between the Client Library and the HTTP Transport Layer. The interfaces to both of these libraries are specified below.
The Socket Transport Layer, HTTP Transport Layer and Named Pipe Transport Layer are all pluggable modules that are loaded by the Agent Controller to manage communications with other components. The pluggable message pipeline strategy is described below. All pluggable transport layers use the same interface with the Agent Controller. This interface is defined below. Each transport layer specifies its own message envelope and low-level commands to which it responds, if any. These details are also described below.
Each transport layer may work with one or more payload normalizers. See the message pipeline section below for a full discussion of the role of the payload normalizer. These payload normalizers are pluggable modules loaded by the Agent Controller and all payload normalizers interact with the Agent Controller using a standard interface. This interface is described below.
Each message pipeline is associated with a single Command Extractor. The Command Extractor breaks the command stream into individual commands and passes them to the Agent Controller for processing. In the current design, all message pipelines share a single Command Extractor. The interface between the Agent Controller and the Command Extractor is described below.
The Agent component in the figure above corresponds to both system agents and data collector agents, as both interact with the rest of the system in the same way. Agents exchange messages with the Agent Controller through the Named Pipe Transport Layer. Among the deliverables for the current project is a set of C++ templates and base classes that standardize the communications through the named pipe mechanism. These classes will be used as the basis for further agent development, including the system agents that will be developed as part of this project. A description of these classes and templates is provided below. Agents are not required to use the base classes. These are provided for convenience.
2.3 Agent Controller Subcomponents
The Agent Controller can be further subdivided in logical subcomponents. These may be broken out into shared libraries, or they may be implemented within the Agent Controller itself. This decision will be made during the low-level design. Because the Agent Controller will be implemented in C, these will not be implemented as classes, but they will be functionally similar to classes. See Figure 2-3 for an overview of the subcomponents. Each of these is described briefly below.
Figure 2-3 – Agent Controller Subcomponents
2.3.1 Configuration Manager
The Agent Controller configures itself and starts agents as defined in the agent configuration file. The startup of the Agent Controller is a multi-step process; the Agent Controller starts itself by reading a configuration file and provides services for servicing clients and agents. The current implementation of the Agent Controller does not recognize modifications to the startup configuration file at runtime. The Agent Controller has to be restarted for the modifications to the startup file to take effect.
2.3.2 Error Handling Services[kac1]
The Error Handling Services component will provide services for error management, detection, propagation and logging mechanisms. The error management will be based on the Common Base Event schema to make it flexible and extensible.
2.3.3 Logging Service[kac2]
The Logging Service component will provide interface for logging an informational message, event or an error.
2.3.4 Agent Manager
The Agent Manager will provide an interface that allows clients and agents to obtain information about the available agents and gain access to instances of agents. Agents also use this interface to register as agents after establishing a connection with the Agent Controller. This interface is described in the Protocol section below. The Agent Manager will manage agent lifecycles and agent locking. The Agent Manager is described in much more detail in the “Strategies” section of this document.
2.3.5 Data Manager
The client requests for data from the agent that is routed through the Agent Controller. The Agent Controller gets the client requested data from shared memory (the agent puts the data on the shared memory) and puts it on the appropriate transport layer to be received by the client.
2.3.6 Connection Manager
The Connection Manager will maintain data structures describing components that have established connections to the Agent Controller. When a component requests a connection, the Connection Manager will validate the protocol being used, assign a connection ID and create an entry in its connection table. The Connection Manager will also provide services to allow other Agent Controller subcomponents to obtain connection information, such as the transport layer associated with a given connection ID.
2.3.7 Message Manager
The Message Manager loads the plug-ins for the message pipeline, provides the interfaces called by these plug-ins and manages the routing of messages through the pipeline. When a command is sent to a subcomponent within the Agent Controller (for instance, to the Agent Manager), the Message Manager will identify the message and call the appropriate internal function for the processing of the command. (Each internal subcomponent will provide a single entry point for all commands sent to that subcomponent.) The interfaces to the various plug-ins involved in the message pipeline are described below.
2.4 Service Agents
In addition to redesigning the Agent Controller component, this project involves the development of several system agents which will provide general services to clients, other agents and the Agent Controller. These system agents are shown in the figure below.
Figure 2-4 – System Agents
2.4.1 Process Controller
Provides ability to create, start, and stop a specified application. It sends notifications concerning process state changes (events) for the application it starts, such as when and how it is terminated. Its purpose is to insulate clients from the OS-specific details of creating/controlling an application.
A standard process controller will be implemented as a service agent. A standard process control interface will be published, allowing developers to create custom controllers (e.g., to support binary instrumentation prior to application launch).
2.4.2 Environment Info
The EnvironmentInfo service agent provides basic system information about the platform on which it is running. Its purpose is to provide clients with identifying information that it might display to a user and/or to use in determining its own analysis of the data. For instance, when collecting performance data it is useful to a user to know which CPU was used when they compare data for an application which was run on multiple systems. It may also be necessary for the client to know CPU info when interpreting data for display.
(A standard interface will be published, allowing developers to create proprietary replacements – but this is not required for initial release. Initial version of this service agent is likely to include Intel proprietary methods for determining CPU names. The open source common interface would include methods to obtain all the other information.)
2.4.3 File Transfer
This agent provides the service to move data files between the client and Agent Controller systems.
In order to move files, the Agent Controller will start up its own separate listening thread. The port number is specified in the “serviceconfig.xml.” The default port number is now set at filePort=”10005.”
At the startup of the Agent Controller, the “File” server will be created and listening to the specified port number.
The client will send in a command request for the file server port connection mechanism (e.g. socket). Once the client holds the socket connection, it can perform the actual file transfer operations:
· get a file from the Agent Controller
· send a file to the Agent Controller
· delete a file in the Agent Controller system
It is noted that the current implementation of this “File” server is using the JVM as the underlying mechanism for processing the file transfer.
2.4.4 Remote File Browsing
This agent provides the service for the client to explore directories and files in the Agent Controller system transparently just like local file system.
The client needs the ability to launch applications in the target system. In order to launch correctly, it has to specify information such as file name and its path name.
In addition, it also needs to verify whether this is an executable file that can be launched or not (e.g. finding out the file attribute info).
As a result, the client requires a programmatic way to query, explore and validate through the directory and file systems in the target system. This agent provides such services.
2.5 Client Library Components
The TPTP client library consists of a set of API that facilitates in providing communication interface between a client application like the Eclipse workbench and the Agent Controller running on the target machine. The TPTP client library can be logically viewed as containing the following listed components also as shown in the fig 2.5.1
Fig 2.5.1 Components of the Client Library
The Client library uses the interfaces provided by the various components of the client library to communicate with the Agent Controller.
2.5.1 Connection Management & Secured Connection Management
The Connection Management and Secured Connection Management components handle the communication with the Agent Controller by establishing the physical connection with the target machine. This connection is called Control Channel and is used for sending control messages from one end to another. This component also includes the representation of the target machine node and processes running on it. This component also starts a thread to listen for the commands coming from the Agent Controller. The Secure Connection Management component provides SSL connection to enable secured communication. This component will handle the details of secured authentication, security key store management, etc. A client can optionally either use regular connection or secured connection.
2.5.2 Session Management
The Session Management component will enable the concept of a user session on the Agent Controller. A client application can create multiple user sessions simultaneously and operate within their contexts.
2.5.3 Message Management
The Message Management component provides interfaces and classes for building the control or event messages that go across the established connection to the Agent Controller or agents. There are several commands defined in this component. Please refer to the next sections for a detailed list. A client application will use the Message Management component to build and issue necessary commands for performing an activity like data collection using the agents.
2.5.4 Agent Management
The Agent Management component consists of interfaces and classes that define the client-side implementation of the agents running on the target machine. These agent classes can be treated as proxies for the actual running agents. The client application will create instances of these agents and use them to communicate with the agents (via Agent Controller) running on the target machine.
2.5.5 Data Management
The Data Manager component provides a data channel for facilitating the data transfer from the agents to the client application. The data channel is dedicated for performing the data transfer. It provides the interface definition of Data Processors. A client application will implement the data processor interface to create data processors. The Data Manager component will hand over the data received over the data channel to the registered data processors for further handling and processing of data.
2.5.6 Event Listeners & Handlers
A client application creates various objects that correspond to the actual objects on the Agent Controller and also the agents themselves. The client application ought to know about the status of these objects from time to time. So, the Event Listeners and Handlers component defines the interfaces for getting information about these objects with the help of Listeners that listen to these objects. The client application can implement these interfaces to provide listeners that can listen to one or more objects, for e.g. like Connection, Process, Agent, etc. Each of these listeners defined are first registered with their respective objects so that when an event command is received over the control channel, they get forwarded to these listeners.
A new design is being proposed (check the design strategy section for more details) in addition to the existing mechanism to use the regular Command Handler interface to receive the event commands and handle them using the defined the command handlers.
2.5.7 Command Handlers
The Command Handler component provides handler interface and handler objects to handle the commands coming from the agents or Agent Controller. The commands could be the responses to the commands issued by the client to the Agent Controller or any other associated components with it. This component defines the interface for a Command Handler. The client application can implement this interface to provide handlers and register them.
2.5.8 Configuration Management – NEW?
The current Hyades client implementation doesn’t have any configuration for the client library as the configuration information is maintained by the Eclipse workbench in the preferencesUI. For the C++ client library implementation, we need to be able to specify the following configuration information (though we could depend on the client application if we decide to)
- Shared library location paths of the various communication libraries, agent libraries
- Target machine IP address
- Target machine port number
2.5.9 File Transfer Management
The file transfer management component provides client-side implementation of the file transfer mechanism. This component provide interface to initiate requests to the file server running on the Agent Controller.
2.6 External Interfaces of Client Library
The external interfaces exposed by the client library are described in the following sections. A client application uses this interface to interact with the client library. An example of how to interact with the client library also is illustrated.
NOTE: This section currently shows the interfaces based on the current java implementation of the client.
2.6.1 Client Library external interfaces
The fig 2.6.2 shows a different view of the components and the interfaces that are exposed to the client by the client library. The interface definitions of the external interfaces of the client library are shown in the next section[kac3] .
Fig 2.6.2 Client Library external interfaces and internal components
2.6.2 Client library Java packages
The java packages that provide the external interfaces and their implementation are listed below.[kac4]
- org.eclipse.hyades.execution.core – this package contains basic interfaces
- org.eclipse.hyades.internal.execution.local.common – this package contains the Message API, Command API, Data Processor API and Command Handler API.
- org.eclipse.hyades.internal.execution.local.control – this package contains the interfaces and implementation for Connection API, Agent API (creating and dealing with agents and agent configuration) and Event Listener API.
- org.eclipse.hyades.execution.local – NOTE: Classes from this package doesn’t show any direct relationship with the classes in the above packages.
2.6.3 Communication API
These interfaces or classes are defined in the org.eclipse.hyades.execution.internal.local.control package of the current Java implementation.[kac5]
2.6.3.1 NodeFactory
The NodeFactory class provides interface for creating a new node that represents the target machine node. A client application can use this Class to create objects of Node to be able to communicate with them. All methods of the NodeFactory class are static members.
2.6.3.1.1 Create Node
This interface allows a client application to create node that acts like a delegate to the target machine. There more than one overloaded interfaces that take different parameters for creating the code. This interface returns an instance of the Node Class. All these methods return an existing instance of Node if a node already exists with the specified IP address or hostname. If a Principal is passed, the Node is associated with the user or application whatever the case may be.
|
Method Signature |
Description |
|
Node createNode(InetAddress address) |
Creates a Node with the IP address specified. Throws an UnknownHostException if the target machine doesn’t exist in the network. |
|
Node createNode(InetAddress address, Principal principal)
|
Creates a Node with the IP address and the Principal specified. Principal here is the user or application who is creating this node. If the same user tries to create this Node again, this method throws DuplicateUserException. |
|
Node createNode(String hostname) |