Eclipse TPTP Version 4.2
File Transfer Service Design

Technical Owner:  Kevin O'Leary

Original contributers:  Hoang M. Nguyen, Guru Naragajan

Version 1.0

June 2006

Copyright © 2005, 2006 Intel Corporation.

Content provided under the terms and conditions of the Eclipse Public License Version 1.0.

Eclipse is a trademark of Eclipse Foundation, Inc.

Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

1  Introduction

This document describes the design and interface of the file transfer service provided by the agent controller (AC) in TPTP 4.2

1.1 Scope

This document is intended for the following users:

•     Developers who will implement the service

•     Tester to understand the external interface and how to use the service

•     Information developers to integrate the service into development guide

•     Contributors who want to understand the implementation and its rationale
  
  
   

1.2      Prerequisites

In order to have enough context and concepts of TPTP, readers should review the following documents and articles before examining this design:

• The high-level design document of the agent controller in TPTP 4.2
  •  http://www.eclipse.org/tptp/platform/documents/design/DataCollectionHLD.htm
• Current implementation details of the file transfer service

  • http://www.eclipse.org/tptp/home/documents/resources/RAC%20Assisted%20File%20Transfer.doc

• Overview of the DIME format

  • http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarxml/html/dimewsattch.asp

• The DIME binary format

  • http://msdn.microsoft.com/msdnmag/issues/02/12/DIME/default.aspx?fig=true#fig2

• The usage of the DIME protocol

  • http://msdn.microsoft.com/msdnmag/issues/02/12/DIME/default.aspx

1.3           Requirements

The requirements of the file transfer service are:

• Be able to provide secure file transfer (e.g. using SSL).   (This requirement is not implemented in TPTP version 4.2.)
   
• Be able to configure the port number used for the file transfer such that it can specify its own port or share a port.  This allows a single port to be used for all communication between the client and a remote system.
   
• Provide the basic file transfer operations, i.e. GET, PUT, and DELETE files.
   
• Be extensible to provide additional operations such as multiple GETs, PUTs, or DELETEs.
   
• Be efficient and provide the best data transfer rate.
   
• Be able to scale (e.g. serving very large data files or many multiple files simultaneously).
   
• Work with the new architecture of the agent controller in TPTP 4.2.

1.4           Why do we need a new design of file transfer service?

It is important to have this service redesigned to address the following constraints and limitations:

• The current file transfer service requires a separate port number to listen to. In some environments, it is not easy to open up additional port numbers.
   
• The current file transfer service is implemented only in Java and requires a JVM to run. It tends to be slower in some platforms since it relies on Java library implementation for that platform. Java may not be available or pre-installed in some platforms.
   
• The current file transfer service uses at least two dedicated separate sockets for each file transfer request, as follows:
  • One socket for the initial transfer request (querying for port number).
  • One socket for the actual transfer of a file

  The current file transfer service will not scale very well if you have to transfer a lot of files.
   

• The current implementation has its own communication code (i.e., it does not use the AC communication infrastructure). Therefore, it is complex and hard to maintain them separately.
   
• The current file transfer service requires a lot of additional work or reimplementation to support new protocols such as HTTP.
   
• The current file transfer service does not support multiplexing data or command. Therefore, it is not possible to limit the number of required sockets to one.
   
• The current file transfer service needs to conform to and make use of the latest architecture of the TPTP agent controller in order to gain the benefits and maintainability that uniformity of design provides.

2  File Transfer Service Design

This section describes the internal components and processing flow of the transfer service.

2.1      Design objectives

The basic design objectives for the file transfer service are as follow:

• Language independence
   
  • Either client or agent can be written in any supported language (i.e., C/C++ or Java)
• Providing best performance
   
  • This implementation should be able to provide equal or better performance than the current implementation
     
  • This implementation should use native data transfer mechanism when possible (e.g. native socket I/O).
• Implementation independence
   
  • This implementation should be able to change the file transfer implementation without affecting the external APIs or the wired protocol.
• Flexible for change
   
  • This implementation should support the open standard and be capable of adding additional new features without breaking the APIs or the basic processing flow.
• Scale
   
  • This implementation should eliminate dedicated port numbers or socket connections for each file transfer request (e.g. sharing socket connections for multiple files/requests).

Some of the important design points to keep in mind when using this document:

• The service should be able to expand to support “local host” transfer transparently. The service should be able to detect when the transfer is happening on the same system and avoid setting up a remote connection when all that is needed is a simple local file copy.  (This local file copy optimization is not implemented in TPTP v4.2.)
   
• Optimal performance of a file transfer operation dictates that the file being transferred be read only once, hence the implementation needs to avoid making any intermediate copy.

Figure 1 presents a high-level view of the file transfer servcie as implemented for the new technology Agent Controller in TPTP version 4.2 and later.

Figure 1.  High-level view of file transfer service

Note that the file transfer service is written and behaves just like any other agent. This means:

• It runs in a separate process
   
• It will be launched by the agent controller and register itself
   
• It will receive commands through the agent controller

Figure 2 shows how the Agent Controller allows transport layers to be configured to use multiple ports.

Figure 2.  Multiple and configurable listening ports

The new design of the AC enables the user to configure multiple pluggable transport layers such that each can listen to its own port number. As a result, the diagram highlights the ability for the file transfer service to be able to use a dedicated port number (like 10005 for compatibility with existing file transfer service) or make use of the same main port number as other agents (thus avoiding opening a new port number for file transfer service only).

In addition, the diagram shows that the agent implementation is shielded from the details of the transport mechanism itself, such as whether it has SSL encryption or not.

Figure 3 illustrates the flow for a basic file tranfer works.

Figure 3.  How the file transfer works

The flow of a basic transfer flow is as follows::

• The file transfer request is submitted through the normal control channel of the AC and routed to the File Transfer agent.
   
• If the File Transfer agent determines it is a GET or PUT request, shared memory is allocated to buffer the contents of the file.
   
• The File Transfer agent then iniates a binary transfer of the file contents through the standard data channel.

There are a couple of important points not illustrated in the diagram:

• The same data channel between the client and the File Transfer agent will be used to transfer multiple files for multiple requests. This is different than the current implementation where a separate socket connection is required for each transfer request.
   
• The two separate channels shown here are conceptual in order to highlight the usage of the connection for transferring files.  Both commands and data can flow through a single physical connection and the separation of the two channels is hidden within the transport layer.

2.2           Transfer Data format: DIME-compliant

There are a couple of important quotes in the mentioned articles (see “Prerequisites” section for links) worth repeating to highlight why we choose to adopt the DIME format:

“The Direct Internet Message Encapsulation (DIME) specification addresses all these issues [data format and packaging] in a minimal and efficient manner.”

“While potentially less flexible than MIME, DIME prescribes a simpler message format, which in turn provides for faster and more efficient message processing.”

“When binding DIME messages to HTTP, the HTTP Content-type header field must specify "application/dime" instead of the usual "application/soap+xml" or "text/xml" defined respectively by the SOAP 1.2 and SOAP 1.1 protocols.

Why did we choose to adopt the DIME format?

• It is an open standard adopted and managed by the Internet Engineering Task Force (IETF)
   
• It does not require the XML parser or parsing time
   
• It supports data chunking
   
• It supports the ability to multiplex the data channel for different purposes and different data types
   
• It is easy to support HTTP communication protocol
   
• It is supported and interoperable with SOAP and web services
   
• It offers an open data representation in an efficient manner that can yield top performance of data transfer,

Figure 4 shows the detail of a typical file transfer flow, illustrating how DIME is used in the system.

Figure 4.  Use of DIME in file transfer

The diagram shows a typical transfer flow and how DIME is used in the system:

• After receiving a GET file “A” request from the client, the File Transfer agent reads the content of local file “A” into shared memory and builds a sequence of DIME chunks. Each file will be split into one or more DIME records based upon the size of the buffers and the size of the file.
   
• This shared memory buffer is used to transfer data from the File Transfer agent to the Agent Controller. A single shared memory buffer exists for this Agent, this buffer is meant to handle multiple file transfer requests.
   
• Once this file transfer data reaches the Agent Controller it is transferred to the client using the standard data transfer mechanism. Each DIME record has a header block that includes a file ID and record indicator on whether it is a first, middle or last record.
   
• These DIME records are sent across the data channel where they are received by the client TPTP library and correlated back into a single file called “A” on the client system.

2.3      Use of DIME records on Data Channel

Figure 5 shows how DIME is used to identify data chunks going across a single data channel.

Figure 5.  DIME records use on a data channel

Above is an example of how DIME is used to identify data chunks going across a single channel. This allows different data types, such as performance data collected by an agent monitoring an application and contents of a file, to be sent across the same channel in an interleaved fashion.  This also shows how different sets of same data type (i.e., multiple files) can be sent across the same channel simulatneously..

The DIME record contains sufficient information to allow the data to be properly grouped and correlated by the receiving component, client or agent, without knowing the form of the data itself.

3  External Interface

This section describes the external APIs that an application uses to access the file transfer service.

3.1       File Transfer APIs

3.1.1      Basic File Transfer Operations

The File Transfer agent provides the following operations.

• getFile – copy a file from the remote system
• putFile – copy a file to the remote system
• deleteFile – delete a file from the remote system

The file transfer agent receives these requests on the command channel (any command responses are returned on the command channel) and subsequently returns the file on the data channel. The establishment of a data path  is handled by the client API. A data path will only be created if one does not currently exist. All file transfer functions check whether a data path has been created, and create one where one does not exit.

Multiple file transfer requests may be served by a single File Transfer agent.

The default File Transfer service agent can be replaced by a custom implementation and accessed just as any other agent.

Table 1 below summarizes the functions that can be called upon using a File Transfer agent object  The references to local and remote systems are relative to where the client (i.e. the calling application) is running.  

3.2            Sample file transfer code

The following example for using the C++ File Transfer API is taken from the Sample code provided in the SDK package.

/**

 *********************************************************

 *                                                      

 * @brief                                               

 *    MyFileTransferClient

 * This is a sample client program that uses the Agent Controller

 * to get access to the FileTransfer agent.

 *

 * The FileTransferAgent is then sent 3 messages.

 *    getFile

 *    deleteFile

 *    putFile

 *

 * Usage : MyFileTransferClient                 <==defaults to localhost, port 10006

 *          : MyFileTransferClient <HostName> <Port>

 *          : For e.g. MyFileTransferClient

 *                            MyFileTransferClient "lab-system-001" 10006

 * Expected Output:

 *    Connected to Agent Controller message

 *    Running getFile test

 *    Get File X From File Y

 *

 *    Running deleteFile test

 *    Delete File X

 *

 *    Running putFile tests

 *    Put file X To File Y

 *    All finished

 *    Press any key to exit...

 *

 *  External Dependencies:

 *    1 file to test file transfers.

 *          The content of this file does not matter, it can be text or binary.

 *          This will be used to test the FileTransfer operations.

 *

 *          This Sample encodes the file path as followed.

 *          On Windows

 *                C:\big_simple.txt

 *          On Linux

 *                /tmp/big_simple.txt

 *   

 * Source Code Files:

 *    MyFileTransferClient.cpp

 *

 *

 *********************************************************/

int main(int argc, char* argv[])

{

      char* hostName = "localhost";

      int portNum = 10006;

      INode* TargetNode = 0;

      AgentController* ACProxy = 0;

      //Start of generic connect to AgentController code.

      if ((argc == 2) || (argc > 3))

      {

            cout<<"Usage: MyFileTransferClient or MyFileTransferClient <HostName> <Port> "<<endl;

            cout<<"Press any key to exit..."<<endl;

            getchar();

            return -1;

      }

      // Use the host name and port number if provided on the command line.

      if (argc == 3)

      {

            hostName = argv[1];

            portNum = atoi(argv[2]);

      }

      // Create a Node object for the target system of choice.

      TargetNode = NodeFactory::createNode(hostName);

      if (!TargetNode)

      {

            cout<<"ERROR: Failed to create the Node object."<<endl;

            cout<<"Press any key to exit..."<<endl;

            getchar();

            return -1;

      }

      // Establish a connection with an Agent Controller on the

      // target system.  The port number specified must match

      // that defined in the configuration file for the desired

      // Agent Controller.

      ACProxy = TargetNode->connect(portNum);

      if (!ACProxy)

      {

            cout<<"ERROR: Unable to connect to the Agent Controller running on "<<hostName<<" at port number "<<portNum<<endl;

            cout<<"Press any key to exit..."<<endl;

            getchar();

            return -1;

      }

      cout<<endl<<"Connected to the Agent Controller on \""<<hostName<<"\" at port number "<<portNum<<endl;

      //End of generic connect to AgentController code.

      //Start of FileTransferAgent specific code.

      // Create a client-side representative of a FileTransfer object from which

      // a MyFileTransferAgent is derived, specifying the name of the desired agent

      // as it has been installed in the agents directory.

      FileTransfer* MyFileTransferAgent

            = new FileTransfer("org.eclipse.tptp.FileTransferAgent");

      // Get a handle to the actual FileTransferAgent, storing it in the MyFileTransferAgent

      // object.  If the call is successful, the FileTransferAgent is running and

      // ready to respond to requests.

      // Note that the Agent Controller manages the life of an agent, so it will start

      // one running if needed.

      // We request controller access to the agent since we want to perform

      // actions like getFile and putFile. However, the FileTransferAgent is not checking

      // access levels on incoming requests at this time (TBD).

      if (-1 == ACProxy->getAgent(MyFileTransferAgent, TPTP_CONTROLLER_ACCESS))

      {

            cout<<"ERROR: Unable to get agent org.eclipse.tptp.FileTransferAgent"<<endl;             

            cout<<"Press any key to exit..."<<endl;

            getchar();

            return -1;

      }

      cout<<endl<<"The FileTransferAgent Agent ID: "<<MyFileTransferAgent->getAgentID()<<endl;

      //These next variables encode the file paths for the transfered files.

#ifdef _WIN32

      char *remoteFile = "C:\\big_simple.txt";          //This is the source file for getFile and putFile.

      char *localFile = "C:\\big_local.txt";            //This is the destination file for getFile and putFile.

                                            //Deleted by deleteFile operation.

#else

      char *remoteFile = "/tmp/big_simple.txt";         //This is the source file for getFile and putFile.

      char *localFile = "/tmp/big_local.txt";           //This is the destination file for getFile and putFile.

                                            //Deleted by deleteFile operation.

#endif

      cout <<"Running getFile test " << endl;

      cout <<"Get file " << localFile << " From File " << remoteFile << endl << endl;

      //Get the file specified in remoteFile to localFile.

      MyFileTransferAgent->getFile(localFile, remoteFile);

      cout<<"Running deleteFile test" <<endl;

      cout <<"Delete file " << localFile << endl << endl;

      MyFileTransferAgent->deleteFile(localFile);

      cout <<"Running putFile test " << endl;

      cout <<"Put file " << remoteFile << " To File " << localFile << endl;

      //Put the file specified in remoteFile to localFile.

      MyFileTransferAgent->putFile(remoteFile, localFile);

      // Wait for data to finish coming over channel.

      // Note: A protocol for recognizing end of data is TBD.

      Sleep(2000);

      // Release resources allocated by the Agent Controller

      MyFileTransferAgent->destroyDataConnection();

      MyFileTransferAgent->releaseAgent();

      ACProxy->disconnect();

      // Release client side resources

      delete MyFileTransferAgent;

      ACProxy->destroy();

      NodeFactory::deleteNode(TargetNode);

      cout<<"All finished"<<endl;

      cout<<"Press any key to exit..."<<endl;

      getchar();

      return 0;

}

3.3           File Transfer Agent  XML Command Protocol

This section describes the XML-formatted commands recognized by the File Transfer agent.  These are the client API uses these in its implement .

3.3.1      Commands

The following commands are supported through the File Transfer interface.    

Note that this is basically maintaining the same command format of the current protocol where there is no new information is required for the request.

Command:        

getFile / putFile / deleteFile

Parameters:

Response:

No Data returned

Alternate Response (Error situation):

SECURE_CONNECTION_REQUIRED_EXCEPTION

FILE_NOT_FOUND_ERROR

IO_ERROR

Description:

In case of an error, one of the above error messages (Alternate Response) will be provided as a CBE formatted response.