Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cdt-dev] Remote C/C++ Project Support Prototype

Hi Jeff,

Cross posting to ptp-dev...

Interestingly enough, it seems like most of what you did is done by PTP/RDT or CDT already in other ways. For instance, PTP has a framework for remote file access and launching remote commands which we use to target both RSE and a lightweight SSH provider called Remote Tools.

Your IRemoteFile proxy does a lot of things that we have been using CDT's EFSExtentions for (i.e. path extraction/mapping), although it does have the nice addition that it tells you something about the OS targeted, which EFS does not, and which we hadn't added to the EFSExtensions but probably could/should have/still ought to. Unfortunately the problem with IPath is that it's really designed to only deal with paths on the local filesystem, and IPaths tend to get passed down deep into the bowels of CDT, at which point you've sort of lost any way of getting back at the EFS objects they came from. We sort of rely on the fact that IPath.toString() uses UNIX path separators and so because we pretty much only care about targeting UNIX/Linux on the back end, things work out, but a current issue with this is that targeting Windows as a remote system is pretty broken.

For determining how to launch the commands, we have gone about this two ways. For makefile builds, we have our own RemoteMakeBuilder that asks the PTP service model for the build service provider, and then uses it to execute the remote builds. For managed build we use the ICommandLauncher coupled with a remote version of the toolchain so that when MBS launches the commands they execute on the remote machine, once again using the build service provider. The downside to your approach as you have done it so far is you don't have the flexibility to build on a different machine than you are editing on, or to build on multiple machines as a part of one build, which the RDT model allows for. Our model also allows for having files reside on your local machine for editing and source control integration but building on the remote machine.

One thing that it looks like you haven't really covered (at least from your email, I haven't looked at your code) is environment variables. If you are using the same local toolchains that CDT provides, then the environment variables that MBS uses to execute the build will be those from the local machine, and that can really mess up builds sometimes. Part of why we have remote toolchains in RDT is so that we can provide environment suppliers that get the environment from the remote machine.

The last thing... if you are using the indexer directly on either a mount or on EFS resources it's going to be significantly slower than if all the files were local or if you used the RDT remote indexer which does all the indexing on the server. There is a significant performance hit to pulling down all the source files over the wire. On a toy project you might not notice much, but on a "real" project and especially a large project, it's very significant. Also along these lines, doing binary parsing over the wire is quite possibly the slowest thing I have ever seen, to the point where in our remote toolchains we remove the specifications for the binary parsers. Even on relatively small projects the binary parser gets way out of hand.

The fact that I have not looked at your code brings up an interesting point... I'm not sure I *CAN* look at it unless it is already EPL licensed. Is it? Of course I know you intend it to be so if you are thinking of submitting it to CDT but are all the license files there, etc.? Normally we don't look at code until there's been a submission in Bugzilla, which implies it's being submitted under the EPL. Kind of a weird artifact of all this distributed source control stuff with Git... I can't look at it till I know it's properly licensed, but I don't know it's licensed properly unless I go look... cyclic!

At any rate, I'm interested in continuing the discussion... it might be worth having a call at some point in the near future.

Chris Recoskie
Team Lead, IBM CDT and RDT
IBM Toronto

Inactive hide details for Jeff Johnston ---01/12/2012 06:51:53 PM---I have recently been working on Remote Project support for Jeff Johnston ---01/12/2012 06:51:53 PM---I have recently been working on Remote Project support for Autotools projects and wanted to start a


Jeff Johnston <jjohnstn@xxxxxxxxxx>


"CDT General developers list." <cdt-dev@xxxxxxxxxxx>


01/12/2012 06:51 PM


[cdt-dev] Remote C/C++ Project Support Prototype

Sent by:


I have recently been working on Remote Project support for Autotools
projects and wanted to start a discussion on what changes I made in the
CDT and Autotools and whether this would be something the CDT would want
to pursue.

To start with, my goal was to eliminate the need for separate Project
types or wizards which IMO there are lots to confuse the user as-is.  To
create the remote project, I simply wanted to specify the location of a
C or C++ project as being remote.  To do this, in the C/C++  Project
Wizards, one can specify a non-default location outside of the workspace
and there is a file-system pull-down where one can choose the rse
filesystem.  From there, there is a browser to look at existing remote
connections and select an existing remote directory.  Trying this out
was relatively easy, but as it turns out the RSE adds the connection
name at the end of the URI it creates for the locaion, but does not add
the project name.  So, to create the project name directory, one has to
add the name of new project ahead of the connection query portion of the
URI.  This obviously could be improved in the future so that the editing
step would not be needed.

For my design, I chose to use RDT.  The beauty of RDT is that it allows
conversion between paths and URIs.  This is an important feature in that
it is possible to leave existing UI elements as-is and there is no real
learning curve for the end-user.  The context of paths can be implied by
the project location.  If the project is remote, the paths are on the
same remote machine and no need to specify the full URI all over the
place.  The user doesn't really need to see the remote machine URI all
over the place and by supporting paths, the data formats do not require
changing (e.g. paths occur in many classes in the CDT and there would be
no need to switch to URIs).

So, the first thing was for me to create the framework.

In my particular experiment, I created a RemoteProxyManager in
org.eclipse.cdt.core.internal.remoteproxy that implemented

public interface IRemoteProxyManager {
        String EXTENSION_POINT_ID = "RemoteProxyManager"; //$NON-NLS-1$
        String MANAGER_NAME = "manager"; //$NON-NLS-1$
        String SCHEME_ID = "scheme"; //$NON-NLS-1$
        public IRemoteFileProxy getFileProxy(IProject project) throws
        public IRemoteCommandLauncher getLauncher(IProject project)
throws CoreException;
        public String getOS(IProject project) throws CoreException;

The job of the proxy manager was to fish out proxies for file and
command access as well as to identify the remote OS.

For file access, there is the IRemoteFileProxy interface:

 public interface IRemoteFileProxy {

        public URI toURI(IPath path);
        public URI toURI(String path);
        public String toPath(URI uri);
        public String getDirectorySeparator();
        public IFileStore getResource(String path);


When accessing a path, one uses the project to get the IRemoteFileProxy
and then paths can be converted to URIs or IFileStore objects directly.
 A path for storing or displaying in the UI can be obtained from the
URI via the API.

The IRemoteCommandLauncher interface allows one to run commands on the
same system as the project:
public interface IRemoteCommandLauncher {

        public final static int OK = 0;
        public final static int COMMAND_CANCELED = 1;
        public final static int ILLEGAL_COMMAND = -1;

        public Process execute(String commandPath, String[] args,
String[] env, String changeToDirectory, IProgressMonitor monitor) throws
        public Process execute(String commandPath, String[] args,
String[] env, String changeToDirectory, boolean usePTY, IProgressMonitor
monitor) throws CoreException;
        public int waitAndRead(OutputStream output, OutputStream err,
IProgressMonitor monitor);
        public String getErrorMessage();
        public String getCommandLine();


In my implementation, I also created a plug-in extension that would
allow specification of an IRemoteProxyManager.  The remote proxy manager
would register itself to handle certain URI schema.  The default
RemoteProxyManager class only handles local references and then if not
local, looks for an extension to handle the URI schema.  It then punts
what ever method call was requested to the IRemoteProxyManager extension
class.  For example, I created a remote proxy manager that uses RDT and
handles URIs starting with rse:// and remotetools:// which handles
projects set up via RSE or RDT.

I created the separate extension because my experiment started by
altering the Autotools plug-ins in Linux Tools.  I did not want to make
Autotools force RDT to be required so I used the extension.  The plug-in
that implements the extension for RSE and RDT connections using RDT is
not required for local project users.  For the CDT, I copied over the
code from Linux Tools for the most part so I also copied over the
extension.  The extension has the benefit that other systems could be
used to plug in and handle other remote system types (e.g. IIRC, RDT
doesn't support Windows as a target).

The design could easily be altered to use RDT directly and the CDT would
then have a dependency on RDT.

Once the framework was in place, I started converting the Autotools and
CDT code to use the proxies and found a few things that needed changing
as a rule:

1. All path accesses that are project located can not use
   Instead, the project is used to get the IRemoteFileProxy and then
   a path can be used to get an IFileStore (EFS file).

2. The getLocation() method can not be used for a resource.  This
   returns null for a remote object.  Instead, one must use
   getLocationURI(), then use the IRemoteFileProxy to convert to a path
   which is what getLocation() returns.

3. Checking for existence requires getting the IFileStore as in 1), then
   using the fetchInfo() method to get the IFileInfo exists()

4. Running a command requires using a IRemoteCommandLauncher grabbed by
   using the project rather than a ICommandLauncher.

Once my changes were in place, I was able to create a remote Autotools
Hello world project using the C Project Wizard (all I did was specify a
remote location).  The remote project configures and builds on the
remote machine.  I made a change so that the Binaries folder shows up
and I am able to execute the binary on the remote system and have the
results show up in the Console.  In addition, I was able to invoke
Autotools commands remotely.

I did not change all code in the CDT as this was a proof of concept.  I
added support to the TranslationUnit so the indexer appears to work but
while it uses the correct remote source file, I did not make it use
external headers so that would require further changes.  There is a
remote indexer and if the indexer changes are not desired, the remote
indexer could be defaulted for the end-user by the C Project wizard.

I didn't actually have to change too much in the Autotools plug-ins nor
the CDT to get all of this to work.

If someone would like to see the changes I made, I have a fork of the
CDT with a branch called "rdt" which you can access read-only here:


The Linux Tools changes for Autotools can be found in the RDT branch of
the org.eclipse.linuxtools git repository.

The Autotool changes are not in the master branch and are not part of
the code move currently under way.

Would there be interest in pursuing this further?  I'm sure there are
things I haven't thought of.

I could also look into posting a screen-cast somewhere so you could see
how it all works.

-- Jeff J.
cdt-dev mailing list

GIF image

GIF image

Back to the top