Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Buckminster » Generic RMAP?
Generic RMAP? [message #693754] Thu, 07 July 2011 07:08 Go to next message
Mike Haney is currently offline Mike HaneyFriend
Messages: 25
Registered: June 2011
Junior Member
We just finished a proof-of-concept using Buckminister for our developer provisioning and build environment. Everything worked great, now I am setting this up for a major project.

We are using Git, and after a lot of back and forth about repository granularity, we decided to use 1 Git repos per Eclipse feature. We have established a naming convention whereby the Git repos name is the name of the feature, and all components in that repos begin with that feature name. So for example, the /myproj/mail repository would contain myproj.mail.feature, myproj.mail.plugin1, etc.

This works out really well for the RMAP, because I can simply build the regex for the locator based on the feature name, as such:

<rm:locator pattern="^myproj.mail(\..+)?" searchPathRef="mail" />
<rm:searchPath name="mail">
<rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="git">
<rm:property key="git.remote.uri" value="ssh://git@myserver/myproj/mail"/>
<rm:uri format="{0}/mail,{1}">

and so on, for each feature.

Now, is there a way I can take this a step further and have a generic locator with a pattern such as "^myproj(\..+)?" and within the searchPath be able to extract the second component of the feature name and use it as a parameter for repos name?

That would allow me to have 1 RMAP that covers the entire project, and not have to update it every time we add a feature repos.

Thanks,

Mike
Re: Generic RMAP? [message #695431 is a reply to message #693754] Mon, 11 July 2011 18:01 Go to previous messageGo to next message
Mike Haney is currently offline Mike HaneyFriend
Messages: 25
Registered: June 2011
Junior Member
I figured this out. I tried to post my solution here, but it complains that I don't have enough posts to add links (must be picking up the namespace uris from the rmap file or something).

Actually, we have setup a nice little workflow using Git and Buckminster that allows us to dynamically create repositories per-feature, add them to our Git server with no additional admin needed, and find them with Buckminster using a generic RMAP. Now I am working on integrating with Jira to automatically create a Jira project when a new repository is created.

Anyway, if anyone is interested in this workflow, let me know and when I get time I'll write it up on the wiki or something.
(no subject) [message #695507 is a reply to message #695431] Mon, 11 July 2011 22:07 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2500
Registered: July 2009
Senior Member
On 7/11/11 8:01 PM, Mike Haney wrote:
> I figured this out. I tried to post my solution here, but it complains
> that I don't have enough posts to add links

Sorry - the forum software is braindamaged...
Just post 5 times in some suitable forum with words to that effect...

>
> Actually, we have setup a nice little workflow using Git and Buckminster
> that allows us to dynamically create repositories per-feature, add them
> to our Git server with no additional admin needed, and find them with
> Buckminster using a generic RMAP. Now I am working on integrating with
> Jira to automatically create a Jira project when a new repository is
> created.
>
Sounds cool.

regards
- henrik
Re: (no subject) [message #696650 is a reply to message #695507] Thu, 14 July 2011 13:46 Go to previous messageGo to next message
Matthew Webber is currently offline Matthew WebberFriend
Messages: 198
Registered: December 2010
Senior Member
Hi Mike,
Are you able to share your solution with us? We have a very similar problem.
Matthew
Re: (no subject) [message #697168 is a reply to message #696650] Fri, 15 July 2011 17:36 Go to previous messageGo to next message
Mike Haney is currently offline Mike HaneyFriend
Messages: 25
Registered: June 2011
Junior Member
OK, I will share it in just a few minutes (this post should get me over the limit so the forum doesn't complain).
Git/Buckminster workflow [message #697196 is a reply to message #697168] Fri, 15 July 2011 18:38 Go to previous messageGo to next message
Mike Haney is currently offline Mike HaneyFriend
Messages: 25
Registered: June 2011
Junior Member
OK, here is our workflow for working with Git and Buckminster.

Overview

We are using Gitolite on our server to manage our git repositories, and we use Pulse to manage Eclipse installations, although I have tried all of this with an unmanaged install of Eclipse and it works fine.

Our app is an RCP/RAP single-sourced app with a few base products and a ton of plugins/features shared between them. Our project is named "teamwork", and by convention we are using 1 git repository per main Eclipse feature. To make this all work, we use a simple naming convention, whereby the repository name is the name of the main feature, and all plugins are in the root folder of the repos and are named as <project-name>.<feature-name>.*

So for example, let's say we create the sample "mail" app in the "teamwork" project, the structure would look like this:

/mail
/mail/.git
/mail/teamwork.mail.app
/mail/teamwork.mail.feature
/mail/teamwork.mail.extra-plugin1
etc...

On our Git server, I have setup Gitolite to use wildcard repositories under the /teamwork namespace. I'll explain more about this next.

Git server setup

I won't get into general Gitolite setup, because it is very well documented and pretty easy to use. The relevant part of gitolite.conf for our purposes looks like this:

repo teamwork/[a-zA-Z0-9].*
   C = mike
   RW+ = mike kyle kyle-work
   R = @all


What this says is that the user mike (that's me) can push any repository to the /teamwork namespace and if it doesn't exist, it will be created. Myself and another developer (kyle and kyle-work - he has 2 different keys for different machines) can push and pull to whatever repositories exist in that namespace, and the rest of the developers have read-only access (we're not giving junior developers commit access, at least until they become more familiar with git).

This is very simple because we are a small team - larger installs would probably want to define user groups, etc. Also note that there are very robust permission capabilities in gitolite, such as the ability to set the creator of a repos as the owner, branch-level permissions, etc. These are beyond the scope of this writeup, but the gitolite documentation is very good and it's worth looking into. For example, we will probably soon go to wildcard branches with permissions such that junior developers can only push to their own branch namespace, and then the senior devs will merge their changes into "master" after review (as opposed to now, where we pull directly from their local repos or have them send us patches).

The Magic Rmap
Given this repository structure, and the naming conventions set out earlier, a generic rmap was pretty simple to create. Here is our actual rmap for the project mentioned earlier (capforge is the name of our internal dev server):

<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
  <rm:locator pattern="^teamwork(\..+)?" searchPathRef="teamwork"/>
  <rm:searchPath name="teamwork">
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="git">
      <rm:property key="git.remote.name" value="origin"/>
      <rm:property key="git.auto.fetch" value="true"/>
      <rm:propertyElement key="git.remote.uri">
        <bc:format format="ssh://git@capforge/teamwork/{0}">
          <bc:format format="{1}">
            <bc:split pattern="\." style="unquoted">
              <bc:propertyRef key="buckminster.component"/>
            </bc:split>
          </bc:format>
        </bc:format>
      </rm:propertyElement>
      <rm:uri format="{0}/{1},{2}">
        <bc:propertyRef key="workspace.root"/>
        <bc:format format="{1}">
          <bc:split pattern="\." style="unquoted">
            <bc:propertyRef key="buckminster.component"/>
          </bc:split>
        </bc:format>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>
    </rm:provider>
  </rm:searchPath>
</rm:rmap>


Easy stuff. I actually thought of taking it to the next level and making the project name generic as well, but couldn't figure out a good way to write the expression for the locator. I guess it would work if we added our company prefix to all of the projects, e.g. com.ebix.teamwork.* instead of just teamwork.*, but we're too lazy. I only have a few top-level project namespaces to deal with, so copying the rmap was easy enough.

Putting it all together

OK, so let's say I'm creating the obligatory mail demo app. First, I would generate the plugins in Eclipse and end up with teamwork.mail.app and teamwork.mail.feature. Then I share those in Eclipse and use EGit to create a new repository named "mail" to contain both projects, add, commit, all that stuff. Now I'm ready to push.

The project doesn't exist in Gitolite yet, but no worries, because our config is setup for that. There are 2 ways to force the creation of a wildcard repos:

1. clone using the desired url (i.e. git@capforge/teamwork/mail) and then pull my local mail repos into the clone. For a smooth workflow, it would be great if we could do this cloning when sharing the projects using EGit, but in what is probably a bug, EGit errors when you try to clone an empty repository.

2. the method I usually use is to just push the new repository (created locally when sharing the projects) to the desired url. I usually do this from the command-line, but I'm pretty sure it works with EGit as well, I'm just a creature of habit. But from the commandline, in the local "mail" repos directory, it's pretty simple:

$git remote add origin ssh://git@capforge/teamwork/mail
$git push origin --all


The --all is important, since the master branch doesn't exist on the server yet (in fact, the entire repository doesn't exist on the server yet!).

One other note - if you are like me and constantly bounce between Eclipse and the command-line for your Git work, it might be a good idea to delete this local clone after you have pushed it. The reason I say this is because I have my default Git repository set to /USERHOME/git in Eclipse, which makes it very convenient for working with repositories. However, by default Buckminster will clone repositories into the workspace root. If you only use Eclipse, this really doesn't matter, but since I bounce around a lot between many projects/repos/etc., it's very easy to forget which clone you are working in.

Sharing projects
When other developers need to work on our new "teamwork.mail" feature, they can now just run a cquery for "teamwork.mail.feature". I put our rmap on our server so everyone can access it easily, and I created cqueries for the common projects and also put them on the server.

Using Pulse, I even went one step further and shared the preferences that save the MRU list for the File/Open Component Query dialog, so when developers create a new workspace, the dropdown list is prepopulated with our most common project queries.

Limitations and Enhancements
The main limitation I am working on right now is with target platform management. I was going to open a separate thread on this, but basically I cannot get my queries to work unless I set the target platform manually before running them. I tried various methods of creating empty targets and none of them worked. So for now I have a small project everyone can access containing just target definitions for different projects. This is a bit cumbersome, so any suggestions would be very welcome.

Right now, we manually document in our project wiki all of the features and their corresponding repositories, required target platform, required Eclipse profile, and the cquery to use to materialize that project. I am thinking about ways to automate some or all of this information.

One enhancement I am anxious to make is to automate project creation in Jira when new repositories are created. We made the decision to create a Jira project per component/feature, since eventually each feature will have it's own lifecycle, version, etc. It should be possible to setup a hook in Gitolite so that when our example "/teamwork/mail" repository is created, it runs the Jira command-line tool and creates a matching Jira project with all our default settings.

I hope this has been helpful, and I welcome any questions or suggestions for improvement.
Re: Git/Buckminster workflow [message #699873 is a reply to message #697196] Fri, 22 July 2011 14:11 Go to previous messageGo to next message
Matthew Webber is currently offline Matthew WebberFriend
Messages: 198
Registered: December 2010
Senior Member
Thanks Mike, I really appreciate the detail you have gone to. We can certainly incorporate some of these ideas in our own build environment.
Matthew
Re: Git/Buckminster workflow [message #704109 is a reply to message #697196] Thu, 28 July 2011 09:30 Go to previous message
Matthew Webber is currently offline Matthew WebberFriend
Messages: 198
Registered: December 2010
Senior Member
We have similar problem to solve, and I'd like to share what we're currently trialling. This is based on Mike's excellent ideas, but needed to solve a slightly different problem. Previously, all our code was in a single Subversion repository. We're moving some (but not all) of it out to a number of Git repositories.

Mike said that "We have established a naming convention whereby the Git repos name is the name of the feature, and all components in that repos begin with that feature name". Unfortunately we don't have that luxury - too much existing code. So this part of Mike's RMap won't quite work for us:

      <rm:propertyElement key="git.remote.uri">
        <bc:format format="ssh://git@capforge/teamwork/{0}">
          <bc:format format="{1}">
            <bc:split pattern="\." style="unquoted">
              <bc:propertyRef key="buckminster.component"/>
            </bc:split>
          </bc:format>
        </bc:format>
      </rm:propertyElement>
      <rm:uri format="{0}/{1},{2}">
        <bc:propertyRef key="workspace.root"/>
        <bc:format format="{1}">
          <bc:split pattern="\." style="unquoted">
            <bc:propertyRef key="buckminster.component"/>
          </bc:split>
        </bc:format>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>


However, I noticed that Mike's RMap contained some duplicated logic which determines the repository name:
            <bc:split pattern="\." style="unquoted">
              <bc:propertyRef key="buckminster.component"/>
            </bc:split>


So what I did was move the logic to determine the repository name out to a property:
  <rm:propertyElement key="internal.git.repository.name">
    <bc:replace>
      <bc:propertyRef key="buckminster.component"/>
      <bc:match pattern="^ncsa\.hdf$" replacement="scisoft-hdf.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.CBFlib$" replacement="scisoft-cbflib.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.scisoft\.(ncd|peema|spectroscopy)(\..+)?" replacement="scisoft-$1.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.scisoft\.(core\.feature|analysis|python)$" replacement="scisoft-core.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.(sda\.product\.feature|scisoft|scisoft\.intro)$" replacement="scisoft-sda.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.(scisoft\.ui\.feature|sda\.polling)$" replacement="scisoft-ui.git"/>
      <bc:match pattern="^uk\.ac\.diamond\.scisoft\.analysis\..+" replacement="scisoft-ui.git"/>
      <bc:match pattern="^.+(?!\.git)$" replacement="NONE"/>
    </bc:replace>
  </rm:propertyElement>

Given a component name, this determines which Git respoitory it is in. If it's not in one, it evaluates to NONE. Note that every "match" element is applied (in order) - it doesn't stop at the first match as you might expect - so that match patterns need to be crafted quite carefully.

Then in the relevant RMap provider, I do this:
    <rm:provider componentTypes="osgi.bundle,eclipse.feature" resolutionFilter="(&amp;(repository.git.download.enabled=true)(!(internal.git.repository.name=NONE)))" readerType="git">
      <rm:property key="git.auto.fetch" value="true"/>
      <rm:propertyElement key="git.remote.uri">
        <bc:format format="${repository.git.download.url.diamond}/{0}">
          <bc:propertyRef key="internal.git.repository.name"/>
        </bc:format>
      </rm:propertyElement>
      <rm:uri format="{0}/{1},{2}">
        <bc:propertyRef key="location.root"/>
        <bc:propertyRef key="internal.git.repository.name"/>
        <bc:propertyRef key="buckminster.component"/>
      </rm:uri>
    </rm:provider>

If the particular component is has not yet been moved to Git, as evidenced by internal.git.repository.name=NONE, then this provider will simply be skipped (and we move on to the next provider, which goes to our Subversion repository).

So although our lack of a rigid naming convention means that the mapping to a respoitory name is complex, it's only done in one place. As we move more plugins over to Git, it's fairly straighforward to update the RMap to reflect that.

I'll keep the thread updated on how this works in practice.

Matthew Webber

Previous Topic:Build failure after upgrading to Bucky 3.7
Next Topic:Installing Buckminster in Helios requires an old version of EGit/JGit
Goto Forum:
  


Current Time: Sun Nov 23 06:03:57 GMT 2014

Powered by FUDForum. Page generated in 0.10213 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software