Home » Archived » Buckminster » Generic RMAP?
| Generic RMAP? [message #693754] |
Thu, 07 July 2011 07:08  |
 Mike Haney
Messages: 25
Registered: June 2011 |
Junior Member |
|
|
We just finished a proof-of-concept using Buckminister for our developer provisioning and build environment. Everything worked great, now I am setting this up for a major project.
We are using Git, and after a lot of back and forth about repository granularity, we decided to use 1 Git repos per Eclipse feature. We have established a naming convention whereby the Git repos name is the name of the feature, and all components in that repos begin with that feature name. So for example, the /myproj/mail repository would contain myproj.mail.feature, myproj.mail.plugin1, etc.
This works out really well for the RMAP, because I can simply build the regex for the locator based on the feature name, as such:
<rm:locator pattern="^myproj.mail(\..+)?" searchPathRef="mail" />
<rm:searchPath name="mail">
<rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="git">
<rm:property key="git.remote.uri" value="ssh://git@myserver/myproj/mail"/>
<rm:uri format="{0}/mail,{1}">
and so on, for each feature.
Now, is there a way I can take this a step further and have a generic locator with a pattern such as "^myproj(\..+)?" and within the searchPath be able to extract the second component of the feature name and use it as a parameter for repos name?
That would allow me to have 1 RMAP that covers the entire project, and not have to update it every time we add a feature repos.
Thanks,
Mike
|
|
| | | | |
| Git/Buckminster workflow [message #697196 is a reply to message #697168] |
Fri, 15 July 2011 18:38  |
Mike Haney
Messages: 25
Registered: June 2011 |
Junior Member |
|
|
OK, here is our workflow for working with Git and Buckminster.
Overview
We are using Gitolite on our server to manage our git repositories, and we use Pulse to manage Eclipse installations, although I have tried all of this with an unmanaged install of Eclipse and it works fine.
Our app is an RCP/RAP single-sourced app with a few base products and a ton of plugins/features shared between them. Our project is named "teamwork", and by convention we are using 1 git repository per main Eclipse feature. To make this all work, we use a simple naming convention, whereby the repository name is the name of the main feature, and all plugins are in the root folder of the repos and are named as <project-name>.<feature-name>.*
So for example, let's say we create the sample "mail" app in the "teamwork" project, the structure would look like this:
/mail
/mail/.git
/mail/teamwork.mail.app
/mail/teamwork.mail.feature
/mail/teamwork.mail.extra-plugin1
etc...
On our Git server, I have setup Gitolite to use wildcard repositories under the /teamwork namespace. I'll explain more about this next.
Git server setup
I won't get into general Gitolite setup, because it is very well documented and pretty easy to use. The relevant part of gitolite.conf for our purposes looks like this:
repo teamwork/[a-zA-Z0-9].*
C = mike
RW+ = mike kyle kyle-work
R = @all
What this says is that the user mike (that's me) can push any repository to the /teamwork namespace and if it doesn't exist, it will be created. Myself and another developer (kyle and kyle-work - he has 2 different keys for different machines) can push and pull to whatever repositories exist in that namespace, and the rest of the developers have read-only access (we're not giving junior developers commit access, at least until they become more familiar with git).
This is very simple because we are a small team - larger installs would probably want to define user groups, etc. Also note that there are very robust permission capabilities in gitolite, such as the ability to set the creator of a repos as the owner, branch-level permissions, etc. These are beyond the scope of this writeup, but the gitolite documentation is very good and it's worth looking into. For example, we will probably soon go to wildcard branches with permissions such that junior developers can only push to their own branch namespace, and then the senior devs will merge their changes into "master" after review (as opposed to now, where we pull directly from their local repos or have them send us patches).
The Magic Rmap
Given this repository structure, and the naming conventions set out earlier, a generic rmap was pretty simple to create. Here is our actual rmap for the project mentioned earlier (capforge is the name of our internal dev server):
<?xml version="1.0" encoding="UTF-8"?>
<rm:rmap xmlns:bc="http://www.eclipse.org/buckminster/Common-1.0" xmlns:rm="http://www.eclipse.org/buckminster/RMap-1.0">
<rm:locator pattern="^teamwork(\..+)?" searchPathRef="teamwork"/>
<rm:searchPath name="teamwork">
<rm:provider componentTypes="osgi.bundle,eclipse.feature" readerType="git">
<rm:property key="git.remote.name" value="origin"/>
<rm:property key="git.auto.fetch" value="true"/>
<rm:propertyElement key="git.remote.uri">
<bc:format format="ssh://git@capforge/teamwork/{0}">
<bc:format format="{1}">
<bc:split pattern="\." style="unquoted">
<bc:propertyRef key="buckminster.component"/>
</bc:split>
</bc:format>
</bc:format>
</rm:propertyElement>
<rm:uri format="{0}/{1},{2}">
<bc:propertyRef key="workspace.root"/>
<bc:format format="{1}">
<bc:split pattern="\." style="unquoted">
<bc:propertyRef key="buckminster.component"/>
</bc:split>
</bc:format>
<bc:propertyRef key="buckminster.component"/>
</rm:uri>
</rm:provider>
</rm:searchPath>
</rm:rmap>
Easy stuff. I actually thought of taking it to the next level and making the project name generic as well, but couldn't figure out a good way to write the expression for the locator. I guess it would work if we added our company prefix to all of the projects, e.g. com.ebix.teamwork.* instead of just teamwork.*, but we're too lazy. I only have a few top-level project namespaces to deal with, so copying the rmap was easy enough.
Putting it all together
OK, so let's say I'm creating the obligatory mail demo app. First, I would generate the plugins in Eclipse and end up with teamwork.mail.app and teamwork.mail.feature. Then I share those in Eclipse and use EGit to create a new repository named "mail" to contain both projects, add, commit, all that stuff. Now I'm ready to push.
The project doesn't exist in Gitolite yet, but no worries, because our config is setup for that. There are 2 ways to force the creation of a wildcard repos:
1. clone using the desired url (i.e. git@capforge/teamwork/mail) and then pull my local mail repos into the clone. For a smooth workflow, it would be great if we could do this cloning when sharing the projects using EGit, but in what is probably a bug, EGit errors when you try to clone an empty repository.
2. the method I usually use is to just push the new repository (created locally when sharing the projects) to the desired url. I usually do this from the command-line, but I'm pretty sure it works with EGit as well, I'm just a creature of habit. But from the commandline, in the local "mail" repos directory, it's pretty simple:
$git remote add origin ssh://git@capforge/teamwork/mail
$git push origin --all
The --all is important, since the master branch doesn't exist on the server yet (in fact, the entire repository doesn't exist on the server yet!).
One other note - if you are like me and constantly bounce between Eclipse and the command-line for your Git work, it might be a good idea to delete this local clone after you have pushed it. The reason I say this is because I have my default Git repository set to /USERHOME/git in Eclipse, which makes it very convenient for working with repositories. However, by default Buckminster will clone repositories into the workspace root. If you only use Eclipse, this really doesn't matter, but since I bounce around a lot between many projects/repos/etc., it's very easy to forget which clone you are working in.
Sharing projects
When other developers need to work on our new "teamwork.mail" feature, they can now just run a cquery for "teamwork.mail.feature". I put our rmap on our server so everyone can access it easily, and I created cqueries for the common projects and also put them on the server.
Using Pulse, I even went one step further and shared the preferences that save the MRU list for the File/Open Component Query dialog, so when developers create a new workspace, the dropdown list is prepopulated with our most common project queries.
Limitations and Enhancements
The main limitation I am working on right now is with target platform management. I was going to open a separate thread on this, but basically I cannot get my queries to work unless I set the target platform manually before running them. I tried various methods of creating empty targets and none of them worked. So for now I have a small project everyone can access containing just target definitions for different projects. This is a bit cumbersome, so any suggestions would be very welcome.
Right now, we manually document in our project wiki all of the features and their corresponding repositories, required target platform, required Eclipse profile, and the cquery to use to materialize that project. I am thinking about ways to automate some or all of this information.
One enhancement I am anxious to make is to automate project creation in Jira when new repositories are created. We made the decision to create a Jira project per component/feature, since eventually each feature will have it's own lifecycle, version, etc. It should be possible to setup a hook in Gitolite so that when our example "/teamwork/mail" repository is created, it runs the Jira command-line tool and creates a matching Jira project with all our default settings.
I hope this has been helpful, and I welcome any questions or suggestions for improvement.
|
|
| |
| Re: Git/Buckminster workflow [message #704109 is a reply to message #697196] |
Thu, 28 July 2011 09:30 |
Matthew Webber
Messages: 198
Registered: December 2010 |
Senior Member |
|
|
We have similar problem to solve, and I'd like to share what we're currently trialling. This is based on Mike's excellent ideas, but needed to solve a slightly different problem. Previously, all our code was in a single Subversion repository. We're moving some (but not all) of it out to a number of Git repositories.
Mike said that "We have established a naming convention whereby the Git repos name is the name of the feature, and all components in that repos begin with that feature name". Unfortunately we don't have that luxury - too much existing code. So this part of Mike's RMap won't quite work for us:
<rm:propertyElement key="git.remote.uri">
<bc:format format="ssh://git@capforge/teamwork/{0}">
<bc:format format="{1}">
<bc:split pattern="\." style="unquoted">
<bc:propertyRef key="buckminster.component"/>
</bc:split>
</bc:format>
</bc:format>
</rm:propertyElement>
<rm:uri format="{0}/{1},{2}">
<bc:propertyRef key="workspace.root"/>
<bc:format format="{1}">
<bc:split pattern="\." style="unquoted">
<bc:propertyRef key="buckminster.component"/>
</bc:split>
</bc:format>
<bc:propertyRef key="buckminster.component"/>
</rm:uri>
However, I noticed that Mike's RMap contained some duplicated logic which determines the repository name:
<bc:split pattern="\." style="unquoted">
<bc:propertyRef key="buckminster.component"/>
</bc:split>
So what I did was move the logic to determine the repository name out to a property:
<rm:propertyElement key="internal.git.repository.name">
<bc:replace>
<bc:propertyRef key="buckminster.component"/>
<bc:match pattern="^ncsa\.hdf$" replacement="scisoft-hdf.git"/>
<bc:match pattern="^uk\.ac\.diamond\.CBFlib$" replacement="scisoft-cbflib.git"/>
<bc:match pattern="^uk\.ac\.diamond\.scisoft\.(ncd|peema|spectroscopy)(\..+)?" replacement="scisoft-$1.git"/>
<bc:match pattern="^uk\.ac\.diamond\.scisoft\.(core\.feature|analysis|python)$" replacement="scisoft-core.git"/>
<bc:match pattern="^uk\.ac\.diamond\.(sda\.product\.feature|scisoft|scisoft\.intro)$" replacement="scisoft-sda.git"/>
<bc:match pattern="^uk\.ac\.diamond\.(scisoft\.ui\.feature|sda\.polling)$" replacement="scisoft-ui.git"/>
<bc:match pattern="^uk\.ac\.diamond\.scisoft\.analysis\..+" replacement="scisoft-ui.git"/>
<bc:match pattern="^.+(?!\.git)$" replacement="NONE"/>
</bc:replace>
</rm:propertyElement>
Given a component name, this determines which Git respoitory it is in. If it's not in one, it evaluates to NONE. Note that every "match" element is applied (in order) - it doesn't stop at the first match as you might expect - so that match patterns need to be crafted quite carefully.
Then in the relevant RMap provider, I do this:
<rm:provider componentTypes="osgi.bundle,eclipse.feature" resolutionFilter="(&(repository.git.download.enabled=true)(!(internal.git.repository.name=NONE)))" readerType="git">
<rm:property key="git.auto.fetch" value="true"/>
<rm:propertyElement key="git.remote.uri">
<bc:format format="${repository.git.download.url.diamond}/{0}">
<bc:propertyRef key="internal.git.repository.name"/>
</bc:format>
</rm:propertyElement>
<rm:uri format="{0}/{1},{2}">
<bc:propertyRef key="location.root"/>
<bc:propertyRef key="internal.git.repository.name"/>
<bc:propertyRef key="buckminster.component"/>
</rm:uri>
</rm:provider>
If the particular component is has not yet been moved to Git, as evidenced by internal.git.repository.name=NONE, then this provider will simply be skipped (and we move on to the next provider, which goes to our Subversion repository).
So although our lack of a rigid naming convention means that the mapping to a respoitory name is complex, it's only done in one place. As we move more plugins over to Git, it's fairly straighforward to update the RMap to reflect that.
I'll keep the thread updated on how this works in practice.
Matthew Webber
|
|
|
Goto Forum:
Current Time: Sat Jan 18 10:09:00 GMT 2025
Powered by FUDForum. Page generated in 0.03750 seconds |
] 
Search
Help
Register
Login
Home
