On 13-Mar-09, at 12:55 PM, Pascal Rapicault wrote:
Moving the conversation to the p2-dev.
I had seen your announcement about Nexus support for p2 repos and I was waiting for EclipseCon to chat with you there.
> Not sure if anyone is interested but we can support building bundles
> with Tycho and deploying the bundle to the remote Nexus.
When would the p2 metadata generation happen?
As the bundles are deployed.
Also what happens to the metadata in case of a copy from repos, I assume that Nexus stores it too?
If by copy you mean a physically host P2 repository, we would regenerate the metadata to make sure it's intact. This is what we do for Maven repositories as well in Nexus. In the case of the Orbit zip we downloaded it didn't seem to have any so we just generated it. Right now we are using the standard P2 metadata generator but we're finding it uses a _lot_ of memory so we're not sure if it's something that's known and will be fixed. We might take a crack at writing one as while we are combining many repositories it's doubling the normal memory requirements of Nexus.
> I'm going to try and expose the Nexus instance we have with P2 support today.
> We can easily allow people to proxy the Orbit P2 repository once
> it's up but we can provide one for folks as an experiment. I really
> just want to see what happens. But I assume things like PDE headless
> build and Buckminster can try this stuff right away so hopefully it
> will be useful.
We can give it a try, I assume also that we should be able to simply browse it as an artifact repository and mirror from it.
Let us know what you have and what to try.
Everything is browsable, from a more webserver type look or through the EXT-JS interface, or via the REST API if you want to do your own thing.
> We also made a bunch of host P2 repositories for AspectJ and
> Subclipse. Basically what we need to create a single P2 "group" that
> allows m2eclipse users to point at a single URL and get the
> aggregated set of update sites for everything they need.
In p2, a "group" is something installable whose all dependencies will be installed.
Instead I think you mean probably composite repository or aggregation of repository no?
Maybe our terminology is not quite the same but in Nexus we have hosted repositories which are physical repositories that you store. They maybe copies as is the case with Orbit, the Subclipse P2 repository we made, and the AspectJ repository we made. We also have proxied repositories so in this case we proxy the Ganymede P2 repository. Repository groups in Nexus can contain any number of hosted and proxied repositories. So for our m2eclipse demonstration we create a group which contains our hosted Subclipse, AspectJ and m2e repositories and a Ganymede proxy.
Inside Eclipse we wipe our all the URLs in the Update Manager and point it at the group repository URL. All the P2 metadata will be merged dynamically so that what appears to be there is a single P2 repository containing all the features we need for a successful m2eclipse install. One stop shopping.
> have a "lockdown list" of sorts which is really to pick the root set
> of IUs which basically hides any other versions for stability. A
> profile essentially.
In p2, a profile is the result of an installation. Do you mean that this is software made to work together like a "distro" (or repo controlled line up), e.g. all the milestones sites for a set of repos?
Yes, it is essentially a versioned distribution. I might have 4 versions of Subclipse in a hosted repository but I don't want the resolver to "see" it. Basically locking down the first level of IUs so that you can definitively control what the client is going to consume. Our use case is that for a Eclipse installation you need something definitive, and for server side your QA people are going to test a very specific set of bundles and say they are OK. You need to make sure that same set of bundles shows up on the server. If you use version ranges in your bundles to allow for dynamic updating you cannot point a resolver at a boundless repository. So the lock down lists are essentially a constraint to deal with the dichotomy of "I must know that what I use is what I tested, but I still must be able to incrementally update bundles. So this lock down (or whatever we end up calling it) is something that a QA engineer could manage. The QA person tests a new bundle and updates the lock down list and creates a new version which might reveal 1..N new versions of bundles. The client on the other side when resolving will now see the new versions and update. We didn't really see a large managed P2 repository being very useful for production purposes without this constraining layer. The case where you are continually deploying to a P2 repository and updating, you definitely do not want a resolver picking up new versions just because they happen to be in the repository. We figured this provides the flexibility so that you have content you can actively manage as opposed to producing static repositories with one-time generated metadata which will soon be out of date. Especially in cases where you are repeatedly building new bundles test over and over again.
What we have is a fully working system, but we've only had it working for a week. But we can use this mechanism to install m2eclipse completely from Nexus and we are moving very fast.
> How are you guys deploying to remote repositories? I've seen mention
> of a P2 publisher which I assume is file system based as I can't see
> any reference to remote repository deployment anywhere. If there is
> such a remote deployment mechanism I'd like to take a look at the
> code to see if I could wire it up to Nexus.
The publisher / generator can not publish to a remote repository. However the blocking / missing part is not the publisher itself, but simply the lack of an implementation of artifact and metadata repository that can be written to remotely. Therefore, right now what we do to publish a new build is copy the files over to the server. To avoid the merging headache we have a concept of composite repository that allows for several repos to be seen as one.
Nexus can understand any protocol at this point so I'm willing to bet during EclipseCon we could make something that deployed to Nexus. For the same reason we can't use Nexus for Maven Central you could not use Nexus to serve out the content to most of the users. But it would be relatively simple to deploy to and manage the cotent in Nexus and then create a lock down list, test against Nexus for the all the projects and then have it eject a static version of the lock down list for serving up with Apache (or NGinx which is what we use on Maven Central because it is so unbelievably fast).
And I'll be completely forthright in that this is a commercial implementation. But we're giving away commercial versions to any OSS organization in perpetuity. Apache and XWiki are already running copies. It's only 3k anyway so it's not like it will break anyone's piggy bank. I think it could help here and I would hope being a commercial product is not a problem.
> Still working toward some useful and functional demos for the OSGi tooling summit.
Andrew "PDE Build" Niefer and I won't be able to make it to the summit, but we can probably chat before that.
I think it would be relatively straight forward to get PDE Build to publish and consume from Nexus if you guys wanted to try and hack it together at EclilpseCon.
PaScaL
> dj
>
> [1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=241427
> [2] https://bugs.eclipse.org/bugs/show_bug.cgi?id=266232
>
>
>
> orbit-dev-bounces@xxxxxxxxxxx wrote on 03/12/2009 06:30:12 AM:
>
> > Hi Jason,
> >
> > DJ is working on it:
> > https://bugs.eclipse.org/bugs/show_bug.cgi?id=241427
> >
> > Cheers,
> > --
> > Martin Oberhuber, Senior Member of Technical Staff, Wind River
> > Target Management Project Lead, DSDP PMC Member
> > http://www.eclipse.org/dsdp/tm
> >
> >
> >
> > > -----Original Message-----
> > > From: orbit-dev-bounces@xxxxxxxxxxx
> > > [mailto:orbit-dev-bounces@xxxxxxxxxxx] On Behalf Of Jason van Zyl
> > > Sent: Donnerstag, 12. März 2009 09:25
> > > To: Orbit Developer discussion
> > > Subject: [orbit-dev] Best way to get a binary copy of the
> > > latest Orbitbundles
> > >
> > > Hi,
> > >
> > > Would there be a nice way to get the latest release build of
> > > Orbit in
> > > an automated way?
> > >
> > > Nexus is an artifact repository manager that now has
> > > prototype support
> > > for proxied P2 repositories, hosted P2 repositories and arbitrary
> > > groupings of both types. P2 repositories in Nexus now have the same
> > > capabilities Maven repositories have so full RBAC access,
> > > staging and
> > > promotion, grouping, ordering, routing tables, RSS feeds, audit, the
> > > whole nine yards. What I'm trying to do is publish as many standard
> > > bundles as I can in a public instance of Nexus so that users
> > > who wish
> > > to try can provision bundles from Nexus in the same way a Maven user
> > > would provision normal JARs.
> > >
> > > Is there a P2 repository somewhere I can just consume what would be
> > > considered the latest release versions of all the bundles? Or even
> > > just the raw bundles are fine, Nexus can generate the necessary P2
> > > metadata as bundles are deployed into Nexus so I'll take anything
> > > really.
> > >
> > > Thanks,
> > >
> > > Jason
> > >
> > > ----------------------------------------------------------
> > > Jason van Zyl
> > > Founder, Apache Maven
> > > http://twitter.com/jvanzyl
> > > ----------------------------------------------------------
> > >
> > > We all have problems. How we deal with them is a measure of our worth.
> > >
> > > -- Unknown
> > >
> > > Thanks,
> > >
> > > Jason
> > >
> > > ----------------------------------------------------------
> > > Jason van Zyl
> > > Founder, Apache Maven
> > > http://twitter.com/jvanzyl
> > > ----------------------------------------------------------
> > >
> > > You are never dedicated to something you have complete confidence in.
> > > No one is fanatically shouting that the sun is going to rise tomorrow.
> > > They know it is going to rise tomorrow. When people are fanatically
> > > dedicated to political or religious faiths or any other kind of
> > > dogmas or goals, it's always because these dogmas or
> > > goals are in doubt.
> > >
> > > -- Robert Pirzig, Zen and the Art of Motorcycle Maintenance
> > >
> > > _______________________________________________
> > > orbit-dev mailing list
> > > orbit-dev@xxxxxxxxxxx
> > > https://dev.eclipse.org/mailman/listinfo/orbit-dev
> > >
> > _______________________________________________
> > orbit-dev mailing list
> > orbit-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/orbit-dev
> _______________________________________________
> orbit-dev mailing list
> orbit-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/orbit-dev
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> ----------------------------------------------------------
>
> happiness is like a butterfly: the more you chase it, the more it will
> elude you, but if you turn your attention to other things, it will come
> and sit softly on your shoulder ...
>
> -- Thoreau
> _______________________________________________
> orbit-dev mailing list
> orbit-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/orbit-dev
p2-dev mailing list
p2-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/p2-dev
