[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [p2-dev] Re: [orbit-dev] Best way to get a binary copy of the latest Orbitbundles

Pascal Rapicault wrote:
 > > Not sure if anyone is interested but we can support building bundles
 > > with Tycho and deploying the bundle to the remote Nexus.
 > When would the p2 metadata generation happen?
 > As the bundles are deployed.
 > Also what happens to the metadata in case of a copy from repos, I
 > assume that Nexus stores it too?

> If by copy you mean a physically host P2 repository, we would
> regenerate the metadata to make sure it's intact.
This is a major problem. Metadata should/can not be regenerated. The main reasons are:
1) An IU should be treated immutably. Id + version == immutable content. The IU is for a particular artifact is the responsibility of the owner of the artifact.
2) The binary can not always be reverse engineered to produce the IU that the owner of the artifact would have created. It happens that you have been successful so far because orbit metadata is a straight transcription of the bundle manifest. However this breaks when a) the generator is changed, b) the owner of the artifact used a p2.inf to specify more information (e.g. overwrite capability, requirement, etc). c) the artifact is not something that p2 metadata gen recognize.
3) Also some IUs are pure metadata, there is no artifact associated with it that needs to be stored and deployed. An example of such a thing is the top leve IU of a product, or the configuration units.

I believe Nexus either already does or can be extended relatively easily to do exactly as you describe

1) Once deployed, artifact metadata does not change, which effectively means Id + version == immutable content. Nexus can also be configured to block redeployment of artifacts with same id/version.

2) Deployment to Nexus is not limited to "real" artifacts, like bundle jar files. Current prototype always generates IU from binaries, we're prepared to support explicit IU and "metadata-only" IU. It is a simple matter of writing code ;-)

> for Maven repositories as well in Nexus. In the case of the Orbit
> zip we downloaded it didn't seem to have any so we just generated
> it. Right now we are using the standard P2 metadata generator but
> we're finding it uses a _lot_ of memory so we're not sure if it's
> something that's known and will be fixed. We might take a crack at
> writing one as while we are combining many repositories it's
> doubling the normal memory requirements of Nexus.
Most of the memory consumption is caused by the vast amount of strings that result from the generation because we do not try to intern at generation time like we do at runtime. But the core issue is the SimpleMetadataRepository implementation that loads the complete repo. If you are really worried about memory consumption rather than changing the generator I recommend you to look at an alternative implementation of the MetadataRepository. But in reality the best solution would be to simply reuse metadata when it is provided.


> > have a "lockdown list" of sorts which is really to pick the root set
> > of IUs which basically hides any other versions for stability. A
> > profile essentially.
> In p2, a profile is the result of an installation. Do you mean that
> this is software made to work together like a "distro" (or repo
> controlled line up), e.g. all the milestones sites for a set of repos?
> Yes, it is essentially a versioned distribution. I might have 4
> versions of Subclipse in a hosted repository but I don't want the
> resolver to "see" it. Basically locking down the first level of IUs
> so that you can definitively control what the client is going to
> consume. Our use case is that for a Eclipse installation you need
> something definitive, and for server side your QA people are going
> to test a very specific set of bundles and say they are OK. You need
> to make sure that same set of bundles shows up on the server. If you
> use version ranges in your bundles to allow for dynamic updating you
> cannot point a resolver at a boundless repository. So the lock down
> lists are essentially a constraint to deal with the dichotomy of "I
> must know that what I use is what I tested, but I still must be able
> to incrementally update bundles. So this lock down (or whatever we
> end up calling it) is something that a QA engineer could manage. The
> QA person tests a new bundle and updates the lock down list and
> creates a new version which might reveal 1..N new versions of
> bundles. The client on the other side when resolving will now see
> the new versions and update. We didn't really see a large managed P2
> repository being very useful for production purposes without this
> constraining layer. The case where you are continually deploying to
> a P2 repository and updating, you definitely do not want a resolver
> picking up new versions just because they happen to be in the
> repository. We figured this provides the flexibility so that you
> have content you can actively manage as opposed to producing static
> repositories with one-time generated metadata which will soon be out
> of date. Especially in cases where you are repeatedly building new
> bundles test over and over again.
I understand the need and see where you are coming from. This is definitely valuable when you have the complete control over the set of repositories that any user can see at anytime. However, it is a bit fuzzy to me on how this will work for people who are adding random repos that they found on web. Eventually would not they be exposed to content they are not supposed to see?
At eclipse, bundle providers usually maintain several sites for different purpose (N build, M build, Release) with Galileo as the main integration hub for the community. This is our cheap version of this lockdown :) Also in the eclipse eco-system, this problem has not been encountered as much because people are still using features which provide a lock down aspect from the producer point of view thus limiting the set of choices and drastically reducing the search space. That said, we are not super happy about this and we will be looking at addressing this in 3.6.

First, some background. "Artifact repository" is one of fundamental concepts of Maven-based development. All build dependencies are resolved from artifact repositories and builds results are deployed to an artifact repository. We think this approach is useful for Eclipse/OSGi applications and P2 actually makes it particularly attractive since P2 allows us to install from repository directly.

With this model, repository content is not static which makes resolution of dependencies with version ranges potentially indeterministic. Not good for reproducible builds or client installations.

The "lock down" list lets us solve this problem. Although we have not figured out all details yet, it looks like p2 repositories in Nexus will operate in two "modes". We'll have "open" repositories, used as deployment target by developers of CI systems. And we will have "locked down" or "versioned" repositories with controlled changes. Locked down repositories most likely will be pure metadata with actual artifacts coming from either open or proxy repositories. It will also be possible to "deploy" locked-down repository to Nexus.

So the intent of lock-down is not to force clients to use one specific P2 repository, but rather to enable flexible yet reproducible repository management mechanism. It makes maintaining "WTP 3.5M5 all-in-one" repository next to trivial, for example, both in terms of configuration effort and disk footprint.

> What we have is a fully working system, but we've only had it
> working for a week. But we can use this mechanism to install
> m2eclipse completely from Nexus and we are moving very fast.
Could you describe a bit more "fully working system"? From the description you've given so far, I fail to see where the configuration information is coming from and how you are integrating with p2. Do you invoke the p2 engine?

As I described above, it is already possible to deploy m2e build to Nexus and install the new build together with all prerequisites from single "virtual" P2 repository maintained by Nexus without any additional intermediate steps.

We use stock P2 engine on the client and P2 engine with some custom bundles on the server (to generate metadata).

-- Regards, Igor