[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [p2-dev] Re: [orbit-dev] Best way to get a binary copy of the latest Orbitbundles
- From: Igor Fedorenko <ifedorenko@xxxxxxxxxxxx>
- Date: Mon, 16 Mar 2009 16:01:20 -0400
- Delivered-to: email@example.com
- User-agent: Thunderbird 18.104.22.168 (X11/20090105)
Pascal Rapicault wrote:
> > Not sure if anyone is interested but we can support building bundles
> > with Tycho and deploying the bundle to the remote Nexus.
> When would the p2 metadata generation happen?
> As the bundles are deployed.
> Also what happens to the metadata in case of a copy from repos, I
> assume that Nexus stores it too?
> If by copy you mean a physically host P2 repository, we would
> regenerate the metadata to make sure it's intact.
This is a major problem. Metadata should/can not be regenerated. The
main reasons are:
1) An IU should be treated immutably. Id + version == immutable content.
The IU is for a particular artifact is the responsibility of the owner
of the artifact.
2) The binary can not always be reverse engineered to produce the IU
that the owner of the artifact would have created. It happens that you
have been successful so far because orbit metadata is a straight
transcription of the bundle manifest. However this breaks when a) the
generator is changed, b) the owner of the artifact used a p2.inf to
specify more information (e.g. overwrite capability, requirement, etc).
c) the artifact is not something that p2 metadata gen recognize.
3) Also some IUs are pure metadata, there is no artifact associated with
it that needs to be stored and deployed. An example of such a thing is
the top leve IU of a product, or the configuration units.
I believe Nexus either already does or can be extended relatively easily
to do exactly as you describe
1) Once deployed, artifact metadata does not change, which effectively
means Id + version == immutable content. Nexus can also be configured to
block redeployment of artifacts with same id/version.
2) Deployment to Nexus is not limited to "real" artifacts, like bundle
jar files. Current prototype always generates IU from binaries, we're
prepared to support explicit IU and "metadata-only" IU. It is a simple
matter of writing code ;-)
> for Maven repositories as well in Nexus. In the case of the Orbit
> zip we downloaded it didn't seem to have any so we just generated
> it. Right now we are using the standard P2 metadata generator but
> we're finding it uses a _lot_ of memory so we're not sure if it's
> something that's known and will be fixed. We might take a crack at
> writing one as while we are combining many repositories it's
> doubling the normal memory requirements of Nexus.
Most of the memory consumption is caused by the vast amount of strings
that result from the generation because we do not try to intern at
generation time like we do at runtime. But the core issue is the
SimpleMetadataRepository implementation that loads the complete repo. If
you are really worried about memory consumption rather than changing the
generator I recommend you to look at an alternative implementation of
the MetadataRepository. But in reality the best solution would be to
simply reuse metadata when it is provided.
> > have a "lockdown list" of sorts which is really to pick the root set
> > of IUs which basically hides any other versions for stability. A
> > profile essentially.
> In p2, a profile is the result of an installation. Do you mean that
> this is software made to work together like a "distro" (or repo
> controlled line up), e.g. all the milestones sites for a set of repos?
> Yes, it is essentially a versioned distribution. I might have 4
> versions of Subclipse in a hosted repository but I don't want the
> resolver to "see" it. Basically locking down the first level of IUs
> so that you can definitively control what the client is going to
> consume. Our use case is that for a Eclipse installation you need
> something definitive, and for server side your QA people are going
> to test a very specific set of bundles and say they are OK. You need
> to make sure that same set of bundles shows up on the server. If you
> use version ranges in your bundles to allow for dynamic updating you
> cannot point a resolver at a boundless repository. So the lock down
> lists are essentially a constraint to deal with the dichotomy of "I
> must know that what I use is what I tested, but I still must be able
> to incrementally update bundles. So this lock down (or whatever we
> end up calling it) is something that a QA engineer could manage. The
> QA person tests a new bundle and updates the lock down list and
> creates a new version which might reveal 1..N new versions of
> bundles. The client on the other side when resolving will now see
> the new versions and update. We didn't really see a large managed P2
> repository being very useful for production purposes without this
> constraining layer. The case where you are continually deploying to
> a P2 repository and updating, you definitely do not want a resolver
> picking up new versions just because they happen to be in the
> repository. We figured this provides the flexibility so that you
> have content you can actively manage as opposed to producing static
> repositories with one-time generated metadata which will soon be out
> of date. Especially in cases where you are repeatedly building new
> bundles test over and over again.
I understand the need and see where you are coming from. This is
definitely valuable when you have the complete control over the set of
repositories that any user can see at anytime. However, it is a bit
fuzzy to me on how this will work for people who are adding random repos
that they found on web. Eventually would not they be exposed to content
they are not supposed to see?
At eclipse, bundle providers usually maintain several sites for
different purpose (N build, M build, Release) with Galileo as the main
integration hub for the community. This is our cheap version of this
lockdown :) Also in the eclipse eco-system, this problem has not been
encountered as much because people are still using features which
provide a lock down aspect from the producer point of view thus limiting
the set of choices and drastically reducing the search space. That said,
we are not super happy about this and we will be looking at addressing
this in 3.6.
First, some background. "Artifact repository" is one of fundamental
concepts of Maven-based development. All build dependencies are resolved
from artifact repositories and builds results are deployed to an
artifact repository. We think this approach is useful for Eclipse/OSGi
applications and P2 actually makes it particularly attractive since P2
allows us to install from repository directly.
With this model, repository content is not static which makes resolution
of dependencies with version ranges potentially indeterministic. Not
good for reproducible builds or client installations.
The "lock down" list lets us solve this problem. Although we have not
figured out all details yet, it looks like p2 repositories in Nexus will
operate in two "modes". We'll have "open" repositories, used as
deployment target by developers of CI systems. And we will have "locked
down" or "versioned" repositories with controlled changes. Locked down
repositories most likely will be pure metadata with actual artifacts
coming from either open or proxy repositories. It will also be possible
to "deploy" locked-down repository to Nexus.
So the intent of lock-down is not to force clients to use one specific
P2 repository, but rather to enable flexible yet reproducible repository
management mechanism. It makes maintaining "WTP 3.5M5 all-in-one"
repository next to trivial, for example, both in terms of configuration
effort and disk footprint.
> What we have is a fully working system, but we've only had it
> working for a week. But we can use this mechanism to install
> m2eclipse completely from Nexus and we are moving very fast.
Could you describe a bit more "fully working system"? From the
description you've given so far, I fail to see where the configuration
information is coming from and how you are integrating with p2. Do you
invoke the p2 engine?
As I described above, it is already possible to deploy m2e build to
Nexus and install the new build together with all prerequisites from
single "virtual" P2 repository maintained by Nexus without any
additional intermediate steps.
We use stock P2 engine on the client and P2 engine with some custom
bundles on the server (to generate metadata).