Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] Builds and repository specificity

By chance, I was lucky enough to discuss this issue with some smart people on the Planning Council call this morning and helped clarify to me how it is a bad idea to provide only "one big repository"  (which below I mentioned was "ok") or even use a generic general purpose one in your own build. Guess we should try and formally document this better, I know I've seen pieces here and there ... maybe we'll capture the result of this discussion.

We, as providers, need to provide repositories for several audiences or types of users. I'll give some mostly hypothetical examples below to  illustrate, and perhaps this list can fill in details or issues I omit.

End Users:

The "update site" URL in a feature.xml file should be specific to one "release stream" or release train. For example, in webtools we use
This is because it would be bad (confusing at least, if not outright wrong) for our helios users to  be presented with indigo content. Either now, or, even when indigo formally releases next June.
But, its goodness if they can automatically get updates to things they have previously installed from our Helios site.

As a practical matter, it is often easiest to provide these release URLs as a composite repository, where the "real" repositories, are more specific, but those more specific repositories are "an implementation detail" and could change, should not be "published", counted on for builds, etc. For example, in webtools, our release repository is a composite that conceptually combines several other repositories:
but normally no one should have to use those specific repositories directly.

Release engineering.

Here, repositories must be provided to be specific to one build. This is so others builds can be reproducible. Some teams provide these as zipped-up p2 repositories, sometimes as a URL specific to exactly one location. For example,
a build repository from Orbit is
Sometimes teams might be able to count on a "release repository" having only one version, so they can count on having reproducible builds from it, such as
but that is pretty variable project to project, and depends on a project's policies, agreements, etc.
So, while you are building your own stuff ... you should be using one exact repository provided by others so your builds will be reproducible.

I know its tempting to use some "general purpose" URL to "get the latest" of whatever someone is providing ... p2 makes that so easy! :) ... but ... especially for "release builds" (which you need to have reproducible) you need to use a specific repository or know how to always get exactly the same bundle versions you got before.

So, my view is, we should "constrain" end users some, to help them get good installs, and keep good installs.  (this is partially since feature versions are complex, wildly variable, and there's no way for them to tell which one is intended for which stream).

And, more relevant to this discussion,  builds should be very constrained, especially release builds, so you get exactly the same thing when you rebuild. Hopefully everyone provides these build specific repositories, but not sure we've ever spelled this out before.

This might be get more complicated by using various tools or APIs, such as for b3 aggregator, or p2 mirror, etc., you _can_ specify an exact versions to copy/mirror ... but that only makes sense if the provider provides the exact version, as we do tor the helios aggregation builds. There's no way a consumer would know exactly which version to get ... right? And bundle manifest constraints are not enough (not nearly enough) since normally you want those "wide" to _run_ with many versions, but for a build, to be reproducible you need to get a specific build (or set of bundles with exact versions) as your prereq.

Does this make sense? Other views? Have I overlooked anything? Stated anything incorrectly? Am I stating the obvious? Or something that's controversial?


From:        David M Williams/Raleigh/IBM@IBMUS
To:        cross-project-issues-dev@xxxxxxxxxxx
Cc:        Frédéric Madiot <fmadiot@xxxxxxxxxxxxxxxx>
Date:        09/15/2010 10:45 AM
Subject:        Re: [cross-project-issues-dev] Build helios failed
Sent by:        cross-project-issues-dev-bounces@xxxxxxxxxxx

You call that simple? :)  (It probably is, and I'm just being dense in "picturing" it).

What's the next step?  How can we correct this in the quickest way? (today is RC4 +2, after all)?

While there's no "standard" way to do it (so far) some projects do keep a "repository for everything" at one URL, but under the covers that is just a composite that "points to" more specific repositories, for specific releases, and clients could use the more specific URLs if they needed to. Is that the case here? It does sound like some separation is required for the different levels, especially for yearly releases.

I'm moving discussion to "cross project" since you imply there's some indigo/helios bleeding-trough for other teams too ... and we need to avoid that!

Also, I'm confused about the comments about "update site" vs. "p2 repository". It is required to provide a p2 repository. Maybe I'm misunderstanding (lots of speed reading this morning).
But I am aware of a bug in one of the common builders that was failing to produce the p2 repository, but am assuming that _has_ to be fixed before final releases/builds.
See bug 324870.
Not sure if that's related or not.


Laurent Goubet <laurent.goubet@xxxxxxx>
David M Williams/Raleigh/IBM@IBMUS
Frédéric Madiot <fmadiot@xxxxxxxxxxxxxxxx>, Hugo Brunelière <hugo.bruneliere@xxxxxxxx>, Nicolas Bros <nbros.mia@xxxxxxxxx>
09/15/2010 10:14 AM
Re: Build helios failed


The problem is quite simple actually :

- Acceleo milestones (RC1, RC2, RC3, RC4) are located on the aggregate
"m2t milestones" update site.
- Said aggregate "m2t milestones" update site contain all versions of
Acceleo : one for Ganymede, another targeted at Helios, and a last one
targeted at Indigo (M1)
- I contribute to the helios aggregator my RC4 milestone through this
update site
- MoDisco depends on Acceleo without specifying a maximum version : it
then does not retrieve my contributed build (Helios RC4) but the very
latest it can find on the update site : Indigo M1

And here is the issue : we get a MoDisco build for Helios that is built
against a dependency targetted at Indigo. What happens next :

- I build my Acceleo "final" release, it is no longer located on the
aggregate "m2t milestones" update site, but rather on the aggregate "m2t
releases" update site
- Said aggregate "m2t releases" site doesn't contain any version
targetted at indigo (M1 is not a release, and it's the only "Indigo"
build for Acceleo)
- I contribute to the helios aggregator my "final" build through this
new update site
- MoDisco was hoping to find an Indigo version of Acceleo, does not, and
the build fails.

I have no idea whether MoDisco is the only project that implicitely
fetches a dependency that is _not_ aimed at the Helios release, but I am
pretty sure it isn't : if anyone else contribute an aggregate update
site to the Helios aggregator, chances are one or more of the built
plugins are built against a dependency that is in fact too recent (and
which API might have been broken since).

What's even worse is that Nicolas seem to have been able to install
MoDisco from the Helios update site even though the version range for
its Acceleo dependency was wrong, which would mean a version of Acceleo
that was meant for Indigo ended up on the Helios update site.

Laurent Goubet

David M Williams wrote:
> I'll admit I don't quite understand who is doing what to whom ... but,
> hope you get it worked out soon!  But I'll make a few comments.
> a) It is normal for a repository to contain many levels of a bundles
> or features (that's one of design goals ... a "repository" contains a
> lot ... a big huge data base of code). But, this can get complicated,
> and requires good communication and perhaps some maintenance to make
> sure what is there is what is intended and "valid" (that is all "fits
> together").
> b) It is normal (nearly required) for bundle's to specify their
> pre-reqs with a minimum and maximum range ... and, yes, we should all
> be following certain rules about what those versions mean.
> I doubt these brief comments will help that much (I'm probably missing
> this issue) .... but, hope if can focus discussion on how to get the
> problem solved.
> If there is a larger issue here that deserves broader discussion or
> agreements, perhaps a not to cross-project list would be best.
> Let me know if I can help,
> Thanks,
> From:        Laurent Goubet <laurent.goubet@xxxxxxx>
> To:        Nicolas Bros <nbros.mia@xxxxxxxxx>
> Cc:        David M Williams/Raleigh/IBM@IBMUS, Frédéric Madiot
> <fmadiot@xxxxxxxxxxxxxxxx>, Hugo Brunelière <hugo.bruneliere@xxxxxxxx>
> Date:        09/15/2010 09:08 AM
> Subject:        Re: Build helios failed
> ------------------------------------------------------------------------
> Nicolas,
> Unfortunately, the m2t update sites (we still use the old build system
> for now, at least until the "promote" from hudson  becomes possible
> without too much hassle) aggregates the versions from europa, ganymede,
> helios, indigo ... indiscriminately. "milestones" include our 0.8.1RC4
> (no longer the 0.8.0, and we had no 0.8.2), 3.0.1RC4 (Helios SR1), and
> 3.1.0M1. The "releases" update site is the same : 0.8.1 final, 3.0.1
> final, and in June 3.1.0 final. I know Acceleo isn't the only one to use
> one of the old aggregate update sites, and I am pretty sure the old
> "project" update sites (m2t, emf, m2m...) aren't the only ones that
> aggregate multiple versions of the same plugin.
> If I understand correctly the requirement, we would need at least 2
> update sites for each simultaneous release :
> Acceleo - Ganymede - milestones
> Acceleo - Ganymede - releases
> Acceleo - Helios - milestones
> Acceleo - Helios - releases
> Acceleo - Indigo - milestones
> Acceleo - Indigo - releases
> ....
> and if we need "integration" update sites, that's one more for each
> simultaneous release ... seems like a lot of work. We'll probably cope
> with it when we finally switch to hudson builds, but I am pretty sure
> there are others in the same case as us for the SR1 release
> (emf/transaction, emf/validation, m2m/atl, gmp/gmf...), thus most likely
> getting a lot of versions in Helios SR1 that weren't intended for SR1
> ... or any Helios release for that matters.
> Specifying closed dependency intervals in the plugins doesn't seem to be
> the right thing to do.
> Just my two cents.
> Laurent Goubet
> Obeo
> Nicolas Bros wrote:
> > I think I now understand what happened:
> > MoDisco plug-ins don't specify a dependency range for Acceleo. So, p2
> > takes the higher version it finds when building MoDisco.
> > The version of MoDisco targeting Helios SR1 is built with the same
> > update sites as those specified in the Helios build model.
> > So, since you specified the "milestones" update site, MoDisco was
> > built using version 3.1.0 of Acceleo from this update site.
> > This dependency then becomes explicit in the MoDisco update site built
> > with p2.
> > But when you switched from "milestones" to "releases" in the Helios
> > build model, this version wasn't available anymore, causing the
> > aggregation failure.
> >
> > I believe the problem could have been avoided by either specifying
> > closed version dependency intervals in MoDisco plug-ins (which
> > requires a knowledge of Acceleo versioning rules), or making sure that
> > the Acceleo update site specified in the Helios build model only
> > contains versions of Acceleo targeting Helios, at all times.
> > For example, for MoDisco we have 2 different update sites, one
> > specified in the Helios build model, and the other in the Indigo build
> > model.
> >
> > On Wed, Sep 15, 2010 at 2:30 PM, Laurent Goubet
> > <laurent.goubet@xxxxxxx <
mailto:laurent.goubet@xxxxxxx>> wrote:
> >
> >     Nicolas,
> >
> >     As far as the Acceleo.b3aggrcon file is concerned, here is the
> >     only change :
> >
> >     -----
> >      <repositories
> >    
> location=""
> >     description="M2T ACCELEO milestones">
> >       <features name=""
> >     versionRange="3.0.1.v201009150356">
> >     -----
> >     changed to
> >     -----
> >     <repositories
> >    
> location=""
> >     description="M2T ACCELEO releases">
> >       <features name=""
> >     versionRange="3.0.1.v201009150438">
> >     -----
> >
> >     The two builds (3.0.1RC4 -on milestones- and 3.0.1 -on releases-)
> >     are exactly identical except for their qualifiers ... yet I
> >     believe the change that caused your build to fail is the switching
> >     from the milestones update site to the releases update site :
> >     3.1.0M1 (Acceleo version for Indigo M1) is present on the m2t
> >     milestones update site, but no 3.1.0* release is present on the
> >     releases update site (and there won't be any before 3.1.0 final,
> >     for Indigo, in June). David, Is it possible that there was a
> >     "leak" that allowed MoDisco to fetch a build from the m2t
> >     milestones (3.1.0M1) that wasn't the one I wanted to contribute to
> >     Helios (3.0.1RC4)? That would mean we haven't got any way to
> >     ensure that the Helios build is really ready for consumers, as
> >     other could have the same problem as MoDisco without being able to
> >     realize it : the only thing that allowed for us to know that
> >     MoDisco was depending on the wrong version of Acceleo was me
> >     switching to the m2t releases update site.
> >
> >     It would be a good idea to tell everyone to double check that
> >     their "final" build really installs without failure on an Helios
> >     platform (and that it installs without wanting to fetch anything
> >     from the Indigo update site).
> >
> >     Laurent Goubet
> >     Obeo
> >
> >     Nicolas Bros wrote:
> >
> >         Thank you for the explanation.
> >         But there is something I don't understand : we haven't changed
> >         the dependencies for SR1, and these dependencies were resolved
> >         in Helios until now.
> >         So, since MoDisco was installable with only the Helios release
> >         update site, does it mean that Acceleo 3.1.0 was available
> >         from the Helios update site?
> >         What exactly changed between the aggregation builds #371 (Sep
> >         15, 2010 4:58:33 AM) and #372 (Sep 15, 2010 7:01:02 AM)?
> >
> >         On Wed, Sep 15, 2010 at 1:16 PM, Laurent Goubet
> >         <laurent.goubet@xxxxxxx <
> >         <
> >         <
mailto:laurent.goubet@xxxxxxx>>> wrote:
> >
> >            Hi,
> >
> >            Just saw that the Helios aggregation build failed because of
> >            MoDisco with message
> >
> >            [exec] Missing requirement:
> >            org.eclipse.gmt.modisco.workflow.driver.acceleo
> >            0.8.1.v201009090831 requires 'bundle
> org.eclipse.acceleo.ide.ui
> >            [3.1.0,4.0.0)' but it could not be found
> >
> >            The change that triggered the build was me contributing the
> >         final
> >            Acceleo SR1 b3aggrcon file for the aggregation, but I
> >         believe this
> >            to be a failure from MoDisco. I just wanted to point out that
> >            version 3.1.0 of Acceleo (minimal requirement from the
> failure
> >            above) is the version that will be contributed to Indigo, the
> >            Acceleo version for Helios being 3.0.1.
> >
> >            Regards,
> >
> >            Laurent Goubet
> >            Obeo
> >
> >
> >
> >
> >         --
> >         Nicolas Bros
> >         R&D
> >         tel: 06 75 09 19 88
> >         nbros@xxxxxxxxxxxxxxxx <
> >         <
mailto:nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>>
> >         nbros.mia@xxxxxxxxx <
> >         <
mailto:nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>>
> >
> >         Mia-Software, 410 clos de la Courtine
> >         93160 Noisy-le-Grand
> > <>
> >         .: model driven agility :.
> >
> >
> >
> >
> >
> > --
> > Nicolas Bros
> > R&D
> > tel: 06 75 09 19 88
> > nbros@xxxxxxxxxxxxxxxx <
> > nbros.mia@xxxxxxxxx <
> > Mia-Software, 410 clos de la Courtine
> > 93160 Noisy-le-Grand
> > <>
> > .: model driven agility :.
> [attachment "laurent_goubet.vcf" deleted by David M Williams/Raleigh/IBM]

[attachment "laurent_goubet.vcf" deleted by David M Williams/Raleigh/IBM] _______________________________________________
cross-project-issues-dev mailing list

Back to the top