Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » B3 » Aggregator with version range support
Aggregator with version range support [message #517009] Thu, 25 February 2010 17:21 Go to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Hello,

I'd like to discuss the algorithm how the aggregator should resolve IU requests. This post is rather long but I wanted to express in detail what the problem is all about. I will appreciate you spending some time on it and helping me find a solution.

Background:

Let's have following aggregation definition:

Contribution A (contributed by user A)
|
+ Mapped Repository R1
|
+ IU Request: id=org.eclipse.myframework; version>=1.0.0

Contribution B (contributed by user B)
|
+ Mapped Repository R2
|
+ IU Request: id=org.eclipse.something.different; version>=1.0.0


Repository R1 contains:
org.eclipse.myframework/0.9.0.v20070415
org.eclipse.myframework/1.0.0.v20080105
org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)

Repository R2 contains:
org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A does not expect and does not want at all)
org.eclipse.something.different/2.5.1.v20100225 (this is what user B expects)

=================================================

Now, what is the expected result? I think that a user would think probably like this:

The contribution "A" yields a version of "org.eclipse.myframework" that is the best from repository R1 and is not in conflict with any other contribution at the same time.
Since "org.eclipse.myframework" is not mapped in any other contribution, the natural resolution of it is "org.eclipse.myframework/1.1.0.v20091220".

The contribution "B" yields a version of "org.eclipse.something.different" that is the best from repository R1 and is not in conflict with any other contribution at the same time.
Since "org.eclipse.something.different" is not mapped in any other contribution, the natural resolution of it is "org.eclipse.something.different/2.5.1.v20100225".

The aggregated repository will contain:
org.eclipse.myframework/1.1.0.v20091220
org.eclipse.something.different/2.5.1.v20100225

OK?

However, the new aggregator (with version range support) will create an aggregated repository containing:
org.eclipse.myframework/1.2.0.beta_for_testing_only
org.eclipse.something.different/2.5.1.v20100225

The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We use a request accross all repositories, regardless where the IU was mapped.

In my view, this is bad.

That's why I tried to change the algorithm:

For each IU request
1) Resolve the request against its parent mapped repository (first level only, no dependencies) and get a set of matching resolutions
2) Create a modified IU request that defines the IU name and version enumeration (of matching resolutions) instead of the original version range
3) Provide the modified request to the global aggregated repository processing

This way, only IUs that are really present in appropriate mapped repository may be passed to the final resolution.

So far so good, but the problem is that the only implementation of IRequirement is the RequiredCapability which does not support version constraints different from version ranges.

As a proof of concept, I tried to hack this all so that the interim repository supports another implementation of IRequirement with more complex version constraints. It works as expected!

Finally, the questions to answer:

1) Do you think that this approach is nature/logical, i.e. is it how it should work?
2) If the answer to 1) is YES, then we need to think about how to do it without too much hacking (I can provide more details on what all I had to do and what I don't like on the "hacking solution")

Current b3 aggregator (available on the b3 update site) uses the "hacking solution" which should later be removed or replaced with something better. This is now there for testing purpose only.

The cleanest solution would be to suggest support for complex version constraints in any p2 repository, but I am afraid that it would be a big problem because of compatibility (old p2 clients would not be able to read all new p2 repos).

PS: Support for complex version constraints would allow us to implement what Henrik has mentioned in one of our discussions - "User wants to map versions from 1.0.0 to 2.0.0 but not 1.3.4 which is known to be buggy"

Thanks for your patience, comments, questions and ideas :-)

Best regards
Filip
Re: Aggregator with version range support [message #517083 is a reply to message #517009] Fri, 26 February 2010 07:01 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
Some comments inline.

On 02/25/2010 06:21 PM, Filip Hrbek wrote:
> Hello,
>
> I'd like to discuss the algorithm how the aggregator should resolve IU
> requests. This post is rather long but I wanted to express in detail
> what the problem is all about. I will appreciate you spending some time
> on it and helping me find a solution.
>
> Background:
>
> Let's have following aggregation definition:
>
> Contribution A (contributed by user A)
> |
> + Mapped Repository R1
> |
> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>
> Contribution B (contributed by user B)
> |
> + Mapped Repository R2
> |
> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>
>
> Repository R1 contains:
> org.eclipse.myframework/0.9.0.v20070415
> org.eclipse.myframework/1.0.0.v20080105
> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>
> Repository R2 contains:
> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
> does not expect and does not want at all)
> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
> expects)
>
> =================================================
>
> Now, what is the expected result? I think that a user would think
> probably like this:
>
> The contribution "A" yields a version of "org.eclipse.myframework" that
> is the best from repository R1 and is not in conflict with any other
> contribution at the same time.
> Since "org.eclipse.myframework" is not mapped in any other contribution,
> the natural resolution of it is "org.eclipse.myframework/1.1.0.v20091220".
>
> The contribution "B" yields a version of
> "org.eclipse.something.different" that is the best from repository R1
> and is not in conflict with any other contribution at the same time.
> Since "org.eclipse.something.different" is not mapped in any other
> contribution, the natural resolution of it is
> "org.eclipse.something.different/2.5.1.v20100225".
>
> The aggregated repository will contain:
> org.eclipse.myframework/1.1.0.v20091220
> org.eclipse.something.different/2.5.1.v20100225
>
> OK?
>
No, I don't think that's OK. If the user is unhappy with versions of the framework that is above 1.1.0, then that should
be stated in contribution A. If the range is open-ended, then A has explicitly declared that 1.2.0 is an OK replacement.

> However, the new aggregator (with version range support) will create an
> aggregated repository containing:
> org.eclipse.myframework/1.2.0.beta_for_testing_only
> org.eclipse.something.different/2.5.1.v20100225
>
Which seems OK, given the circumstances.

Try another use-case:

Contribution A is exactly as in your example. Contribution B contains updates contains a limited subset of newer
versions of bundles also found in contribution A. This can easily happen if A is a release and the combination of A and
B is a service release.

> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
> use a request accross all repositories, regardless where the IU was mapped.
>
> In my view, this is bad.
>
In my view, this is good and indeed necessary.

> That's why I tried to change the algorithm:
>
> For each IU request
> 1) Resolve the request against its parent mapped repository (first level
> only, no dependencies) and get a set of matching resolutions
> 2) Create a modified IU request that defines the IU name and version
> enumeration (of matching resolutions) instead of the original version range
> 3) Provide the modified request to the global aggregated repository
> processing
>
> This way, only IUs that are really present in appropriate mapped
> repository may be passed to the final resolution.
>
Why is the IU found in the same repository more "appropriate" then an IU found in another? I can see that for some
specific use-case that might be the case, but it's not a general truth.

> So far so good, but the problem is that the only implementation of
> IRequirement is the RequiredCapability which does not support version
> constraints different from version ranges.
>
> As a proof of concept, I tried to hack this all so that the interim
> repository supports another implementation of IRequirement with more
> complex version constraints. It works as expected!
>
> Finally, the questions to answer:
>
> 1) Do you think that this approach is nature/logical, i.e. is it how it
> should work?

No. I think this is wrong. I think the right approach must be to consciously insert a rule that limits the dependency.
It should not be automatic.

- thomas
Re: Aggregator with version range support [message #517089 is a reply to message #517083] Fri, 26 February 2010 03:44 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Hi Thomas,

thanks for your comments.

I understand what you mean. It is also very easy to implement what you suggest, but I have a different view on the problem.

I understand the "contribution" as a unit which someone is responsible for. When I talk about responsibility, I mean that the one who contributes also guarantees that only approved versions from his approved source may be included (new aggregator also shows these versions interactively in the mapping).

You are right that sometimes it would be good to include "service packs" above "releases". IMHO it should not be automatic though. I would rather see a little change in the model to achieve this - to allow map repositories as virtual composites:

Contribution A (contributed by user A)
|
+ Mapped Repository - Virtual Composite of [R1,R2]
|
+ IU Request: id=org.eclipse.myframework; version>=1.0.0

This would be the conscious rule to add more available (approved) resolutions. The basic idea is "be strict by default, allow being less strict on user's conscious action".

I'd like to see more people contributing to this discussion to avoid an endless dialog of two hard-headed guys :-)

Filip

Thomas Hallgren wrote:
> Hi Filip,
> Some comments inline.
>
> On 02/25/2010 06:21 PM, Filip Hrbek wrote:
>> Hello,
>>
>> I'd like to discuss the algorithm how the aggregator should resolve IU
>> requests. This post is rather long but I wanted to express in detail
>> what the problem is all about. I will appreciate you spending some time
>> on it and helping me find a solution.
>>
>> Background:
>>
>> Let's have following aggregation definition:
>>
>> Contribution A (contributed by user A)
>> |
>> + Mapped Repository R1
>> |
>> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>>
>> Contribution B (contributed by user B)
>> |
>> + Mapped Repository R2
>> |
>> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>>
>>
>> Repository R1 contains:
>> org.eclipse.myframework/0.9.0.v20070415
>> org.eclipse.myframework/1.0.0.v20080105
>> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>>
>> Repository R2 contains:
>> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
>> does not expect and does not want at all)
>> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
>> expects)
>>
>> =================================================
>>
>> Now, what is the expected result? I think that a user would think
>> probably like this:
>>
>> The contribution "A" yields a version of "org.eclipse.myframework" that
>> is the best from repository R1 and is not in conflict with any other
>> contribution at the same time.
>> Since "org.eclipse.myframework" is not mapped in any other contribution,
>> the natural resolution of it is
>> "org.eclipse.myframework/1.1.0.v20091220".
>>
>> The contribution "B" yields a version of
>> "org.eclipse.something.different" that is the best from repository R1
>> and is not in conflict with any other contribution at the same time.
>> Since "org.eclipse.something.different" is not mapped in any other
>> contribution, the natural resolution of it is
>> "org.eclipse.something.different/2.5.1.v20100225".
>>
>> The aggregated repository will contain:
>> org.eclipse.myframework/1.1.0.v20091220
>> org.eclipse.something.different/2.5.1.v20100225
>>
>> OK?
>>
> No, I don't think that's OK. If the user is unhappy with versions of the
> framework that is above 1.1.0, then that should be stated in
> contribution A. If the range is open-ended, then A has explicitly
> declared that 1.2.0 is an OK replacement.
>
>> However, the new aggregator (with version range support) will create an
>> aggregated repository containing:
>> org.eclipse.myframework/1.2.0.beta_for_testing_only
>> org.eclipse.something.different/2.5.1.v20100225
>>
> Which seems OK, given the circumstances.
>
> Try another use-case:
>
> Contribution A is exactly as in your example. Contribution B contains
> updates contains a limited subset of newer versions of bundles also
> found in contribution A. This can easily happen if A is a release and
> the combination of A and B is a service release.
>
>> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
>> use a request accross all repositories, regardless where the IU was
>> mapped.
>>
>> In my view, this is bad.
>>
> In my view, this is good and indeed necessary.
>
>> That's why I tried to change the algorithm:
>>
>> For each IU request
>> 1) Resolve the request against its parent mapped repository (first level
>> only, no dependencies) and get a set of matching resolutions
>> 2) Create a modified IU request that defines the IU name and version
>> enumeration (of matching resolutions) instead of the original version
>> range
>> 3) Provide the modified request to the global aggregated repository
>> processing
>>
>> This way, only IUs that are really present in appropriate mapped
>> repository may be passed to the final resolution.
>>
> Why is the IU found in the same repository more "appropriate" then an IU
> found in another? I can see that for some specific use-case that might
> be the case, but it's not a general truth.
>
>> So far so good, but the problem is that the only implementation of
>> IRequirement is the RequiredCapability which does not support version
>> constraints different from version ranges.
>>
>> As a proof of concept, I tried to hack this all so that the interim
>> repository supports another implementation of IRequirement with more
>> complex version constraints. It works as expected!
>>
>> Finally, the questions to answer:
>>
>> 1) Do you think that this approach is nature/logical, i.e. is it how it
>> should work?
>
> No. I think this is wrong. I think the right approach must be to
> consciously insert a rule that limits the dependency. It should not be
> automatic.
>
> - thomas
Re: Aggregator with version range support [message #517100 is a reply to message #517089] Fri, 26 February 2010 09:08 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
What do you propose should happen if contribution A contains:

org.eclipse.myframework/1.1.0.alpha

and contribution B contains:

org.eclipse.myframework/1.1.0.final

?

What algorithm is used to determine the best one?

- thomas



On 02/26/2010 09:30 AM, Filip Hrbek wrote:
> Hi Thomas,
>
> thanks for your comments.
>
> I understand what you mean. It is also very easy to implement what you
> suggest, but I have a different view on the problem.
>
> I understand the "contribution" as a unit which someone is responsible
> for. When I talk about responsibility, I mean that the one who
> contributes also guarantees that only approved versions from his
> approved source may be included (new aggregator also shows these
> versions interactively in the mapping).
>
> You are right that sometimes it would be good to include "service packs"
> above "releases". IMHO it should not be automatic though. I would rather
> see a little change in the model to achieve this - to allow map
> repositories as virtual composites:
>
> Contribution A (contributed by user A)
> |
> + Mapped Repository - Virtual Composite of [R1,R2]
> |
> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>
> This would be the conscious rule to add more available (approved)
> resolutions. The basic idea is "be strict by default, allow being less
> strict on user's conscious action".
>
> I'd like to see more people contributing to this discussion to avoid an
> endless dialog of two hard-headed guys :-)
>
> Filip
>
> Thomas Hallgren wrote:
>> Hi Filip,
>> Some comments inline.
>>
>> On 02/25/2010 06:21 PM, Filip Hrbek wrote:
>>> Hello,
>>>
>>> I'd like to discuss the algorithm how the aggregator should resolve IU
>>> requests. This post is rather long but I wanted to express in detail
>>> what the problem is all about. I will appreciate you spending some time
>>> on it and helping me find a solution.
>>>
>>> Background:
>>>
>>> Let's have following aggregation definition:
>>>
>>> Contribution A (contributed by user A)
>>> |
>>> + Mapped Repository R1
>>> |
>>> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>>>
>>> Contribution B (contributed by user B)
>>> |
>>> + Mapped Repository R2
>>> |
>>> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>>>
>>>
>>> Repository R1 contains:
>>> org.eclipse.myframework/0.9.0.v20070415
>>> org.eclipse.myframework/1.0.0.v20080105
>>> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>>>
>>> Repository R2 contains:
>>> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
>>> does not expect and does not want at all)
>>> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
>>> expects)
>>>
>>> =================================================
>>>
>>> Now, what is the expected result? I think that a user would think
>>> probably like this:
>>>
>>> The contribution "A" yields a version of "org.eclipse.myframework" that
>>> is the best from repository R1 and is not in conflict with any other
>>> contribution at the same time.
>>> Since "org.eclipse.myframework" is not mapped in any other contribution,
>>> the natural resolution of it is
>>> "org.eclipse.myframework/1.1.0.v20091220".
>>>
>>> The contribution "B" yields a version of
>>> "org.eclipse.something.different" that is the best from repository R1
>>> and is not in conflict with any other contribution at the same time.
>>> Since "org.eclipse.something.different" is not mapped in any other
>>> contribution, the natural resolution of it is
>>> "org.eclipse.something.different/2.5.1.v20100225".
>>>
>>> The aggregated repository will contain:
>>> org.eclipse.myframework/1.1.0.v20091220
>>> org.eclipse.something.different/2.5.1.v20100225
>>>
>>> OK?
>>>
>> No, I don't think that's OK. If the user is unhappy with versions of
>> the framework that is above 1.1.0, then that should be stated in
>> contribution A. If the range is open-ended, then A has explicitly
>> declared that 1.2.0 is an OK replacement.
>>
>>> However, the new aggregator (with version range support) will create an
>>> aggregated repository containing:
>>> org.eclipse.myframework/1.2.0.beta_for_testing_only
>>> org.eclipse.something.different/2.5.1.v20100225
>>>
>> Which seems OK, given the circumstances.
>>
>> Try another use-case:
>>
>> Contribution A is exactly as in your example. Contribution B contains
>> updates contains a limited subset of newer versions of bundles also
>> found in contribution A. This can easily happen if A is a release and
>> the combination of A and B is a service release.
>>
>>> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
>>> use a request accross all repositories, regardless where the IU was
>>> mapped.
>>>
>>> In my view, this is bad.
>>>
>> In my view, this is good and indeed necessary.
>>
>>> That's why I tried to change the algorithm:
>>>
>>> For each IU request
>>> 1) Resolve the request against its parent mapped repository (first level
>>> only, no dependencies) and get a set of matching resolutions
>>> 2) Create a modified IU request that defines the IU name and version
>>> enumeration (of matching resolutions) instead of the original version
>>> range
>>> 3) Provide the modified request to the global aggregated repository
>>> processing
>>>
>>> This way, only IUs that are really present in appropriate mapped
>>> repository may be passed to the final resolution.
>>>
>> Why is the IU found in the same repository more "appropriate" then an
>> IU found in another? I can see that for some specific use-case that
>> might be the case, but it's not a general truth.
>>
>>> So far so good, but the problem is that the only implementation of
>>> IRequirement is the RequiredCapability which does not support version
>>> constraints different from version ranges.
>>>
>>> As a proof of concept, I tried to hack this all so that the interim
>>> repository supports another implementation of IRequirement with more
>>> complex version constraints. It works as expected!
>>>
>>> Finally, the questions to answer:
>>>
>>> 1) Do you think that this approach is nature/logical, i.e. is it how it
>>> should work?
>>
>> No. I think this is wrong. I think the right approach must be to
>> consciously insert a rule that limits the dependency. It should not be
>> automatic.
>>
>> - thomas
Re: Aggregator with version range support [message #517104 is a reply to message #517089] Fri, 26 February 2010 09:31 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
A note about contributor responsibilities.

On 02/26/2010 09:30 AM, Filip Hrbek wrote:
> I understand the "contribution" as a unit which someone is responsible
> for. When I talk about responsibility, I mean that the one who
> contributes also guarantees that only approved versions from his
> approved source may be included (new aggregator also shows these
> versions interactively in the mapping).
>
That is a good understanding of a "contribution" but the contributor has one other very important responsibility. He
must absolutely not use the repository itself as a mechanism to limit the scope of resolution. If he does, he is missing
the whole point of p2!

An author of a p2 repository must always be aware that his repository can, and will, be used together with other
repositories. This can happen in the Eclipse IDE Install wizard as well as in our Aggregator. In addition to that, any
composite repository may blend his repositories with others and use no other mechanism then p2 to resolve.

What you propose is not only that we release the repository publisher from this responsibility, you also suggest that
this should be the default behavior in the Aggregator. This is in strong contrast with how p2 is supposed to work.
That's why I don't approve.

I do think we should have a mechanism that help us to "fix" repositories where this consideration has not been honored
and that causes problems.

- thomas
Re: Aggregator with version range support [message #517105 is a reply to message #517100] Fri, 26 February 2010 09:33 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> Hi Filip,
> What do you propose should happen if contribution A contains:
>
> org.eclipse.myframework/1.1.0.alpha
>
> and contribution B contains:
>
> org.eclipse.myframework/1.1.0.final
>
> ?
>
> What algorithm is used to determine the best one?

If there is no intersection of resolutions from contributions A and B, then this is a conflict.
Contributors must contact each other to modify the rules. It's exactly what I said about the policy "be strict by default".

To answer your question - if contributor A decides to accept the "final" version, he should add the repository containing the "final" version to his contribution.

Filip
Re: Aggregator with version range support [message #517106 is a reply to message #517105] Fri, 26 February 2010 09:37 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 10:33 AM, Filip Hrbek wrote:
>
> Thomas Hallgren wrote:
>> Hi Filip,
>> What do you propose should happen if contribution A contains:
>>
>> org.eclipse.myframework/1.1.0.alpha
>>
>> and contribution B contains:
>>
>> org.eclipse.myframework/1.1.0.final
>>
>> ?
>>
>> What algorithm is used to determine the best one?
>
> If there is no intersection of resolutions from contributions A and B,
> then this is a conflict.

I assume the same scenario as you outlined in your first post. The only difference is that in B, it's not a beta, it's a
final. And in A, it's just alphas.

- thomas
Re: Aggregator with version range support [message #517111 is a reply to message #517106] Fri, 26 February 2010 09:59 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 10:33 AM, Filip Hrbek wrote:
>>
>> Thomas Hallgren wrote:
>>> Hi Filip,
>>> What do you propose should happen if contribution A contains:
>>>
>>> org.eclipse.myframework/1.1.0.alpha
>>>
>>> and contribution B contains:
>>>
>>> org.eclipse.myframework/1.1.0.final
>>>
>>> ?
>>>
>>> What algorithm is used to determine the best one?
>>
>> If there is no intersection of resolutions from contributions A and B,
>> then this is a conflict.
>
> I assume the same scenario as you outlined in your first post. The only
> difference is that in B, it's not a beta, it's a final. And in A, it's
> just alphas.
>
> - thomas

Then yes, the "alpha" wins. If contributor "A" discovers that a better version is available, he should add the repo to his contribution.

If you really think that what I say breaks the p2 world, well, then our model is bad! We shouldn't map things from a mapped repository.

To make what you say intuitive, then a contribution should look like this:

Contribution A
|
+ Repository hints
|
+ Repo A
+ Repo B
+ Repo C
|
+ Mappings
|
+ IU a.b.c, version >=1.0.0
+ Everything from Repo B

The "Repository hints" would serve as possible sources that the contributor "recommends" to use for the final resolution. We could also remove current "Validation repositories" since it would not make sense any more (in my example, "Repo C" is actually a validation repository since nothing is mapped in contribution A).

With such a model, I could agree with you. Not with current model though since it is very confusing - one thinks he maps something within the resolution scope of the mapped repository but he may get unexpectedly something else.

Filip
Re: Aggregator with version range support [message #517115 is a reply to message #517111] Fri, 26 February 2010 05:26 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 10:59 AM, Filip Hrbek wrote:

> Then yes, the "alpha" wins. If contributor "A" discovers that a better
> version is available, he should add the repo to his contribution.
>
I'm trying to map this to our current use cases. In Helios, each contribution is just a fragment of the final result
that has lot of dependencies not contained in that fragment. The contributors do this, well aware that when the
aggregation happens, everything will be there. The contribution inter-dependencies are very intentional.

Miles at MS first creates a big aggregation containing a lot of things from Galileo and other places. He then uses this
as a contribution when he creates the final aggregation together with other contributions that contains all in-house
developed features. He is well aware that the in-house contributions all have inter-dependencies and that they depend
heavily on the aggregation he created in the first step.

> If you really think that what I say breaks the p2 world, well, then our
> model is bad! We shouldn't map things from a mapped repository.
>
Not sure what you mean by that. How else can we map things?

> To make what you say intuitive, then a contribution should look like this:
>
> Contribution A
> |
> + Repository hints
> |
> + Repo A
> + Repo B
> + Repo C
> |
> + Mappings
> |
> + IU a.b.c, version >=1.0.0
> + Everything from Repo B
>
> The "Repository hints" would serve as possible sources that the
> contributor "recommends" to use for the final resolution. We could also
> remove current "Validation repositories" since it would not make sense
> any more (in my example, "Repo C" is actually a validation repository
> since nothing is mapped in contribution A).
>
That assumes that all contributions are self sufficient. And sure, there are perhaps use-cases where this could be good.
I haven't seen them yet though. On the contrary, the use-cases at hand always have contributions that depend on other
contributions.

> With such a model, I could agree with you. Not with current model though
> since it is very confusing - one thinks he maps something within the
> resolution scope of the mapped repository but he may get unexpectedly
> something else.
>
Really? I have never made that assumption. We create a verified aggregation and of course everything is verified in
scope of the aggregation. I think you put too much weight on the contribution concept.

- thomas
Re: Aggregator with version range support [message #517120 is a reply to message #517115] Fri, 26 February 2010 10:35 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
....
>
>> To make what you say intuitive, then a contribution should look like
>> this:
>>
>> Contribution A
>> |
>> + Repository hints
>> |
>> + Repo A
>> + Repo B
>> + Repo C
>> |
>> + Mappings
>> |
>> + IU a.b.c, version >=1.0.0
>> + Everything from Repo B
>>
>> The "Repository hints" would serve as possible sources that the
>> contributor "recommends" to use for the final resolution. We could also
>> remove current "Validation repositories" since it would not make sense
>> any more (in my example, "Repo C" is actually a validation repository
>> since nothing is mapped in contribution A).
>>
> That assumes that all contributions are self sufficient. And sure, there
> are perhaps use-cases where this could be good. I haven't seen them yet
> though. On the contrary, the use-cases at hand always have contributions
> that depend on other contributions.
>

No, you are missing the point. The contributions may transitively depend on other contributions of course.
I just want to express what you say with the model - just to avoid confusion. I am still convinced that
if you map IUs by choosing them from a mapped repository, you naturally expect to get them within the scope of that
repository. The result must either follow this rule, or the model must be changed somehow (I provided a draft of a suggestion above).

What my contribution depends on does not have to be included in my "repository hints" at all.


Filip
Re: Aggregator with version range support [message #517121 is a reply to message #517115] Fri, 26 February 2010 10:37 Go to previous messageGo to next message
Karel Brezina is currently offline Karel BrezinaFriend
Messages: 68
Registered: July 2009
Member
>> With such a model, I could agree with you. Not with current model though
>> since it is very confusing - one thinks he maps something within the
>> resolution scope of the mapped repository but he may get unexpectedly
>> something else.
>>
> Really? I have never made that assumption. We create a verified
> aggregation and of course everything is verified in scope of the
> aggregation. I think you put too much weight on the contribution concept.
>
> - thomas

Well, I agree with Filip. Each model should reflect the real world. Why
should we model a relation between a concrete repository and a concrete
IU, when in fact the relation does not exist? Wouldn't it be better to
model it as a heap of requests and a heap of resources? AFAIK,
buckminster works exactly the same way (CQuery vs. RMap)

Karel
Re: Aggregator with version range support [message #517125 is a reply to message #517120] Fri, 26 February 2010 06:06 Go to previous messageGo to next message
Achim Demelt is currently offline Achim DemeltFriend
Messages: 160
Registered: July 2009
Senior Member
Hi guys,

Sorry for jumping into this discussion so late, but you truly are hard-
headed guys and are exchanging arguments at such a fast pace that I've
hardly had time to catch up ;-)

Now here's my point of view, but before you read on, please note two things:

* I'm a hard-headed guy myself ;-)
* I have no practical experience with the aggregator (yet), and my
theoretical knowledge of it is, of course, not as deep as yours.

That said, here's my use-case, and I think it supports Filip's position:

We're pulling in everything from the Galileo release site. This includes
Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
repo hosted at itemis. This repo happens to include a 0.8.x version of
Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
Especially not if this leads to the 0.7.x version being excluded in the
result, because then I end up with a completely inconsistent repository.
(Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).

Now if I understand you guys correctly, Thomas would argue that I would have
to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
There is, however, no need to do that, because I assume the Galileo repo to
never, ever contain that version. It's only as a side-effect from pulling
org.antlr that this ends up in my result. But I could never foresee this
effect when maintaining the contribution from Galileo.

Cheers,
Achim


Filip Hrbek wrote:

> Thomas Hallgren wrote:
> ...
>>
>>> To make what you say intuitive, then a contribution should look like
>>> this:
>>>
>>> Contribution A
>>> |
>>> + Repository hints
>>> |
>>> + Repo A
>>> + Repo B
>>> + Repo C
>>> |
>>> + Mappings
>>> |
>>> + IU a.b.c, version >=1.0.0
>>> + Everything from Repo B
>>>
>>> The "Repository hints" would serve as possible sources that the
>>> contributor "recommends" to use for the final resolution. We could also
>>> remove current "Validation repositories" since it would not make sense
>>> any more (in my example, "Repo C" is actually a validation repository
>>> since nothing is mapped in contribution A).
>>>
>> That assumes that all contributions are self sufficient. And sure, there
>> are perhaps use-cases where this could be good. I haven't seen them yet
>> though. On the contrary, the use-cases at hand always have contributions
>> that depend on other contributions.
>>
>
> No, you are missing the point. The contributions may transitively depend
> on other contributions of course. I just want to express what you say with
> the model - just to avoid confusion. I am still convinced that if you map
> IUs by choosing them from a mapped repository, you naturally expect to get
> them within the scope of that repository. The result must either follow
> this rule, or the model must be changed somehow (I provided a draft of a
> suggestion above).
>
> What my contribution depends on does not have to be included in my
> "repository hints" at all.
>
>
> Filip
Re: Aggregator with version range support [message #517129 is a reply to message #517121] Fri, 26 February 2010 11:03 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:37 AM, Karel Brezina wrote:
> Well, I agree with Filip. Each model should reflect the real world. Why
> should we model a relation between a concrete repository and a concrete
> IU, when in fact the relation does not exist?

Sure it exists. It's just not what ultimately is selected. That's a version range problem and it should not be resolved
by making the contribution as such the scope of resolution.


> Wouldn't it be better to
> model it as a heap of requests and a heap of resources? AFAIK,
> buckminster works exactly the same way (CQuery vs. RMap)
>
We already do model it as you suggest. You are trying to move that model down to each contribution which makes the
scoping all wrong. The contribution can never be the scope of anything. The 'heap of resources' is the sum of all
contributions. That has to be the default. I'm fine with adding rules to force deviations from that default in case such
rules are missing from a contribution and things break. I.e. a rule that you add to a contribution that says, "iu x.y.z.
should have an upper bound 1.2.3", but we cannot make the contribution itself a scope of resolution.

- thomas
Re: Aggregator with version range support [message #517134 is a reply to message #517125] Fri, 26 February 2010 06:30 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Hi Achim,

you don't have to apologize for jumping into our discussion, you are welcome of course!

I am hard-headed but I am able listen to others and finally change my mind, but I prefer listening to more than one guy (Thomas - it does not mean that I don't listen to you of course). It always yields better results.

I think that your use-case exactly reflects my concern. Thomas probably knows other use-cases.

Our goal is to provide mechanisms for all of them of course. I am not against adding more options to the aggregator that would allow a user to do what he really wants.

All I want is

* to have a model that reflects reality
* to have a model that reflects user's natural thinking
* to have an engine that works in conformance with the model (not doing unexpected things)
* to have reasonable default settings (i.e. for use-cases that are fine to most users)
* to allow customization of the behavior by setting specific options/flags/rules...

I am sure we will find a solution. Again, the more opinions I collect now the better the solution will be.

Filip


Achim Demelt wrote:

> Hi guys,
>
> Sorry for jumping into this discussion so late, but you truly are hard-
> headed guys and are exchanging arguments at such a fast pace that I've
> hardly had time to catch up ;-)
>
> Now here's my point of view, but before you read on, please note two things:
>
> * I'm a hard-headed guy myself ;-)
> * I have no practical experience with the aggregator (yet), and my
> theoretical knowledge of it is, of course, not as deep as yours.
>
> That said, here's my use-case, and I think it supports Filip's position:
>
> We're pulling in everything from the Galileo release site. This includes
> Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
> repo hosted at itemis. This repo happens to include a 0.8.x version of
> Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
> Especially not if this leads to the 0.7.x version being excluded in the
> result, because then I end up with a completely inconsistent repository.
> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>
> Now if I understand you guys correctly, Thomas would argue that I would have
> to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
> There is, however, no need to do that, because I assume the Galileo repo to
> never, ever contain that version. It's only as a side-effect from pulling
> org.antlr that this ends up in my result. But I could never foresee this
> effect when maintaining the contribution from Galileo.
>
> Cheers,
> Achim
>
Re: Aggregator with version range support [message #517135 is a reply to message #517120] Fri, 26 February 2010 06:30 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:35 AM, Filip Hrbek wrote:
> I am still convinced that if you map IUs by choosing them from a mapped
> repository, you naturally expect to get them within the scope of that
> repository. The result must either follow this rule, or the model must
> be changed somehow (I provided a draft of a suggestion above).
>
I think it's the false assumption that a mapping creates a hard link to a specific repository that must change. That's
never the case in p2 and if possible, we should make that more clear to the user.

This is actually the same discussion that we had a some time ago [1]. The user has one view of things but ultimately,
the planner makes changes to that view. The solution is not to prevent that from happening. The problem is, how do we
visualize that to the user?

[1] http://www.eclipse.org/forums/index.php?t=msg&&th=15 9023&goto=502689#msg_502689

- thomas
Re: Aggregator with version range support [message #517139 is a reply to message #517125] Fri, 26 February 2010 11:29 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:58 AM, Achim Demelt wrote:
> Hi guys,
>
> Sorry for jumping into this discussion so late, but you truly are hard-
> headed guys and are exchanging arguments at such a fast pace that I've
> hardly had time to catch up ;-)
>
> Now here's my point of view, but before you read on, please note two things:
>
> * I'm a hard-headed guy myself ;-)
> * I have no practical experience with the aggregator (yet), and my
> theoretical knowledge of it is, of course, not as deep as yours.
>
> That said, here's my use-case, and I think it supports Filip's position:
>
> We're pulling in everything from the Galileo release site. This includes
> Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
> repo hosted at itemis. This repo happens to include a 0.8.x version of
> Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
> Especially not if this leads to the 0.7.x version being excluded in the
> result, because then I end up with a completely inconsistent repository.
> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>
> Now if I understand you guys correctly, Thomas would argue that I would have
> to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
> There is, however, no need to do that, because I assume the Galileo repo to
> never, ever contain that version. It's only as a side-effect from pulling
> org.antlr that this ends up in my result. But I could never foresee this
> effect when maintaining the contribution from Galileo.
>
Good point. You don't own the Galileo contribution. So there are three solutions to this problem:

1. The most likely one is that there is no problem. If EMF 2.6 isn't included, then the planner will give up on XText
0.8.0 and try something else. If no solution at all can be found given all input, it will give up. It will _never_
create something that is inconsistent.

2. Your aggregation fails since no resolution is possible so you (the aggregator maintainer) sees that a contribution
has a problem and report this to the contributor. The contributor must then address this problem. Perhaps by adding an
upper limit to a bundle requirement. This is what currently happens on a day to day basis when problems occur in Helios.

3. Your aggregation fails and you add a rule that is above all contributions that says "In the final result, I don't
want Xtext of higher version then 0.7.x". This rule is propagated as a constraint to the planner. This possibility does
not yet exist, but I wouldn't be adverse to adding it.

- thomas
Re: Aggregator with version range support [message #517142 is a reply to message #517139] Fri, 26 February 2010 11:39 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
+1
finally a description of the problem/solution that I understood the
implications of.

The next issue is how to make user understand what went wrong and how to
fix it.

- henrik
> Good point. You don't own the Galileo contribution. So there are three
> solutions to this problem:
>
> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
> included, then the planner will give up on XText 0.8.0 and try
> something else. If no solution at all can be found given all input, it
> will give up. It will _never_ create something that is inconsistent.
>
> 2. Your aggregation fails since no resolution is possible so you (the
> aggregator maintainer) sees that a contribution has a problem and
> report this to the contributor. The contributor must then address this
> problem. Perhaps by adding an upper limit to a bundle requirement.
> This is what currently happens on a day to day basis when problems
> occur in Helios.
>
> 3. Your aggregation fails and you add a rule that is above all
> contributions that says "In the final result, I don't want Xtext of
> higher version then 0.7.x". This rule is propagated as a constraint to
> the planner. This possibility does not yet exist, but I wouldn't be
> adverse to adding it.
>
> - thomas
>


--
- henrik
Re: Aggregator with version range support [message #517145 is a reply to message #517129] Fri, 26 February 2010 11:45 Go to previous messageGo to next message
Karel Brezina is currently offline Karel BrezinaFriend
Messages: 68
Registered: July 2009
Member
Thomas Hallgren napsal(a):
> On 02/26/2010 11:37 AM, Karel Brezina wrote:
>> Well, I agree with Filip. Each model should reflect the real world. Why
>> should we model a relation between a concrete repository and a concrete
>> IU, when in fact the relation does not exist?
>
> Sure it exists. It's just not what ultimately is selected. That's a
> version range problem and it should not be resolved by making the
> contribution as such the scope of resolution.

Maybe I feel the parent child relationship a bit stronger. I have two
sons, I don't want to ultimately realize that they don't belong to me. :-)

In my thread I haven't mentioned contributions at all. I just wanted to
generalized it a little bit. I still feel that when the engine considers
it as two separate heaps (requests and resources) then the UI should
reflect it, otherwise the result of the aggreation might be a big surprise.

Karel
Re: Aggregator with version range support [message #517146 is a reply to message #517139] Fri, 26 February 2010 11:47 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Yes, Thomas is right here. The aggregator always yields consistent result, or it fails.
If EMF 2.6 is not available in any of mapped repositories, it will choose xtext 0.7.x.

But if EMF 2.6 is found somewhere else, you may get 0.8.x which you don't want for some reason.
In that case, you must add a version constraint to your contribution.

I think that limiting the scope of resolution for selected mapping (which does not have to be the default) would be better than adding an explicit constraint.

It should be up to each user how he wants the aggregator to behave.

We must talk about changing the model which still does not reflect the reality - that's the biggest issue here.

Filip



Thomas Hallgren wrote:
....
>> Especially not if this leads to the 0.7.x version being excluded in the
>> result, because then I end up with a completely inconsistent repository.
>> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>>
>> Now if I understand you guys correctly, Thomas would argue that I
>> would have
>> to explicitly exclude the 0.8.x version when fetching from the Galileo
>> repo.
>> There is, however, no need to do that, because I assume the Galileo
>> repo to
>> never, ever contain that version. It's only as a side-effect from pulling
>> org.antlr that this ends up in my result. But I could never foresee this
>> effect when maintaining the contribution from Galileo.
>>
> Good point. You don't own the Galileo contribution. So there are three
> solutions to this problem:
>
> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
> included, then the planner will give up on XText 0.8.0 and try something
> else. If no solution at all can be found given all input, it will give
> up. It will _never_ create something that is inconsistent.
>
....
Re: Aggregator with version range support [message #517154 is a reply to message #517145] Fri, 26 February 2010 07:14 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 12:45 PM, Karel Brezina wrote:
> Maybe I feel the parent child relationship a bit stronger. I have two
> sons, I don't want to ultimately realize that they don't belong to me. :-)
>
I have three, and I think the saying is - you only have your kids as a loan. ;-)

- thomas
Re: Aggregator with version range support [message #517155 is a reply to message #517142] Fri, 26 February 2010 07:14 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Henrik, what do you think of the model?

That's what makes me worried the most. It does not reflect what the engine finally does.

I think we should concentrate on this issue now.

If you didn't want to go through all the posts, here's the summary:

- user specifies a mapped repository
- user selects a mapped UI *from* this repository (in the UI, it is a *child node* of that mapped repository)
- user specifies a version range (which may be open)
- user can check interactively which versions from that repository match his request
- user runs the aggregation

I think that user expects that the aggregation will contain one of the versions he had seen in the interactive checklist. What a surprise if there's something else?

I have suggested a change to that model so that we would avoid this confusion (see former posts).

In addition to this, I think that optional limiting of the resolution scope for explicitly added IUs would be helpful too, as well as introducing more complex version constraints than version ranges (which is another big issue).

Filip


Henrik Lindberg wrote:
> +1
> finally a description of the problem/solution that I understood the
> implications of.
>
> The next issue is how to make user understand what went wrong and how to
> fix it.
>
> - henrik
>> Good point. You don't own the Galileo contribution. So there are three
>> solutions to this problem:
>>
>> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
>> included, then the planner will give up on XText 0.8.0 and try
>> something else. If no solution at all can be found given all input, it
>> will give up. It will _never_ create something that is inconsistent.
>>
>> 2. Your aggregation fails since no resolution is possible so you (the
>> aggregator maintainer) sees that a contribution has a problem and
>> report this to the contributor. The contributor must then address this
>> problem. Perhaps by adding an upper limit to a bundle requirement.
>> This is what currently happens on a day to day basis when problems
>> occur in Helios.
>>
>> 3. Your aggregation fails and you add a rule that is above all
>> contributions that says "In the final result, I don't want Xtext of
>> higher version then 0.7.x". This rule is propagated as a constraint to
>> the planner. This possibility does not yet exist, but I wouldn't be
>> adverse to adding it.
>>
>> - thomas
>>
>
>
Re: Aggregator with version range support [message #517160 is a reply to message #517155] Fri, 26 February 2010 07:36 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
I did read all of the posts on this subject.

It is just that I have not used the aggregator enough to have a good
feel for what would be intuitive.

When picking something from a repository, what does it mean?
a) pick things matching this requirement, and then use all available
repositories to satisfy all dependencies.
b) pick things matching this requirement from this repository, and then
use all available respositories *but* this repository to satisfy the
dependencies.

b) is good for "OMG there is all sorts of crap in there in addition to
the few things I actually do need".

When using b) it is easy to to create a non valid result, because the
thing picked from the problem repo needs more stuff not in other
repositories. The user would then need to add those explicitly until a
valid result is obtained (or gives up because of conflicts).

I can see option b) as a flag on a repo - "only pick the explicitly
requested IUs".

A question - as a user I would assume that if I request A with a range,
(from one repo, as I perceive it as a user), and there exists A's in
this range in that repo, that they are indeed included in the final
result even if there are newer versions of A in other repositories that
are mapped with other rules. Is this true?

Regards
- henrik

On 2/26/10 1:10 PM, Filip Hrbek wrote:
> Henrik, what do you think of the model?
>
> That's what makes me worried the most. It does not reflect what the
> engine finally does.
>
> I think we should concentrate on this issue now.
>
> If you didn't want to go through all the posts, here's the summary:
>
> - user specifies a mapped repository
> - user selects a mapped UI *from* this repository (in the UI, it is a
> *child node* of that mapped repository)
> - user specifies a version range (which may be open)
> - user can check interactively which versions from that repository match
> his request
> - user runs the aggregation
>
> I think that user expects that the aggregation will contain one of the
> versions he had seen in the interactive checklist. What a surprise if
> there's something else?
>
> I have suggested a change to that model so that we would avoid this
> confusion (see former posts).
>
> In addition to this, I think that optional limiting of the resolution
> scope for explicitly added IUs would be helpful too, as well as
> introducing more complex version constraints than version ranges (which
> is another big issue).
>
> Filip
>
>
> Henrik Lindberg wrote:
>> +1
>> finally a description of the problem/solution that I understood the
>> implications of.
>>
>> The next issue is how to make user understand what went wrong and how to
>> fix it.
>> - henrik
>>> Good point. You don't own the Galileo contribution. So there are three
>>> solutions to this problem:
>>>
>>> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
>>> included, then the planner will give up on XText 0.8.0 and try
>>> something else. If no solution at all can be found given all input, it
>>> will give up. It will _never_ create something that is inconsistent.
>>>
>>> 2. Your aggregation fails since no resolution is possible so you (the
>>> aggregator maintainer) sees that a contribution has a problem and
>>> report this to the contributor. The contributor must then address this
>>> problem. Perhaps by adding an upper limit to a bundle requirement.
>>> This is what currently happens on a day to day basis when problems
>>> occur in Helios.
>>>
>>> 3. Your aggregation fails and you add a rule that is above all
>>> contributions that says "In the final result, I don't want Xtext of
>>> higher version then 0.7.x". This rule is propagated as a constraint to
>>> the planner. This possibility does not yet exist, but I wouldn't be
>>> adverse to adding it.
>>>
>>> - thomas
>>>
>>
>>
Re: Aggregator with version range support [message #517164 is a reply to message #517160] Fri, 26 February 2010 12:46 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
> A question - as a user I would assume that if I request A with a range,
> (from one repo, as I perceive it as a user), and there exists A's in
> this range in that repo, that they are indeed included in the final
> result even if there are newer versions of A in other repositories that
> are mapped with other rules. Is this true?
>
Nope. The planner consider all repositories as one source of input. It never considers what individual repository a
specific requirement originated from. It will create a plan where all requirement ranges are fulfilled but the actual
IU's included can stem from any repository.

- thomas
Re: Aggregator with version range support [message #517168 is a reply to message #517164] Fri, 26 February 2010 13:09 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>> A question - as a user I would assume that if I request A with a range,
>> (from one repo, as I perceive it as a user), and there exists A's in
>> this range in that repo, that they are indeed included in the final
>> result even if there are newer versions of A in other repositories that
>> are mapped with other rules. Is this true?
>>
> Nope. The planner consider all repositories as one source of input. It
> never considers what individual repository a specific requirement
> originated from. It will create a plan where all requirement ranges are
> fulfilled but the actual IU's included can stem from any repository.
>


....and this it the issue!

What Henrik stated is IMHO intuitive from the user's perspective (unless the user is familiar with all our discussions :-) ).

I want to change the model at least so that a user wouldn't think that it would work like this. Ideally, I would add more options to change the behavior as needed.

Question (to move forward):

Is the model I suggested better or worse than current? If it is bad, why?

Filip
Re: Aggregator with version range support [message #517174 is a reply to message #517168] Fri, 26 February 2010 08:46 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:09 PM, Filip Hrbek wrote:
> Thomas Hallgren wrote:
>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>> A question - as a user I would assume that if I request A with a range,
>>> (from one repo, as I perceive it as a user), and there exists A's in
>>> this range in that repo, that they are indeed included in the final
>>> result even if there are newer versions of A in other repositories that
>>> are mapped with other rules. Is this true?
>>>
>> Nope. The planner consider all repositories as one source of input. It
>> never considers what individual repository a specific requirement
>> originated from. It will create a plan where all requirement ranges
>> are fulfilled but the actual IU's included can stem from any repository.
>>
>
>
> ...and this it the issue!
>
> What Henrik stated is IMHO intuitive from the user's perspective (unless
> the user is familiar with all our discussions :-) ).
>
> I want to change the model at least so that a user wouldn't think that
> it would work like this. Ideally, I would add more options to change the
> behavior as needed.
>
> Question (to move forward):
>
> Is the model I suggested better or worse than current? If it is bad, why?
>
I think its worse because it doesn't really solve the problem. The contribution can not be the scope of resolution so
adding more repositories at that level doesn't help. The final solution may have chosen IUs from other repositories
regardless.

The validation repository needs to stay because it validates the final aggregation, not each contribution.

I think the current model is fine given the requirements. We really don't want to add more things then necessary to each
contribution. The biggest problem I see, is that we have no way of visualizing that the planner might choose an IU that
is different from what the user has in his view. The user needs to be made aware of that this can happen during
authoring (the tool should make him aware that his selection isn't necessarily resolved from the current repository) and
also when the planner induced change occur (as warning or info markers once the planning has completed).

- thomas
Re: Aggregator with version range support [message #517177 is a reply to message #517174] Fri, 26 February 2010 13:55 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
....
>>
>> Question (to move forward):
>>
>> Is the model I suggested better or worse than current? If it is bad, why?
>>
> I think its worse because it doesn't really solve the problem. The
> contribution can not be the scope of resolution so adding more
> repositories at that level doesn't help. The final solution may have
> chosen IUs from other repositories regardless.

1) It does not have to be the scope by default but it would be interesting to have such an option (perhaps in future).
The scope (if requested) would only affect the first level, i.e. the components the user is explicitly mapping.

2) If it is *not* the resolution scope (which we talk about as the default right now), then we can't
have the selected IUs *under* the mapped repository because it is confusing. That is why I moved the
repository hints away from the mapping requests - just to make it clear.

The reason why I named the node *hints* is to make it clear that it's nothing more than a hint, i.e. which repositories
to store to the global heap of available repositories.

>
> The validation repository needs to stay because it validates the final
> aggregation, not each contribution.

If the validation repository belongs to the aggregation root, why don't all mapped repositories belong there?
If they don't mean the resolution scope for mapped IUs, they should also be placed in the aggregator root
since they affect the final aggregation, not the individual mappings.

However, I think that even the validation repositories should be moved under the contribution anyway.
If someone knows that his contribution needs another repository for validation, he can freely contribute
a repository hint for validation.


>
> I think the current model is fine given the requirements. We really
> don't want to add more things then necessary to each contribution. The
> biggest problem I see, is that we have no way of visualizing that the
> planner might choose an IU that is different from what the user has in
> his view. The user needs to be made aware of that this can happen during
> authoring (the tool should make him aware that his selection isn't
> necessarily resolved from the current repository) and also when the
> planner induced change occur (as warning or info markers once the
> planning has completed).

If we change the model so that user understands that he does not choose from the mapped repository, then
I can see a simple solution. We already have an interactive list of available versions in the UI. All we need to do
is to show *all* versions from *all* repositories (not only those from contribution's hints) and also show
which repository the resolved IU comes from. Then the user is not confused - he can see all potentially valid matches of
his selection before the (long running) verification is run.

Filip
Re: Aggregator with version range support [message #517180 is a reply to message #517177] Fri, 26 February 2010 14:10 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:55 PM, Filip Hrbek wrote:
> If the validation repository belongs to the aggregation root, why don't
> all mapped repositories belong there?

A contributor typically contributes repositories. He optionally filters what should be contributed from those
repositories by adding mappings. No mappings == everything. So of course all contributed repositories are listed in
their respective contributions.

The aggregator manager decides what contributions to aggregate and how to validate that aggregation. The validation has
nothing to do with each contribution. The contributions are neither self sufficient and not validated. Validation is on
the aggregation as a whole and thus, cannot be in a contribution.

HTH,
- thomas
Re: Aggregator with version range support [message #517183 is a reply to message #517177] Fri, 26 February 2010 14:14 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:55 PM, Filip Hrbek wrote:

> If we change the model so that user understands that he does not choose
> from the mapped repository, then
> I can see a simple solution.

Consider the time when a contributor edits his contribution. He contributes one or several repositories. The optionally
mapped things that he might add, should be considered filters. By adding them he says, don't automatically map other
things from my contributed repositories.

At the time of editing the contribution, the contributor does not always have access to all other repositories. He
should not really be aware of how the final aggregation is constituted.

- thomas
Re: Aggregator with version range support [message #517184 is a reply to message #517180] Fri, 26 February 2010 14:26 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 02:55 PM, Filip Hrbek wrote:
>> If the validation repository belongs to the aggregation root, why don't
>> all mapped repositories belong there?
>
> A contributor typically contributes repositories. He optionally filters
> what should be contributed from those repositories by adding mappings.
> No mappings == everything. So of course all contributed repositories are
> listed in their respective contributions.
>

Of course I agree with this, although "no mappings = everything" is not
intuitive enough for me. Btw., it does not mean "everything from this repo"
since you may actually finally map something else (newer versions from other
repositories).

I came to the conclusion that all repos should
be under aggregator root using your argument - that it does not create the
resolution scope.


> The aggregator manager decides what contributions to aggregate and how
> to validate that aggregation. The validation has nothing to do with each
> contribution. The contributions are neither self sufficient and not
> validated. Validation is on the aggregation as a whole and thus, cannot
> be in a contribution.

Really? Why?

I can imagine a context menu action that validates a contribution or a set of
selected contributions. This would be a good base for the aggregator manager
if the indivitual contributions were pre-validated by contribution owners.
I think that the contributor owner can imagine what repositores he needs to validate
his contribution, and can add them as repository hints. Duplicate entries don't matter,
they finally share the same resource. And, it does not prevent us from keeping
global validation repositories as an option.


Taking all your arguments into account, I still think that having the mapped IUs
under the mapper repository *is confusing* and needs to be refactored.

Filip
Re: Aggregator with version range support [message #517197 is a reply to message #517184] Fri, 26 February 2010 15:07 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Filip,
I think it all comes back to the requirements and the current use-cases.

1. In Helios and Galileo:
1a. each contributor is responsible for providing their update site(s) as input.
1b. each contributor is responsible for the integrity of the content of the contributed repositories.
1c. each contributor might select specific roots (features) to be mapped if he doesn't want the whole thing included.
The selection he makes here is of course limited to what he provides.
1d. the aggregation manager never edits a contribution. He merely aggregates them with some validation (the platform)
and disable/enable contributions as he see fit based on their ability to provide functional input.

For this use-case, since each contributor only deals with his own contribution, there cannot be any confusion caused by
having the mapped IU's in the contribution.


2. For Morgan Stanley
2a. One person is creating the aggregation.
2b. He builds one single contribution with everything in it since he is the sole contributor and owner of the whole
process. No need for additional email contacts for "blame" email, nor to spread the responsibility.

For this use-case, since there is only one contribution, there cannot be any confusion caused by having the mapped IU's
in that contribution.

The scenario you describe attempts to solve a use-case that sits somewhere in between the Eclipse release train use-case
and the MS use-case where the contributor is not limited to his own contribution and/or the aggregator manager uses
multiple contributions and is concerned with what each contribution contains. I don't understand why we need this. What
problem is it you are trying to solve?

If there was an easy way to refactor the model and resolve the issue once and for all, I would be all for that. But I
can't see how we can do that without disrupting the above use-cases. And doing it just to satisfy a use-case that
currently only exists in theory is not something I think we are prepared to spend resources on.

- thomas
Re: Aggregator with version range support [message #517222 is a reply to message #517197] Fri, 26 February 2010 15:56 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
OK, we need a conclusion, I don't like conclusionless discussions:

1) I removed the hack that was implementing what I though was natural
2) Now it works as it worked before (although it needs a lot of testing since not everything could be done with version ranges in the same way as with fixed versions)
3) I am very unsatisfied with the model-to-reality relationship, I think we did a bad job here.

I still think we could refactor the model without much effort if we spent some more time on discussions with more people. I don't think it's wise to create
software with only two use-cases in one's mind - it's not professional.


I wish you a nice weekend.

Good news is that we suppose to have a functional b3 aggregator (both integrated in Eclipse IDE and standalone headless application) which will be capable of
converting old helios build files into new b3 aggregator files and running the build of the Helios release.

Filip


Thomas Hallgren wrote:
> Filip,
> I think it all comes back to the requirements and the current use-cases.
>
> 1. In Helios and Galileo:
> 1a. each contributor is responsible for providing their update site(s)
> as input.
> 1b. each contributor is responsible for the integrity of the content of
> the contributed repositories.
> 1c. each contributor might select specific roots (features) to be mapped
> if he doesn't want the whole thing included. The selection he makes here
> is of course limited to what he provides.
> 1d. the aggregation manager never edits a contribution. He merely
> aggregates them with some validation (the platform) and disable/enable
> contributions as he see fit based on their ability to provide functional
> input.
>
> For this use-case, since each contributor only deals with his own
> contribution, there cannot be any confusion caused by having the mapped
> IU's in the contribution.
>
>
> 2. For Morgan Stanley
> 2a. One person is creating the aggregation.
> 2b. He builds one single contribution with everything in it since he is
> the sole contributor and owner of the whole process. No need for
> additional email contacts for "blame" email, nor to spread the
> responsibility.
>
> For this use-case, since there is only one contribution, there cannot be
> any confusion caused by having the mapped IU's in that contribution.
>
> The scenario you describe attempts to solve a use-case that sits
> somewhere in between the Eclipse release train use-case and the MS
> use-case where the contributor is not limited to his own contribution
> and/or the aggregator manager uses multiple contributions and is
> concerned with what each contribution contains. I don't understand why
> we need this. What problem is it you are trying to solve?
>
> If there was an easy way to refactor the model and resolve the issue
> once and for all, I would be all for that. But I can't see how we can do
> that without disrupting the above use-cases. And doing it just to
> satisfy a use-case that currently only exists in theory is not something
> I think we are prepared to spend resources on.
>
> - thomas
Re: Aggregator with version range support [message #517246 is a reply to message #517222] Fri, 26 February 2010 11:42 Go to previous messageGo to next message
Achim Demelt is currently offline Achim DemeltFriend
Messages: 160
Registered: July 2009
Senior Member
I must admit that I could not follow all of your arguments due to a lack of
in-depth aggregator knowledge. But I have the feeling that you really
*should* continue this discussion at another time, maybe over a beer at
EclipseCon. And, yes, include as many people (and as much beer) as it takes
to find a solution that leaves noone with bad feelings.

One more thing from my side: p2 tends to do things in a way that is
incomprehensible to the layman, and this includes many talented software
engineers who just didn't have the good fortune to study the p2 code base
themselves. It would be a shame if the aggregator acquired a similar
reputation.

Cheers,
Achim

Filip Hrbek wrote:

> OK, we need a conclusion, I don't like conclusionless discussions:
>
> 1) I removed the hack that was implementing what I though was natural
> 2) Now it works as it worked before (although it needs a lot of testing
> since not everything could be done with version ranges in the same way as
> with fixed versions) 3) I am very unsatisfied with the model-to-reality
> relationship, I think we did a bad job here.
>
> I still think we could refactor the model without much effort if we spent
> some more time on discussions with more people. I don't think it's wise to
> create software with only two use-cases in one's mind - it's not
> professional.
>
>
> I wish you a nice weekend.
>
> Good news is that we suppose to have a functional b3 aggregator (both
> integrated in Eclipse IDE and standalone headless application) which will
> be capable of converting old helios build files into new b3 aggregator
> files and running the build of the Helios release.
>
> Filip
>
>
> Thomas Hallgren wrote:
>> Filip,
>> I think it all comes back to the requirements and the current use-cases.
>>
>> 1. In Helios and Galileo:
>> 1a. each contributor is responsible for providing their update site(s)
>> as input.
>> 1b. each contributor is responsible for the integrity of the content of
>> the contributed repositories.
>> 1c. each contributor might select specific roots (features) to be mapped
>> if he doesn't want the whole thing included. The selection he makes here
>> is of course limited to what he provides.
>> 1d. the aggregation manager never edits a contribution. He merely
>> aggregates them with some validation (the platform) and disable/enable
>> contributions as he see fit based on their ability to provide functional
>> input.
>>
>> For this use-case, since each contributor only deals with his own
>> contribution, there cannot be any confusion caused by having the mapped
>> IU's in the contribution.
>>
>>
>> 2. For Morgan Stanley
>> 2a. One person is creating the aggregation.
>> 2b. He builds one single contribution with everything in it since he is
>> the sole contributor and owner of the whole process. No need for
>> additional email contacts for "blame" email, nor to spread the
>> responsibility.
>>
>> For this use-case, since there is only one contribution, there cannot be
>> any confusion caused by having the mapped IU's in that contribution.
>>
>> The scenario you describe attempts to solve a use-case that sits
>> somewhere in between the Eclipse release train use-case and the MS
>> use-case where the contributor is not limited to his own contribution
>> and/or the aggregator manager uses multiple contributions and is
>> concerned with what each contribution contains. I don't understand why
>> we need this. What problem is it you are trying to solve?
>>
>> If there was an easy way to refactor the model and resolve the issue
>> once and for all, I would be all for that. But I can't see how we can do
>> that without disrupting the above use-cases. And doing it just to
>> satisfy a use-case that currently only exists in theory is not something
>> I think we are prepared to spend resources on.
>>
>> - thomas
Re: Aggregator with version range support [message #517297 is a reply to message #517246] Fri, 26 February 2010 20:21 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Thanks Achim. You deserve a diplomacy badge ;) i must confess that I
don't get the arguments presented either. And even worse, I don't think
I even understand the problem except that there is a case that is
confusing for a user...

I think it is important to go back to the beginning and look at how a
user is making statements, what the user think they mean, and what they
really mean. Mixing this with implementation detail makes my head
explode...

- henrik

Achim Demelt <a.demelt@exxcellent.de> wrote:
> I must admit that I could not follow all of your arguments due to a
> lack of
> in-depth aggregator knowledge. But I have the feeling that you really
> *should* continue this discussion at another time, maybe over a beer
> at
> EclipseCon. And, yes, include as many people (and as much beer) as it
> takes
> to find a solution that leaves noone with bad feelings.
>
> One more thing from my side: p2 tends to do things in a way that is
> incomprehensible to the layman, and this includes many talented
> software
> engineers who just didn't have the good fortune to study the p2 code
> base
> themselves. It would be a shame if the aggregator acquired a similar
> reputation.
>
> Cheers,
> Achim
>
> Filip Hrbek wrote:
>
> > OK, we need a conclusion, I don't like conclusionless discussions:
> >
> > 1) I removed the hack that was implementing what I though was
> > natural
> > 2) Now it works as it worked before (although it needs a lot of
> > testing
> > since not everything could be done with version ranges in the same
> > way as
> > with fixed versions) 3) I am very unsatisfied with the
> > model-to-reality
> > relationship, I think we did a bad job here.
> >
> > I still think we could refactor the model without much effort if we
> > spent
> > some more time on discussions with more people. I don't think it's
> > wise to
> > create software with only two use-cases in one's mind - it's not
> > professional.
> >
> >
> > I wish you a nice weekend.
> >
> > Good news is that we suppose to have a functional b3 aggregator
> > (both
> > integrated in Eclipse IDE and standalone headless application) which
> > will
> > be capable of converting old helios build files into new b3
> > aggregator
> > files and running the build of the Helios release.
> >
> > Filip
> >
> >
> > Thomas Hallgren wrote:
> >> Filip,
> >> I think it all comes back to the requirements and the current
> > > use-cases.
> >>
> >> 1. In Helios and Galileo:
> >> 1a. each contributor is responsible for providing their update
> > > site(s)
> >> as input.
> >> 1b. each contributor is responsible for the integrity of the
> > > content of
> >> the contributed repositories.
> >> 1c. each contributor might select specific roots (features) to be
> > > mapped
> >> if he doesn't want the whole thing included. The selection he makes
> > > here
> >> is of course limited to what he provides.
> >> 1d. the aggregation manager never edits a contribution. He merely
> >> aggregates them with some validation (the platform) and
> > > disable/enable
> >> contributions as he see fit based on their ability to provide
> > > functional
> >> input.
> >>
> >> For this use-case, since each contributor only deals with his own
> >> contribution, there cannot be any confusion caused by having the
> > > mapped
> >> IU's in the contribution.
> >>
> >>
> >> 2. For Morgan Stanley
> >> 2a. One person is creating the aggregation.
> >> 2b. He builds one single contribution with everything in it since
> > > he is
> >> the sole contributor and owner of the whole process. No need for
> >> additional email contacts for "blame" email, nor to spread the
> >> responsibility.
> >>
> >> For this use-case, since there is only one contribution, there
> > > cannot be
> >> any confusion caused by having the mapped IU's in that
> > > contribution.
> >>
> >> The scenario you describe attempts to solve a use-case that sits
> >> somewhere in between the Eclipse release train use-case and the MS
> >> use-case where the contributor is not limited to his own
> > > contribution
> >> and/or the aggregator manager uses multiple contributions and is
> >> concerned with what each contribution contains. I don't understand
> > > why
> >> we need this. What problem is it you are trying to solve?
> >>
> >> If there was an easy way to refactor the model and resolve the
> > > issue
> >> once and for all, I would be all for that. But I can't see how we
> > > can do
> >> that without disrupting the above use-cases. And doing it just to
> >> satisfy a use-case that currently only exists in theory is not
> > > something
> >> I think we are prepared to spend resources on.
> >>
> >> - thomas
>
>


--
- henrik
Re: Aggregator with version range support [message #517298 is a reply to message #517164] Fri, 26 February 2010 20:21 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
That is not what i asked. I asked if it only picked the latest.
- henrik
Thomas Hallgren <thomas@tada.se> wrote:
> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
> > A question - as a user I would assume that if I request A with a
> > range,
> > (from one repo, as I perceive it as a user), and there exists A's in
> > this range in that repo, that they are indeed included in the final
> > result even if there are newer versions of A in other repositories
> > that
> > are mapped with other rules. Is this true?
> >
> Nope. The planner consider all repositories as one source of input. It
> never considers what individual repository a specific requirement
> originated from. It will create a plan where all requirement ranges
> are fulfilled but the actual IU's included can stem from any
> repository.
>
> - thomas
>


--
- henrik
Re: Aggregator with version range support [message #517330 is a reply to message #517298] Fri, 26 February 2010 23:19 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
> That is not what i asked. I asked if it only picked the latest.
> - henrik

I'm reading your question again and I come up with the same answer :-)

But let me try and rephrase:

The fact that there are newer versions of A in other repositories, and still within the requested range, might well mean
that these newer versions are chosen. The planner will pick the best solution possible for A in scope of the aggregation
as a whole. No consideration whatsoever is given to in which source repositories an IU is found.

OK?

- thomas

> Thomas Hallgren<thomas@tada.se> wrote:
>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>> A question - as a user I would assume that if I request A with a range,
>>> (from one repo, as I perceive it as a user), and there exists A's in
>>> this range in that repo, that they are indeed included in the final
>>> result even if there are newer versions of A in other repositories
>>> that are mapped with other rules. Is this true?
>>>
>> Nope. The planner consider all repositories as one source of input. It
>> never considers what individual repository a specific requirement
>> originated from. It will create a plan where all requirement ranges
>> are fulfilled but the actual IU's included can stem from any
>> repository.
>>
>> - thomas
>>
>
>
Re: Aggregator with version range support [message #517338 is a reply to message #517330] Fri, 26 February 2010 19:56 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Well, what I want to know is still not clear.
I understand that:
- a request to include A [fromA1, toA1] does not mean that some other
request of A [fromA2, toA2 > toA1] is restricted.
- a request to include A [some range] is not tied to a particular repository

What I wonder what the resulting repository will contain if there are
many versions of A that comply. Say that fromA1 < fromA2, and toA1 > toA2.

If I have these requests in my map, I may have different desires:
a) I want all versions i.e. fromA1 (the smallest requested) to toA1 (the
largest requested).
b) I want the latest version that complies with all requests.

I would desire a) if I want to include all "old version" (in stated
range), and desire b) if I just want a slimmed repository.

This was my question - what does the aggregator mirror if I add the two
requests to A?

- henrik

On 2/27/10 12:19 AM, Thomas Hallgren wrote:
> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>> That is not what i asked. I asked if it only picked the latest.
>> - henrik
>
> I'm reading your question again and I come up with the same answer :-)
>
> But let me try and rephrase:
>
> The fact that there are newer versions of A in other repositories, and
> still within the requested range, might well mean that these newer
> versions are chosen. The planner will pick the best solution possible
> for A in scope of the aggregation as a whole. No consideration
> whatsoever is given to in which source repositories an IU is found.
>
> OK?
>
> - thomas
>
>> Thomas Hallgren<thomas@tada.se> wrote:
>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>> A question - as a user I would assume that if I request A with a range,
>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>> this range in that repo, that they are indeed included in the final
>>>> result even if there are newer versions of A in other repositories
>>>> that are mapped with other rules. Is this true?
>>>>
>>> Nope. The planner consider all repositories as one source of input. It
>>> never considers what individual repository a specific requirement
>>> originated from. It will create a plan where all requirement ranges
>>> are fulfilled but the actual IU's included can stem from any
>>> repository.
>>>
>>> - thomas
>>>
>>
>>
>
Re: Aggregator with version range support [message #517370 is a reply to message #517338] Sat, 27 February 2010 09:59 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
If there are two requests:

A [fromA1, toA1]
A [fromA2, toA2]

where fromA1 > fromA2 and toA1 > toA2

then the planner will choose a version that matches the range [fromA1, toA2], i.e. both original request must be satisfied.
If no such a match is found, the aggregation fails.

In addition to this, it does not matter where the chosen match comes from, i.e. if A [fromA1, toA1] was defined "from repo X"
where the highest version of A is Vx, then the aggregator may choose version Vy from repo Y if Vy > Vx && Vy <= toA2.

That's why I don't like the "graphical picture" of the thing in the UI, where user might think that he gets only things from
the repository's resolution scope.

Filip


Henrik Lindberg wrote:
> Well, what I want to know is still not clear.
> I understand that:
> - a request to include A [fromA1, toA1] does not mean that some other
> request of A [fromA2, toA2 > toA1] is restricted.
> - a request to include A [some range] is not tied to a particular
> repository
>
> What I wonder what the resulting repository will contain if there are
> many versions of A that comply. Say that fromA1 < fromA2, and toA1 > toA2.
>
> If I have these requests in my map, I may have different desires:
> a) I want all versions i.e. fromA1 (the smallest requested) to toA1 (the
> largest requested).
> b) I want the latest version that complies with all requests.
>
> I would desire a) if I want to include all "old version" (in stated
> range), and desire b) if I just want a slimmed repository.
>
> This was my question - what does the aggregator mirror if I add the two
> requests to A?
>
> - henrik
>
> On 2/27/10 12:19 AM, Thomas Hallgren wrote:
>> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>>> That is not what i asked. I asked if it only picked the latest.
>>> - henrik
>>
>> I'm reading your question again and I come up with the same answer :-)
>>
>> But let me try and rephrase:
>>
>> The fact that there are newer versions of A in other repositories, and
>> still within the requested range, might well mean that these newer
>> versions are chosen. The planner will pick the best solution possible
>> for A in scope of the aggregation as a whole. No consideration
>> whatsoever is given to in which source repositories an IU is found.
>>
>> OK?
>>
>> - thomas
>>
>>> Thomas Hallgren<thomas@tada.se> wrote:
>>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>>> A question - as a user I would assume that if I request A with a
>>>>> range,
>>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>>> this range in that repo, that they are indeed included in the final
>>>>> result even if there are newer versions of A in other repositories
>>>>> that are mapped with other rules. Is this true?
>>>>>
>>>> Nope. The planner consider all repositories as one source of input. It
>>>> never considers what individual repository a specific requirement
>>>> originated from. It will create a plan where all requirement ranges
>>>> are fulfilled but the actual IU's included can stem from any
>>>> repository.
>>>>
>>>> - thomas
>>>>
>>>
>>>
>>
>
Re: Aggregator with version range support [message #517403 is a reply to message #517370] Sat, 27 February 2010 15:08 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Boy, this is a tough question to answer...

When aggregating - what do I get in my aggregated (mirrored) result:
Only one version of A, or several?
Can I get serveral?

- henrik

On 2/27/10 10:59 AM, Filip Hrbek wrote:
> If there are two requests:
>
> A [fromA1, toA1]
> A [fromA2, toA2]
>
> where fromA1 > fromA2 and toA1 > toA2
>
> then the planner will choose a version that matches the range [fromA1,
> toA2], i.e. both original request must be satisfied.
> If no such a match is found, the aggregation fails.
>
> In addition to this, it does not matter where the chosen match comes
> from, i.e. if A [fromA1, toA1] was defined "from repo X"
> where the highest version of A is Vx, then the aggregator may choose
> version Vy from repo Y if Vy > Vx && Vy <= toA2.
>
> That's why I don't like the "graphical picture" of the thing in the UI,
> where user might think that he gets only things from
> the repository's resolution scope.
>
> Filip
>
>
> Henrik Lindberg wrote:
>> Well, what I want to know is still not clear.
>> I understand that:
>> - a request to include A [fromA1, toA1] does not mean that some other
>> request of A [fromA2, toA2 > toA1] is restricted.
>> - a request to include A [some range] is not tied to a particular
>> repository
>>
>> What I wonder what the resulting repository will contain if there are
>> many versions of A that comply. Say that fromA1 < fromA2, and toA1 >
>> toA2.
>>
>> If I have these requests in my map, I may have different desires:
>> a) I want all versions i.e. fromA1 (the smallest requested) to toA1
>> (the largest requested).
>> b) I want the latest version that complies with all requests.
>>
>> I would desire a) if I want to include all "old version" (in stated
>> range), and desire b) if I just want a slimmed repository.
>>
>> This was my question - what does the aggregator mirror if I add the
>> two requests to A?
>>
>> - henrik
>>
>> On 2/27/10 12:19 AM, Thomas Hallgren wrote:
>>> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>>>> That is not what i asked. I asked if it only picked the latest.
>>>> - henrik
>>>
>>> I'm reading your question again and I come up with the same answer :-)
>>>
>>> But let me try and rephrase:
>>>
>>> The fact that there are newer versions of A in other repositories, and
>>> still within the requested range, might well mean that these newer
>>> versions are chosen. The planner will pick the best solution possible
>>> for A in scope of the aggregation as a whole. No consideration
>>> whatsoever is given to in which source repositories an IU is found.
>>>
>>> OK?
>>>
>>> - thomas
>>>
>>>> Thomas Hallgren<thomas@tada.se> wrote:
>>>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>>>> A question - as a user I would assume that if I request A with a
>>>>>> range,
>>>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>>>> this range in that repo, that they are indeed included in the final
>>>>>> result even if there are newer versions of A in other repositories
>>>>>> that are mapped with other rules. Is this true?
>>>>>>
>>>>> Nope. The planner consider all repositories as one source of input. It
>>>>> never considers what individual repository a specific requirement
>>>>> originated from. It will create a plan where all requirement ranges
>>>>> are fulfilled but the actual IU's included can stem from any
>>>>> repository.
>>>>>
>>>>> - thomas
>>>>>
>>>>
>>>>
>>>
>>
Re: Aggregator with version range support [message #517418 is a reply to message #517403] Sat, 27 February 2010 16:59 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/27/2010 04:08 PM, Henrik Lindberg wrote:
> Boy, this is a tough question to answer...
>
> When aggregating - what do I get in my aggregated (mirrored) result:
> Only one version of A, or several?
> Can I get serveral?
>
You can, but it would be fairly rare. If the planner finds no solution where only one version of A is included and A is
not a singleton (which it would be under most circumstances), then the planner might decide to bring in two versions.

- thomas
Re: Aggregator with version range support [message #517422 is a reply to message #517418] Sat, 27 February 2010 17:29 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Thanks.

I noticed there is an enhancement request to enable including multiple
versions. I assume that is valid when creating repositories to be used
in development (you want all to be included because you don't know what
people are building against).

Can you explain how that is acheived?

- henrik

On 2/27/10 5:59 PM, Thomas Hallgren wrote:
> On 02/27/2010 04:08 PM, Henrik Lindberg wrote:
>> Boy, this is a tough question to answer...
>>
>> When aggregating - what do I get in my aggregated (mirrored) result:
>> Only one version of A, or several?
>> Can I get serveral?
>>
> You can, but it would be fairly rare. If the planner finds no solution
> where only one version of A is included and A is not a singleton (which
> it would be under most circumstances), then the planner might decide to
> bring in two versions.
>
> - thomas
Re: Aggregator with version range support [message #517426 is a reply to message #517422] Sat, 27 February 2010 13:36 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/27/2010 06:29 PM, Henrik Lindberg wrote:
> Thanks.
>
> I noticed there is an enhancement request to enable including multiple
> versions. I assume that is valid when creating repositories to be used
> in development (you want all to be included because you don't know what
> people are building against).
>
> Can you explain how that is acheived?
>
We can accomplish this by using a target consisting of one shared artifact repository and a composite meta-data
repository. Each aggregation run will then add one new child repository to the composite. This child is 100% verified
(using the planner) but when all children are brought together using the composite, there will no longer be a guarantee
that everything is consistent.

This is very similar to how many repositories at eclipse.org are constituted today. Platform, Galileo, and Helios
included. The only difference is that they don't currently utilize a shared artifact repository.

In a typical scenario this will always work since the composite contains permutations of the same aggregation where each
permutation introduces new components or new versions of old components. The planner will then satisfy an install from
the last permutation when installing "the latest" or when upgrading.

The problem that can arise is when you install earlier versions and those versions have requirements that are too
loosely specified. When that happens, the client will be forced to install from a selected repository instead of from
the composite. Creating the validation utility capable of detecting such problems while spending a reasonable amount of
time and CPU is challenging but I don't think it is impossible.

- thomas
Re: Aggregator with version range support [message #596327 is a reply to message #517009] Fri, 26 February 2010 07:01 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
Some comments inline.

On 02/25/2010 06:21 PM, Filip Hrbek wrote:
> Hello,
>
> I'd like to discuss the algorithm how the aggregator should resolve IU
> requests. This post is rather long but I wanted to express in detail
> what the problem is all about. I will appreciate you spending some time
> on it and helping me find a solution.
>
> Background:
>
> Let's have following aggregation definition:
>
> Contribution A (contributed by user A)
> |
> + Mapped Repository R1
> |
> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>
> Contribution B (contributed by user B)
> |
> + Mapped Repository R2
> |
> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>
>
> Repository R1 contains:
> org.eclipse.myframework/0.9.0.v20070415
> org.eclipse.myframework/1.0.0.v20080105
> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>
> Repository R2 contains:
> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
> does not expect and does not want at all)
> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
> expects)
>
> =================================================
>
> Now, what is the expected result? I think that a user would think
> probably like this:
>
> The contribution "A" yields a version of "org.eclipse.myframework" that
> is the best from repository R1 and is not in conflict with any other
> contribution at the same time.
> Since "org.eclipse.myframework" is not mapped in any other contribution,
> the natural resolution of it is "org.eclipse.myframework/1.1.0.v20091220".
>
> The contribution "B" yields a version of
> "org.eclipse.something.different" that is the best from repository R1
> and is not in conflict with any other contribution at the same time.
> Since "org.eclipse.something.different" is not mapped in any other
> contribution, the natural resolution of it is
> "org.eclipse.something.different/2.5.1.v20100225".
>
> The aggregated repository will contain:
> org.eclipse.myframework/1.1.0.v20091220
> org.eclipse.something.different/2.5.1.v20100225
>
> OK?
>
No, I don't think that's OK. If the user is unhappy with versions of the framework that is above 1.1.0, then that should
be stated in contribution A. If the range is open-ended, then A has explicitly declared that 1.2.0 is an OK replacement.

> However, the new aggregator (with version range support) will create an
> aggregated repository containing:
> org.eclipse.myframework/1.2.0.beta_for_testing_only
> org.eclipse.something.different/2.5.1.v20100225
>
Which seems OK, given the circumstances.

Try another use-case:

Contribution A is exactly as in your example. Contribution B contains updates contains a limited subset of newer
versions of bundles also found in contribution A. This can easily happen if A is a release and the combination of A and
B is a service release.

> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
> use a request accross all repositories, regardless where the IU was mapped.
>
> In my view, this is bad.
>
In my view, this is good and indeed necessary.

> That's why I tried to change the algorithm:
>
> For each IU request
> 1) Resolve the request against its parent mapped repository (first level
> only, no dependencies) and get a set of matching resolutions
> 2) Create a modified IU request that defines the IU name and version
> enumeration (of matching resolutions) instead of the original version range
> 3) Provide the modified request to the global aggregated repository
> processing
>
> This way, only IUs that are really present in appropriate mapped
> repository may be passed to the final resolution.
>
Why is the IU found in the same repository more "appropriate" then an IU found in another? I can see that for some
specific use-case that might be the case, but it's not a general truth.

> So far so good, but the problem is that the only implementation of
> IRequirement is the RequiredCapability which does not support version
> constraints different from version ranges.
>
> As a proof of concept, I tried to hack this all so that the interim
> repository supports another implementation of IRequirement with more
> complex version constraints. It works as expected!
>
> Finally, the questions to answer:
>
> 1) Do you think that this approach is nature/logical, i.e. is it how it
> should work?

No. I think this is wrong. I think the right approach must be to consciously insert a rule that limits the dependency.
It should not be automatic.

- thomas
Re: Aggregator with version range support [message #596335 is a reply to message #517083] Fri, 26 February 2010 08:30 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Hi Thomas,

thanks for your comments.

I understand what you mean. It is also very easy to implement what you suggest, but I have a different view on the problem.

I understand the "contribution" as a unit which someone is responsible for. When I talk about responsibility, I mean that the one who contributes also guarantees that only approved versions from his approved source may be included (new aggregator also shows these versions interactively in the mapping).

You are right that sometimes it would be good to include "service packs" above "releases". IMHO it should not be automatic though. I would rather see a little change in the model to achieve this - to allow map repositories as virtual composites:

Contribution A (contributed by user A)
|
+ Mapped Repository - Virtual Composite of [R1,R2]
|
+ IU Request: id=org.eclipse.myframework; version>=1.0.0

This would be the conscious rule to add more available (approved) resolutions. The basic idea is "be strict by default, allow being less strict on user's conscious action".

I'd like to see more people contributing to this discussion to avoid an endless dialog of two hard-headed guys :-)

Filip

Thomas Hallgren wrote:
> Hi Filip,
> Some comments inline.
>
> On 02/25/2010 06:21 PM, Filip Hrbek wrote:
>> Hello,
>>
>> I'd like to discuss the algorithm how the aggregator should resolve IU
>> requests. This post is rather long but I wanted to express in detail
>> what the problem is all about. I will appreciate you spending some time
>> on it and helping me find a solution.
>>
>> Background:
>>
>> Let's have following aggregation definition:
>>
>> Contribution A (contributed by user A)
>> |
>> + Mapped Repository R1
>> |
>> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>>
>> Contribution B (contributed by user B)
>> |
>> + Mapped Repository R2
>> |
>> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>>
>>
>> Repository R1 contains:
>> org.eclipse.myframework/0.9.0.v20070415
>> org.eclipse.myframework/1.0.0.v20080105
>> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>>
>> Repository R2 contains:
>> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
>> does not expect and does not want at all)
>> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
>> expects)
>>
>> =================================================
>>
>> Now, what is the expected result? I think that a user would think
>> probably like this:
>>
>> The contribution "A" yields a version of "org.eclipse.myframework" that
>> is the best from repository R1 and is not in conflict with any other
>> contribution at the same time.
>> Since "org.eclipse.myframework" is not mapped in any other contribution,
>> the natural resolution of it is
>> "org.eclipse.myframework/1.1.0.v20091220".
>>
>> The contribution "B" yields a version of
>> "org.eclipse.something.different" that is the best from repository R1
>> and is not in conflict with any other contribution at the same time.
>> Since "org.eclipse.something.different" is not mapped in any other
>> contribution, the natural resolution of it is
>> "org.eclipse.something.different/2.5.1.v20100225".
>>
>> The aggregated repository will contain:
>> org.eclipse.myframework/1.1.0.v20091220
>> org.eclipse.something.different/2.5.1.v20100225
>>
>> OK?
>>
> No, I don't think that's OK. If the user is unhappy with versions of the
> framework that is above 1.1.0, then that should be stated in
> contribution A. If the range is open-ended, then A has explicitly
> declared that 1.2.0 is an OK replacement.
>
>> However, the new aggregator (with version range support) will create an
>> aggregated repository containing:
>> org.eclipse.myframework/1.2.0.beta_for_testing_only
>> org.eclipse.something.different/2.5.1.v20100225
>>
> Which seems OK, given the circumstances.
>
> Try another use-case:
>
> Contribution A is exactly as in your example. Contribution B contains
> updates contains a limited subset of newer versions of bundles also
> found in contribution A. This can easily happen if A is a release and
> the combination of A and B is a service release.
>
>> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
>> use a request accross all repositories, regardless where the IU was
>> mapped.
>>
>> In my view, this is bad.
>>
> In my view, this is good and indeed necessary.
>
>> That's why I tried to change the algorithm:
>>
>> For each IU request
>> 1) Resolve the request against its parent mapped repository (first level
>> only, no dependencies) and get a set of matching resolutions
>> 2) Create a modified IU request that defines the IU name and version
>> enumeration (of matching resolutions) instead of the original version
>> range
>> 3) Provide the modified request to the global aggregated repository
>> processing
>>
>> This way, only IUs that are really present in appropriate mapped
>> repository may be passed to the final resolution.
>>
> Why is the IU found in the same repository more "appropriate" then an IU
> found in another? I can see that for some specific use-case that might
> be the case, but it's not a general truth.
>
>> So far so good, but the problem is that the only implementation of
>> IRequirement is the RequiredCapability which does not support version
>> constraints different from version ranges.
>>
>> As a proof of concept, I tried to hack this all so that the interim
>> repository supports another implementation of IRequirement with more
>> complex version constraints. It works as expected!
>>
>> Finally, the questions to answer:
>>
>> 1) Do you think that this approach is nature/logical, i.e. is it how it
>> should work?
>
> No. I think this is wrong. I think the right approach must be to
> consciously insert a rule that limits the dependency. It should not be
> automatic.
>
> - thomas
Re: Aggregator with version range support [message #596342 is a reply to message #596335] Fri, 26 February 2010 09:08 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
What do you propose should happen if contribution A contains:

org.eclipse.myframework/1.1.0.alpha

and contribution B contains:

org.eclipse.myframework/1.1.0.final

?

What algorithm is used to determine the best one?

- thomas



On 02/26/2010 09:30 AM, Filip Hrbek wrote:
> Hi Thomas,
>
> thanks for your comments.
>
> I understand what you mean. It is also very easy to implement what you
> suggest, but I have a different view on the problem.
>
> I understand the "contribution" as a unit which someone is responsible
> for. When I talk about responsibility, I mean that the one who
> contributes also guarantees that only approved versions from his
> approved source may be included (new aggregator also shows these
> versions interactively in the mapping).
>
> You are right that sometimes it would be good to include "service packs"
> above "releases". IMHO it should not be automatic though. I would rather
> see a little change in the model to achieve this - to allow map
> repositories as virtual composites:
>
> Contribution A (contributed by user A)
> |
> + Mapped Repository - Virtual Composite of [R1,R2]
> |
> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>
> This would be the conscious rule to add more available (approved)
> resolutions. The basic idea is "be strict by default, allow being less
> strict on user's conscious action".
>
> I'd like to see more people contributing to this discussion to avoid an
> endless dialog of two hard-headed guys :-)
>
> Filip
>
> Thomas Hallgren wrote:
>> Hi Filip,
>> Some comments inline.
>>
>> On 02/25/2010 06:21 PM, Filip Hrbek wrote:
>>> Hello,
>>>
>>> I'd like to discuss the algorithm how the aggregator should resolve IU
>>> requests. This post is rather long but I wanted to express in detail
>>> what the problem is all about. I will appreciate you spending some time
>>> on it and helping me find a solution.
>>>
>>> Background:
>>>
>>> Let's have following aggregation definition:
>>>
>>> Contribution A (contributed by user A)
>>> |
>>> + Mapped Repository R1
>>> |
>>> + IU Request: id=org.eclipse.myframework; version>=1.0.0
>>>
>>> Contribution B (contributed by user B)
>>> |
>>> + Mapped Repository R2
>>> |
>>> + IU Request: id=org.eclipse.something.different; version>=1.0.0
>>>
>>>
>>> Repository R1 contains:
>>> org.eclipse.myframework/0.9.0.v20070415
>>> org.eclipse.myframework/1.0.0.v20080105
>>> org.eclipse.myframework/1.1.0.v20091220 (this is what user A expects)
>>>
>>> Repository R2 contains:
>>> org.eclipse.myframework/1.2.0.beta_for_testing_only (this is what user A
>>> does not expect and does not want at all)
>>> org.eclipse.something.different/2.5.1.v20100225 (this is what user B
>>> expects)
>>>
>>> =================================================
>>>
>>> Now, what is the expected result? I think that a user would think
>>> probably like this:
>>>
>>> The contribution "A" yields a version of "org.eclipse.myframework" that
>>> is the best from repository R1 and is not in conflict with any other
>>> contribution at the same time.
>>> Since "org.eclipse.myframework" is not mapped in any other contribution,
>>> the natural resolution of it is
>>> "org.eclipse.myframework/1.1.0.v20091220".
>>>
>>> The contribution "B" yields a version of
>>> "org.eclipse.something.different" that is the best from repository R1
>>> and is not in conflict with any other contribution at the same time.
>>> Since "org.eclipse.something.different" is not mapped in any other
>>> contribution, the natural resolution of it is
>>> "org.eclipse.something.different/2.5.1.v20100225".
>>>
>>> The aggregated repository will contain:
>>> org.eclipse.myframework/1.1.0.v20091220
>>> org.eclipse.something.different/2.5.1.v20100225
>>>
>>> OK?
>>>
>> No, I don't think that's OK. If the user is unhappy with versions of
>> the framework that is above 1.1.0, then that should be stated in
>> contribution A. If the range is open-ended, then A has explicitly
>> declared that 1.2.0 is an OK replacement.
>>
>>> However, the new aggregator (with version range support) will create an
>>> aggregated repository containing:
>>> org.eclipse.myframework/1.2.0.beta_for_testing_only
>>> org.eclipse.something.different/2.5.1.v20100225
>>>
>> Which seems OK, given the circumstances.
>>
>> Try another use-case:
>>
>> Contribution A is exactly as in your example. Contribution B contains
>> updates contains a limited subset of newer versions of bundles also
>> found in contribution A. This can easily happen if A is a release and
>> the combination of A and B is a service release.
>>
>>> The reason why "1.2.0.beta_for_testing_only" was chosen is simple: We
>>> use a request accross all repositories, regardless where the IU was
>>> mapped.
>>>
>>> In my view, this is bad.
>>>
>> In my view, this is good and indeed necessary.
>>
>>> That's why I tried to change the algorithm:
>>>
>>> For each IU request
>>> 1) Resolve the request against its parent mapped repository (first level
>>> only, no dependencies) and get a set of matching resolutions
>>> 2) Create a modified IU request that defines the IU name and version
>>> enumeration (of matching resolutions) instead of the original version
>>> range
>>> 3) Provide the modified request to the global aggregated repository
>>> processing
>>>
>>> This way, only IUs that are really present in appropriate mapped
>>> repository may be passed to the final resolution.
>>>
>> Why is the IU found in the same repository more "appropriate" then an
>> IU found in another? I can see that for some specific use-case that
>> might be the case, but it's not a general truth.
>>
>>> So far so good, but the problem is that the only implementation of
>>> IRequirement is the RequiredCapability which does not support version
>>> constraints different from version ranges.
>>>
>>> As a proof of concept, I tried to hack this all so that the interim
>>> repository supports another implementation of IRequirement with more
>>> complex version constraints. It works as expected!
>>>
>>> Finally, the questions to answer:
>>>
>>> 1) Do you think that this approach is nature/logical, i.e. is it how it
>>> should work?
>>
>> No. I think this is wrong. I think the right approach must be to
>> consciously insert a rule that limits the dependency. It should not be
>> automatic.
>>
>> - thomas
Re: Aggregator with version range support [message #596350 is a reply to message #596335] Fri, 26 February 2010 09:31 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Filip,
A note about contributor responsibilities.

On 02/26/2010 09:30 AM, Filip Hrbek wrote:
> I understand the "contribution" as a unit which someone is responsible
> for. When I talk about responsibility, I mean that the one who
> contributes also guarantees that only approved versions from his
> approved source may be included (new aggregator also shows these
> versions interactively in the mapping).
>
That is a good understanding of a "contribution" but the contributor has one other very important responsibility. He
must absolutely not use the repository itself as a mechanism to limit the scope of resolution. If he does, he is missing
the whole point of p2!

An author of a p2 repository must always be aware that his repository can, and will, be used together with other
repositories. This can happen in the Eclipse IDE Install wizard as well as in our Aggregator. In addition to that, any
composite repository may blend his repositories with others and use no other mechanism then p2 to resolve.

What you propose is not only that we release the repository publisher from this responsibility, you also suggest that
this should be the default behavior in the Aggregator. This is in strong contrast with how p2 is supposed to work.
That's why I don't approve.

I do think we should have a mechanism that help us to "fix" repositories where this consideration has not been honored
and that causes problems.

- thomas
Re: Aggregator with version range support [message #596356 is a reply to message #517100] Fri, 26 February 2010 09:33 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> Hi Filip,
> What do you propose should happen if contribution A contains:
>
> org.eclipse.myframework/1.1.0.alpha
>
> and contribution B contains:
>
> org.eclipse.myframework/1.1.0.final
>
> ?
>
> What algorithm is used to determine the best one?

If there is no intersection of resolutions from contributions A and B, then this is a conflict.
Contributors must contact each other to modify the rules. It's exactly what I said about the policy "be strict by default".

To answer your question - if contributor A decides to accept the "final" version, he should add the repository containing the "final" version to his contribution.

Filip
Re: Aggregator with version range support [message #596362 is a reply to message #517105] Fri, 26 February 2010 09:37 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 10:33 AM, Filip Hrbek wrote:
>
> Thomas Hallgren wrote:
>> Hi Filip,
>> What do you propose should happen if contribution A contains:
>>
>> org.eclipse.myframework/1.1.0.alpha
>>
>> and contribution B contains:
>>
>> org.eclipse.myframework/1.1.0.final
>>
>> ?
>>
>> What algorithm is used to determine the best one?
>
> If there is no intersection of resolutions from contributions A and B,
> then this is a conflict.

I assume the same scenario as you outlined in your first post. The only difference is that in B, it's not a beta, it's a
final. And in A, it's just alphas.

- thomas
Re: Aggregator with version range support [message #596368 is a reply to message #517106] Fri, 26 February 2010 09:59 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 10:33 AM, Filip Hrbek wrote:
>>
>> Thomas Hallgren wrote:
>>> Hi Filip,
>>> What do you propose should happen if contribution A contains:
>>>
>>> org.eclipse.myframework/1.1.0.alpha
>>>
>>> and contribution B contains:
>>>
>>> org.eclipse.myframework/1.1.0.final
>>>
>>> ?
>>>
>>> What algorithm is used to determine the best one?
>>
>> If there is no intersection of resolutions from contributions A and B,
>> then this is a conflict.
>
> I assume the same scenario as you outlined in your first post. The only
> difference is that in B, it's not a beta, it's a final. And in A, it's
> just alphas.
>
> - thomas

Then yes, the "alpha" wins. If contributor "A" discovers that a better version is available, he should add the repo to his contribution.

If you really think that what I say breaks the p2 world, well, then our model is bad! We shouldn't map things from a mapped repository.

To make what you say intuitive, then a contribution should look like this:

Contribution A
|
+ Repository hints
|
+ Repo A
+ Repo B
+ Repo C
|
+ Mappings
|
+ IU a.b.c, version >=1.0.0
+ Everything from Repo B

The "Repository hints" would serve as possible sources that the contributor "recommends" to use for the final resolution. We could also remove current "Validation repositories" since it would not make sense any more (in my example, "Repo C" is actually a validation repository since nothing is mapped in contribution A).

With such a model, I could agree with you. Not with current model though since it is very confusing - one thinks he maps something within the resolution scope of the mapped repository but he may get unexpectedly something else.

Filip
Re: Aggregator with version range support [message #596375 is a reply to message #517111] Fri, 26 February 2010 10:21 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 10:59 AM, Filip Hrbek wrote:

> Then yes, the "alpha" wins. If contributor "A" discovers that a better
> version is available, he should add the repo to his contribution.
>
I'm trying to map this to our current use cases. In Helios, each contribution is just a fragment of the final result
that has lot of dependencies not contained in that fragment. The contributors do this, well aware that when the
aggregation happens, everything will be there. The contribution inter-dependencies are very intentional.

Miles at MS first creates a big aggregation containing a lot of things from Galileo and other places. He then uses this
as a contribution when he creates the final aggregation together with other contributions that contains all in-house
developed features. He is well aware that the in-house contributions all have inter-dependencies and that they depend
heavily on the aggregation he created in the first step.

> If you really think that what I say breaks the p2 world, well, then our
> model is bad! We shouldn't map things from a mapped repository.
>
Not sure what you mean by that. How else can we map things?

> To make what you say intuitive, then a contribution should look like this:
>
> Contribution A
> |
> + Repository hints
> |
> + Repo A
> + Repo B
> + Repo C
> |
> + Mappings
> |
> + IU a.b.c, version >=1.0.0
> + Everything from Repo B
>
> The "Repository hints" would serve as possible sources that the
> contributor "recommends" to use for the final resolution. We could also
> remove current "Validation repositories" since it would not make sense
> any more (in my example, "Repo C" is actually a validation repository
> since nothing is mapped in contribution A).
>
That assumes that all contributions are self sufficient. And sure, there are perhaps use-cases where this could be good.
I haven't seen them yet though. On the contrary, the use-cases at hand always have contributions that depend on other
contributions.

> With such a model, I could agree with you. Not with current model though
> since it is very confusing - one thinks he maps something within the
> resolution scope of the mapped repository but he may get unexpectedly
> something else.
>
Really? I have never made that assumption. We create a verified aggregation and of course everything is verified in
scope of the aggregation. I think you put too much weight on the contribution concept.

- thomas
Re: Aggregator with version range support [message #596382 is a reply to message #596375] Fri, 26 February 2010 10:35 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
....
>
>> To make what you say intuitive, then a contribution should look like
>> this:
>>
>> Contribution A
>> |
>> + Repository hints
>> |
>> + Repo A
>> + Repo B
>> + Repo C
>> |
>> + Mappings
>> |
>> + IU a.b.c, version >=1.0.0
>> + Everything from Repo B
>>
>> The "Repository hints" would serve as possible sources that the
>> contributor "recommends" to use for the final resolution. We could also
>> remove current "Validation repositories" since it would not make sense
>> any more (in my example, "Repo C" is actually a validation repository
>> since nothing is mapped in contribution A).
>>
> That assumes that all contributions are self sufficient. And sure, there
> are perhaps use-cases where this could be good. I haven't seen them yet
> though. On the contrary, the use-cases at hand always have contributions
> that depend on other contributions.
>

No, you are missing the point. The contributions may transitively depend on other contributions of course.
I just want to express what you say with the model - just to avoid confusion. I am still convinced that
if you map IUs by choosing them from a mapped repository, you naturally expect to get them within the scope of that
repository. The result must either follow this rule, or the model must be changed somehow (I provided a draft of a suggestion above).

What my contribution depends on does not have to be included in my "repository hints" at all.


Filip
Re: Aggregator with version range support [message #596386 is a reply to message #596375] Fri, 26 February 2010 10:37 Go to previous messageGo to next message
Karel Brezina is currently offline Karel BrezinaFriend
Messages: 68
Registered: July 2009
Member
>> With such a model, I could agree with you. Not with current model though
>> since it is very confusing - one thinks he maps something within the
>> resolution scope of the mapped repository but he may get unexpectedly
>> something else.
>>
> Really? I have never made that assumption. We create a verified
> aggregation and of course everything is verified in scope of the
> aggregation. I think you put too much weight on the contribution concept.
>
> - thomas

Well, I agree with Filip. Each model should reflect the real world. Why
should we model a relation between a concrete repository and a concrete
IU, when in fact the relation does not exist? Wouldn't it be better to
model it as a heap of requests and a heap of resources? AFAIK,
buckminster works exactly the same way (CQuery vs. RMap)

Karel
Re: Aggregator with version range support [message #597942 is a reply to message #517120] Fri, 26 February 2010 10:58 Go to previous messageGo to next message
Achim Demelt is currently offline Achim DemeltFriend
Messages: 160
Registered: July 2009
Senior Member
Hi guys,

Sorry for jumping into this discussion so late, but you truly are hard-
headed guys and are exchanging arguments at such a fast pace that I've
hardly had time to catch up ;-)

Now here's my point of view, but before you read on, please note two things:

* I'm a hard-headed guy myself ;-)
* I have no practical experience with the aggregator (yet), and my
theoretical knowledge of it is, of course, not as deep as yours.

That said, here's my use-case, and I think it supports Filip's position:

We're pulling in everything from the Galileo release site. This includes
Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
repo hosted at itemis. This repo happens to include a 0.8.x version of
Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
Especially not if this leads to the 0.7.x version being excluded in the
result, because then I end up with a completely inconsistent repository.
(Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).

Now if I understand you guys correctly, Thomas would argue that I would have
to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
There is, however, no need to do that, because I assume the Galileo repo to
never, ever contain that version. It's only as a side-effect from pulling
org.antlr that this ends up in my result. But I could never foresee this
effect when maintaining the contribution from Galileo.

Cheers,
Achim


Filip Hrbek wrote:

> Thomas Hallgren wrote:
> ...
>>
>>> To make what you say intuitive, then a contribution should look like
>>> this:
>>>
>>> Contribution A
>>> |
>>> + Repository hints
>>> |
>>> + Repo A
>>> + Repo B
>>> + Repo C
>>> |
>>> + Mappings
>>> |
>>> + IU a.b.c, version >=1.0.0
>>> + Everything from Repo B
>>>
>>> The "Repository hints" would serve as possible sources that the
>>> contributor "recommends" to use for the final resolution. We could also
>>> remove current "Validation repositories" since it would not make sense
>>> any more (in my example, "Repo C" is actually a validation repository
>>> since nothing is mapped in contribution A).
>>>
>> That assumes that all contributions are self sufficient. And sure, there
>> are perhaps use-cases where this could be good. I haven't seen them yet
>> though. On the contrary, the use-cases at hand always have contributions
>> that depend on other contributions.
>>
>
> No, you are missing the point. The contributions may transitively depend
> on other contributions of course. I just want to express what you say with
> the model - just to avoid confusion. I am still convinced that if you map
> IUs by choosing them from a mapped repository, you naturally expect to get
> them within the scope of that repository. The result must either follow
> this rule, or the model must be changed somehow (I provided a draft of a
> suggestion above).
>
> What my contribution depends on does not have to be included in my
> "repository hints" at all.
>
>
> Filip
Re: Aggregator with version range support [message #597947 is a reply to message #517121] Fri, 26 February 2010 11:03 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:37 AM, Karel Brezina wrote:
> Well, I agree with Filip. Each model should reflect the real world. Why
> should we model a relation between a concrete repository and a concrete
> IU, when in fact the relation does not exist?

Sure it exists. It's just not what ultimately is selected. That's a version range problem and it should not be resolved
by making the contribution as such the scope of resolution.


> Wouldn't it be better to
> model it as a heap of requests and a heap of resources? AFAIK,
> buckminster works exactly the same way (CQuery vs. RMap)
>
We already do model it as you suggest. You are trying to move that model down to each contribution which makes the
scoping all wrong. The contribution can never be the scope of anything. The 'heap of resources' is the sum of all
contributions. That has to be the default. I'm fine with adding rules to force deviations from that default in case such
rules are missing from a contribution and things break. I.e. a rule that you add to a contribution that says, "iu x.y.z.
should have an upper bound 1.2.3", but we cannot make the contribution itself a scope of resolution.

- thomas
Re: Aggregator with version range support [message #597953 is a reply to message #597942] Fri, 26 February 2010 11:16 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Hi Achim,

you don't have to apologize for jumping into our discussion, you are welcome of course!

I am hard-headed but I am able listen to others and finally change my mind, but I prefer listening to more than one guy (Thomas - it does not mean that I don't listen to you of course). It always yields better results.

I think that your use-case exactly reflects my concern. Thomas probably knows other use-cases.

Our goal is to provide mechanisms for all of them of course. I am not against adding more options to the aggregator that would allow a user to do what he really wants.

All I want is

* to have a model that reflects reality
* to have a model that reflects user's natural thinking
* to have an engine that works in conformance with the model (not doing unexpected things)
* to have reasonable default settings (i.e. for use-cases that are fine to most users)
* to allow customization of the behavior by setting specific options/flags/rules...

I am sure we will find a solution. Again, the more opinions I collect now the better the solution will be.

Filip


Achim Demelt wrote:

> Hi guys,
>
> Sorry for jumping into this discussion so late, but you truly are hard-
> headed guys and are exchanging arguments at such a fast pace that I've
> hardly had time to catch up ;-)
>
> Now here's my point of view, but before you read on, please note two things:
>
> * I'm a hard-headed guy myself ;-)
> * I have no practical experience with the aggregator (yet), and my
> theoretical knowledge of it is, of course, not as deep as yours.
>
> That said, here's my use-case, and I think it supports Filip's position:
>
> We're pulling in everything from the Galileo release site. This includes
> Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
> repo hosted at itemis. This repo happens to include a 0.8.x version of
> Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
> Especially not if this leads to the 0.7.x version being excluded in the
> result, because then I end up with a completely inconsistent repository.
> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>
> Now if I understand you guys correctly, Thomas would argue that I would have
> to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
> There is, however, no need to do that, because I assume the Galileo repo to
> never, ever contain that version. It's only as a side-effect from pulling
> org.antlr that this ends up in my result. But I could never foresee this
> effect when maintaining the contribution from Galileo.
>
> Cheers,
> Achim
>
Re: Aggregator with version range support [message #597954 is a reply to message #517120] Fri, 26 February 2010 11:18 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:35 AM, Filip Hrbek wrote:
> I am still convinced that if you map IUs by choosing them from a mapped
> repository, you naturally expect to get them within the scope of that
> repository. The result must either follow this rule, or the model must
> be changed somehow (I provided a draft of a suggestion above).
>
I think it's the false assumption that a mapping creates a hard link to a specific repository that must change. That's
never the case in p2 and if possible, we should make that more clear to the user.

This is actually the same discussion that we had a some time ago [1]. The user has one view of things but ultimately,
the planner makes changes to that view. The solution is not to prevent that from happening. The problem is, how do we
visualize that to the user?

[1] http://www.eclipse.org/forums/index.php?t=msg&&th=15 9023&goto=502689#msg_502689

- thomas
Re: Aggregator with version range support [message #597959 is a reply to message #597942] Fri, 26 February 2010 11:29 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 11:58 AM, Achim Demelt wrote:
> Hi guys,
>
> Sorry for jumping into this discussion so late, but you truly are hard-
> headed guys and are exchanging arguments at such a fast pace that I've
> hardly had time to catch up ;-)
>
> Now here's my point of view, but before you read on, please note two things:
>
> * I'm a hard-headed guy myself ;-)
> * I have no practical experience with the aggregator (yet), and my
> theoretical knowledge of it is, of course, not as deep as yours.
>
> That said, here's my use-case, and I think it supports Filip's position:
>
> We're pulling in everything from the Galileo release site. This includes
> Xtext 0.7.x. We also require one additional artifact (org.antlr) from the p2
> repo hosted at itemis. This repo happens to include a 0.8.x version of
> Xtext. I absolutely do *not* want that 0.8.x version to end up in my result.
> Especially not if this leads to the 0.7.x version being excluded in the
> result, because then I end up with a completely inconsistent repository.
> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>
> Now if I understand you guys correctly, Thomas would argue that I would have
> to explicitly exclude the 0.8.x version when fetching from the Galileo repo.
> There is, however, no need to do that, because I assume the Galileo repo to
> never, ever contain that version. It's only as a side-effect from pulling
> org.antlr that this ends up in my result. But I could never foresee this
> effect when maintaining the contribution from Galileo.
>
Good point. You don't own the Galileo contribution. So there are three solutions to this problem:

1. The most likely one is that there is no problem. If EMF 2.6 isn't included, then the planner will give up on XText
0.8.0 and try something else. If no solution at all can be found given all input, it will give up. It will _never_
create something that is inconsistent.

2. Your aggregation fails since no resolution is possible so you (the aggregator maintainer) sees that a contribution
has a problem and report this to the contributor. The contributor must then address this problem. Perhaps by adding an
upper limit to a bundle requirement. This is what currently happens on a day to day basis when problems occur in Helios.

3. Your aggregation fails and you add a rule that is above all contributions that says "In the final result, I don't
want Xtext of higher version then 0.7.x". This rule is propagated as a constraint to the planner. This possibility does
not yet exist, but I wouldn't be adverse to adding it.

- thomas
Re: Aggregator with version range support [message #597964 is a reply to message #517139] Fri, 26 February 2010 11:39 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
+1
finally a description of the problem/solution that I understood the
implications of.

The next issue is how to make user understand what went wrong and how to
fix it.

- henrik
> Good point. You don't own the Galileo contribution. So there are three
> solutions to this problem:
>
> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
> included, then the planner will give up on XText 0.8.0 and try
> something else. If no solution at all can be found given all input, it
> will give up. It will _never_ create something that is inconsistent.
>
> 2. Your aggregation fails since no resolution is possible so you (the
> aggregator maintainer) sees that a contribution has a problem and
> report this to the contributor. The contributor must then address this
> problem. Perhaps by adding an upper limit to a bundle requirement.
> This is what currently happens on a day to day basis when problems
> occur in Helios.
>
> 3. Your aggregation fails and you add a rule that is above all
> contributions that says "In the final result, I don't want Xtext of
> higher version then 0.7.x". This rule is propagated as a constraint to
> the planner. This possibility does not yet exist, but I wouldn't be
> adverse to adding it.
>
> - thomas
>


--
- henrik
Re: Aggregator with version range support [message #597967 is a reply to message #517129] Fri, 26 February 2010 11:45 Go to previous messageGo to next message
Karel Brezina is currently offline Karel BrezinaFriend
Messages: 68
Registered: July 2009
Member
Thomas Hallgren napsal(a):
> On 02/26/2010 11:37 AM, Karel Brezina wrote:
>> Well, I agree with Filip. Each model should reflect the real world. Why
>> should we model a relation between a concrete repository and a concrete
>> IU, when in fact the relation does not exist?
>
> Sure it exists. It's just not what ultimately is selected. That's a
> version range problem and it should not be resolved by making the
> contribution as such the scope of resolution.

Maybe I feel the parent child relationship a bit stronger. I have two
sons, I don't want to ultimately realize that they don't belong to me. :-)

In my thread I haven't mentioned contributions at all. I just wanted to
generalized it a little bit. I still feel that when the engine considers
it as two separate heaps (requests and resources) then the UI should
reflect it, otherwise the result of the aggreation might be a big surprise.

Karel
Re: Aggregator with version range support [message #597972 is a reply to message #517139] Fri, 26 February 2010 11:47 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Yes, Thomas is right here. The aggregator always yields consistent result, or it fails.
If EMF 2.6 is not available in any of mapped repositories, it will choose xtext 0.7.x.

But if EMF 2.6 is found somewhere else, you may get 0.8.x which you don't want for some reason.
In that case, you must add a version constraint to your contribution.

I think that limiting the scope of resolution for selected mapping (which does not have to be the default) would be better than adding an explicit constraint.

It should be up to each user how he wants the aggregator to behave.

We must talk about changing the model which still does not reflect the reality - that's the biggest issue here.

Filip



Thomas Hallgren wrote:
....
>> Especially not if this leads to the 0.7.x version being excluded in the
>> result, because then I end up with a completely inconsistent repository.
>> (Since Xtext 0.8 requires EMF 2.6, which is not contained in Galileo).
>>
>> Now if I understand you guys correctly, Thomas would argue that I
>> would have
>> to explicitly exclude the 0.8.x version when fetching from the Galileo
>> repo.
>> There is, however, no need to do that, because I assume the Galileo
>> repo to
>> never, ever contain that version. It's only as a side-effect from pulling
>> org.antlr that this ends up in my result. But I could never foresee this
>> effect when maintaining the contribution from Galileo.
>>
> Good point. You don't own the Galileo contribution. So there are three
> solutions to this problem:
>
> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
> included, then the planner will give up on XText 0.8.0 and try something
> else. If no solution at all can be found given all input, it will give
> up. It will _never_ create something that is inconsistent.
>
....
Re: Aggregator with version range support [message #597976 is a reply to message #517145] Fri, 26 February 2010 12:03 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 12:45 PM, Karel Brezina wrote:
> Maybe I feel the parent child relationship a bit stronger. I have two
> sons, I don't want to ultimately realize that they don't belong to me. :-)
>
I have three, and I think the saying is - you only have your kids as a loan. ;-)

- thomas
Re: Aggregator with version range support [message #597980 is a reply to message #517142] Fri, 26 February 2010 12:10 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Henrik, what do you think of the model?

That's what makes me worried the most. It does not reflect what the engine finally does.

I think we should concentrate on this issue now.

If you didn't want to go through all the posts, here's the summary:

- user specifies a mapped repository
- user selects a mapped UI *from* this repository (in the UI, it is a *child node* of that mapped repository)
- user specifies a version range (which may be open)
- user can check interactively which versions from that repository match his request
- user runs the aggregation

I think that user expects that the aggregation will contain one of the versions he had seen in the interactive checklist. What a surprise if there's something else?

I have suggested a change to that model so that we would avoid this confusion (see former posts).

In addition to this, I think that optional limiting of the resolution scope for explicitly added IUs would be helpful too, as well as introducing more complex version constraints than version ranges (which is another big issue).

Filip


Henrik Lindberg wrote:
> +1
> finally a description of the problem/solution that I understood the
> implications of.
>
> The next issue is how to make user understand what went wrong and how to
> fix it.
>
> - henrik
>> Good point. You don't own the Galileo contribution. So there are three
>> solutions to this problem:
>>
>> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
>> included, then the planner will give up on XText 0.8.0 and try
>> something else. If no solution at all can be found given all input, it
>> will give up. It will _never_ create something that is inconsistent.
>>
>> 2. Your aggregation fails since no resolution is possible so you (the
>> aggregator maintainer) sees that a contribution has a problem and
>> report this to the contributor. The contributor must then address this
>> problem. Perhaps by adding an upper limit to a bundle requirement.
>> This is what currently happens on a day to day basis when problems
>> occur in Helios.
>>
>> 3. Your aggregation fails and you add a rule that is above all
>> contributions that says "In the final result, I don't want Xtext of
>> higher version then 0.7.x". This rule is propagated as a constraint to
>> the planner. This possibility does not yet exist, but I wouldn't be
>> adverse to adding it.
>>
>> - thomas
>>
>
>
Re: Aggregator with version range support [message #597984 is a reply to message #597980] Fri, 26 February 2010 12:30 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
I did read all of the posts on this subject.

It is just that I have not used the aggregator enough to have a good
feel for what would be intuitive.

When picking something from a repository, what does it mean?
a) pick things matching this requirement, and then use all available
repositories to satisfy all dependencies.
b) pick things matching this requirement from this repository, and then
use all available respositories *but* this repository to satisfy the
dependencies.

b) is good for "OMG there is all sorts of crap in there in addition to
the few things I actually do need".

When using b) it is easy to to create a non valid result, because the
thing picked from the problem repo needs more stuff not in other
repositories. The user would then need to add those explicitly until a
valid result is obtained (or gives up because of conflicts).

I can see option b) as a flag on a repo - "only pick the explicitly
requested IUs".

A question - as a user I would assume that if I request A with a range,
(from one repo, as I perceive it as a user), and there exists A's in
this range in that repo, that they are indeed included in the final
result even if there are newer versions of A in other repositories that
are mapped with other rules. Is this true?

Regards
- henrik

On 2/26/10 1:10 PM, Filip Hrbek wrote:
> Henrik, what do you think of the model?
>
> That's what makes me worried the most. It does not reflect what the
> engine finally does.
>
> I think we should concentrate on this issue now.
>
> If you didn't want to go through all the posts, here's the summary:
>
> - user specifies a mapped repository
> - user selects a mapped UI *from* this repository (in the UI, it is a
> *child node* of that mapped repository)
> - user specifies a version range (which may be open)
> - user can check interactively which versions from that repository match
> his request
> - user runs the aggregation
>
> I think that user expects that the aggregation will contain one of the
> versions he had seen in the interactive checklist. What a surprise if
> there's something else?
>
> I have suggested a change to that model so that we would avoid this
> confusion (see former posts).
>
> In addition to this, I think that optional limiting of the resolution
> scope for explicitly added IUs would be helpful too, as well as
> introducing more complex version constraints than version ranges (which
> is another big issue).
>
> Filip
>
>
> Henrik Lindberg wrote:
>> +1
>> finally a description of the problem/solution that I understood the
>> implications of.
>>
>> The next issue is how to make user understand what went wrong and how to
>> fix it.
>> - henrik
>>> Good point. You don't own the Galileo contribution. So there are three
>>> solutions to this problem:
>>>
>>> 1. The most likely one is that there is no problem. If EMF 2.6 isn't
>>> included, then the planner will give up on XText 0.8.0 and try
>>> something else. If no solution at all can be found given all input, it
>>> will give up. It will _never_ create something that is inconsistent.
>>>
>>> 2. Your aggregation fails since no resolution is possible so you (the
>>> aggregator maintainer) sees that a contribution has a problem and
>>> report this to the contributor. The contributor must then address this
>>> problem. Perhaps by adding an upper limit to a bundle requirement.
>>> This is what currently happens on a day to day basis when problems
>>> occur in Helios.
>>>
>>> 3. Your aggregation fails and you add a rule that is above all
>>> contributions that says "In the final result, I don't want Xtext of
>>> higher version then 0.7.x". This rule is propagated as a constraint to
>>> the planner. This possibility does not yet exist, but I wouldn't be
>>> adverse to adding it.
>>>
>>> - thomas
>>>
>>
>>
Re: Aggregator with version range support [message #597988 is a reply to message #597984] Fri, 26 February 2010 12:46 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
> A question - as a user I would assume that if I request A with a range,
> (from one repo, as I perceive it as a user), and there exists A's in
> this range in that repo, that they are indeed included in the final
> result even if there are newer versions of A in other repositories that
> are mapped with other rules. Is this true?
>
Nope. The planner consider all repositories as one source of input. It never considers what individual repository a
specific requirement originated from. It will create a plan where all requirement ranges are fulfilled but the actual
IU's included can stem from any repository.

- thomas
Re: Aggregator with version range support [message #597997 is a reply to message #517164] Fri, 26 February 2010 13:09 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>> A question - as a user I would assume that if I request A with a range,
>> (from one repo, as I perceive it as a user), and there exists A's in
>> this range in that repo, that they are indeed included in the final
>> result even if there are newer versions of A in other repositories that
>> are mapped with other rules. Is this true?
>>
> Nope. The planner consider all repositories as one source of input. It
> never considers what individual repository a specific requirement
> originated from. It will create a plan where all requirement ranges are
> fulfilled but the actual IU's included can stem from any repository.
>


....and this it the issue!

What Henrik stated is IMHO intuitive from the user's perspective (unless the user is familiar with all our discussions :-) ).

I want to change the model at least so that a user wouldn't think that it would work like this. Ideally, I would add more options to change the behavior as needed.

Question (to move forward):

Is the model I suggested better or worse than current? If it is bad, why?

Filip
Re: Aggregator with version range support [message #598003 is a reply to message #517168] Fri, 26 February 2010 13:31 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:09 PM, Filip Hrbek wrote:
> Thomas Hallgren wrote:
>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>> A question - as a user I would assume that if I request A with a range,
>>> (from one repo, as I perceive it as a user), and there exists A's in
>>> this range in that repo, that they are indeed included in the final
>>> result even if there are newer versions of A in other repositories that
>>> are mapped with other rules. Is this true?
>>>
>> Nope. The planner consider all repositories as one source of input. It
>> never considers what individual repository a specific requirement
>> originated from. It will create a plan where all requirement ranges
>> are fulfilled but the actual IU's included can stem from any repository.
>>
>
>
> ...and this it the issue!
>
> What Henrik stated is IMHO intuitive from the user's perspective (unless
> the user is familiar with all our discussions :-) ).
>
> I want to change the model at least so that a user wouldn't think that
> it would work like this. Ideally, I would add more options to change the
> behavior as needed.
>
> Question (to move forward):
>
> Is the model I suggested better or worse than current? If it is bad, why?
>
I think its worse because it doesn't really solve the problem. The contribution can not be the scope of resolution so
adding more repositories at that level doesn't help. The final solution may have chosen IUs from other repositories
regardless.

The validation repository needs to stay because it validates the final aggregation, not each contribution.

I think the current model is fine given the requirements. We really don't want to add more things then necessary to each
contribution. The biggest problem I see, is that we have no way of visualizing that the planner might choose an IU that
is different from what the user has in his view. The user needs to be made aware of that this can happen during
authoring (the tool should make him aware that his selection isn't necessarily resolved from the current repository) and
also when the planner induced change occur (as warning or info markers once the planning has completed).

- thomas
Re: Aggregator with version range support [message #598014 is a reply to message #598003] Fri, 26 February 2010 13:55 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
....
>>
>> Question (to move forward):
>>
>> Is the model I suggested better or worse than current? If it is bad, why?
>>
> I think its worse because it doesn't really solve the problem. The
> contribution can not be the scope of resolution so adding more
> repositories at that level doesn't help. The final solution may have
> chosen IUs from other repositories regardless.

1) It does not have to be the scope by default but it would be interesting to have such an option (perhaps in future).
The scope (if requested) would only affect the first level, i.e. the components the user is explicitly mapping.

2) If it is *not* the resolution scope (which we talk about as the default right now), then we can't
have the selected IUs *under* the mapped repository because it is confusing. That is why I moved the
repository hints away from the mapping requests - just to make it clear.

The reason why I named the node *hints* is to make it clear that it's nothing more than a hint, i.e. which repositories
to store to the global heap of available repositories.

>
> The validation repository needs to stay because it validates the final
> aggregation, not each contribution.

If the validation repository belongs to the aggregation root, why don't all mapped repositories belong there?
If they don't mean the resolution scope for mapped IUs, they should also be placed in the aggregator root
since they affect the final aggregation, not the individual mappings.

However, I think that even the validation repositories should be moved under the contribution anyway.
If someone knows that his contribution needs another repository for validation, he can freely contribute
a repository hint for validation.


>
> I think the current model is fine given the requirements. We really
> don't want to add more things then necessary to each contribution. The
> biggest problem I see, is that we have no way of visualizing that the
> planner might choose an IU that is different from what the user has in
> his view. The user needs to be made aware of that this can happen during
> authoring (the tool should make him aware that his selection isn't
> necessarily resolved from the current repository) and also when the
> planner induced change occur (as warning or info markers once the
> planning has completed).

If we change the model so that user understands that he does not choose from the mapped repository, then
I can see a simple solution. We already have an interactive list of available versions in the UI. All we need to do
is to show *all* versions from *all* repositories (not only those from contribution's hints) and also show
which repository the resolved IU comes from. Then the user is not confused - he can see all potentially valid matches of
his selection before the (long running) verification is run.

Filip
Re: Aggregator with version range support [message #598017 is a reply to message #517177] Fri, 26 February 2010 14:10 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:55 PM, Filip Hrbek wrote:
> If the validation repository belongs to the aggregation root, why don't
> all mapped repositories belong there?

A contributor typically contributes repositories. He optionally filters what should be contributed from those
repositories by adding mappings. No mappings == everything. So of course all contributed repositories are listed in
their respective contributions.

The aggregator manager decides what contributions to aggregate and how to validate that aggregation. The validation has
nothing to do with each contribution. The contributions are neither self sufficient and not validated. Validation is on
the aggregation as a whole and thus, cannot be in a contribution.

HTH,
- thomas
Re: Aggregator with version range support [message #598024 is a reply to message #517177] Fri, 26 February 2010 14:14 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 02:55 PM, Filip Hrbek wrote:

> If we change the model so that user understands that he does not choose
> from the mapped repository, then
> I can see a simple solution.

Consider the time when a contributor edits his contribution. He contributes one or several repositories. The optionally
mapped things that he might add, should be considered filters. By adding them he says, don't automatically map other
things from my contributed repositories.

At the time of editing the contribution, the contributor does not always have access to all other repositories. He
should not really be aware of how the final aggregation is constituted.

- thomas
Re: Aggregator with version range support [message #598030 is a reply to message #517180] Fri, 26 February 2010 14:26 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
Thomas Hallgren wrote:
> On 02/26/2010 02:55 PM, Filip Hrbek wrote:
>> If the validation repository belongs to the aggregation root, why don't
>> all mapped repositories belong there?
>
> A contributor typically contributes repositories. He optionally filters
> what should be contributed from those repositories by adding mappings.
> No mappings == everything. So of course all contributed repositories are
> listed in their respective contributions.
>

Of course I agree with this, although "no mappings = everything" is not
intuitive enough for me. Btw., it does not mean "everything from this repo"
since you may actually finally map something else (newer versions from other
repositories).

I came to the conclusion that all repos should
be under aggregator root using your argument - that it does not create the
resolution scope.


> The aggregator manager decides what contributions to aggregate and how
> to validate that aggregation. The validation has nothing to do with each
> contribution. The contributions are neither self sufficient and not
> validated. Validation is on the aggregation as a whole and thus, cannot
> be in a contribution.

Really? Why?

I can imagine a context menu action that validates a contribution or a set of
selected contributions. This would be a good base for the aggregator manager
if the indivitual contributions were pre-validated by contribution owners.
I think that the contributor owner can imagine what repositores he needs to validate
his contribution, and can add them as repository hints. Duplicate entries don't matter,
they finally share the same resource. And, it does not prevent us from keeping
global validation repositories as an option.


Taking all your arguments into account, I still think that having the mapped IUs
under the mapper repository *is confusing* and needs to be refactored.

Filip
Re: Aggregator with version range support [message #598034 is a reply to message #517184] Fri, 26 February 2010 15:07 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Filip,
I think it all comes back to the requirements and the current use-cases.

1. In Helios and Galileo:
1a. each contributor is responsible for providing their update site(s) as input.
1b. each contributor is responsible for the integrity of the content of the contributed repositories.
1c. each contributor might select specific roots (features) to be mapped if he doesn't want the whole thing included.
The selection he makes here is of course limited to what he provides.
1d. the aggregation manager never edits a contribution. He merely aggregates them with some validation (the platform)
and disable/enable contributions as he see fit based on their ability to provide functional input.

For this use-case, since each contributor only deals with his own contribution, there cannot be any confusion caused by
having the mapped IU's in the contribution.


2. For Morgan Stanley
2a. One person is creating the aggregation.
2b. He builds one single contribution with everything in it since he is the sole contributor and owner of the whole
process. No need for additional email contacts for "blame" email, nor to spread the responsibility.

For this use-case, since there is only one contribution, there cannot be any confusion caused by having the mapped IU's
in that contribution.

The scenario you describe attempts to solve a use-case that sits somewhere in between the Eclipse release train use-case
and the MS use-case where the contributor is not limited to his own contribution and/or the aggregator manager uses
multiple contributions and is concerned with what each contribution contains. I don't understand why we need this. What
problem is it you are trying to solve?

If there was an easy way to refactor the model and resolve the issue once and for all, I would be all for that. But I
can't see how we can do that without disrupting the above use-cases. And doing it just to satisfy a use-case that
currently only exists in theory is not something I think we are prepared to spend resources on.

- thomas
Re: Aggregator with version range support [message #598038 is a reply to message #517197] Fri, 26 February 2010 15:56 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
OK, we need a conclusion, I don't like conclusionless discussions:

1) I removed the hack that was implementing what I though was natural
2) Now it works as it worked before (although it needs a lot of testing since not everything could be done with version ranges in the same way as with fixed versions)
3) I am very unsatisfied with the model-to-reality relationship, I think we did a bad job here.

I still think we could refactor the model without much effort if we spent some more time on discussions with more people. I don't think it's wise to create
software with only two use-cases in one's mind - it's not professional.


I wish you a nice weekend.

Good news is that we suppose to have a functional b3 aggregator (both integrated in Eclipse IDE and standalone headless application) which will be capable of
converting old helios build files into new b3 aggregator files and running the build of the Helios release.

Filip


Thomas Hallgren wrote:
> Filip,
> I think it all comes back to the requirements and the current use-cases.
>
> 1. In Helios and Galileo:
> 1a. each contributor is responsible for providing their update site(s)
> as input.
> 1b. each contributor is responsible for the integrity of the content of
> the contributed repositories.
> 1c. each contributor might select specific roots (features) to be mapped
> if he doesn't want the whole thing included. The selection he makes here
> is of course limited to what he provides.
> 1d. the aggregation manager never edits a contribution. He merely
> aggregates them with some validation (the platform) and disable/enable
> contributions as he see fit based on their ability to provide functional
> input.
>
> For this use-case, since each contributor only deals with his own
> contribution, there cannot be any confusion caused by having the mapped
> IU's in the contribution.
>
>
> 2. For Morgan Stanley
> 2a. One person is creating the aggregation.
> 2b. He builds one single contribution with everything in it since he is
> the sole contributor and owner of the whole process. No need for
> additional email contacts for "blame" email, nor to spread the
> responsibility.
>
> For this use-case, since there is only one contribution, there cannot be
> any confusion caused by having the mapped IU's in that contribution.
>
> The scenario you describe attempts to solve a use-case that sits
> somewhere in between the Eclipse release train use-case and the MS
> use-case where the contributor is not limited to his own contribution
> and/or the aggregator manager uses multiple contributions and is
> concerned with what each contribution contains. I don't understand why
> we need this. What problem is it you are trying to solve?
>
> If there was an easy way to refactor the model and resolve the issue
> once and for all, I would be all for that. But I can't see how we can do
> that without disrupting the above use-cases. And doing it just to
> satisfy a use-case that currently only exists in theory is not something
> I think we are prepared to spend resources on.
>
> - thomas
Re: Aggregator with version range support [message #600172 is a reply to message #517222] Fri, 26 February 2010 16:34 Go to previous messageGo to next message
Achim Demelt is currently offline Achim DemeltFriend
Messages: 160
Registered: July 2009
Senior Member
I must admit that I could not follow all of your arguments due to a lack of
in-depth aggregator knowledge. But I have the feeling that you really
*should* continue this discussion at another time, maybe over a beer at
EclipseCon. And, yes, include as many people (and as much beer) as it takes
to find a solution that leaves noone with bad feelings.

One more thing from my side: p2 tends to do things in a way that is
incomprehensible to the layman, and this includes many talented software
engineers who just didn't have the good fortune to study the p2 code base
themselves. It would be a shame if the aggregator acquired a similar
reputation.

Cheers,
Achim

Filip Hrbek wrote:

> OK, we need a conclusion, I don't like conclusionless discussions:
>
> 1) I removed the hack that was implementing what I though was natural
> 2) Now it works as it worked before (although it needs a lot of testing
> since not everything could be done with version ranges in the same way as
> with fixed versions) 3) I am very unsatisfied with the model-to-reality
> relationship, I think we did a bad job here.
>
> I still think we could refactor the model without much effort if we spent
> some more time on discussions with more people. I don't think it's wise to
> create software with only two use-cases in one's mind - it's not
> professional.
>
>
> I wish you a nice weekend.
>
> Good news is that we suppose to have a functional b3 aggregator (both
> integrated in Eclipse IDE and standalone headless application) which will
> be capable of converting old helios build files into new b3 aggregator
> files and running the build of the Helios release.
>
> Filip
>
>
> Thomas Hallgren wrote:
>> Filip,
>> I think it all comes back to the requirements and the current use-cases.
>>
>> 1. In Helios and Galileo:
>> 1a. each contributor is responsible for providing their update site(s)
>> as input.
>> 1b. each contributor is responsible for the integrity of the content of
>> the contributed repositories.
>> 1c. each contributor might select specific roots (features) to be mapped
>> if he doesn't want the whole thing included. The selection he makes here
>> is of course limited to what he provides.
>> 1d. the aggregation manager never edits a contribution. He merely
>> aggregates them with some validation (the platform) and disable/enable
>> contributions as he see fit based on their ability to provide functional
>> input.
>>
>> For this use-case, since each contributor only deals with his own
>> contribution, there cannot be any confusion caused by having the mapped
>> IU's in the contribution.
>>
>>
>> 2. For Morgan Stanley
>> 2a. One person is creating the aggregation.
>> 2b. He builds one single contribution with everything in it since he is
>> the sole contributor and owner of the whole process. No need for
>> additional email contacts for "blame" email, nor to spread the
>> responsibility.
>>
>> For this use-case, since there is only one contribution, there cannot be
>> any confusion caused by having the mapped IU's in that contribution.
>>
>> The scenario you describe attempts to solve a use-case that sits
>> somewhere in between the Eclipse release train use-case and the MS
>> use-case where the contributor is not limited to his own contribution
>> and/or the aggregator manager uses multiple contributions and is
>> concerned with what each contribution contains. I don't understand why
>> we need this. What problem is it you are trying to solve?
>>
>> If there was an easy way to refactor the model and resolve the issue
>> once and for all, I would be all for that. But I can't see how we can do
>> that without disrupting the above use-cases. And doing it just to
>> satisfy a use-case that currently only exists in theory is not something
>> I think we are prepared to spend resources on.
>>
>> - thomas
Re: Aggregator with version range support [message #600179 is a reply to message #600172] Fri, 26 February 2010 20:21 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Thanks Achim. You deserve a diplomacy badge ;) i must confess that I
don't get the arguments presented either. And even worse, I don't think
I even understand the problem except that there is a case that is
confusing for a user...

I think it is important to go back to the beginning and look at how a
user is making statements, what the user think they mean, and what they
really mean. Mixing this with implementation detail makes my head
explode...

- henrik

Achim Demelt <a.demelt@exxcellent.de> wrote:
> I must admit that I could not follow all of your arguments due to a
> lack of
> in-depth aggregator knowledge. But I have the feeling that you really
> *should* continue this discussion at another time, maybe over a beer
> at
> EclipseCon. And, yes, include as many people (and as much beer) as it
> takes
> to find a solution that leaves noone with bad feelings.
>
> One more thing from my side: p2 tends to do things in a way that is
> incomprehensible to the layman, and this includes many talented
> software
> engineers who just didn't have the good fortune to study the p2 code
> base
> themselves. It would be a shame if the aggregator acquired a similar
> reputation.
>
> Cheers,
> Achim
>
> Filip Hrbek wrote:
>
> > OK, we need a conclusion, I don't like conclusionless discussions:
> >
> > 1) I removed the hack that was implementing what I though was
> > natural
> > 2) Now it works as it worked before (although it needs a lot of
> > testing
> > since not everything could be done with version ranges in the same
> > way as
> > with fixed versions) 3) I am very unsatisfied with the
> > model-to-reality
> > relationship, I think we did a bad job here.
> >
> > I still think we could refactor the model without much effort if we
> > spent
> > some more time on discussions with more people. I don't think it's
> > wise to
> > create software with only two use-cases in one's mind - it's not
> > professional.
> >
> >
> > I wish you a nice weekend.
> >
> > Good news is that we suppose to have a functional b3 aggregator
> > (both
> > integrated in Eclipse IDE and standalone headless application) which
> > will
> > be capable of converting old helios build files into new b3
> > aggregator
> > files and running the build of the Helios release.
> >
> > Filip
> >
> >
> > Thomas Hallgren wrote:
> >> Filip,
> >> I think it all comes back to the requirements and the current
> > > use-cases.
> >>
> >> 1. In Helios and Galileo:
> >> 1a. each contributor is responsible for providing their update
> > > site(s)
> >> as input.
> >> 1b. each contributor is responsible for the integrity of the
> > > content of
> >> the contributed repositories.
> >> 1c. each contributor might select specific roots (features) to be
> > > mapped
> >> if he doesn't want the whole thing included. The selection he makes
> > > here
> >> is of course limited to what he provides.
> >> 1d. the aggregation manager never edits a contribution. He merely
> >> aggregates them with some validation (the platform) and
> > > disable/enable
> >> contributions as he see fit based on their ability to provide
> > > functional
> >> input.
> >>
> >> For this use-case, since each contributor only deals with his own
> >> contribution, there cannot be any confusion caused by having the
> > > mapped
> >> IU's in the contribution.
> >>
> >>
> >> 2. For Morgan Stanley
> >> 2a. One person is creating the aggregation.
> >> 2b. He builds one single contribution with everything in it since
> > > he is
> >> the sole contributor and owner of the whole process. No need for
> >> additional email contacts for "blame" email, nor to spread the
> >> responsibility.
> >>
> >> For this use-case, since there is only one contribution, there
> > > cannot be
> >> any confusion caused by having the mapped IU's in that
> > > contribution.
> >>
> >> The scenario you describe attempts to solve a use-case that sits
> >> somewhere in between the Eclipse release train use-case and the MS
> >> use-case where the contributor is not limited to his own
> > > contribution
> >> and/or the aggregator manager uses multiple contributions and is
> >> concerned with what each contribution contains. I don't understand
> > > why
> >> we need this. What problem is it you are trying to solve?
> >>
> >> If there was an easy way to refactor the model and resolve the
> > > issue
> >> once and for all, I would be all for that. But I can't see how we
> > > can do
> >> that without disrupting the above use-cases. And doing it just to
> >> satisfy a use-case that currently only exists in theory is not
> > > something
> >> I think we are prepared to spend resources on.
> >>
> >> - thomas
>
>


--
- henrik
Re: Aggregator with version range support [message #600187 is a reply to message #517164] Fri, 26 February 2010 20:21 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
That is not what i asked. I asked if it only picked the latest.
- henrik
Thomas Hallgren <thomas@tada.se> wrote:
> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
> > A question - as a user I would assume that if I request A with a
> > range,
> > (from one repo, as I perceive it as a user), and there exists A's in
> > this range in that repo, that they are indeed included in the final
> > result even if there are newer versions of A in other repositories
> > that
> > are mapped with other rules. Is this true?
> >
> Nope. The planner consider all repositories as one source of input. It
> never considers what individual repository a specific requirement
> originated from. It will create a plan where all requirement ranges
> are fulfilled but the actual IU's included can stem from any
> repository.
>
> - thomas
>


--
- henrik
Re: Aggregator with version range support [message #600194 is a reply to message #517298] Fri, 26 February 2010 23:19 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
> That is not what i asked. I asked if it only picked the latest.
> - henrik

I'm reading your question again and I come up with the same answer :-)

But let me try and rephrase:

The fact that there are newer versions of A in other repositories, and still within the requested range, might well mean
that these newer versions are chosen. The planner will pick the best solution possible for A in scope of the aggregation
as a whole. No consideration whatsoever is given to in which source repositories an IU is found.

OK?

- thomas

> Thomas Hallgren<thomas@tada.se> wrote:
>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>> A question - as a user I would assume that if I request A with a range,
>>> (from one repo, as I perceive it as a user), and there exists A's in
>>> this range in that repo, that they are indeed included in the final
>>> result even if there are newer versions of A in other repositories
>>> that are mapped with other rules. Is this true?
>>>
>> Nope. The planner consider all repositories as one source of input. It
>> never considers what individual repository a specific requirement
>> originated from. It will create a plan where all requirement ranges
>> are fulfilled but the actual IU's included can stem from any
>> repository.
>>
>> - thomas
>>
>
>
Re: Aggregator with version range support [message #600200 is a reply to message #517330] Sat, 27 February 2010 00:49 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Well, what I want to know is still not clear.
I understand that:
- a request to include A [fromA1, toA1] does not mean that some other
request of A [fromA2, toA2 > toA1] is restricted.
- a request to include A [some range] is not tied to a particular repository

What I wonder what the resulting repository will contain if there are
many versions of A that comply. Say that fromA1 < fromA2, and toA1 > toA2.

If I have these requests in my map, I may have different desires:
a) I want all versions i.e. fromA1 (the smallest requested) to toA1 (the
largest requested).
b) I want the latest version that complies with all requests.

I would desire a) if I want to include all "old version" (in stated
range), and desire b) if I just want a slimmed repository.

This was my question - what does the aggregator mirror if I add the two
requests to A?

- henrik

On 2/27/10 12:19 AM, Thomas Hallgren wrote:
> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>> That is not what i asked. I asked if it only picked the latest.
>> - henrik
>
> I'm reading your question again and I come up with the same answer :-)
>
> But let me try and rephrase:
>
> The fact that there are newer versions of A in other repositories, and
> still within the requested range, might well mean that these newer
> versions are chosen. The planner will pick the best solution possible
> for A in scope of the aggregation as a whole. No consideration
> whatsoever is given to in which source repositories an IU is found.
>
> OK?
>
> - thomas
>
>> Thomas Hallgren<thomas@tada.se> wrote:
>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>> A question - as a user I would assume that if I request A with a range,
>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>> this range in that repo, that they are indeed included in the final
>>>> result even if there are newer versions of A in other repositories
>>>> that are mapped with other rules. Is this true?
>>>>
>>> Nope. The planner consider all repositories as one source of input. It
>>> never considers what individual repository a specific requirement
>>> originated from. It will create a plan where all requirement ranges
>>> are fulfilled but the actual IU's included can stem from any
>>> repository.
>>>
>>> - thomas
>>>
>>
>>
>
Re: Aggregator with version range support [message #600223 is a reply to message #600200] Sat, 27 February 2010 09:59 Go to previous messageGo to next message
Filip Hrbek is currently offline Filip HrbekFriend
Messages: 233
Registered: July 2009
Senior Member
If there are two requests:

A [fromA1, toA1]
A [fromA2, toA2]

where fromA1 > fromA2 and toA1 > toA2

then the planner will choose a version that matches the range [fromA1, toA2], i.e. both original request must be satisfied.
If no such a match is found, the aggregation fails.

In addition to this, it does not matter where the chosen match comes from, i.e. if A [fromA1, toA1] was defined "from repo X"
where the highest version of A is Vx, then the aggregator may choose version Vy from repo Y if Vy > Vx && Vy <= toA2.

That's why I don't like the "graphical picture" of the thing in the UI, where user might think that he gets only things from
the repository's resolution scope.

Filip


Henrik Lindberg wrote:
> Well, what I want to know is still not clear.
> I understand that:
> - a request to include A [fromA1, toA1] does not mean that some other
> request of A [fromA2, toA2 > toA1] is restricted.
> - a request to include A [some range] is not tied to a particular
> repository
>
> What I wonder what the resulting repository will contain if there are
> many versions of A that comply. Say that fromA1 < fromA2, and toA1 > toA2.
>
> If I have these requests in my map, I may have different desires:
> a) I want all versions i.e. fromA1 (the smallest requested) to toA1 (the
> largest requested).
> b) I want the latest version that complies with all requests.
>
> I would desire a) if I want to include all "old version" (in stated
> range), and desire b) if I just want a slimmed repository.
>
> This was my question - what does the aggregator mirror if I add the two
> requests to A?
>
> - henrik
>
> On 2/27/10 12:19 AM, Thomas Hallgren wrote:
>> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>>> That is not what i asked. I asked if it only picked the latest.
>>> - henrik
>>
>> I'm reading your question again and I come up with the same answer :-)
>>
>> But let me try and rephrase:
>>
>> The fact that there are newer versions of A in other repositories, and
>> still within the requested range, might well mean that these newer
>> versions are chosen. The planner will pick the best solution possible
>> for A in scope of the aggregation as a whole. No consideration
>> whatsoever is given to in which source repositories an IU is found.
>>
>> OK?
>>
>> - thomas
>>
>>> Thomas Hallgren<thomas@tada.se> wrote:
>>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>>> A question - as a user I would assume that if I request A with a
>>>>> range,
>>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>>> this range in that repo, that they are indeed included in the final
>>>>> result even if there are newer versions of A in other repositories
>>>>> that are mapped with other rules. Is this true?
>>>>>
>>>> Nope. The planner consider all repositories as one source of input. It
>>>> never considers what individual repository a specific requirement
>>>> originated from. It will create a plan where all requirement ranges
>>>> are fulfilled but the actual IU's included can stem from any
>>>> repository.
>>>>
>>>> - thomas
>>>>
>>>
>>>
>>
>
Re: Aggregator with version range support [message #600245 is a reply to message #517370] Sat, 27 February 2010 15:08 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Boy, this is a tough question to answer...

When aggregating - what do I get in my aggregated (mirrored) result:
Only one version of A, or several?
Can I get serveral?

- henrik

On 2/27/10 10:59 AM, Filip Hrbek wrote:
> If there are two requests:
>
> A [fromA1, toA1]
> A [fromA2, toA2]
>
> where fromA1 > fromA2 and toA1 > toA2
>
> then the planner will choose a version that matches the range [fromA1,
> toA2], i.e. both original request must be satisfied.
> If no such a match is found, the aggregation fails.
>
> In addition to this, it does not matter where the chosen match comes
> from, i.e. if A [fromA1, toA1] was defined "from repo X"
> where the highest version of A is Vx, then the aggregator may choose
> version Vy from repo Y if Vy > Vx && Vy <= toA2.
>
> That's why I don't like the "graphical picture" of the thing in the UI,
> where user might think that he gets only things from
> the repository's resolution scope.
>
> Filip
>
>
> Henrik Lindberg wrote:
>> Well, what I want to know is still not clear.
>> I understand that:
>> - a request to include A [fromA1, toA1] does not mean that some other
>> request of A [fromA2, toA2 > toA1] is restricted.
>> - a request to include A [some range] is not tied to a particular
>> repository
>>
>> What I wonder what the resulting repository will contain if there are
>> many versions of A that comply. Say that fromA1 < fromA2, and toA1 >
>> toA2.
>>
>> If I have these requests in my map, I may have different desires:
>> a) I want all versions i.e. fromA1 (the smallest requested) to toA1
>> (the largest requested).
>> b) I want the latest version that complies with all requests.
>>
>> I would desire a) if I want to include all "old version" (in stated
>> range), and desire b) if I just want a slimmed repository.
>>
>> This was my question - what does the aggregator mirror if I add the
>> two requests to A?
>>
>> - henrik
>>
>> On 2/27/10 12:19 AM, Thomas Hallgren wrote:
>>> On 02/26/2010 09:21 PM, Henrik Lindberg wrote:
>>>> That is not what i asked. I asked if it only picked the latest.
>>>> - henrik
>>>
>>> I'm reading your question again and I come up with the same answer :-)
>>>
>>> But let me try and rephrase:
>>>
>>> The fact that there are newer versions of A in other repositories, and
>>> still within the requested range, might well mean that these newer
>>> versions are chosen. The planner will pick the best solution possible
>>> for A in scope of the aggregation as a whole. No consideration
>>> whatsoever is given to in which source repositories an IU is found.
>>>
>>> OK?
>>>
>>> - thomas
>>>
>>>> Thomas Hallgren<thomas@tada.se> wrote:
>>>>> On 02/26/2010 01:30 PM, Henrik Lindberg wrote:
>>>>>> A question - as a user I would assume that if I request A with a
>>>>>> range,
>>>>>> (from one repo, as I perceive it as a user), and there exists A's in
>>>>>> this range in that repo, that they are indeed included in the final
>>>>>> result even if there are newer versions of A in other repositories
>>>>>> that are mapped with other rules. Is this true?
>>>>>>
>>>>> Nope. The planner consider all repositories as one source of input. It
>>>>> never considers what individual repository a specific requirement
>>>>> originated from. It will create a plan where all requirement ranges
>>>>> are fulfilled but the actual IU's included can stem from any
>>>>> repository.
>>>>>
>>>>> - thomas
>>>>>
>>>>
>>>>
>>>
>>
Re: Aggregator with version range support [message #600260 is a reply to message #517403] Sat, 27 February 2010 16:59 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/27/2010 04:08 PM, Henrik Lindberg wrote:
> Boy, this is a tough question to answer...
>
> When aggregating - what do I get in my aggregated (mirrored) result:
> Only one version of A, or several?
> Can I get serveral?
>
You can, but it would be fairly rare. If the planner finds no solution where only one version of A is included and A is
not a singleton (which it would be under most circumstances), then the planner might decide to bring in two versions.

- thomas
Re: Aggregator with version range support [message #600266 is a reply to message #517418] Sat, 27 February 2010 17:29 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Thanks.

I noticed there is an enhancement request to enable including multiple
versions. I assume that is valid when creating repositories to be used
in development (you want all to be included because you don't know what
people are building against).

Can you explain how that is acheived?

- henrik

On 2/27/10 5:59 PM, Thomas Hallgren wrote:
> On 02/27/2010 04:08 PM, Henrik Lindberg wrote:
>> Boy, this is a tough question to answer...
>>
>> When aggregating - what do I get in my aggregated (mirrored) result:
>> Only one version of A, or several?
>> Can I get serveral?
>>
> You can, but it would be fairly rare. If the planner finds no solution
> where only one version of A is included and A is not a singleton (which
> it would be under most circumstances), then the planner might decide to
> bring in two versions.
>
> - thomas
Re: Aggregator with version range support [message #600272 is a reply to message #517422] Sat, 27 February 2010 17:58 Go to previous message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 02/27/2010 06:29 PM, Henrik Lindberg wrote:
> Thanks.
>
> I noticed there is an enhancement request to enable including multiple
> versions. I assume that is valid when creating repositories to be used
> in development (you want all to be included because you don't know what
> people are building against).
>
> Can you explain how that is acheived?
>
We can accomplish this by using a target consisting of one shared artifact repository and a composite meta-data
repository. Each aggregation run will then add one new child repository to the composite. This child is 100% verified
(using the planner) but when all children are brought together using the composite, there will no longer be a guarantee
that everything is consistent.

This is very similar to how many repositories at eclipse.org are constituted today. Platform, Galileo, and Helios
included. The only difference is that they don't currently utilize a shared artifact repository.

In a typical scenario this will always work since the composite contains permutations of the same aggregation where each
permutation introduces new components or new versions of old components. The planner will then satisfy an install from
the last permutation when installing "the latest" or when upgrading.

The problem that can arise is when you install earlier versions and those versions have requirements that are too
loosely specified. When that happens, the client will be forced to install from a selected repository instead of from
the composite. Creating the validation utility capable of detecting such problems while spending a reasonable amount of
time and CPU is challenging but I don't think it is impossible.

- thomas
Previous Topic:Aggregator debate - take 2
Next Topic:Problems with DS on startup
Goto Forum:
  


Current Time: Thu Mar 28 18:30:44 GMT 2024

Powered by FUDForum. Page generated in 0.07378 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top