Home » Archived » B3 » Repositories and Providers
| Repositories and Providers [message #538592] |
Tue, 08 June 2010 07:16  |
|
I've done some thinking about the scenarios that we want to support with
respect to SCM's.
1. CI performs check out of a root folder. b3 providers performs
discovery below the root.
2. Like #1 but b3 performs the check out.
3. Root is huge and it is not desirable to check it out. Provides use
team project sets, maps, or Buckminster style discovery to make a more
precise check out.
We need to be able to specify what the expectations are between b3 and
the CI system, and between the repositories and the providers. #1 and #2
is of course very similar. It boils down to that b3 finds something that
is already checked out, i.e. an policy:
mustExist:
b3 should make no attempt to check out and fail if it's not present.
trusted:
Trust what is there (i.e. no update) but perform a new check out if it's
missing.
update:
Update what is there or perform a new check out if it's missing.
overwrite:
Remove what is there and then perform a new check out.
mustNotExist:
Fail if something is found at the intended check-out location.
The question is, how do we combine this with #3? Do we simply add a
boolean flag such as "providerCheckout" and then push the policy down to
each provider?
- thomas
|
|
|
| Re: Repositories and Providers [message #538886 is a reply to message #538592] |
Tue, 08 June 2010 22:11  |
 Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Hi,
In b3, what we decided so far is that the syntax for repository branches
has a specification of what should be excluded from / included in the
resulting repository "view". Exclude / Include is a list of verbatim
paths, or patterns.
We also have a specification of "update-policy" that can be one of
"default" (i.e. default for the combination of repository type and type
or branch), "fail-modified", "keep-modified", "merge-modified",
"no-update", and "replace-modified". Don't know if these cover all of
the cases you listed, but this is the list we came up with when you and
I designed this.
All of this is pushed down to the repository implementation(s). They
will later be asked to produce BuildUnit instances given a request for a
capability.
One possibility is to inject different implementations in the CI
scenario - they would know that they are never supposed to check
something out. The specifications in the b3 script then acts as sanity
checks - did the CI server actually check out what is required. Such a
solution allows the same script to run anywhere and it then performs the
checkout as an integrated part.
The CI "checkout" could likewise read the b3 script, perform the
checkout as directed by the script, and then finish (and let other jobs
run with reuse of the checked out material).
An elaborate CI setup could also combine the requirements on
repositories from multiple b3 scripts, perhaps merge the requirements to
come up with a "sum of everything that needs to be checked out".
Regards
- henrik
On 6/8/10 9:16 AM, Thomas Hallgren wrote:
> I've done some thinking about the scenarios that we want to support with
> respect to SCM's.
>
> 1. CI performs check out of a root folder. b3 providers performs
> discovery below the root.
>
> 2. Like #1 but b3 performs the check out.
>
> 3. Root is huge and it is not desirable to check it out. Provides use
> team project sets, maps, or Buckminster style discovery to make a more
> precise check out.
>
> We need to be able to specify what the expectations are between b3 and
> the CI system, and between the repositories and the providers. #1 and #2
> is of course very similar. It boils down to that b3 finds something that
> is already checked out, i.e. an policy:
>
> mustExist:
> b3 should make no attempt to check out and fail if it's not present.
>
> trusted:
> Trust what is there (i.e. no update) but perform a new check out if it's
> missing.
>
> update:
> Update what is there or perform a new check out if it's missing.
>
> overwrite:
> Remove what is there and then perform a new check out.
>
> mustNotExist:
> Fail if something is found at the intended check-out location.
>
> The question is, how do we combine this with #3? Do we simply add a
> boolean flag such as "providerCheckout" and then push the policy down to
> each provider?
>
> - thomas
>
|
|
|
| Re: Repositories and Providers [message #538920 is a reply to message #538886] |
Wed, 09 June 2010 06:08 |
|
On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
> Hi,
> In b3, what we decided so far is that the syntax for repository branches
> has a specification of what should be excluded from / included in the
> resulting repository "view". Exclude / Include is a list of verbatim
> paths, or patterns.
>
That's all good, but it doesn't really cover the case when you want to use an existing "team project set" or similar in
a provider. They have been around for a long time and are very commonly used. The same is true for the PDE 'maps'. We
could of course require that such files are abandoned in favor of the b3 syntax but that might make adoption more painful.
> We also have a specification of "update-policy" that can be one of
> "default" (i.e. default for the combination of repository type and type
> or branch), "fail-modified", "keep-modified", "merge-modified",
> "no-update", and "replace-modified". Don't know if these cover all of
> the cases you listed, but this is the list we came up with when you and
> I designed this.
>
Right. But these names seem to describe what to do when a merge-conflict arises and can really only be used with what I
refer to as 'update'. What I'm after is what b3 should do when it encounters already existing material (or not) at the
intended location. This is how I would map the current names to what I'm after:
fail-modified = update (and fail on need to merge)
keep-modified = update (and keep working copy on need to merge)
merge-modified = update (and merge if needed)
replace-modified = update (overwrite local copy instead of merge)
no-update = trusted
??? = overwrite
??? = mustExist
??? = mustNotExist
So perhaps adding the last three to the list would be enough. On the other hand, many names in an enum might make it
hard to understand and a solution could be to have two enums. Something like this:
exists-policy { fail, use, merge, overwrite }
merge-policy { fail, keep, merge, replace }
There's still nothing that enables me to declare that I expect the material to be present. I.e. if it's missing, should
I do a initial check out or should I fail?
initial-checkout true/false
(defaults to true)
exists-policy = fail cannot be combined with initial-checkout = false.
- thomas
> All of this is pushed down to the repository implementation(s). They
> will later be asked to produce BuildUnit instances given a request for a
> capability.
>
> One possibility is to inject different implementations in the CI
> scenario - they would know that they are never supposed to check
> something out. The specifications in the b3 script then acts as sanity
> checks - did the CI server actually check out what is required. Such a
> solution allows the same script to run anywhere and it then performs the
> checkout as an integrated part.
>
Sure, that's possible but I will not be able to use the exact same script in the IDE and the CI.
- thomas
|
|
|
| Re: Repositories and Providers [message #538934 is a reply to message #538920] |
Wed, 09 June 2010 07:05 |
|
On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
> exists-policy { fail, use, merge, overwrite }
> merge-policy { fail, keep, merge, replace }
>
Thinking more about the names. This might be less confusing:
exists-policy { fail, keep, replace, merge }
merge-policy { fail, use-local, use-remote, merge }
- thomas
|
|
| |
| Re: Repositories and Providers [message #539006 is a reply to message #538934] |
Wed, 09 June 2010 10:21 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I like that proposal bettet. IT is also vetter Tham The current "update
policy".
- henrik
Thomas Hallgren <thomas@tada.se> wrote:
> On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
>> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
>> exists-policy { fail, use, merge, overwrite }
>> merge-policy { fail, keep, merge, replace }
>>
>
> Thinking more about the names. This might be less confusing:
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
- thomas
--
- henrik
|
|
|
| Re: Repositories and Providers [message #539011 is a reply to message #539006] |
Wed, 09 June 2010 10:25 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Doh!
Reminder: remember to switch to English auto correct when typing in
English.
I think the message came through tnough :)
- henrik
Henrik Lindberg <henrik.lindberg@cloudsmith.com> wrote:
> I like that proposal bettet. IT is also vetter Tham The current
> "update
> policy".
> - henrik
>
> Thomas Hallgren <thomas@tada.se> wrote:
>> On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
>>> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
>>> exists-policy { fail, use, merge, overwrite }
>>> merge-policy { fail, keep, merge, replace }
>>>
>>
>> Thinking more about the names. This might be less confusing:
>>
>> exists-policy { fail, keep, replace, merge }
>> merge-policy { fail, use-local, use-remote, merge }
- thomas
--
- henrik
|
|
|
| Re: Repositories and Providers [message #539026 is a reply to message #539005] |
Wed, 09 June 2010 11:18 |
|
On 06/09/2010 12:21 PM, Henrik Lindberg wrote:
>> Sure, that's possible but I will not be able to use the exact same
>> script in the IDE and the CI.
>
> Interested in why why that is? Seems like a big benefit to only maintain
> repository info in one place. If it is not as simple as using the same
> policy for all repos if the same type, then we should consider
> advicing/weaving per repository.
>
What I meant was that if injections are needed in the CI scenario, then those needs to be maintained somewhere. Which I
guess, calls for a different b3 script (or different parameters that controls the injection somehow). No big deal.
> Regarding .psf and buckminster rmaps; think these are listed as
> repositories of "psf" and "rmap" type.
Yes, that's an elegant solution. Under the covers, it will just delegate to whatever repositories that are referenced.
> That is probably enough, but does
> not work if there is a need to specify different policies for different
> things referenced by these external maps.
>
Nobody has ever expressed a need to do that so that won't be a problem. Should someone like to do it, well, then it's
time to abandon the psf and instead use the b3 syntax.
- thomas
|
|
|
| Re: Repositories and Providers [message #539056 is a reply to message #539026] |
Wed, 09 June 2010 13:26 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
So, trying to sum up:
1. Change the "update-policy" to the two policies you suggested.
exists-policy { fail, keep, replace, merge }
merge-policy { fail, use-local, use-remote, merge }
I downside is that the enum's become keywords (and keep, replace, and
merge are very likely to clash with other things). (The current solution
was crafted so the enums all have "-" in them to reduce clashes).
Suggestion; have all enums everywhere have a special leading character
like '@'. (Yet another more complicated impl is to treat the enum values
as IDs and have data conversion rules per enum - not sure how well that
works though, I tried something similar earlier and it was not possible
to return an EObject supposed to be part of the model built by the
grammar this way).
The only solution I can think of would use a reference to an Enum
instead of containing one. This way I can use an ID, and have a scope
provider that provides the enums to link to. (I use this technique to
reference implicit variables like "request" in resolver expressions).
Using @ is however much much simpler...
proposal example:
exists-policy : @keep;
2. psf and buckminster maps
Handled as repository types in b3 (i.e. nothings special needed in b3,
it is up to the impl of these). Any advanced handling of these is also
up to the respective impl (i.e. parameters passed from the script etc).
3. Regarding CI, and using different policies.
I think the best solution is to have the list of repositories in a
separate .b3 file (i.e. separate from the resolution specificaion). It
is not only the policies that need to differ, it is also things like
where to place the local/mirrored repositories - the list of things to
change is likely just as long/complicated as the original list.
Complicating the repository declaration with conditionals
(when(${on.ci.server}) ... etc) is also messy).
The resolution part needs to know about available repositories (they
refer to the repositories via names) - if the specification is imported,
it is possible to get different compatible configurations by modifying
the "classpath". There is currently only limited support for imports in
the b3 impl, so until that is done, different b3 scripts will need to be
used when different policies are wanted.
- henrik
|
|
|
| Re: Repositories and Providers [message #539210 is a reply to message #539056] |
Thu, 10 June 2010 00:19 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Hi,
I looked at this again, and I think it is further simplified if thought
of as three separate things:
1. what to do if there is nothing at all
2. what to do if there is something there
3. what to do if there is a need to merge
I also believe that we did not cover all the cases.
1. what to do when location is empty:
on-empty-location: checkout | accept-empty | fail ;
The default is "checkout".
checkout
Clone or checkout the "remote" repository.
accept-empty
Needed if we want to have repositories that may be empty - i.e. not
return anything if nothing was checked out. (Perhaps, in situations
where some big checkout of everything may be done by the ci server, if
it is not there - fine we will use the other repositories).
Not sure if this option is needed.
fail
It is expected that someone else has performed a checkout to the location.
2. what to do if there is something there
(and it was not put there by on-empty-location : checkout in the same
invocation).
update-policy
: default
| no-update
| fail-uncommitted
| fail-localchange
| update
| replace;
;
default
Depends on use of branch/tag/main etc. (And is the default if nothing is
stated). Don't remember exactly what we decided, but something like main
= update, tag/timestamp = fail-localchange.
no-update
trust whatever is there to be correct
fail-uncomitted, fail-localchange
if the content is dirty - no update, just check and fail if dirty.
In git there are actually two separate dirty cases - if there is
uncommitted changes (i.e. in workspace only), or changes not pushed to
the cloned repository. So we may need two separate states:
fail-uncommitted, fail-localchange (where fail-localchange is the same
as fail-uncommitted if the repo does not distinguish between them).
We could use git specific terms, but then it is perhaps more confusing
when used with other types of repositories.
update
perform an update (the merge-policy specifies how to handle the update).
replace
result is the same as removing everything and checking out from scratch
(may be done more efficiently naturally if result is guaranteed to be
the same).
(Possible addition to update-policy is to allow specification of a delay
between update checks. This could speed up building if performed over
and over again manually triggered in a local scenario. Proposal
update-hourly, update-daily, or a separate attribute 'update-interval :
00:00:30', but this could be added by a repository impl).
3. what to do if there is a need to merge
(A merge-policy can only be used if the update-policy is 'update'.)
merge-policy
: default
| fail-uncommitted
| fail-localchange
| fail-merge
| fail-conflicts
| use-local
| use-remote
| accept-conflicts
;
default
This default only kicks in if main == update by default, or user changed
setting to update for the others. May depend on use of branch/tag/main
etc. but I am not sure if that is valid - seems reasonable with
fail-conflicts as the default in all cases. (We don't need a "default"
in that case).
fail-uncomitted, fail-localchange
Fail if local content is 'dirty' (same rule as for update-policy) even
if there is (potentially) no need to merge because the remote is
unchanged. This is essentially the same as the corresponding
specification for update-policy but with the difference that a merge is
performed if not dirty. Rationale - I am supposed to run a clean build,
reproducable by others, any changes are done by mistake.
fail-merge
Fail if there is any need to merge (local/remote change) even if it can
be done without manual intervention. i.e. accept clean updates, added
and removed files but not anything requiring a merge. Rationale, I am
not anticipating change in anything I am also changing.
fail-conflicts
Fail if merge results in conflicts requiring a manual-merge. Rationale,
I don't want to clean up conflicting merges for everything all at once.
(could be called fail-manual-merge).
use-local
If a merge results in conflict requiring manual-merge - keep the
local/dirty copy. Rationale - I will deal with merges later, I want the
stuff I am working on to build now with the latest except where others
have fixed the old and bad and I am working on the next better solution.
use-remote
If a merge results in conflict requiring manual-merge - replace the
local dirty copy with the remote. Rationale - oops, I must have been
experimenting, my mistake - fix it.
accept-conflicts
Perform merge and if there are conflicts (requiring manual
intervention), accept these (i.e. perform the merge), and let the build
fail (or b3 for that matter, once the merge is completed). Rationale - I
probably need my head examined, but this is what I want - I do not
expect any conflicts and if they occur then I will fix them.
- henrik
On 6/9/10 3:26 PM, Henrik Lindberg wrote:
> So, trying to sum up:
>
> 1. Change the "update-policy" to the two policies you suggested.
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
>
> I downside is that the enum's become keywords (and keep, replace, and
> merge are very likely to clash with other things). (The current solution
> was crafted so the enums all have "-" in them to reduce clashes).
> Suggestion; have all enums everywhere have a special leading character
> like '@'. (Yet another more complicated impl is to treat the enum values
> as IDs and have data conversion rules per enum - not sure how well that
> works though, I tried something similar earlier and it was not possible
> to return an EObject supposed to be part of the model built by the
> grammar this way).
> The only solution I can think of would use a reference to an Enum
> instead of containing one. This way I can use an ID, and have a scope
> provider that provides the enums to link to. (I use this technique to
> reference implicit variables like "request" in resolver expressions).
>
> Using @ is however much much simpler...
>
> proposal example:
> exists-policy : @keep;
>
>
> 2. psf and buckminster maps
>
> Handled as repository types in b3 (i.e. nothings special needed in b3,
> it is up to the impl of these). Any advanced handling of these is also
> up to the respective impl (i.e. parameters passed from the script etc).
>
> 3. Regarding CI, and using different policies.
>
> I think the best solution is to have the list of repositories in a
> separate .b3 file (i.e. separate from the resolution specificaion). It
> is not only the policies that need to differ, it is also things like
> where to place the local/mirrored repositories - the list of things to
> change is likely just as long/complicated as the original list.
> Complicating the repository declaration with conditionals
> (when(${on.ci.server}) ... etc) is also messy).
>
> The resolution part needs to know about available repositories (they
> refer to the repositories via names) - if the specification is imported,
> it is possible to get different compatible configurations by modifying
> the "classpath". There is currently only limited support for imports in
> the b3 impl, so until that is done, different b3 scripts will need to be
> used when different policies are wanted.
>
> - henrik
|
|
|
| Re: Repositories and Providers [message #539264 is a reply to message #539210] |
Thu, 10 June 2010 08:35 |
|
Comments inline.
On 06/10/2010 02:19 AM, Henrik Lindberg wrote:
> Hi,
> I looked at this again, and I think it is further simplified if thought
> of as three separate things:
>
> 1. what to do if there is nothing at all
> 2. what to do if there is something there
> 3. what to do if there is a need to merge
>
> I also believe that we did not cover all the cases.
>
> 1. what to do when location is empty:
>
> on-empty-location: checkout | accept-empty | fail ;
>
> The default is "checkout".
>
> checkout
> Clone or checkout the "remote" repository.
>
> accept-empty
> Needed if we want to have repositories that may be empty - i.e. not
> return anything if nothing was checked out. (Perhaps, in situations
> where some big checkout of everything may be done by the ci server, if
> it is not there - fine we will use the other repositories).
> Not sure if this option is needed.
>
I think this is confusing and not really needed. Confusing because it implies a checkout with a subsequent control
rather then one simple directive. Not really needed because IMO, it's covered by the 'checkout'. If it proceeds without
errors and the result is an empty repository, well, then that's the current state of things. We might consider some kind
of validation that asserts that a checkout gives some expected result of course, but that's a different thing altogether.
What does empty mean? Is it physically empty or a folder with SCM meta-data that indicates no content? In my view, this
enum should be about the former. I.e. "what to do if there is nothing at all" is to be interpreted as "what to do if the
location is physically empty" (hence my boolean 'initial-checkout').
I agree that
on-empty-location: checkout | fail
is clearer.
> fail
> It is expected that someone else has performed a checkout to the location.
>
> 2. what to do if there is something there
> (and it was not put there by on-empty-location : checkout in the same
> invocation).
>
> update-policy
> : default
> | no-update
> | fail-uncommitted
> | fail-localchange
> | update
> | replace;
> ;
>
> default
> Depends on use of branch/tag/main etc. (And is the default if nothing is
> stated). Don't remember exactly what we decided, but something like main
> = update, tag/timestamp = fail-localchange.
>
head of any branch (not just main) = update.
tag/timestamp/revision (i.e. fixed material) = trust-clean (see New proposal below)
> no-update
> trust whatever is there to be correct
>
I like 'trust' better because it declares what to do rather then what not to do.
> fail-uncomitted, fail-localchange
> if the content is dirty - no update, just check and fail if dirty.
> In git there are actually two separate dirty cases - if there is
> uncommitted changes (i.e. in workspace only), or changes not pushed to
> the cloned repository. So we may need two separate states:
> fail-uncommitted, fail-localchange (where fail-localchange is the same
> as fail-uncommitted if the repo does not distinguish between them).
>
> We could use git specific terms, but then it is perhaps more confusing
> when used with other types of repositories.
>
The distinction is needed but problem arise when it is expressed as multiple ways to fail instead of what to do on
success. What if it does not fail? Do you update then? Or trust? Or replace even (since you may want to erase the
checked out content and replace it with what you have in your local clone)? How do you express that you want to fail
both on local changes and on uncommitted changes versus just one of them?
New proposal:
Let repository implementations like git have an additional enum:
uncommitted-policy: accept | fail
Remove fail-uncommitted from update-policy.
Rename fail-localchange to trust-clean
Rename noUpdate to trust-dirty
update-policy
: default
| trust-clean
| trust-dirty
| update
| replace;
;
> update
> perform an update (the merge-policy specifies how to handle the update).
>
> replace
> result is the same as removing everything and checking out from scratch
> (may be done more efficiently naturally if result is guaranteed to be
> the same).
>
Perhaps. But I think the main reason to do this is that you want to make sure that no optimization where made and that
everything indeed is from scratch so I'd rather document this enum as doing just that.
> (Possible addition to update-policy is to allow specification of a delay
> between update checks. This could speed up building if performed over
> and over again manually triggered in a local scenario. Proposal
> update-hourly, update-daily, or a separate attribute 'update-interval :
> 00:00:30', but this could be added by a repository impl).
>
Not in favor of that since it introduces a behavior that is hard to track/reproduce. If you want to control when updates
are made you usually have other reasons then intervals. Besides, most repositories today are relatively fast in
discovering that content is unchanged when an update is requested. I suggest we postpone this until we see that there's
a need for it.
> 3. what to do if there is a need to merge
> (A merge-policy can only be used if the update-policy is 'update'.)
>
> merge-policy
> : default
> | fail-uncommitted
> | fail-localchange
> | fail-merge
> | fail-conflicts
> | use-local
> | use-remote
> | accept-conflicts
> ;
>
> default
> This default only kicks in if main == update by default, or user changed
> setting to update for the others. May depend on use of branch/tag/main
> etc. but I am not sure if that is valid - seems reasonable with
> fail-conflicts as the default in all cases. (We don't need a "default"
> in that case).
>
I don't think we need this.
> fail-uncomitted, fail-localchange
> Fail if local content is 'dirty' (same rule as for update-policy) even
> if there is (potentially) no need to merge because the remote is
> unchanged. This is essentially the same as the corresponding
> specification for update-policy but with the difference that a merge is
> performed if not dirty. Rationale - I am supposed to run a clean build,
> reproducable by others, any changes are done by mistake.
>
This can be simplified if the handling of uncommitted is moved to a separate enum.
> fail-merge
> Fail if there is any need to merge (local/remote change) even if it can
> be done without manual intervention. i.e. accept clean updates, added
> and removed files but not anything requiring a merge. Rationale, I am
> not anticipating change in anything I am also changing.
>
> fail-conflicts
> Fail if merge results in conflicts requiring a manual-merge. Rationale,
> I don't want to clean up conflicting merges for everything all at once.
> (could be called fail-manual-merge).
>
> use-local
> If a merge results in conflict requiring manual-merge - keep the
> local/dirty copy. Rationale - I will deal with merges later, I want the
> stuff I am working on to build now with the latest except where others
> have fixed the old and bad and I am working on the next better solution.
>
> use-remote
> If a merge results in conflict requiring manual-merge - replace the
> local dirty copy with the remote. Rationale - oops, I must have been
> experimenting, my mistake - fix it.
>
> accept-conflicts
> Perform merge and if there are conflicts (requiring manual
> intervention), accept these (i.e. perform the merge), and let the build
> fail (or b3 for that matter, once the merge is completed). Rationale - I
> probably need my head examined, but this is what I want - I do not
> expect any conflicts and if they occur then I will fix them.
>
We introduce a new dimension here that might open up a new can of worms. 'accept-conflicts' will yield a workspace that
cannot be built so even if the repository succeeds, it actually fails to produce something that can be used by
subsequent build actions. Still, it isn't good enough to just fail the repository checkout since there might be other
repositories involved and it might be desirable for them to continue with a best effort until everything has been
checked out. Also, what should the subsequent invocations of the providers that use this repository do? Should they
proceed? What if the conflict touches files that have an effect on the transitive scope of the resolution? And here it
gets really complicated ...
In some sense, I'd say that accept/fail conflicts doesn't belong in b3. That kind of handling is useful in the IDE and
there it's very likely that the user just do 'Team' -> 'Update to Head' or similar in the explorer anyway and get all
the bells and whistles associated with conflict management through the UI.
I think I'm back to:
merge-policy { fail, use-local, use-remote, merge }
perhaps with one minor change:
merge-policy { fail, use-workspace, use-scm, merge }
It's easy to understand and it goes without saying that conflicts requiring manual intervention are failures. The
message is: If you want to really edit your conflicts, then use the UI tools that the SCM provider makes available.
The difference between fail-merge and fail-local is already handled by using trust-clean versus update. If you want to
fail on any local change regardless if a merge is needed or not, use trust-clean. If you want to fail local changes only
when a merge is needed, then use update + fail.
One case remains and that is that you want to fail when a non-conflicting merge is attempted on a local change but other
merges should succeed. Why would you ever want that? Besides, merges on files that have no local changes can only happen
when you check things out from a branch other then the one you currently have. So it boils down to applying some
change-set to a workspace with a desire to fail if that change-set cause non-conflicting merges with workspace changes.
No use-case for that IMO and such actions calls for a UI anyway.
- thomas
|
|
|
| Re: Repositories and Providers [message #539291 is a reply to message #539056] |
Thu, 10 June 2010 09:44 |
|
On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
> So, trying to sum up:
>
> 1. Change the "update-policy" to the two policies you suggested.
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
>
> I downside is that the enum's become keywords (and keep, replace, and
> merge are very likely to clash with other things). (The current solution
> was crafted so the enums all have "-" in them to reduce clashes).
> Suggestion; have all enums everywhere have a special leading character
> like '@'. (Yet another more complicated impl is to treat the enum values
> as IDs and have data conversion rules per enum - not sure how well that
> works though, I tried something similar earlier and it was not possible
> to return an EObject supposed to be part of the model built by the
> grammar this way).
> The only solution I can think of would use a reference to an Enum
> instead of containing one. This way I can use an ID, and have a scope
> provider that provides the enums to link to. (I use this technique to
> reference implicit variables like "request" in resolver expressions).
>
> Using @ is however much much simpler...
>
> proposal example:
> exists-policy : @keep;
>
Albeit uglier. I would much prefer if we could do without the @.
- thomas
|
|
|
| Use @ for enums? [message #539321 is a reply to message #539291] |
Thu, 10 June 2010 12:25 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
It is a lot more work if done as a feature containment, but a lot easier
if done as a link - but I have not tried a reference to an Enum instance.
As we are changing these enums frequently, if they were done with @,
there is just one place to change them.
But I will experiment a bit.
I changed the Subject BTW...
- henrik
On 6/10/10 11:44 AM, Thomas Hallgren wrote:
> On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
>> So, trying to sum up:
>>
>> 1. Change the "update-policy" to the two policies you suggested.
>>
>> exists-policy { fail, keep, replace, merge }
>> merge-policy { fail, use-local, use-remote, merge }
>>
>> I downside is that the enum's become keywords (and keep, replace, and
>> merge are very likely to clash with other things). (The current solution
>> was crafted so the enums all have "-" in them to reduce clashes).
>> Suggestion; have all enums everywhere have a special leading character
>> like '@'. (Yet another more complicated impl is to treat the enum values
>> as IDs and have data conversion rules per enum - not sure how well that
>> works though, I tried something similar earlier and it was not possible
>> to return an EObject supposed to be part of the model built by the
>> grammar this way).
>> The only solution I can think of would use a reference to an Enum
>> instead of containing one. This way I can use an ID, and have a scope
>> provider that provides the enums to link to. (I use this technique to
>> reference implicit variables like "request" in resolver expressions).
>>
>> Using @ is however much much simpler...
>>
>> proposal example:
>> exists-policy : @keep;
>>
>
> - thomas
|
|
|
| Re: Repositories and Providers [message #539369 is a reply to message #539264] |
Thu, 10 June 2010 14:28 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Almost there...
comments inline. But summary here:
on-empty-location: checkout | fail
update-policy
: default
| check-clean
| accept-dirty
| update
| merge
| replace;
;
conflict-policy
: use-local
| use-scm
| fail
;
But there are some unfinished discussions... and questions/issues to
address... - read on...
- henrik
On 6/10/10 10:35 AM, Thomas Hallgren wrote:
> On 06/10/2010 02:19 AM, Henrik Lindberg wrote:
>> 1. what to do when location is empty:
>>
>> on-empty-location: checkout | accept-empty | fail ;
>>
> What does empty mean?
I meant completely empty as in directory does not exist, or does not
contain anything. Essentially this provides a sort of "overlay". And if
it was empty, and accepted as such, this also means "no update" (and
essentially == 'skip this repo if it was not checked out by CI or
manually'). But as I said, I don't know if it is valuable to have such
variability. I can live without it.
The same can be achieved with logic in the providers and setting
properties to include/skip resolution from certain repositories. So,
lets keep it simple and not have this option in "on-empty..."
> I agree that
> on-empty-location: checkout | fail
> is clearer.
>
ok, deal.
>> update-policy
>> : default
>> | no-update
>> | fail-uncommitted
>> | fail-localchange
>> | update
>> | replace;
>> ;
>>
>> default
>> ...
>>
> head of any branch (not just main) = update.
> tag/timestamp/revision (i.e. fixed material) = trust-clean (see New
> proposal below)
>
ok.
>> no-update
>> trust whatever is there to be correct
>>
> I like 'trust' better because it declares what to do rather then what
> not to do.
>
ok. see new proposal further down.
>> fail-uncomitted, fail-localchange
>>...
>> We could use git specific terms, but then it is perhaps more confusing
>> when used with other types of repositories.
>>
> The distinction is needed but problem arise when it is expressed as
> multiple ways to fail instead of what to do on success. What if it does
> not fail? Do you update then? Or trust? Or replace even (since you may
> want to erase the checked out content and replace it with what you have
> in your local clone)? How do you express that you want to fail both on
> local changes and on uncommitted changes versus just one of them?
>
The combination as failing on committed local change, but not on dirty
workspace did not occur to me as a combination that made sense.
> New proposal:
>
> Let repository implementations like git have an additional enum:
>
> uncommitted-policy: accept | fail
>
I don't understand - "uncommitted" == "dirty in workspace", "committed"
== in local repo clone, "pushed" is "in the cloned repo". So,
"uncomitted-policy : accept " == trust-dirty ?!?
How about:
distribution-policy : accept-local-commit | fail ;
(only applies to Distributed SCMs).
>
> update-policy
> : default
> | trust-clean
> | trust-dirty
> | update
> | replace;
> ;
>
I don't like the "trust" word, how about "check-clean", "accept-dirty"
instead?
>> replace
>> result is the same as removing everything and checking out from scratch
>> (may be done more efficiently naturally if result is guaranteed to be
>> the same).
>>
> Perhaps. But I think the main reason to do this is that you want to make
> sure that no optimization where made and that everything indeed is from
> scratch so I'd rather document this enum as doing just that.
>
ok, fine.
>> (Possible addition to update-policy is to allow specification of a delay
>> between update checks. ...
>>
> Not in favor of that ...
>
ok.
>> 3. what to do if there is a need to merge
>> (A merge-policy can only be used if the update-policy is 'update'.)
>>
>> merge-policy
>> : default
>> | fail-uncommitted
>> | fail-localchange
>> | fail-merge
>> | fail-conflicts
>> | use-local
>> | use-remote
>> | accept-conflicts
>> ;
>>
>> default
> I don't think we need this.
>
ok.
>> fail-uncomitted, fail-localchange
>> ...
> This can be simplified if the handling of uncommitted is moved to a
> separate enum.
>
yes.
>> fail-merge
>> fail-conflicts
>> use-local
>> use-remote
>> accept-conflicts
> In some sense, I'd say that accept/fail conflicts doesn't belong in b3.
> That kind of handling is useful in the IDE and there it's very likely
> that the user just do 'Team' -> 'Update to Head' or similar in the
> explorer anyway and get all the bells and whistles associated with
> conflict management through the UI.
>
That works for me too.
> I think I'm back to:
> merge-policy { fail, use-local, use-remote, merge }
> perhaps with one minor change:
> merge-policy { fail, use-workspace, use-scm, merge }
>
ok, however...
this may not be enough if git is used and there are:
a) dirty stuff in workspace
b) local commits
c) changes in cloned repository
Don't know what really happens in this situation - I can imagine seeing
it as a progression, i.e. use workspace, throw away workspace, throw
away workspace and local commit... But I don't know if that is how you
think about it when using git. I assume it will merge what is in the
workspace and leave the local commit until the workspace change is
committed. So I wonder if there is a difference in practice?
The "use-workspace" is a misnomer in the special case that there are no
changes in the workspace, but there are locally committed changes -
"use-local" works better in that case, but "use-scm" seems ambiguous.
I am still struggling with the enums - don't find them intuitive.
merge-policy : fail; // means "fail-if-merge-is-required"
merge-policy : use-workspace; // means "merge, and if there is a
conflict use workspace"
merge-policy : use-scm; // means "merge, and if there is a conflict use
the scm"
merge-policy : merge; // means "merge and fail-if-there are conflicts"
"merge" is implied in all but the first. If we move "merge" to
update-policy, it becomes:
update-policy
: default
| check-clean
| accept-dirty
| update
| merge
| replace;
;
With the meaning that "update" only accepts clean updates (i.e. the same
as "update + merge-policy : fail").
merge-policy, is then all about conflicts, and can be renamed:
conflict-policy : use-local | use-scm | fail ;
> ...it goes without saying that conflicts
> requiring manual intervention are failures. The message is: If you want
> to really edit your conflicts, then use the UI tools that the SCM
> provider makes available.
>
ok, I accept that conflicting merges always lead to fail.
> The difference between fail-merge and fail-local is already handled by
> using trust-clean versus update. If you want to fail on any local change
> regardless if a merge is needed or not, use trust-clean. If you want to
> fail local changes only when a merge is needed, then use update + fail.
>
There is a difference, but it not an important use case.
I can live without having this distinction, and handle what I was
thinking about in the IDE - it is not a valid automated/headless/ci feature.
> One case remains and that is that you want to fail when a
> non-conflicting merge is attempted on a local change but other merges
> should succeed.
This is better done in the IDE with a UI, so just failing is better.
>
> - thomas
|
|
|
| Re: Repositories and Providers [message #539424 is a reply to message #539369] |
Thu, 10 June 2010 16:19 |
|
Some minors...
> How about:
> distribution-policy : accept-local-commit | fail ;
> (only applies to Distributed SCMs).
>
Yes, perhaps. But there are more things to consider when we discuss how to manage the clone. So much in fact, that I
feel it should be a separate topic altogether. Can we table that aspect of the discussion for now and concentrate on how
the repository (local clone in case of git or similar), relates to the workspace? I think it might be easier if we treat
the clones relationship to its source (or sources) as a separate thing with its own set of enums.
>>
>> update-policy
>> : default
>> | trust-clean
>> | trust-dirty
>> | update
>> | replace;
>> ;
>>
> I don't like the "trust" word, how about "check-clean", "accept-dirty"
> instead?
>
I don't like the check word. Ok, so you check. Then what? With the new meaning of update and merge, you must first check
if the workspace is clean. The merge however is an update of dirty material. What about:
update-policy
: default
| keep-clean
| keep-dirty
| update-clean
| update-dirty
| replace
;
> this may not be enough if git is used and there are:
> a) dirty stuff in workspace
> b) local commits
> c) changes in cloned repository
>
It is enough if we treat clone management separately.
> The "use-workspace" is a misnomer in the special case that there are no
> changes in the workspace, but there are locally committed changes -
> "use-local" works better in that case, but "use-scm" seems ambiguous.
>
The scm ambiguity is resolved if we treat clone management separately. The word local however, is still ambiguous (local
clone versus local workspace changes). Since this is now a pure conflict-policy, the use-workspace is not a misnomer at
all. So I'm back to workspace versus scm.
> With the meaning that "update" only accepts clean updates (i.e. the same
> as "update + merge-policy : fail").
>
> merge-policy, is then all about conflicts, and can be renamed:
>
> conflict-policy : use-local | use-scm | fail ;
>
Agreed. The separation is cleaner. Not sure if it's better to rename to conflict-policy since it's the policy that is
used for update-policy 'merge'. It feels natural to call it 'merge-policy'. But I can buy both.
- thomas
|
|
|
| Re: Use @ for enums? [message #539438 is a reply to message #539321] |
Thu, 10 June 2010 17:40 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
My experminet worked out just fine, the enum values does not have to be
keywords, but I need them to use '_' instead of '-' as separators (I
could use '.'), or we could use camelCase.
The amound of work is not unreasonable - instead of just declaring the
enum in the grammar:
- write a value converter (10 lines or so)
- write a content assist proposal provider (10 lines or so)
- write syntactic coloring (as they are no longer keyword tokens)
I think we are going to have several enums that needs this - and I like
to keep the code for one enum type together, so will do a bit more work
to make it easier to maintain (or right now to change them a couple of
times :)
- henrik
On 6/10/10 2:25 PM, Henrik Lindberg wrote:
> It is a lot more work if done as a feature containment, but a lot easier
> if done as a link - but I have not tried a reference to an Enum instance.
>
> As we are changing these enums frequently, if they were done with @,
> there is just one place to change them.
>
> But I will experiment a bit.
>
> I changed the Subject BTW...
> - henrik
>
> On 6/10/10 11:44 AM, Thomas Hallgren wrote:
>> On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
>>> So, trying to sum up:
>>>
>>> 1. Change the "update-policy" to the two policies you suggested.
>>>
>>> exists-policy { fail, keep, replace, merge }
>>> merge-policy { fail, use-local, use-remote, merge }
>>>
>>> I downside is that the enum's become keywords (and keep, replace, and
>>> merge are very likely to clash with other things). (The current solution
>>> was crafted so the enums all have "-" in them to reduce clashes).
>>> Suggestion; have all enums everywhere have a special leading character
>>> like '@'. (Yet another more complicated impl is to treat the enum values
>>> as IDs and have data conversion rules per enum - not sure how well that
>>> works though, I tried something similar earlier and it was not possible
>>> to return an EObject supposed to be part of the model built by the
>>> grammar this way).
>>> The only solution I can think of would use a reference to an Enum
>>> instead of containing one. This way I can use an ID, and have a scope
>>> provider that provides the enums to link to. (I use this technique to
>>> reference implicit variables like "request" in resolver expressions).
>>>
>>> Using @ is however much much simpler...
>>>
>>> proposal example:
>>> exists-policy : @keep;
>>>
>
>>
>> - thomas
>
|
|
|
| Re: Repositories and Providers [message #539464 is a reply to message #539424] |
Thu, 10 June 2010 20:17 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I think we are trying to squeeze too many dimensions into the same enum,
and that is why it feels confusing. Better to break them apart:
- do I want a checkout or not
- is it ok to build dirty material or not
- do I want to (try) to stay current
Grammar:
'policy' '{'
('checkout' ':' boolean ';')?
(('accept-dirty' ':' boolean ';')?
('update' ':' boolean; )?
('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
) | ('replace' ';' )? )?
'}'
e.g.
// checkout if empty, when building it is ok if state is dirty, perform
// update, and it is ok if this leaves workspace in dirty state.
// (A typical developer policy)
policy {
checkout : true;
accept-dirty : true ;
update : true;
on-merge-conflict : use-workspace;
}
// ci server checked out build
policy {
checkout : false ;
accept-dirty : false;
update : false;
}
// ci, or manually checked out, keep it updated (but clean)
policy {
checkout : false;
accept-dirty : false;
update : true ;
}
Defaults are:
no policy = defaults determined by branch point type
policy stated - defaults determined by branch point type, overridden by
what is stated.
i.e. if update is performed and the result is dirty content (because of
merge), the "on-dirty-content" will dictate what to do. The state is the
same as if user manually made the same state (well almost, as the scm
may remember stuff).
I think this is much cleaner.
- henrik
|
|
|
| Re: Repositories and Providers [message #539471 is a reply to message #539464] |
Thu, 10 June 2010 21:19 |
|
This is excellent.
- thomas
On 06/10/2010 10:17 PM, Henrik Lindberg wrote:
> I think we are trying to squeeze too many dimensions into the same enum,
> and that is why it feels confusing. Better to break them apart:
>
> - do I want a checkout or not
> - is it ok to build dirty material or not
> - do I want to (try) to stay current
>
> Grammar:
>
> 'policy' '{'
> ('checkout' ':' boolean ';')?
> (('accept-dirty' ':' boolean ';')?
> ('update' ':' boolean; )?
> ('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
> ) | ('replace' ';' )? )?
> '}'
>
> e.g.
> // checkout if empty, when building it is ok if state is dirty, perform
> // update, and it is ok if this leaves workspace in dirty state.
> // (A typical developer policy)
> policy {
> checkout : true;
> accept-dirty : true ;
> update : true;
> on-merge-conflict : use-workspace;
> }
>
> // ci server checked out build
> policy {
> checkout : false ;
> accept-dirty : false;
> update : false;
> }
>
> // ci, or manually checked out, keep it updated (but clean)
> policy {
> checkout : false;
> accept-dirty : false;
> update : true ;
> }
>
> Defaults are:
> no policy = defaults determined by branch point type
> policy stated - defaults determined by branch point type, overridden by
> what is stated.
>
>
> i.e. if update is performed and the result is dirty content (because of
> merge), the "on-dirty-content" will dictate what to do. The state is the
> same as if user manually made the same state (well almost, as the scm
> may remember stuff).
>
> I think this is much cleaner.
>
> - henrik
|
|
|
| Re: Repositories and Providers [message #539762 is a reply to message #539471] |
Sat, 12 June 2010 17:03 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Checked in an implementation of this to 'head', with a small variation:
I used the following keywords to avoid conflicts:
- checkout-content
- update-content
- replace-content
Regards
- henrik
On 6/10/10 11:19 PM, Thomas Hallgren wrote:
> This is excellent.
>
> - thomas
>
> On 06/10/2010 10:17 PM, Henrik Lindberg wrote:
>> I think we are trying to squeeze too many dimensions into the same enum,
>> and that is why it feels confusing. Better to break them apart:
>>
>> - do I want a checkout or not
>> - is it ok to build dirty material or not
>> - do I want to (try) to stay current
>>
>> Grammar:
>>
>> 'policy' '{'
>> ('checkout' ':' boolean ';')?
>> (('accept-dirty' ':' boolean ';')?
>> ('update' ':' boolean; )?
>> ('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
>> ) | ('replace' ';' )? )?
>> '}'
>>
>> e.g.
>> // checkout if empty, when building it is ok if state is dirty, perform
>> // update, and it is ok if this leaves workspace in dirty state.
>> // (A typical developer policy)
>> policy {
>> checkout : true;
>> accept-dirty : true ;
>> update : true;
>> on-merge-conflict : use-workspace;
>> }
>>
>> // ci server checked out build
>> policy {
>> checkout : false ;
>> accept-dirty : false;
>> update : false;
>> }
>>
>> // ci, or manually checked out, keep it updated (but clean)
>> policy {
>> checkout : false;
>> accept-dirty : false;
>> update : true ;
>> }
>>
>> Defaults are:
>> no policy = defaults determined by branch point type
>> policy stated - defaults determined by branch point type, overridden by
>> what is stated.
>>
>>
>> i.e. if update is performed and the result is dirty content (because of
>> merge), the "on-dirty-content" will dictate what to do. The state is the
>> same as if user manually made the same state (well almost, as the scm
>> may remember stuff).
>>
>> I think this is much cleaner.
>>
>> - henrik
>
|
|
|
| Re: Repositories and Providers [message #608341 is a reply to message #538592] |
Tue, 08 June 2010 22:11 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Hi,
In b3, what we decided so far is that the syntax for repository branches
has a specification of what should be excluded from / included in the
resulting repository "view". Exclude / Include is a list of verbatim
paths, or patterns.
We also have a specification of "update-policy" that can be one of
"default" (i.e. default for the combination of repository type and type
or branch), "fail-modified", "keep-modified", "merge-modified",
"no-update", and "replace-modified". Don't know if these cover all of
the cases you listed, but this is the list we came up with when you and
I designed this.
All of this is pushed down to the repository implementation(s). They
will later be asked to produce BuildUnit instances given a request for a
capability.
One possibility is to inject different implementations in the CI
scenario - they would know that they are never supposed to check
something out. The specifications in the b3 script then acts as sanity
checks - did the CI server actually check out what is required. Such a
solution allows the same script to run anywhere and it then performs the
checkout as an integrated part.
The CI "checkout" could likewise read the b3 script, perform the
checkout as directed by the script, and then finish (and let other jobs
run with reuse of the checked out material).
An elaborate CI setup could also combine the requirements on
repositories from multiple b3 scripts, perhaps merge the requirements to
come up with a "sum of everything that needs to be checked out".
Regards
- henrik
On 6/8/10 9:16 AM, Thomas Hallgren wrote:
> I've done some thinking about the scenarios that we want to support with
> respect to SCM's.
>
> 1. CI performs check out of a root folder. b3 providers performs
> discovery below the root.
>
> 2. Like #1 but b3 performs the check out.
>
> 3. Root is huge and it is not desirable to check it out. Provides use
> team project sets, maps, or Buckminster style discovery to make a more
> precise check out.
>
> We need to be able to specify what the expectations are between b3 and
> the CI system, and between the repositories and the providers. #1 and #2
> is of course very similar. It boils down to that b3 finds something that
> is already checked out, i.e. an policy:
>
> mustExist:
> b3 should make no attempt to check out and fail if it's not present.
>
> trusted:
> Trust what is there (i.e. no update) but perform a new check out if it's
> missing.
>
> update:
> Update what is there or perform a new check out if it's missing.
>
> overwrite:
> Remove what is there and then perform a new check out.
>
> mustNotExist:
> Fail if something is found at the intended check-out location.
>
> The question is, how do we combine this with #3? Do we simply add a
> boolean flag such as "providerCheckout" and then push the policy down to
> each provider?
>
> - thomas
>
|
|
|
| Re: Repositories and Providers [message #608343 is a reply to message #538886] |
Wed, 09 June 2010 06:08 |
|
On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
> Hi,
> In b3, what we decided so far is that the syntax for repository branches
> has a specification of what should be excluded from / included in the
> resulting repository "view". Exclude / Include is a list of verbatim
> paths, or patterns.
>
That's all good, but it doesn't really cover the case when you want to use an existing "team project set" or similar in
a provider. They have been around for a long time and are very commonly used. The same is true for the PDE 'maps'. We
could of course require that such files are abandoned in favor of the b3 syntax but that might make adoption more painful.
> We also have a specification of "update-policy" that can be one of
> "default" (i.e. default for the combination of repository type and type
> or branch), "fail-modified", "keep-modified", "merge-modified",
> "no-update", and "replace-modified". Don't know if these cover all of
> the cases you listed, but this is the list we came up with when you and
> I designed this.
>
Right. But these names seem to describe what to do when a merge-conflict arises and can really only be used with what I
refer to as 'update'. What I'm after is what b3 should do when it encounters already existing material (or not) at the
intended location. This is how I would map the current names to what I'm after:
fail-modified = update (and fail on need to merge)
keep-modified = update (and keep working copy on need to merge)
merge-modified = update (and merge if needed)
replace-modified = update (overwrite local copy instead of merge)
no-update = trusted
??? = overwrite
??? = mustExist
??? = mustNotExist
So perhaps adding the last three to the list would be enough. On the other hand, many names in an enum might make it
hard to understand and a solution could be to have two enums. Something like this:
exists-policy { fail, use, merge, overwrite }
merge-policy { fail, keep, merge, replace }
There's still nothing that enables me to declare that I expect the material to be present. I.e. if it's missing, should
I do a initial check out or should I fail?
initial-checkout true/false
(defaults to true)
exists-policy = fail cannot be combined with initial-checkout = false.
- thomas
> All of this is pushed down to the repository implementation(s). They
> will later be asked to produce BuildUnit instances given a request for a
> capability.
>
> One possibility is to inject different implementations in the CI
> scenario - they would know that they are never supposed to check
> something out. The specifications in the b3 script then acts as sanity
> checks - did the CI server actually check out what is required. Such a
> solution allows the same script to run anywhere and it then performs the
> checkout as an integrated part.
>
Sure, that's possible but I will not be able to use the exact same script in the IDE and the CI.
- thomas
|
|
|
| Re: Repositories and Providers [message #608344 is a reply to message #538920] |
Wed, 09 June 2010 07:05 |
|
On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
> exists-policy { fail, use, merge, overwrite }
> merge-policy { fail, keep, merge, replace }
>
Thinking more about the names. This might be less confusing:
exists-policy { fail, keep, replace, merge }
merge-policy { fail, use-local, use-remote, merge }
- thomas
|
|
| |
| Re: Repositories and Providers [message #608347 is a reply to message #538934] |
Wed, 09 June 2010 10:21 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I like that proposal bettet. IT is also vetter Tham The current "update
policy".
- henrik
Thomas Hallgren <thomas@tada.se> wrote:
> On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
>> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
>> exists-policy { fail, use, merge, overwrite }
>> merge-policy { fail, keep, merge, replace }
>>
>
> Thinking more about the names. This might be less confusing:
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
- thomas
--
- henrik
|
|
|
| Re: Repositories and Providers [message #608349 is a reply to message #539006] |
Wed, 09 June 2010 10:25 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Doh!
Reminder: remember to switch to English auto correct when typing in
English.
I think the message came through tnough :)
- henrik
Henrik Lindberg <henrik.lindberg@cloudsmith.com> wrote:
> I like that proposal bettet. IT is also vetter Tham The current
> "update
> policy".
> - henrik
>
> Thomas Hallgren <thomas@tada.se> wrote:
>> On 06/09/2010 08:08 AM, Thomas Hallgren wrote:
>>> On 06/09/2010 12:11 AM, Henrik Lindberg wrote:
>>> exists-policy { fail, use, merge, overwrite }
>>> merge-policy { fail, keep, merge, replace }
>>>
>>
>> Thinking more about the names. This might be less confusing:
>>
>> exists-policy { fail, keep, replace, merge }
>> merge-policy { fail, use-local, use-remote, merge }
- thomas
--
- henrik
|
|
|
| Re: Repositories and Providers [message #608351 is a reply to message #539005] |
Wed, 09 June 2010 11:18 |
|
On 06/09/2010 12:21 PM, Henrik Lindberg wrote:
>> Sure, that's possible but I will not be able to use the exact same
>> script in the IDE and the CI.
>
> Interested in why why that is? Seems like a big benefit to only maintain
> repository info in one place. If it is not as simple as using the same
> policy for all repos if the same type, then we should consider
> advicing/weaving per repository.
>
What I meant was that if injections are needed in the CI scenario, then those needs to be maintained somewhere. Which I
guess, calls for a different b3 script (or different parameters that controls the injection somehow). No big deal.
> Regarding .psf and buckminster rmaps; think these are listed as
> repositories of "psf" and "rmap" type.
Yes, that's an elegant solution. Under the covers, it will just delegate to whatever repositories that are referenced.
> That is probably enough, but does
> not work if there is a need to specify different policies for different
> things referenced by these external maps.
>
Nobody has ever expressed a need to do that so that won't be a problem. Should someone like to do it, well, then it's
time to abandon the psf and instead use the b3 syntax.
- thomas
|
|
|
| Re: Repositories and Providers [message #608353 is a reply to message #539026] |
Wed, 09 June 2010 13:26 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
So, trying to sum up:
1. Change the "update-policy" to the two policies you suggested.
exists-policy { fail, keep, replace, merge }
merge-policy { fail, use-local, use-remote, merge }
I downside is that the enum's become keywords (and keep, replace, and
merge are very likely to clash with other things). (The current solution
was crafted so the enums all have "-" in them to reduce clashes).
Suggestion; have all enums everywhere have a special leading character
like '@'. (Yet another more complicated impl is to treat the enum values
as IDs and have data conversion rules per enum - not sure how well that
works though, I tried something similar earlier and it was not possible
to return an EObject supposed to be part of the model built by the
grammar this way).
The only solution I can think of would use a reference to an Enum
instead of containing one. This way I can use an ID, and have a scope
provider that provides the enums to link to. (I use this technique to
reference implicit variables like "request" in resolver expressions).
Using @ is however much much simpler...
proposal example:
exists-policy : @keep;
2. psf and buckminster maps
Handled as repository types in b3 (i.e. nothings special needed in b3,
it is up to the impl of these). Any advanced handling of these is also
up to the respective impl (i.e. parameters passed from the script etc).
3. Regarding CI, and using different policies.
I think the best solution is to have the list of repositories in a
separate .b3 file (i.e. separate from the resolution specificaion). It
is not only the policies that need to differ, it is also things like
where to place the local/mirrored repositories - the list of things to
change is likely just as long/complicated as the original list.
Complicating the repository declaration with conditionals
(when(${on.ci.server}) ... etc) is also messy).
The resolution part needs to know about available repositories (they
refer to the repositories via names) - if the specification is imported,
it is possible to get different compatible configurations by modifying
the "classpath". There is currently only limited support for imports in
the b3 impl, so until that is done, different b3 scripts will need to be
used when different policies are wanted.
- henrik
|
|
|
| Re: Repositories and Providers [message #608354 is a reply to message #539056] |
Thu, 10 June 2010 00:19 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Hi,
I looked at this again, and I think it is further simplified if thought
of as three separate things:
1. what to do if there is nothing at all
2. what to do if there is something there
3. what to do if there is a need to merge
I also believe that we did not cover all the cases.
1. what to do when location is empty:
on-empty-location: checkout | accept-empty | fail ;
The default is "checkout".
checkout
Clone or checkout the "remote" repository.
accept-empty
Needed if we want to have repositories that may be empty - i.e. not
return anything if nothing was checked out. (Perhaps, in situations
where some big checkout of everything may be done by the ci server, if
it is not there - fine we will use the other repositories).
Not sure if this option is needed.
fail
It is expected that someone else has performed a checkout to the location.
2. what to do if there is something there
(and it was not put there by on-empty-location : checkout in the same
invocation).
update-policy
: default
| no-update
| fail-uncommitted
| fail-localchange
| update
| replace;
;
default
Depends on use of branch/tag/main etc. (And is the default if nothing is
stated). Don't remember exactly what we decided, but something like main
= update, tag/timestamp = fail-localchange.
no-update
trust whatever is there to be correct
fail-uncomitted, fail-localchange
if the content is dirty - no update, just check and fail if dirty.
In git there are actually two separate dirty cases - if there is
uncommitted changes (i.e. in workspace only), or changes not pushed to
the cloned repository. So we may need two separate states:
fail-uncommitted, fail-localchange (where fail-localchange is the same
as fail-uncommitted if the repo does not distinguish between them).
We could use git specific terms, but then it is perhaps more confusing
when used with other types of repositories.
update
perform an update (the merge-policy specifies how to handle the update).
replace
result is the same as removing everything and checking out from scratch
(may be done more efficiently naturally if result is guaranteed to be
the same).
(Possible addition to update-policy is to allow specification of a delay
between update checks. This could speed up building if performed over
and over again manually triggered in a local scenario. Proposal
update-hourly, update-daily, or a separate attribute 'update-interval :
00:00:30', but this could be added by a repository impl).
3. what to do if there is a need to merge
(A merge-policy can only be used if the update-policy is 'update'.)
merge-policy
: default
| fail-uncommitted
| fail-localchange
| fail-merge
| fail-conflicts
| use-local
| use-remote
| accept-conflicts
;
default
This default only kicks in if main == update by default, or user changed
setting to update for the others. May depend on use of branch/tag/main
etc. but I am not sure if that is valid - seems reasonable with
fail-conflicts as the default in all cases. (We don't need a "default"
in that case).
fail-uncomitted, fail-localchange
Fail if local content is 'dirty' (same rule as for update-policy) even
if there is (potentially) no need to merge because the remote is
unchanged. This is essentially the same as the corresponding
specification for update-policy but with the difference that a merge is
performed if not dirty. Rationale - I am supposed to run a clean build,
reproducable by others, any changes are done by mistake.
fail-merge
Fail if there is any need to merge (local/remote change) even if it can
be done without manual intervention. i.e. accept clean updates, added
and removed files but not anything requiring a merge. Rationale, I am
not anticipating change in anything I am also changing.
fail-conflicts
Fail if merge results in conflicts requiring a manual-merge. Rationale,
I don't want to clean up conflicting merges for everything all at once.
(could be called fail-manual-merge).
use-local
If a merge results in conflict requiring manual-merge - keep the
local/dirty copy. Rationale - I will deal with merges later, I want the
stuff I am working on to build now with the latest except where others
have fixed the old and bad and I am working on the next better solution.
use-remote
If a merge results in conflict requiring manual-merge - replace the
local dirty copy with the remote. Rationale - oops, I must have been
experimenting, my mistake - fix it.
accept-conflicts
Perform merge and if there are conflicts (requiring manual
intervention), accept these (i.e. perform the merge), and let the build
fail (or b3 for that matter, once the merge is completed). Rationale - I
probably need my head examined, but this is what I want - I do not
expect any conflicts and if they occur then I will fix them.
- henrik
On 6/9/10 3:26 PM, Henrik Lindberg wrote:
> So, trying to sum up:
>
> 1. Change the "update-policy" to the two policies you suggested.
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
>
> I downside is that the enum's become keywords (and keep, replace, and
> merge are very likely to clash with other things). (The current solution
> was crafted so the enums all have "-" in them to reduce clashes).
> Suggestion; have all enums everywhere have a special leading character
> like '@'. (Yet another more complicated impl is to treat the enum values
> as IDs and have data conversion rules per enum - not sure how well that
> works though, I tried something similar earlier and it was not possible
> to return an EObject supposed to be part of the model built by the
> grammar this way).
> The only solution I can think of would use a reference to an Enum
> instead of containing one. This way I can use an ID, and have a scope
> provider that provides the enums to link to. (I use this technique to
> reference implicit variables like "request" in resolver expressions).
>
> Using @ is however much much simpler...
>
> proposal example:
> exists-policy : @keep;
>
>
> 2. psf and buckminster maps
>
> Handled as repository types in b3 (i.e. nothings special needed in b3,
> it is up to the impl of these). Any advanced handling of these is also
> up to the respective impl (i.e. parameters passed from the script etc).
>
> 3. Regarding CI, and using different policies.
>
> I think the best solution is to have the list of repositories in a
> separate .b3 file (i.e. separate from the resolution specificaion). It
> is not only the policies that need to differ, it is also things like
> where to place the local/mirrored repositories - the list of things to
> change is likely just as long/complicated as the original list.
> Complicating the repository declaration with conditionals
> (when(${on.ci.server}) ... etc) is also messy).
>
> The resolution part needs to know about available repositories (they
> refer to the repositories via names) - if the specification is imported,
> it is possible to get different compatible configurations by modifying
> the "classpath". There is currently only limited support for imports in
> the b3 impl, so until that is done, different b3 scripts will need to be
> used when different policies are wanted.
>
> - henrik
|
|
|
| Re: Repositories and Providers [message #608356 is a reply to message #539210] |
Thu, 10 June 2010 08:35 |
|
Comments inline.
On 06/10/2010 02:19 AM, Henrik Lindberg wrote:
> Hi,
> I looked at this again, and I think it is further simplified if thought
> of as three separate things:
>
> 1. what to do if there is nothing at all
> 2. what to do if there is something there
> 3. what to do if there is a need to merge
>
> I also believe that we did not cover all the cases.
>
> 1. what to do when location is empty:
>
> on-empty-location: checkout | accept-empty | fail ;
>
> The default is "checkout".
>
> checkout
> Clone or checkout the "remote" repository.
>
> accept-empty
> Needed if we want to have repositories that may be empty - i.e. not
> return anything if nothing was checked out. (Perhaps, in situations
> where some big checkout of everything may be done by the ci server, if
> it is not there - fine we will use the other repositories).
> Not sure if this option is needed.
>
I think this is confusing and not really needed. Confusing because it implies a checkout with a subsequent control
rather then one simple directive. Not really needed because IMO, it's covered by the 'checkout'. If it proceeds without
errors and the result is an empty repository, well, then that's the current state of things. We might consider some kind
of validation that asserts that a checkout gives some expected result of course, but that's a different thing altogether.
What does empty mean? Is it physically empty or a folder with SCM meta-data that indicates no content? In my view, this
enum should be about the former. I.e. "what to do if there is nothing at all" is to be interpreted as "what to do if the
location is physically empty" (hence my boolean 'initial-checkout').
I agree that
on-empty-location: checkout | fail
is clearer.
> fail
> It is expected that someone else has performed a checkout to the location.
>
> 2. what to do if there is something there
> (and it was not put there by on-empty-location : checkout in the same
> invocation).
>
> update-policy
> : default
> | no-update
> | fail-uncommitted
> | fail-localchange
> | update
> | replace;
> ;
>
> default
> Depends on use of branch/tag/main etc. (And is the default if nothing is
> stated). Don't remember exactly what we decided, but something like main
> = update, tag/timestamp = fail-localchange.
>
head of any branch (not just main) = update.
tag/timestamp/revision (i.e. fixed material) = trust-clean (see New proposal below)
> no-update
> trust whatever is there to be correct
>
I like 'trust' better because it declares what to do rather then what not to do.
> fail-uncomitted, fail-localchange
> if the content is dirty - no update, just check and fail if dirty.
> In git there are actually two separate dirty cases - if there is
> uncommitted changes (i.e. in workspace only), or changes not pushed to
> the cloned repository. So we may need two separate states:
> fail-uncommitted, fail-localchange (where fail-localchange is the same
> as fail-uncommitted if the repo does not distinguish between them).
>
> We could use git specific terms, but then it is perhaps more confusing
> when used with other types of repositories.
>
The distinction is needed but problem arise when it is expressed as multiple ways to fail instead of what to do on
success. What if it does not fail? Do you update then? Or trust? Or replace even (since you may want to erase the
checked out content and replace it with what you have in your local clone)? How do you express that you want to fail
both on local changes and on uncommitted changes versus just one of them?
New proposal:
Let repository implementations like git have an additional enum:
uncommitted-policy: accept | fail
Remove fail-uncommitted from update-policy.
Rename fail-localchange to trust-clean
Rename noUpdate to trust-dirty
update-policy
: default
| trust-clean
| trust-dirty
| update
| replace;
;
> update
> perform an update (the merge-policy specifies how to handle the update).
>
> replace
> result is the same as removing everything and checking out from scratch
> (may be done more efficiently naturally if result is guaranteed to be
> the same).
>
Perhaps. But I think the main reason to do this is that you want to make sure that no optimization where made and that
everything indeed is from scratch so I'd rather document this enum as doing just that.
> (Possible addition to update-policy is to allow specification of a delay
> between update checks. This could speed up building if performed over
> and over again manually triggered in a local scenario. Proposal
> update-hourly, update-daily, or a separate attribute 'update-interval :
> 00:00:30', but this could be added by a repository impl).
>
Not in favor of that since it introduces a behavior that is hard to track/reproduce. If you want to control when updates
are made you usually have other reasons then intervals. Besides, most repositories today are relatively fast in
discovering that content is unchanged when an update is requested. I suggest we postpone this until we see that there's
a need for it.
> 3. what to do if there is a need to merge
> (A merge-policy can only be used if the update-policy is 'update'.)
>
> merge-policy
> : default
> | fail-uncommitted
> | fail-localchange
> | fail-merge
> | fail-conflicts
> | use-local
> | use-remote
> | accept-conflicts
> ;
>
> default
> This default only kicks in if main == update by default, or user changed
> setting to update for the others. May depend on use of branch/tag/main
> etc. but I am not sure if that is valid - seems reasonable with
> fail-conflicts as the default in all cases. (We don't need a "default"
> in that case).
>
I don't think we need this.
> fail-uncomitted, fail-localchange
> Fail if local content is 'dirty' (same rule as for update-policy) even
> if there is (potentially) no need to merge because the remote is
> unchanged. This is essentially the same as the corresponding
> specification for update-policy but with the difference that a merge is
> performed if not dirty. Rationale - I am supposed to run a clean build,
> reproducable by others, any changes are done by mistake.
>
This can be simplified if the handling of uncommitted is moved to a separate enum.
> fail-merge
> Fail if there is any need to merge (local/remote change) even if it can
> be done without manual intervention. i.e. accept clean updates, added
> and removed files but not anything requiring a merge. Rationale, I am
> not anticipating change in anything I am also changing.
>
> fail-conflicts
> Fail if merge results in conflicts requiring a manual-merge. Rationale,
> I don't want to clean up conflicting merges for everything all at once.
> (could be called fail-manual-merge).
>
> use-local
> If a merge results in conflict requiring manual-merge - keep the
> local/dirty copy. Rationale - I will deal with merges later, I want the
> stuff I am working on to build now with the latest except where others
> have fixed the old and bad and I am working on the next better solution.
>
> use-remote
> If a merge results in conflict requiring manual-merge - replace the
> local dirty copy with the remote. Rationale - oops, I must have been
> experimenting, my mistake - fix it.
>
> accept-conflicts
> Perform merge and if there are conflicts (requiring manual
> intervention), accept these (i.e. perform the merge), and let the build
> fail (or b3 for that matter, once the merge is completed). Rationale - I
> probably need my head examined, but this is what I want - I do not
> expect any conflicts and if they occur then I will fix them.
>
We introduce a new dimension here that might open up a new can of worms. 'accept-conflicts' will yield a workspace that
cannot be built so even if the repository succeeds, it actually fails to produce something that can be used by
subsequent build actions. Still, it isn't good enough to just fail the repository checkout since there might be other
repositories involved and it might be desirable for them to continue with a best effort until everything has been
checked out. Also, what should the subsequent invocations of the providers that use this repository do? Should they
proceed? What if the conflict touches files that have an effect on the transitive scope of the resolution? And here it
gets really complicated ...
In some sense, I'd say that accept/fail conflicts doesn't belong in b3. That kind of handling is useful in the IDE and
there it's very likely that the user just do 'Team' -> 'Update to Head' or similar in the explorer anyway and get all
the bells and whistles associated with conflict management through the UI.
I think I'm back to:
merge-policy { fail, use-local, use-remote, merge }
perhaps with one minor change:
merge-policy { fail, use-workspace, use-scm, merge }
It's easy to understand and it goes without saying that conflicts requiring manual intervention are failures. The
message is: If you want to really edit your conflicts, then use the UI tools that the SCM provider makes available.
The difference between fail-merge and fail-local is already handled by using trust-clean versus update. If you want to
fail on any local change regardless if a merge is needed or not, use trust-clean. If you want to fail local changes only
when a merge is needed, then use update + fail.
One case remains and that is that you want to fail when a non-conflicting merge is attempted on a local change but other
merges should succeed. Why would you ever want that? Besides, merges on files that have no local changes can only happen
when you check things out from a branch other then the one you currently have. So it boils down to applying some
change-set to a workspace with a desire to fail if that change-set cause non-conflicting merges with workspace changes.
No use-case for that IMO and such actions calls for a UI anyway.
- thomas
|
|
|
| Re: Repositories and Providers [message #608358 is a reply to message #539056] |
Thu, 10 June 2010 09:44 |
|
On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
> So, trying to sum up:
>
> 1. Change the "update-policy" to the two policies you suggested.
>
> exists-policy { fail, keep, replace, merge }
> merge-policy { fail, use-local, use-remote, merge }
>
> I downside is that the enum's become keywords (and keep, replace, and
> merge are very likely to clash with other things). (The current solution
> was crafted so the enums all have "-" in them to reduce clashes).
> Suggestion; have all enums everywhere have a special leading character
> like '@'. (Yet another more complicated impl is to treat the enum values
> as IDs and have data conversion rules per enum - not sure how well that
> works though, I tried something similar earlier and it was not possible
> to return an EObject supposed to be part of the model built by the
> grammar this way).
> The only solution I can think of would use a reference to an Enum
> instead of containing one. This way I can use an ID, and have a scope
> provider that provides the enums to link to. (I use this technique to
> reference implicit variables like "request" in resolver expressions).
>
> Using @ is however much much simpler...
>
> proposal example:
> exists-policy : @keep;
>
Albeit uglier. I would much prefer if we could do without the @.
- thomas
|
|
|
| Use @ for enums? [message #608361 is a reply to message #539291] |
Thu, 10 June 2010 12:25 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
It is a lot more work if done as a feature containment, but a lot easier
if done as a link - but I have not tried a reference to an Enum instance.
As we are changing these enums frequently, if they were done with @,
there is just one place to change them.
But I will experiment a bit.
I changed the Subject BTW...
- henrik
On 6/10/10 11:44 AM, Thomas Hallgren wrote:
> On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
>> So, trying to sum up:
>>
>> 1. Change the "update-policy" to the two policies you suggested.
>>
>> exists-policy { fail, keep, replace, merge }
>> merge-policy { fail, use-local, use-remote, merge }
>>
>> I downside is that the enum's become keywords (and keep, replace, and
>> merge are very likely to clash with other things). (The current solution
>> was crafted so the enums all have "-" in them to reduce clashes).
>> Suggestion; have all enums everywhere have a special leading character
>> like '@'. (Yet another more complicated impl is to treat the enum values
>> as IDs and have data conversion rules per enum - not sure how well that
>> works though, I tried something similar earlier and it was not possible
>> to return an EObject supposed to be part of the model built by the
>> grammar this way).
>> The only solution I can think of would use a reference to an Enum
>> instead of containing one. This way I can use an ID, and have a scope
>> provider that provides the enums to link to. (I use this technique to
>> reference implicit variables like "request" in resolver expressions).
>>
>> Using @ is however much much simpler...
>>
>> proposal example:
>> exists-policy : @keep;
>>
>
> - thomas
|
|
|
| Re: Repositories and Providers [message #608362 is a reply to message #539264] |
Thu, 10 June 2010 14:28 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Almost there...
comments inline. But summary here:
on-empty-location: checkout | fail
update-policy
: default
| check-clean
| accept-dirty
| update
| merge
| replace;
;
conflict-policy
: use-local
| use-scm
| fail
;
But there are some unfinished discussions... and questions/issues to
address... - read on...
- henrik
On 6/10/10 10:35 AM, Thomas Hallgren wrote:
> On 06/10/2010 02:19 AM, Henrik Lindberg wrote:
>> 1. what to do when location is empty:
>>
>> on-empty-location: checkout | accept-empty | fail ;
>>
> What does empty mean?
I meant completely empty as in directory does not exist, or does not
contain anything. Essentially this provides a sort of "overlay". And if
it was empty, and accepted as such, this also means "no update" (and
essentially == 'skip this repo if it was not checked out by CI or
manually'). But as I said, I don't know if it is valuable to have such
variability. I can live without it.
The same can be achieved with logic in the providers and setting
properties to include/skip resolution from certain repositories. So,
lets keep it simple and not have this option in "on-empty..."
> I agree that
> on-empty-location: checkout | fail
> is clearer.
>
ok, deal.
>> update-policy
>> : default
>> | no-update
>> | fail-uncommitted
>> | fail-localchange
>> | update
>> | replace;
>> ;
>>
>> default
>> ...
>>
> head of any branch (not just main) = update.
> tag/timestamp/revision (i.e. fixed material) = trust-clean (see New
> proposal below)
>
ok.
>> no-update
>> trust whatever is there to be correct
>>
> I like 'trust' better because it declares what to do rather then what
> not to do.
>
ok. see new proposal further down.
>> fail-uncomitted, fail-localchange
>>...
>> We could use git specific terms, but then it is perhaps more confusing
>> when used with other types of repositories.
>>
> The distinction is needed but problem arise when it is expressed as
> multiple ways to fail instead of what to do on success. What if it does
> not fail? Do you update then? Or trust? Or replace even (since you may
> want to erase the checked out content and replace it with what you have
> in your local clone)? How do you express that you want to fail both on
> local changes and on uncommitted changes versus just one of them?
>
The combination as failing on committed local change, but not on dirty
workspace did not occur to me as a combination that made sense.
> New proposal:
>
> Let repository implementations like git have an additional enum:
>
> uncommitted-policy: accept | fail
>
I don't understand - "uncommitted" == "dirty in workspace", "committed"
== in local repo clone, "pushed" is "in the cloned repo". So,
"uncomitted-policy : accept " == trust-dirty ?!?
How about:
distribution-policy : accept-local-commit | fail ;
(only applies to Distributed SCMs).
>
> update-policy
> : default
> | trust-clean
> | trust-dirty
> | update
> | replace;
> ;
>
I don't like the "trust" word, how about "check-clean", "accept-dirty"
instead?
>> replace
>> result is the same as removing everything and checking out from scratch
>> (may be done more efficiently naturally if result is guaranteed to be
>> the same).
>>
> Perhaps. But I think the main reason to do this is that you want to make
> sure that no optimization where made and that everything indeed is from
> scratch so I'd rather document this enum as doing just that.
>
ok, fine.
>> (Possible addition to update-policy is to allow specification of a delay
>> between update checks. ...
>>
> Not in favor of that ...
>
ok.
>> 3. what to do if there is a need to merge
>> (A merge-policy can only be used if the update-policy is 'update'.)
>>
>> merge-policy
>> : default
>> | fail-uncommitted
>> | fail-localchange
>> | fail-merge
>> | fail-conflicts
>> | use-local
>> | use-remote
>> | accept-conflicts
>> ;
>>
>> default
> I don't think we need this.
>
ok.
>> fail-uncomitted, fail-localchange
>> ...
> This can be simplified if the handling of uncommitted is moved to a
> separate enum.
>
yes.
>> fail-merge
>> fail-conflicts
>> use-local
>> use-remote
>> accept-conflicts
> In some sense, I'd say that accept/fail conflicts doesn't belong in b3.
> That kind of handling is useful in the IDE and there it's very likely
> that the user just do 'Team' -> 'Update to Head' or similar in the
> explorer anyway and get all the bells and whistles associated with
> conflict management through the UI.
>
That works for me too.
> I think I'm back to:
> merge-policy { fail, use-local, use-remote, merge }
> perhaps with one minor change:
> merge-policy { fail, use-workspace, use-scm, merge }
>
ok, however...
this may not be enough if git is used and there are:
a) dirty stuff in workspace
b) local commits
c) changes in cloned repository
Don't know what really happens in this situation - I can imagine seeing
it as a progression, i.e. use workspace, throw away workspace, throw
away workspace and local commit... But I don't know if that is how you
think about it when using git. I assume it will merge what is in the
workspace and leave the local commit until the workspace change is
committed. So I wonder if there is a difference in practice?
The "use-workspace" is a misnomer in the special case that there are no
changes in the workspace, but there are locally committed changes -
"use-local" works better in that case, but "use-scm" seems ambiguous.
I am still struggling with the enums - don't find them intuitive.
merge-policy : fail; // means "fail-if-merge-is-required"
merge-policy : use-workspace; // means "merge, and if there is a
conflict use workspace"
merge-policy : use-scm; // means "merge, and if there is a conflict use
the scm"
merge-policy : merge; // means "merge and fail-if-there are conflicts"
"merge" is implied in all but the first. If we move "merge" to
update-policy, it becomes:
update-policy
: default
| check-clean
| accept-dirty
| update
| merge
| replace;
;
With the meaning that "update" only accepts clean updates (i.e. the same
as "update + merge-policy : fail").
merge-policy, is then all about conflicts, and can be renamed:
conflict-policy : use-local | use-scm | fail ;
> ...it goes without saying that conflicts
> requiring manual intervention are failures. The message is: If you want
> to really edit your conflicts, then use the UI tools that the SCM
> provider makes available.
>
ok, I accept that conflicting merges always lead to fail.
> The difference between fail-merge and fail-local is already handled by
> using trust-clean versus update. If you want to fail on any local change
> regardless if a merge is needed or not, use trust-clean. If you want to
> fail local changes only when a merge is needed, then use update + fail.
>
There is a difference, but it not an important use case.
I can live without having this distinction, and handle what I was
thinking about in the IDE - it is not a valid automated/headless/ci feature.
> One case remains and that is that you want to fail when a
> non-conflicting merge is attempted on a local change but other merges
> should succeed.
This is better done in the IDE with a UI, so just failing is better.
>
> - thomas
|
|
|
| Re: Repositories and Providers [message #608537 is a reply to message #539369] |
Thu, 10 June 2010 16:19 |
|
Some minors...
> How about:
> distribution-policy : accept-local-commit | fail ;
> (only applies to Distributed SCMs).
>
Yes, perhaps. But there are more things to consider when we discuss how to manage the clone. So much in fact, that I
feel it should be a separate topic altogether. Can we table that aspect of the discussion for now and concentrate on how
the repository (local clone in case of git or similar), relates to the workspace? I think it might be easier if we treat
the clones relationship to its source (or sources) as a separate thing with its own set of enums.
>>
>> update-policy
>> : default
>> | trust-clean
>> | trust-dirty
>> | update
>> | replace;
>> ;
>>
> I don't like the "trust" word, how about "check-clean", "accept-dirty"
> instead?
>
I don't like the check word. Ok, so you check. Then what? With the new meaning of update and merge, you must first check
if the workspace is clean. The merge however is an update of dirty material. What about:
update-policy
: default
| keep-clean
| keep-dirty
| update-clean
| update-dirty
| replace
;
> this may not be enough if git is used and there are:
> a) dirty stuff in workspace
> b) local commits
> c) changes in cloned repository
>
It is enough if we treat clone management separately.
> The "use-workspace" is a misnomer in the special case that there are no
> changes in the workspace, but there are locally committed changes -
> "use-local" works better in that case, but "use-scm" seems ambiguous.
>
The scm ambiguity is resolved if we treat clone management separately. The word local however, is still ambiguous (local
clone versus local workspace changes). Since this is now a pure conflict-policy, the use-workspace is not a misnomer at
all. So I'm back to workspace versus scm.
> With the meaning that "update" only accepts clean updates (i.e. the same
> as "update + merge-policy : fail").
>
> merge-policy, is then all about conflicts, and can be renamed:
>
> conflict-policy : use-local | use-scm | fail ;
>
Agreed. The separation is cleaner. Not sure if it's better to rename to conflict-policy since it's the policy that is
used for update-policy 'merge'. It feels natural to call it 'merge-policy'. But I can buy both.
- thomas
|
|
|
| Re: Use @ for enums? [message #608539 is a reply to message #539321] |
Thu, 10 June 2010 17:40 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
My experminet worked out just fine, the enum values does not have to be
keywords, but I need them to use '_' instead of '-' as separators (I
could use '.'), or we could use camelCase.
The amound of work is not unreasonable - instead of just declaring the
enum in the grammar:
- write a value converter (10 lines or so)
- write a content assist proposal provider (10 lines or so)
- write syntactic coloring (as they are no longer keyword tokens)
I think we are going to have several enums that needs this - and I like
to keep the code for one enum type together, so will do a bit more work
to make it easier to maintain (or right now to change them a couple of
times :)
- henrik
On 6/10/10 2:25 PM, Henrik Lindberg wrote:
> It is a lot more work if done as a feature containment, but a lot easier
> if done as a link - but I have not tried a reference to an Enum instance.
>
> As we are changing these enums frequently, if they were done with @,
> there is just one place to change them.
>
> But I will experiment a bit.
>
> I changed the Subject BTW...
> - henrik
>
> On 6/10/10 11:44 AM, Thomas Hallgren wrote:
>> On 06/09/2010 03:26 PM, Henrik Lindberg wrote:
>>> So, trying to sum up:
>>>
>>> 1. Change the "update-policy" to the two policies you suggested.
>>>
>>> exists-policy { fail, keep, replace, merge }
>>> merge-policy { fail, use-local, use-remote, merge }
>>>
>>> I downside is that the enum's become keywords (and keep, replace, and
>>> merge are very likely to clash with other things). (The current solution
>>> was crafted so the enums all have "-" in them to reduce clashes).
>>> Suggestion; have all enums everywhere have a special leading character
>>> like '@'. (Yet another more complicated impl is to treat the enum values
>>> as IDs and have data conversion rules per enum - not sure how well that
>>> works though, I tried something similar earlier and it was not possible
>>> to return an EObject supposed to be part of the model built by the
>>> grammar this way).
>>> The only solution I can think of would use a reference to an Enum
>>> instead of containing one. This way I can use an ID, and have a scope
>>> provider that provides the enums to link to. (I use this technique to
>>> reference implicit variables like "request" in resolver expressions).
>>>
>>> Using @ is however much much simpler...
>>>
>>> proposal example:
>>> exists-policy : @keep;
>>>
>
>>
>> - thomas
>
|
|
|
| Re: Repositories and Providers [message #608541 is a reply to message #539424] |
Thu, 10 June 2010 20:17 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I think we are trying to squeeze too many dimensions into the same enum,
and that is why it feels confusing. Better to break them apart:
- do I want a checkout or not
- is it ok to build dirty material or not
- do I want to (try) to stay current
Grammar:
'policy' '{'
('checkout' ':' boolean ';')?
(('accept-dirty' ':' boolean ';')?
('update' ':' boolean; )?
('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
) | ('replace' ';' )? )?
'}'
e.g.
// checkout if empty, when building it is ok if state is dirty, perform
// update, and it is ok if this leaves workspace in dirty state.
// (A typical developer policy)
policy {
checkout : true;
accept-dirty : true ;
update : true;
on-merge-conflict : use-workspace;
}
// ci server checked out build
policy {
checkout : false ;
accept-dirty : false;
update : false;
}
// ci, or manually checked out, keep it updated (but clean)
policy {
checkout : false;
accept-dirty : false;
update : true ;
}
Defaults are:
no policy = defaults determined by branch point type
policy stated - defaults determined by branch point type, overridden by
what is stated.
i.e. if update is performed and the result is dirty content (because of
merge), the "on-dirty-content" will dictate what to do. The state is the
same as if user manually made the same state (well almost, as the scm
may remember stuff).
I think this is much cleaner.
- henrik
|
|
|
| Re: Repositories and Providers [message #608543 is a reply to message #539464] |
Thu, 10 June 2010 21:19 |
|
This is excellent.
- thomas
On 06/10/2010 10:17 PM, Henrik Lindberg wrote:
> I think we are trying to squeeze too many dimensions into the same enum,
> and that is why it feels confusing. Better to break them apart:
>
> - do I want a checkout or not
> - is it ok to build dirty material or not
> - do I want to (try) to stay current
>
> Grammar:
>
> 'policy' '{'
> ('checkout' ':' boolean ';')?
> (('accept-dirty' ':' boolean ';')?
> ('update' ':' boolean; )?
> ('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
> ) | ('replace' ';' )? )?
> '}'
>
> e.g.
> // checkout if empty, when building it is ok if state is dirty, perform
> // update, and it is ok if this leaves workspace in dirty state.
> // (A typical developer policy)
> policy {
> checkout : true;
> accept-dirty : true ;
> update : true;
> on-merge-conflict : use-workspace;
> }
>
> // ci server checked out build
> policy {
> checkout : false ;
> accept-dirty : false;
> update : false;
> }
>
> // ci, or manually checked out, keep it updated (but clean)
> policy {
> checkout : false;
> accept-dirty : false;
> update : true ;
> }
>
> Defaults are:
> no policy = defaults determined by branch point type
> policy stated - defaults determined by branch point type, overridden by
> what is stated.
>
>
> i.e. if update is performed and the result is dirty content (because of
> merge), the "on-dirty-content" will dictate what to do. The state is the
> same as if user manually made the same state (well almost, as the scm
> may remember stuff).
>
> I think this is much cleaner.
>
> - henrik
|
|
|
| Re: Repositories and Providers [message #608546 is a reply to message #539471] |
Sat, 12 June 2010 17:03 |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Checked in an implementation of this to 'head', with a small variation:
I used the following keywords to avoid conflicts:
- checkout-content
- update-content
- replace-content
Regards
- henrik
On 6/10/10 11:19 PM, Thomas Hallgren wrote:
> This is excellent.
>
> - thomas
>
> On 06/10/2010 10:17 PM, Henrik Lindberg wrote:
>> I think we are trying to squeeze too many dimensions into the same enum,
>> and that is why it feels confusing. Better to break them apart:
>>
>> - do I want a checkout or not
>> - is it ok to build dirty material or not
>> - do I want to (try) to stay current
>>
>> Grammar:
>>
>> 'policy' '{'
>> ('checkout' ':' boolean ';')?
>> (('accept-dirty' ':' boolean ';')?
>> ('update' ':' boolean; )?
>> ('on-merge-conflict' : 'use-workspace' | 'use-scm' | 'fail';)?
>> ) | ('replace' ';' )? )?
>> '}'
>>
>> e.g.
>> // checkout if empty, when building it is ok if state is dirty, perform
>> // update, and it is ok if this leaves workspace in dirty state.
>> // (A typical developer policy)
>> policy {
>> checkout : true;
>> accept-dirty : true ;
>> update : true;
>> on-merge-conflict : use-workspace;
>> }
>>
>> // ci server checked out build
>> policy {
>> checkout : false ;
>> accept-dirty : false;
>> update : false;
>> }
>>
>> // ci, or manually checked out, keep it updated (but clean)
>> policy {
>> checkout : false;
>> accept-dirty : false;
>> update : true ;
>> }
>>
>> Defaults are:
>> no policy = defaults determined by branch point type
>> policy stated - defaults determined by branch point type, overridden by
>> what is stated.
>>
>>
>> i.e. if update is performed and the result is dirty content (because of
>> merge), the "on-dirty-content" will dictate what to do. The state is the
>> same as if user manually made the same state (well almost, as the scm
>> may remember stuff).
>>
>> I think this is much cleaner.
>>
>> - henrik
>
|
|
|
Goto Forum:
Current Time: Sat Jul 27 10:40:00 GMT 2024
Powered by FUDForum. Page generated in 0.10184 seconds |
] 
Search
Help
Register
Login
Home
