[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakarta.ee-spec.committee] Votes and Staging Repo Expirations
|
I think the fundamental question of how much is it worth is a good one.
Our status quo for this release:
- We have repos expiring or expired already before vote completion
- We have up to 5 staged repos for some projects
So whatever it is we think is the right thing, it definitely is not happening now.
It took tools to find the above issues and backup the right binaries. If we opt for 'no change' in procedures, then those tools need to get cleaned up and made permanent.
I don't really see any silver bullets.
One additional option to throw on the table for (re)consideration is if we had our own Nexus Pro install.
- Right now our virtual "staging" view is literally the entire world of open source projects. It would be possible for us to create the view using our own rules. It could be as narrow as we want it.
- We could control the retention period and therefore never have an expiration issue
- We could use the UI to login and browse so a human could more easily see mistakes.
All of these would have impact that could prevent our current set of problems, but not require tools. Nexus would be our tool.
--
David Blevins
http://twitter.com/dblevins
http://www.tomitribe.com
310-633-3852
> On Aug 26, 2019, at 3:08 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
>
> Thanks, David, that explanation was helpful.
>
> It seems that we have a number of competing requirements. The most basic
> requirement is that we ship exactly what we review and approve. There
> doesn't seem to be an easy way to enforce this, so the question is how
> much effort we need to go to to ensure that mistakes are not made.
>
> Ideally, during the review period, there would be exactly one staging
> repository for each API, which would not change, and after approval that
> staging repository would be released to Maven Central. If that were the
> case, it wouldn't matter whether we reviewed the "view" URL or the
> "numbered" URL.
>
> One known issue that makes the above difficult is the expiration of
> staging repositories. 30 days is enough for any one API, but it's not
> really long enough for the collection of all APIs going into Jakarta EE.
> Again, ideally, this would be increased to 60 days. Lacking that, we might
> need the "backup and restore" tools for the staging repositories.
>
> I think we agreed that having things like GlassFish or the Jakarta EE API
> project explicitly list every "numbered" staging repository is unworkable.
> So it seems like we need to make the "view" URL work for us. This is the
> primary reason I thought we should include the "view" URL in the PR - it
> has to work, and this is a way for us to check that it's referring to the
> right content.
>
> The primary reason for including the "numbered" URL seems to be that it's
> a fixed reference to the artifacts that can not change, and that if we
> always operate on that numbered repository we can ensure that we get what
> we want. But I don't see that there's any way to ensure that the "view"
> URL will only find artifacts in that numbered repository, unless we assume
> or ensure that there is only one such repository for each API.
>
> Maybe we should create an audit tool that can check that there is only
> one staging repository for each API, and that the artifacts visible
> through the "view" URL are the same artifacts that are in that one "numbered"
> staging repository?
>
> So, again, how much effort is this worth?
>
> I don't see any simple operational steps that will ensure we get exactly
> what we want. We either need to make some assumptions or create some tools.
>
>
> David Blevins wrote on 8/24/19 2:15 PM:
>> The most important thing for us to know -- and I've tried and failed to explain this before -- is that this repository is basically like a database view:
>>
>> - https://oss.sonatype.org/content/repositories/staging/
>>
>> Nexus has the ability to sew together multiple physical repositories as one logical view. When those physical repositories are not located in that exact Nexus install, Nexus will act like a caching proxy. It's possible to create a view that basically consists of all actual staging repositories, plus a cache of external repositories like Maven Central.
>>
>> So for example, this is the actual physical staging repository for our deployment 1.7.2 binaries up for vote:
>>
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1002/
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1002/jakarta/enterprise/deploy/jakarta.enterprise.deploy-api/1.7.2/
>>
>> An actual staging repository (not a view) cannot be modified once closed. So there's nothing we can do to keep it fresh or trick it into staying around. Once the above `jakartaenterprisedeploy-1002` staging repository is deleted, we'll see it's contents are no longer available in the "view" here:
>>
>> - https://oss.sonatype.org/content/repositories/staging/jakarta/enterprise/deploy/jakarta.enterprise.deploy-api/1.7.2/
>>
>> Now as I mentioned the view also will allow you to "see" things in Maven Central. When we do releases, we enable `https://oss.sonatype.org/content/repositories/staging/` as a repository in our builds and our Maven Central dependencies end up flowing through the view and being cached. For example here's a chunk of the log output from the JAXR release James did 3 days ago:
>>
>> ----
>> Downloaded from sonatype-nexus-staging: https://oss.sonatype.org/content/repositories/staging/com/headius/options/1.4/options-1.4.jar (14 kB at 6.7 kB/s)
>> Downloading from sonatype-nexus-staging: https://oss.sonatype.org/content/repositories/staging/com/martiansoftware/nailgun-server/0.9.1/nailgun-server-0.9.1.jar
>> Downloading from sonatype-nexus-staging: https://oss.sonatype.org/content/repositories/staging/com/jcraft/jzlib/1.1.3/jzlib-1.1.3.jar
>> ----
>>
>> Each one of these is now sitting cached in Sonatype's staging view:
>>
>> - https://oss.sonatype.org/content/repositories/staging/com/headius/options/1.4/
>> - https://oss.sonatype.org/content/repositories/staging/com/martiansoftware/nailgun-server/0.9.1/
>> - https://oss.sonatype.org/content/repositories/staging/com/jcraft/jzlib/1.1.3/
>>
>> Where I tried to explain this was when I recommended we use the actual physical staging repo names in our PRs because these are the repos we must take action on when a release completes. For deployment there are actually several physical staged repos. I found them via URL hacking:
>>
>> ----
>> 1000 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1000/
>> 1001 200 Jul 25, 2019 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1001/
>> 1002 200 Jul 25, 2019 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1002/
>> 1003 200 Jul 27, 2019 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1003/
>> 1004 200 Jul 27, 2019 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1004/
>> 1005 200 Aug 02, 2019 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1005/
>> 1006 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1006/
>> 1007 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1007/
>> 1008 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1008/
>> 1009 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1009/
>> 1010 404 https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1010/
>> ----
>>
>> The 1.7.2 binaries are in 1002 and 1.7.3 is in 1005.
>>
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1002/jakarta/enterprise/deploy/jakarta.enterprise.deploy-api/1.7.2/
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1005/jakarta/enterprise/deploy/jakarta.enterprise.deploy-api/1.7.3/
>>
>> Both of these will be gone by the time their vote is schedule to complete. Any builds that pulled them in via the local staging view will begin to fail. There are some garbage repos we should just drop:
>>
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1001/jakarta/enterprise/deploy/deployment-spec/1.7/
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1003/jakarta/enterprise/deploy/deployment-spec/1.0/
>> - https://oss.sonatype.org/content/repositories/jakartaenterprisedeploy-1004/jakarta/enterprise/deploy/jakarta.enterprise.deploy-api/1.0.9/
>>
>> Here's Bill's mail release (again found via url hacking). Once the vote concludes on the 28th, he'll have about 3 days to publish the jakartamail-1011 repo which should be enough.
>>
>> ----
>> 1010 404 https://oss.sonatype.org/content/repositories/jakartamail-1010/
>> 1011 200 Aug 01, 2019 https://oss.sonatype.org/content/repositories/jakartamail-1011/
>> 1012 404 https://oss.sonatype.org/content/repositories/jakartamail-1012/
>> 1013 404 https://oss.sonatype.org/content/repositories/jakartamail-1013/
>> ----
>>
>> Here's the transaction release. This vote completes Sept 2, so we're going to potentially lose this binary right at the finish line.
>>
>> ----
>> 1021 404 https://oss.sonatype.org/content/repositories/jakartatransaction-1021/
>> 1022 200 Aug 02, 2019 https://oss.sonatype.org/content/repositories/jakartatransaction-1022/
>> 1023 404 https://oss.sonatype.org/content/repositories/jakartatransaction-1023/
>> ----
>>
>> Here's the xmlrpc release. This vote also completes on Sept 2 and we risk losing this one at the finish line as well.
>>
>> ----
>> 1003 404 https://oss.sonatype.org/content/repositories/jakartaxmlrpc-1003/
>> 1004 200 Aug 02, 2019 https://oss.sonatype.org/content/repositories/jakartaxmlrpc-1004/
>> 1005 404 https://oss.sonatype.org/content/repositories/jakartaxmlrpc-1005/
>> ----
>>
>> Here are the JSP and JSTL repos, which are 1017 and 1019 respectively. The JSP vote concludes Sept 2, so we might possibly make this one if we're extremely fast. JSTL concludes Sept 9, so we have plenty of time to act before that one would be gone.
>>
>> ----
>> 1016 404 https://oss.sonatype.org/content/repositories/jakartaservletjsp-1016/
>> 1017 200 Aug 03, 2019 https://oss.sonatype.org/content/repositories/jakartaservletjsp-1017/
>> 1018 404 https://oss.sonatype.org/content/repositories/jakartaservletjsp-1018/
>> 1019 200 Aug 19, 2019 https://oss.sonatype.org/content/repositories/jakartaservletjsp-1019/
>> 1020 404 https://oss.sonatype.org/content/repositories/jakartaservletjsp-1020/
>> ----
>>
>> Ultimately, this is why I was very against people using `https://oss.sonatype.org/content/repositories/staging/` urls in their PRs. Because it's too many levels removed from reality, perpetuates confusion and creates too many opportunities for the wrong thing to happen.
>>
>> We haven't seen it this release, but there could actually be several physical staging repos containing the same version of an API. Basically every attempt sticks around till it is either dropped or expires. Were this to happen we would still see only one copy in the logical view. The gotcha would be that as each expires and is deleted the one that wins will change. So the logical URL of `https://oss.sonatype.org/content/repositories/staging/jakarta/wombat/2.0/`, which is effectively a symbolic link, might map to `jakartawombat-1003/jakarta/wombat/2.0/` at the start of a vote and `jakartawombat-1006/jakarta/wombat/2.0/` at the end of a vote. The people who voted at the beginning and the people who voted at the end would have voted for completely different binaries.
>>
>> I appreciate this is a lot of information to digest.
>>
>>
