|Re: [jgit-dev] GerritForge's JGit E2E tests with Gerrit in 2022|
Hi Matthias, thanks for sharing your feedback.See my comments below.On 20 Jan 2022, at 00:05, Matthias Sohn <matthias.sohn@xxxxxxxxx> wrote:On Tue, Jan 18, 2022 at 4:37 PM Luca Milanesio <luca.milanesio@xxxxxxxxx> wrote:Dear Gerrit and JGit developers’ community,
I would like to share with you the GerritForge’s plans for adding more E2E tests to JGit changes in 2022 .
Opinions, suggestions, objects, any sort of feedback is more than welcome :-)
 https://gitenterprise.me/2022/01/18/new-year-new-jgit-contributions-coming/I would prefer to see your proposals posted on this list instead of your website since it would be easierto respond and follow the conversation.Sure, all proposals for the JGit project and associated discussion will happen here.The  above is a GerritForge’s initiative announcement, like the JGit E2E tests with Gerrit we introduced a long ago and published in the past .I think the main reason why some contributions to jgit stay in review for a long time is a lack of active reviewers.True, I agree with your analysis. However, some of the reviewers and committers who wrote a large part of the code-base are inactive and won't be back to help us out.The only solution is to increase the volume of reviews and get more changes merged. We do need more people with more knowledge and experience with the code.
Currently, I can only observe the numbers that show a yearly decline in the number of merged changes. Also, the number of active committers is in evident decline.GerritForge wants to help here, provide more testing, more input, more reviews, more ideas, more changes, and make things better for everyone.This won't improve by adding yet another branch in a fork. I think the existing jgit master branch is thedevelopment branch.The focus is on making JGit + Gerrit tests work E2E, not creating any fork.The workflow of using a “runway” of changes all cherry-picked (or rebased) and getting extra integration tests *before* merging them is pretty common.Also, Zuul implements *that* workflow (see  for an overview of how the process works).We spent quite some time in the past on reviewing and testing patches for older jgit releases andmerging them up to master since Gerrit was reluctant to update older Gerrit maintenance branches touse the latest JGit maintenance branch. This added overhead and ate some of the limited capacity we have.Hence I propose we stop this to better use our limited resources by
- limiting development of new features to the master branch
- limiting maintenance of releases to the latest release (currently 6.0)
- maybe as an exception to this rule allow fixes for the last minor release of the last major release (currently 5.13).
Some users asked for this since they still depend on Java 8 which is no longer supported since jgit 6.0
- declaring EOL on all older releasesI fully agree with all of the above: I believe it deserves a separate discussion thread because of its importance and increased visibility and discoverability. This topic is about testing JGit and Gerrit together on master branch.Gerrit actively maintains its last 3 releases (currently 3.3, 3.4, 3.5). Last week the Gerrit Engineering Steering Committeedefined a new policy for updating JGit  on Gerrit maintenance branches supporting this proposal to reduce the numberof active JGit maintenance branches.Yes, I recall the discussion, and there is a global consensus on that.I believe that having E2E tests of JGit + Gerrit executed on the “changes on the runway” to be merged would also help the Gerrit side update the submodule pointer to JGit master soon as possible.P.S. Currently, JGit master and Gerrit master don’t even build together until Gerrit change  is merged.
I appreciate the proposal to run more tests like Gerrit integration tests against open jgit changes.+1I don't understand why another branch is needed for implementing that.Instead these tests should be runon the open changes.You do need a "stream of changes together" (to highlight that the focus is NOT the branch) because when you run the integration tests, you need to see and check things working together, not in isolation.
Let me give you a simple example:
- Change Ifoo removes a foo() method in JGit that is not used by anyone
- Change Ibar introduces a new bar() method based on the foo() method
- You verify Change Ifoo and Ibar in isolation (NOT streamlined) and they both get the Verified +1 and merged.Verifying all individual changes in isolations won't tell you if they work together.With the "stream of changes", you will do instead:
- Change Ifoo removes a foo() method in JGit that no one uses.
- Change Ibar introduces a new bar() method based on the foo() method.
- Change Ifoo and Ibar are put (cherry-picked or rebased) to the "validation stream" and tested together. The build breaks, and none of the two changes are validated.The above example is very simplistic. The problem is a lot more complex than that; however, it shows that creating a "stream of changes" for validation is key when you want to integrate and validate multiple components.There is also an issue with execution times and tradeoffs with the speed of validating incoming changes.Running E2E tests takes a long time (hours) and is very expensive (hundreds of $ of infrastructure time): you want to limit that to ONLY those changes that:- Have passed the normal verification process (e.g. Eclipse's CI)- Have been reviewed and have no vetos from being approved
The two aspects depicted above represent the reason why the trigger for running the E2E tests *cannot* be applied for all open changes targeting master.
Running more tests can help to find more problems and prevent regressions butdoesn't fix the problem that we need more active reviewers.Sure, that is crystal clear. Having more E2E tests of JGit with Gerrit will increase the confidence in merging changes that have been stuck in review for a very long time.We address the confidence in merging changes where current active reviewers and commits have a lack of knowledge of the underlying code.IMHO projects should not be stuck because the original author of the code isn’t around anymore to maintain it.Open-Source is great because allows the code initially written by one author to be reviewed, modified, extended by the whole community.The lack of knowledge should never impede contributions, reviews or approvals.If I am not confident about one change, I’ll write more tests to verify its correctness.If I don’t understand some parts of the source code, I’ll spend more time reading it more carefully.
Our efforts in improving the E2E testing of JGit with Gerrit is directed at *improving* the confidence of current and new reviewers and contributors.I hope that the JGit community would appreciate our initiative and help with our endeavour.The JGit CI  uses Eclipse CBI  to run builds and tests on Jenkins running in Kubernetes.I guess the proposed tests could also be run there. If we need more resources there we can add some more.The tests are E2E with a real Gerrit setup, including:- load-balancers- LDAP authentication- Shared filesystem on NFS- multiple Gerrit primaries
You know better than me that only executing in a production environment we could discover some very nasty JGit bugs, and we (actually you) managed to fix them before the Gerrit release.We want to facilitate and help with the process and adding more testing.
As proposed by David Ostrovsky we should also aim at verifying the bazel build and tests in JGit CI.Unfortunately it can't fully replace the Maven build since there is no support in bazel for building OSGi bundles,Eclipse features and p2 repository which is required for installing it in Eclipse IDE.Sure, that is something we should do regardless.If someone has time to help improving JGit CI we'd appreciate that. Thomas Wolf implemented a pipelinelibrary for the egit builds , but we didn't find time yet to do something similar for the jgit builds.I’d be happy to help with that, as I already do it for the Gerrit project.Do you know where the JGit pipeline is stored and maintained?
I am looking forward to your and your colleagues' contributions. If they are coming as you said we may havesome new jgit committers soon. If possible please also plan time to help with reviewing changes from othercontributors.Sounds good, looking forward to it !Thanks again for taking the time to provide your valuable feedback and reading about initiative.Luca. https://gerrit-review.googlesource.com/c/gerrit/+/327079
Back to the top