| Re: [rdf4j-dev] 3.0.0 release, code freeze, and mono-repo migration |
Running all the benchmarks takes several hours. In a perfect world we would have a Benkins (benchmark Jenkins) that would run all the benchmarks and diff the results from before and after so you would get a “build failure” if you made something in RDF4J considerably slower with your PR._______________________________________________But since they take so long and aren’t run automatically I have experienced that I’ve broken some of the benchmarks and they’ve been broken for a while.So the solution was to make (most of) the benchmarks run exactly one iteration step as part of a junit test. This just verifies that they still work, but not if you’ve made anything slower since last compile.The reason why I like Travis so much is because each build server is your own clean vm/container so you don’t have to worry about things like network ports or if one jmh benchmark can run at the same time as another jmh benchmark.HåvardOn Tue, Aug 27, 2019 at 9:16 PM Håvard Ottestad <hmottestad@xxxxxxxxx> wrote:Wrt. The SHACL tests, some can be marked as part of the compliance test run, so that they are run only after merging instead of on every commit (to a PR).That would still cause problems with parallel builds though (the JMH process doesn't like it when there's two builds running on the same machine, apparently, see the issue for details).Do we use an annotation for marking a test/class as part of the compliance run?No, compliance tests are "marked" by being part of the compliance module (or one of its submodules). This module is only touched by Jenkins when the '-Pcompliance' flag is set on the maven build (which is what the branch verify job does, but the PR verify job does not). However, I'm not sure moving these tests really solves anything. I'll be honest: I don't quite understand why we need these benchmarks to be run as part of the build process at all. Unless I misunderstand, their use is mostly just in local testing / debugging. So what value do we get from running them as part of the regular build?Jeen_______________________________________________HåvardOh one more thing: I have not yet configured any of the jenkins jobs to send mail if builds fail. I'll take care of that once we've shaken things out a bit, but until then I'd appreciate it if you could keep an eye on Jenkins yourself as well.Cheers,JeenOn Tue, Aug 27, 2019 at 8:40 PM Jeen Broekstra <jeen.broekstra@xxxxxxxxx> wrote:I'm reasonably happy with the current setup. We have a master branch (pom version set to 3.0.1-SNAPSHOT), and a develop branch (pom version set to 3.1.0-SNAPSHOT). There's Jenkins jobs in place to run verification and snapshot deployments of each branch.I've opened a new ticket (https://github.com/eclipse/rdf4j/issues/1515) to look into slimming down our test suite somewhat, to make the builds run more quickly. I think there are some easy wins, especially in the SHACL tests (which take almost 8 minutes on their own), and in the Sail and Repository compliance tests (which together take about 20 minutes).I'm also not 100% sure that everything will work correctly come release-time, so we will have to play it by ear a little bit when that rolls around. For now, happy to open up the repo for business again though, and I'm open to suggestions for further improvements/restructures.Cheers,JeenPS you may notice two modules rdf4j-client and rdf4j-storage in the root of the repo. These are pom projects that create shaded jars of the modules that were previously in rdf4j and in rdf4j-storage, respectively. Although I don't think we ever advertised their existence I didn't immediately want to get rid of them, but I must admit they make my OCD itch a little bit. Happy to take opinions on what we should do with them.On Sun, Aug 25, 2019 at 5:35 PM Jeen Broekstra <jeen.broekstra@xxxxxxxxx> wrote:There's a PR up for the monorepo migration. It's a big one so I don't expect a full review but if you have the time, take a peek and let me know what you think.Cheers,JeenOn Thu, 22 Aug. 2019, 17:50 Jeen Broekstra, <jeen.broekstra@xxxxxxxxx> wrote:Absolutely, no worries, the other repos will be around for quite a while longer, at the very least in read-only mode.Cheers,JeenOn Wed, Aug 21, 2019 at 9:22 PM Håvard Ottestad <hmottestad@xxxxxxxxx> wrote:Hi Jeen,_______________________________________________Thanks for doing all this work :)I’ve got a couple of branches for RDF4J-storage that I want to keep. One is support for sh:xone, another is a new memory store (two actually, one hash based and another is tree based). These branches aren’t ready to be merged.Can you keep the RDF4J-storage repo around for a while after you’re done with porting to a mono-repo so I can port those branches myself?Håvardhi folks,Just a heads up that tomorrow we release rdf4j 3.0.0. To get a bit of a head start I'd like to tag the release in github already, so that I can start working on migration to the mono-repo. I will do this in an hour or so. The actual release (uploading to central, news announcement, SDK upload, etc) won't happen until tomorrow though.Immediately after, I want to start work on migrating us back to a mono-repo. Here's how I plan to do this:1. merge all open (stable) PRs that didn't make the cut for the 3.0 release (for example https://github.com/eclipse/rdf4j-storage/pull/279) into develop.2. merge develop into master and close the develop branch. At this point I should only have a single branch to worry about for migration: master. All PRs are merged, no feature branch will be transferred (I've got an open PR myself that is not yet stable, I will leave that out and just replay it when we're done with migration).3. in the rdf4j repo, reorganize directory structure (probably move the top-level modules into a 'core' directory or something like that - details may change depending on what I find works well).4. copy over master branch of rdf4j-storage (including history and tags) to rdf4j, to a storage/ directory.4. copy over master branch of rdf4j-tools (including history and tags) to rdf4j, to a tools/ directory.5. copy over master branch of rdf4j-testsuites (including history and tags) to rdf4j, to a testsuites/ directory.6. reorganize rdf4j master branch further to get maven setup consistent (edit poms, dependencies, move a few (test) modules around again).7. commit and push the whole shebang. Watch Jenkins freak out.8. Tweak Jenkins setup to deal with the new reality.9. Create a new develop branch.Until the above procedure is completed (or at least until step 7 is complete), there is a complete code freeze in place. I am hoping to get things in shape in a day or two, but there's a good chance that it will take until after the weekend to get everything right.The issue tracking progress of the migration is here: https://github.com/eclipse/rdf4j/issues/1467Cheers,Jeen_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev
_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev
_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/rdf4j-dev