Re: [geogig-dev] SQLite backend for GeoGig

[Erik Merkle <emerkle@xxxxxxxxxxxxxxxx>]

First off, all the testing I'm doing is with V2 of the Xerial/SQLite backend. I haven't looked at V1 much, but I can apply the same changes if necessary.

With regard to the parallel tests, I'm referring to the Surfire configuration in the main POM here: https://github.com/emerkle826/geogig/blob/dev/src/parent/pom.xml#L402-L422

Since it is configured with "forkCount" set to "1C", maven will create a process per CPU to run the tests. I know SQLite has issues with more than one process writing to the database, but the tests should be creating separate database files as they run. There may not be a problem with this, but I was having better results by setting the forkCount to "1" locally. Once I get the CLI tests running correctly with an SQLite backend, I'll put the forkCount back to "1C" and see if I still have issues with the tests.

I think I've addressed al the issues with the XerialConflictsDatabaseV2, so I'm close to a PR for this. Hopefully this week, we'll be ready to switch the default backend from BDB to SQLite.

-Erik

Erik Merkle
Software Engineer | Boundless

On Tue, Dec 15, 2015 at 7:58 AM, Gabriel Roldan <groldan@xxxxxxxxxxxxxxxx> wrote:

Hi Erik, that all sounds good.

Lets move the discussion to the mailing list.

Wonder what's going on with the parallel maven tests. How're you running them? And the ones that fail are for Xerial V2 or V1 implementation?

On Sat, Dec 12, 2015 at 2:52 AM, Erik Merkle <emerkle@xxxxxxxxxxxxxxxx> wrote:

OK, I think I have the test issues fixed (running them again to make sure).

I had a couple other issues with the XerialConflictsDatabaseV2 impl (probably in XerialConflictsDatabase but I haven't checked yet). Both the "get" and "remove" methods were building SQL strings assuming namesapce and path were never null. I changed the SQL string being built there to omit the "WHERE" clauses if the namespace and/or path values passed in are null. I'll push it up once I make sure it isn't gonna break anything else and get a PR over to you.

This has, however, led me to the next hurdle with SQLite, which is the multithreading issues. The tests pass if I restrict Maven to a single process. But when I let Maven run the tests in parallel, they start to fail. But I think this was expected.

Also, what I'm going to have in this first PR is a few Maven profiles to allow us to enable the "large tests" that are currently annotated with @Ignore. The profiles set a system property when Maven runs that the test can check. I use JUnit's Assume.assumeTrue(boolean) in those specific tests that way the tests are marked "skipped" if the profile is not active, and the tests are executed if the profile is active.

I think that's enough for one PR. I don't want to mix too many changes in at once.

Have a good weekend and I hope your daughter's performance and her upcoming summer were/are wonderful!

Erik Merkle
Software Engineer | Boundless