Re: [jgit-dev] Refs creation slowness with Jgit

[Matthias Sohn <matthias.sohn@xxxxxxxxx>]

On Wed, Sep 23, 2020 at 8:05 PM <kaushikl@xxxxxxxxxxxxxx> wrote:

CC: repo-discuss@xxxxxxxxxxxxxxxx

On 2020-09-23 09:13, kaushikl@xxxxxxxxxxxxxx wrote:
> Hello,
>
> When investigating slow Gerrit NoteDB migration times, we noticed a
> slowness with jgit creating refs. During the migration a ref is
> created for each Gerrit change, for example ‘changes/01/1/meta’,
> ‘changes/02/2/meta’, ‘changes/03/3/meta’ and so on. We see the
> slowness when creating refs in a new path. This behaviour is
> repeatable, i.e each time a ref is created in a new path, it is slow.
> We noticed it is fast on some machines, but we couldn't quite nail
> down why.
>
> I have a small sample program[1] which illustrates the slowness. It
> creates two refs 'refs/heads/test_simple' and 'refs/heads/test/foo'.
> On executing the program, I see output:
>
> test/foo: 325 ms
> test_simple: 4 ms
>
> My expectation is that 'test/foo' will also be created in order of
> milliseconds and not in order of few hundred milliseconds. The
> slowness seems to be stemming from the Files.getFileStore(dir) call in
> FS.FileStoreAttributes.getFileStoreAttributes(Path dir).

Can you run the LookupFileStoreBenchmark [1] on this system ?
On my Mac this yields 63us to lookup a FileStore for a new file path.

After running the Maven build the benchmarks can be run using Maven:

$ cd jgit

$ mvn clean install -DskipTests -Dmaven.javadoc.skip=true

$ java -jar org.eclipse.jgit.benchmarks/target/benchmarks.jar

or in Eclipse run the main method of the LookupFileStoreBenchmark class.

[1] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/master/org.eclipse.jgit.benchmarks/src/org/eclipse/jgit/benchmarks/LookupFileStoreBenchmark.java

 

-Matthias

> Specs:
>
> Jgit version is 5.9
>
> $ java -version # also tried with version 1.8.0_252
> openjdk version "1.8.0_232"
> OpenJDK Runtime Environment (build
> 1.8.0_232-8u232-b09-0ubuntu1~16.04.1-b09)
> OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
>
> $ uname -r
> 4.15.0-54-generic
>
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 16.04.6 LTS
> Release:        16.04
> Codename:       xenial
>
> Filesystem type is ext4
>
>
>
> [1]
> package test;
>
> import java.io.IOException;
> import java.nio.charset.StandardCharsets;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.Paths;
> import java.util.Arrays;
> import java.util.concurrent.TimeUnit;
>
> import org.eclipse.jgit.api.Git;
> import org.eclipse.jgit.api.errors.GitAPIException;
> import org.eclipse.jgit.lib.Repository;
> import org.eclipse.jgit.revwalk.RevCommit;
> import org.eclipse.jgit.revwalk.RevWalk;
>
> public class Test {
>   public static void main(String[] args) {
>     try {
>
>       String path = null;
>      if (args.length == 1) {
>         path = args[0];
>       } else {
>         System.out.println("Repo path must be specified.");
>         System.exit(1);
>       }
>       Path repoPath = Paths.get(path);
>       long startTimeInNanoSecs;
>       long estimatedTimeInNanoSecs;
>
>       Git git = Git.init().setDirectory(repoPath.toFile()).call();
>       Repository repo = git.getRepository();
>       RevWalk walk = new RevWalk(repo);
>
>       Files.write(repoPath.resolve("file"), Arrays.asList("line"),
> StandardCharsets.UTF_8);
>       git.add().addFilepattern("file").call();
>       git.commit().setMessage("create file").setAuthor("author",
> "author@xxxxxxxxx").call();
>       RevCommit commit =
> walk.parseCommit(repo.exactRef("refs/heads/master").getObjectId());
>
>       String branch = "test/foo";
>       startTimeInNanoSecs = System.nanoTime();
>       git.branchCreate().setName(branch).setStartPoint(commit).call();
>       estimatedTimeInNanoSecs = System.nanoTime() -
> startTimeInNanoSecs;
>       System.out.println(branch + ": " +
> TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
>
>       branch = "test_simple";
>       startTimeInNanoSecs = System.nanoTime();
>       git.branchCreate().setName(branch).setStartPoint(commit).call();
>       estimatedTimeInNanoSecs = System.nanoTime() -
> startTimeInNanoSecs;
>       System.out.println(branch + ": " +
> TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
>     } catch (IllegalStateException | GitAPIException | IOException e) {
>      e.printStackTrace();
>     }
>   }
> }
> _______________________________________________
> jgit-dev mailing list
> jgit-dev@xxxxxxxxxxx
> To unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jgit-dev
_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jgit-dev