Re: [jgit-dev] Creating new refs on a repository with 100k packed-refs i

[kaushikl@xxxxxxxxxxxxxx]

On 2020-10-06 12:50, Luca Milanesio wrote:
> > On 6 Oct 2020, at 20:48, kaushikl@xxxxxxxxxxxxxx wrote:
> > On 2020-10-06 12:32, Luca Milanesio wrote:
> > On 6 Oct 2020, at 20:16, kaushikl@xxxxxxxxxxxxxx wrote:
> > Hi,
> > We have noticed that creating a new ref takes ~600ms on a repository
> > with around 100k packed-refs.
> > My test repository:
> > $ git count-objects -v
> > count: 0
> > size: 0
> > in-pack: 100042
> > packs: 1
> > size-pack: 9734
> > prune-packable: 0
> > garbage: 0
> > size-garbage: 0
> > $ find refs/ -type f | wc -l
> > 0
> > Specs:
> > Machine has 32 cores and 250G RAM.
> > $ uname -r
> > 4.4.0-165-generic
> > $ lsb_release -a
> > No LSB modules are available.
> > Distributor ID:    Ubuntu
> > Description:    Ubuntu 16.04.6 LTS
> > Release:    16.04
> > Codename:    xenial
> > Jgit version is 5.9
> > $ java -version
> > openjdk version "1.8.0_242"
> > OpenJDK Runtime Environment (build
> > 1.8.0_242-8u242-b08-0ubuntu3~16.04-b08)
> > OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
> > Consider a small program[1] which creates a ref 'simple'. On
> > executing the program on my test repository, I see output:
> > simple: 677 ms
> > This seems slow. Is this expected behavior with jgit?
> > Can you share the generated .gitconfig?
> > (When you use JGit on a filesystem, it performs the computation of
> > the
> > filesystem latency)
>
> $ cat config
> [core]
>  repositoryformatversion = 0
>  filemode = true
>  bare = false
>  logallrefupdates = true
>  trustFolderStat = false
> [repack]
>  usedeltabaseoffset = true
> [pack]
>  compression = 9
>  indexversion = 2
>  threads = 6
>  windowmemory = 2g
>  window = 250
>  depth = 50
> [gc]
>  autopacklimit = 4
>  packrefs = true
>  reflogexpire = never
>  reflogexpireunreachable = never
>  auto = 0
> [receive]
>  denyNonFastForwards = false
>
> No, I mean the ~/.gitconfig of the user. If you are running Gerrit
> v3.1 or later, then it is $GERRIT_SITE/etc/jgit.config.

Ah, OK, I see below in ~/.gitconfig.

[filesystem "Private Build|1.8.0_232|/dev/mapper/xxxxxx_local_mnt"]
        timestampResolution = 1001 microseconds
        minRacyThreshold = 5289 microseconds
[filesystem "Private 
Build|1.8.0_242|/dev/mapper/xxxxxxx-xxxxxxx_local_mnt"]
        timestampResolution = 1001 microseconds
        minRacyThreshold = 5486 microseconds

> > > Is this a local disk or a mounted NFS share?
> > > SSD or spinning?
> >
> > This is on a local spinning disk.
>
> Is the disk fragmented?

I don't know how to check that, however, I copied the repository to 
another machine with similar configuration and noticed that it was 
showing similar performance( i.e greater than ~600ms) for ref creation.

> Luca.
>
> > Luca.
> > [1]
> > package test;
> > import java.io.File;
> > import java.io.IOException;
> > import java.util.concurrent.TimeUnit;
> > import org.eclipse.jgit.api.Git;
> > import org.eclipse.jgit.api.errors.GitAPIException;
> > import org.eclipse.jgit.lib.Repository;
> > import org.eclipse.jgit.revwalk.RevCommit;
> > import org.eclipse.jgit.revwalk.RevWalk;
> > public class Test {
> > public static void main(String[] args) {
> > try {
> > String path = null;
> > if (args.length == 1) {
> > path = args[0];
> > } else {
> > System.out.println("Repo path must be specified.");
> > System.exit(1);
> > }
> > String branch = "simple";
> > try (Git git = Git.open(new File(path))) {
> > Repository repo = git.getRepository();
> > RevWalk walk = new RevWalk(repo);
> > RevCommit commit =
> > walk.parseCommit(repo.exactRef("refs/heads/master").getObjectId());
> > long startTimeInNanoSecs = System.nanoTime();
> >
> > git.branchCreate().setName(branch).setStartPoint(commit).call();
> > long estimatedTimeInNanoSecs = System.nanoTime() -
> > startTimeInNanoSecs;
> > System.out.println(branch + ": " +
> > TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
> > }
> > } catch (IllegalStateException | GitAPIException | IOException e)
> > {
> > e.printStackTrace();
> > }
> > }
> > }
> > _______________________________________________
> > jgit-dev mailing list
> > jgit-dev@xxxxxxxxxxx
> > To unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jgit-dev