Re: [jgit-dev] jgit push on http perpetually running gc before push afte

[Matthias Sohn <matthias.sohn@xxxxxxxxx>]

By default JGit gc creates two pack files, the first one containing all objects reachable from local branches and tags

and the second one the remaining objects reachable from remote tracking branches and other refs,

e.g. symbolic refs and the many refs/changes/* created by Gerrit on the server side for changes in review.

I found by trying your test on the ruby repository that when configuring the JGit gc to only pack a single

pack file by setting pack.singlePack=true there is no performance degradation on push.

I improved your test case to specify the refspec defining which branches to push where.

Since your test checks out the local default branch "trunk" I changed your test to set the refspec "HEAD:refs/heads/trunk".

Though this didn't have a noticable effect on push performance.

When repacking to a single pack either using native git's gc or jgit's gc push takes for me around 3,5sec.

When using the default jgit gc configuration creating two pack files the push takes around 4,5sec.

In both cases I configured the repo to create a bitmap index (jgit does this by default, for native git set

repack.writebitmaps=true in order to create a bitmap index).

I'll look into the details why this has such a large effect on push performance.

-Matthias

On Fri, Mar 15, 2019 at 9:22 AM Thibault Kruse <tkruse@xxxxxxxxxx> wrote:

Hi all,

so I can reproduce the problem with public github repositories alone, using the ruby programming language as source.

To reproduce, fork ruby (https://github.com/ruby/ruby) to your account, and setup the project https://github.com/tkruse/jgit-gc-push with your github credentials.

For me running the JUnit test with the gc line enabled once causes all future jgit pushes from this local repository to github to cause 8s, whereas it takes 6 seconds if jgit.gc() is not called or after calls to "git gc" in the shell.

Also with the "problematic" state, all pushes have these additional lines:

DEBUG [main][2019-03-15 17:10:55,228] [jgitdemo.JGitPushAfterGcDemoTest.reproducePushSlowdownAfterGc] - Push
DEBUG [main][2019-03-15 17:10:57,225] [jgitdemo.LoggerPrintWriter$InternalLoggerWriter.write] - Counting objects:       1
DEBUG [main][2019-03-15 17:10:58,230] [jgitdemo.LoggerPrintWriter$InternalLoggerWriter.write] - Counting objects:       37633
DEBUG [main][2019-03-15 17:10:59,065] [jgitdemo.LoggerPrintWriter$InternalLoggerWriter.write] - Counting objects:       81504

while the slowdown is less dramatic in terms of lost time than what we observe internally, I assume that it's the same problem.

As I said before, I tried the same with react (https://github.com/facebook/react), but could not reproduce, so not every repo seems to be affected in the same way.

Also, not sure if this is relevant in any way, but our internal repo has a lot of files names the same way, like sub01/definition.json, sub02/definition.json, sub03/definition.json... Might not be relevant at all though, just thought it's worth mentioning.

On Fri, Mar 15, 2019 at 3:08 PM Thibault Kruse <tkruse@xxxxxxxxxx> wrote:

Small update, I published on github java code with which I can reproduce the issue for our internal target repo: https://github.com/tkruse/jgit-gc-push/blob/master/src/test/java/jgitdemo/JGitPushAfterGcDemoTest.java

However, this only reproduces it for our own repo on our internal gitlab. I have yet to reproduce with a repo on a public server, my first attempt using facebook/react source code as a target repo did not reproduce, so not sure what additional factors are required to reproduce.

As you can see, there is no significant local configuration of jgit involved.

On Fri, Mar 15, 2019 at 11:25 AM Thibault Kruse <tkruse@xxxxxxxxxx> wrote:

> During push the number of references which the receiving repo knows of plays a role and also how

> on sending side refs and objects are packed. Would of course be nice if you can really reproduce that effect with a up to date jgit and a

> open-source repo but I am not sure how good your chances are. Feel free to try.

We can reproduce this with latest jgit 5.2.0.201812061821-r, hopefully on github later today.

And what we see is that after running "git gc" in the shell on the same repo, running jgit push on the same repo "repairs" the jgit performance problem immediately and consistently (so far).

Let me try to set up a public reproducible example today...

> how large is the max. Java heap size ?

We can reproduce this without modifying it

> what's the effective git / jgit configuration you are using ?

We can reproduce this on Ubuntu Linux without any explicit system/global git configuration (defaults).

> how did you configure gc.pruneexpire and gc.prunepackexpire ?

We did not set those.

Some other stats:

show-ref in either state:

  $ git show-ref | wc -l
  43

count-objects in "slow-push-state" repo:

  $ git count-objects -v
  count: 0
  size: 0
  in-pack: 1652188
  packs: 3
  size-pack: 477043
  prune-packable: 0
  garbage: 0
  size-garbage: 0

count-objects in "healthy-push-state" repo:

  $ git count-objects -v
  count: 0
  size: 0
  in-pack: 826094
  packs: 1
  size-pack: 237808
  prune-packable: 0
  garbage: 0
  size-garbage: 0

TRACE_PACKET log (not sure if you also need to compare between both states for this):

11:13:36.719854 pkt-line.c:80           packet:          git< # service=git-receive-pack
11:13:36.719901 pkt-line.c:80           packet:          git< 0000
11:13:36.719916 pkt-line.c:80           packet:          git< 62df589b6bda698f7fae853423e1a94e75d3210e refs/heads/jfranklin/revert_passinternational\0report-status delete-refs side-band-64k quiet atomic ofs-delta push-options agent=git/2.18.1
11:13:36.719936 pkt-line.c:80           packet:          git< 2ce2d6659ab537799fdf153adbd0f3fbb5448763 refs/heads/jira/COMP-8803
11:13:36.719953 pkt-line.c:80           packet:          git< 2502f33cb09e0bf6e0b2c76d1dae7ab1e522b4f0 refs/heads/jira/COMP-8804
11:13:36.719965 pkt-line.c:80           packet:          git< 672617fb9bcaecf25580dd5d5d40c7a69de384b2 refs/heads/jira/IDX-1862
11:13:36.719979 pkt-line.c:80           packet:          git< 4fc5d90c75bff7ea019bd445780fd09379113fa3 refs/heads/jira/MOBILE-16727
11:13:36.719993 pkt-line.c:80           packet:          git< 64cc7d18e417b4f0491b6c24d43cb1faf8c3a648 refs/heads/jira/PASS-4072
11:13:36.720011 pkt-line.c:80           packet:          git< a09a5c78eb80fe5bc886fbb38d35a85e0b644fa8 refs/heads/jira/PROC-220
11:13:36.720024 pkt-line.c:80           packet:          git< 7499f115952551d7c129de767df1383e7f22af10 refs/heads/jira/PROC-224
11:13:36.720038 pkt-line.c:80           packet:          git< 0c042b75b4f38e3a9dd88e8e2d14102c726ac70a refs/heads/jira/PROC-262-qa
11:13:36.720051 pkt-line.c:80           packet:          git< 449a89c177e3f72d05185d47bb30777782c5aa00 refs/heads/jira/PROC-265
11:13:36.720067 pkt-line.c:80           packet:          git< f1a132a9aa2ae2f3206ebac53d84ef25a35a5fbf refs/heads/jira/PROC-318
11:13:36.720077 pkt-line.c:80           packet:          git< 88437d4f07ccc0c638e83148ca7092afee59167a refs/heads/jira/PROC-354-production
11:13:36.720093 pkt-line.c:80           packet:          git< 4c45306db87a15793591f01606373c4047b6b327 refs/heads/jira/PROC-354-qa
11:13:36.720107 pkt-line.c:80           packet:          git< 3d6aa07cff6d73a6269e26f137bce28d584dd420 refs/heads/jira/PROC-354-trunk
11:13:36.720118 pkt-line.c:80           packet:          git< 9c30014d674060ff2033a0e5926b53d25f08e8ae refs/heads/jira/PROC-355-production
11:13:36.720131 pkt-line.c:80           packet:          git< efb2a343fed0bdc20aaf8d661c121f46fe07cb64 refs/heads/jira/PROC-355-qa
11:13:36.720146 pkt-line.c:80           packet:          git< 7b45d3d2c7db66afb592eb43c59f153cc9140267 refs/heads/jira/PROC-355-trunk
11:13:36.720160 pkt-line.c:80           packet:          git< 61e3af2490f2cf471687144312f1da8c1b04120e refs/heads/jira/PROC-388/production
11:13:36.720171 pkt-line.c:80           packet:          git< 79cc7754b0ce5021c357d96f12cf6e5d0dfb66b1 refs/heads/jira/PROC-388/qa
11:13:36.720184 pkt-line.c:80           packet:          git< 5fbab52bc82475100faad389f6ecb36dfd72766c refs/heads/jira/PROC-426
11:13:36.720198 pkt-line.c:80           packet:          git< 8469032709969ba0a31630cd78de569108cb9c97 refs/heads/jira/PROC-427-production
11:13:36.720211 pkt-line.c:80           packet:          git< 567730cb1f73092d7fd2d6e3a35fb9887188a89c refs/heads/jira/PROC-427-qa
11:13:36.720225 pkt-line.c:80           packet:          git< 6504c92d2eefb88ca3b3dabc2d8583cfe9c7502f refs/heads/jira/PROC-427-trunk
11:13:36.720238 pkt-line.c:80           packet:          git< 7836625d6143a5f08c60d6d477912580fefb45c8 refs/heads/jira/RADCLIENT-211
11:13:36.720252 pkt-line.c:80           packet:          git< 95ab197a89d7bf86c583273d21b687b23c0c3940 refs/heads/jira/RECOS-1777
11:13:36.720265 pkt-line.c:80           packet:          git< 6c5abf7abd42fab8a3a54d01461b20850f48687d refs/heads/jira/chongy/DRAT-2736
11:13:36.720279 pkt-line.c:80           packet:          git< 18b12f4e894918d930582097088a4a5ba885cbdf refs/heads/jira/jlamine/MOBILE-9335
11:13:36.720295 pkt-line.c:80           packet:          git< 12ba8ced1017fb4458a4dafb1dce6a0660a1f142 refs/heads/jira/jlamine/PRIME-226
11:13:36.720308 pkt-line.c:80           packet:          git< 0192f65025981a9f461e4ed422e5f32fb557cc61 refs/heads/jira/jlamine/PRIME-238
11:13:36.720321 pkt-line.c:80           packet:          git< 99fa2af06dbc382b3df32e0e63b5d2fa12f8ae9c refs/heads/jira/jlamine/PRIME-239
11:13:36.720334 pkt-line.c:80           packet:          git< 65260005b6c5fdaee4c6ab0ab0114ba45a89b761 refs/heads/jira/jlamine/PRIME-240
11:13:36.720347 pkt-line.c:80           packet:          git< 442229200132930f1697e8c3c4d0a270dd13e8bf refs/heads/proc-220-qa
11:13:36.720361 pkt-line.c:80           packet:          git< 7524d51eb16b3c16817bbaf01a5f9c23cf4999cc refs/heads/production
11:13:36.720374 pkt-line.c:80           packet:          git< 16a9317da2714f0a0db8c3eda0c24a17bee9d596 refs/heads/qa
11:13:36.720387 pkt-line.c:80           packet:          git< 15ef72afc10ad1a112423db56a1e1300b1f32ccd refs/heads/qa-proc-239
11:13:36.720400 pkt-line.c:80           packet:          git< 7f6be184b370d9f948c6c20bbd62757f15ea5e33 refs/heads/qa-rad
11:13:36.720413 pkt-line.c:80           packet:          git< 49a2db6c42c17864ec8036c0fe8cc5a54e866d0a refs/heads/revert-1da2e89a
11:13:36.720426 pkt-line.c:80           packet:          git< d4517ef096e3229b1c59234d9cac52163ff069f6 refs/heads/revert-5af93ee7
11:13:36.720439 pkt-line.c:80           packet:          git< f6a7013cdd6e19234da7a9d91321ba2cb201dfa7 refs/heads/revert-ac92f0cb
11:13:36.720452 pkt-line.c:80           packet:          git< 3ad9e711a5aa485f7a21df6d8a2195992434d5d5 refs/heads/revert/adssl-48
11:13:36.720465 pkt-line.c:80           packet:          git< caf6b9ac16e91bd28177336e4e70001ad9d6c18b refs/heads/trunk
11:13:36.720478 pkt-line.c:80           packet:          git< 2945b32ebcc11f8dbb67ee323557af9a18f88a02 refs/heads/will/hunch
11:13:36.720490 pkt-line.c:80           packet:          git< 0000
11:13:36.721433 pkt-line.c:80           packet:          git> refs/heads/trunk:refs/heads/trunk
11:13:36.721466 pkt-line.c:80           packet:          git> 0000
11:13:36.727364 pkt-line.c:80           packet:          git< refs/heads/trunk:refs/heads/trunk
11:13:36.727462 pkt-line.c:80           packet:          git< 0000
11:13:36.727494 pkt-line.c:80           packet:          git< 62df589b6bda698f7fae853423e1a94e75d3210e refs/heads/jfranklin/revert_passinternational\0report-status delete-refs side-band-64k quiet atomic ofs-delta push-options agent=git/2.18.1
11:13:36.727524 pkt-line.c:80           packet:          git< 2ce2d6659ab537799fdf153adbd0f3fbb5448763 refs/heads/jira/COMP-8803
11:13:36.727546 pkt-line.c:80           packet:          git< 2502f33cb09e0bf6e0b2c76d1dae7ab1e522b4f0 refs/heads/jira/COMP-8804
11:13:36.727568 pkt-line.c:80           packet:          git< 672617fb9bcaecf25580dd5d5d40c7a69de384b2 refs/heads/jira/IDX-1862
11:13:36.727590 pkt-line.c:80           packet:          git< 4fc5d90c75bff7ea019bd445780fd09379113fa3 refs/heads/jira/MOBILE-16727
11:13:36.727612 pkt-line.c:80           packet:          git< 64cc7d18e417b4f0491b6c24d43cb1faf8c3a648 refs/heads/jira/PASS-4072
11:13:36.727633 pkt-line.c:80           packet:          git< a09a5c78eb80fe5bc886fbb38d35a85e0b644fa8 refs/heads/jira/PROC-220
11:13:36.727652 pkt-line.c:80           packet:          git< 7499f115952551d7c129de767df1383e7f22af10 refs/heads/jira/PROC-224
11:13:36.727671 pkt-line.c:80           packet:          git< 0c042b75b4f38e3a9dd88e8e2d14102c726ac70a refs/heads/jira/PROC-262-qa
11:13:36.727691 pkt-line.c:80           packet:          git< 449a89c177e3f72d05185d47bb30777782c5aa00 refs/heads/jira/PROC-265
11:13:36.727710 pkt-line.c:80           packet:          git< f1a132a9aa2ae2f3206ebac53d84ef25a35a5fbf refs/heads/jira/PROC-318
11:13:36.727730 pkt-line.c:80           packet:          git< 88437d4f07ccc0c638e83148ca7092afee59167a refs/heads/jira/PROC-354-production
11:13:36.727750 pkt-line.c:80           packet:          git< 4c45306db87a15793591f01606373c4047b6b327 refs/heads/jira/PROC-354-qa
11:13:36.727770 pkt-line.c:80           packet:          git< 3d6aa07cff6d73a6269e26f137bce28d584dd420 refs/heads/jira/PROC-354-trunk
11:13:36.727789 pkt-line.c:80           packet:          git< 9c30014d674060ff2033a0e5926b53d25f08e8ae refs/heads/jira/PROC-355-production
11:13:36.727809 pkt-line.c:80           packet:          git< efb2a343fed0bdc20aaf8d661c121f46fe07cb64 refs/heads/jira/PROC-355-qa
11:13:36.727829 pkt-line.c:80           packet:          git< 7b45d3d2c7db66afb592eb43c59f153cc9140267 refs/heads/jira/PROC-355-trunk
11:13:36.727848 pkt-line.c:80           packet:          git< 61e3af2490f2cf471687144312f1da8c1b04120e refs/heads/jira/PROC-388/production
11:13:36.727867 pkt-line.c:80           packet:          git< 79cc7754b0ce5021c357d96f12cf6e5d0dfb66b1 refs/heads/jira/PROC-388/qa
11:13:36.727887 pkt-line.c:80           packet:          git< 5fbab52bc82475100faad389f6ecb36dfd72766c refs/heads/jira/PROC-426
11:13:36.727906 pkt-line.c:80           packet:          git< 8469032709969ba0a31630cd78de569108cb9c97 refs/heads/jira/PROC-427-production
11:13:36.727925 pkt-line.c:80           packet:          git< 567730cb1f73092d7fd2d6e3a35fb9887188a89c refs/heads/jira/PROC-427-qa
11:13:36.727945 pkt-line.c:80           packet:          git< 6504c92d2eefb88ca3b3dabc2d8583cfe9c7502f refs/heads/jira/PROC-427-trunk
11:13:36.727964 pkt-line.c:80           packet:          git< 7836625d6143a5f08c60d6d477912580fefb45c8 refs/heads/jira/RADCLIENT-211
11:13:36.727983 pkt-line.c:80           packet:          git< 95ab197a89d7bf86c583273d21b687b23c0c3940 refs/heads/jira/RECOS-1777
11:13:36.728002 pkt-line.c:80           packet:          git< 6c5abf7abd42fab8a3a54d01461b20850f48687d refs/heads/jira/chongy/DRAT-2736
11:13:36.728034 pkt-line.c:80           packet:          git< 18b12f4e894918d930582097088a4a5ba885cbdf refs/heads/jira/jlamine/MOBILE-9335
11:13:36.728053 pkt-line.c:80           packet:          git< 12ba8ced1017fb4458a4dafb1dce6a0660a1f142 refs/heads/jira/jlamine/PRIME-226
11:13:36.728073 pkt-line.c:80           packet:          git< 0192f65025981a9f461e4ed422e5f32fb557cc61 refs/heads/jira/jlamine/PRIME-238
11:13:36.728092 pkt-line.c:80           packet:          git< 99fa2af06dbc382b3df32e0e63b5d2fa12f8ae9c refs/heads/jira/jlamine/PRIME-239
11:13:36.728111 pkt-line.c:80           packet:          git< 65260005b6c5fdaee4c6ab0ab0114ba45a89b761 refs/heads/jira/jlamine/PRIME-240
11:13:36.728131 pkt-line.c:80           packet:          git< 442229200132930f1697e8c3c4d0a270dd13e8bf refs/heads/proc-220-qa
11:13:36.728152 pkt-line.c:80           packet:          git< 7524d51eb16b3c16817bbaf01a5f9c23cf4999cc refs/heads/production
11:13:36.728171 pkt-line.c:80           packet:          git< 16a9317da2714f0a0db8c3eda0c24a17bee9d596 refs/heads/qa
11:13:36.728190 pkt-line.c:80           packet:          git< 15ef72afc10ad1a112423db56a1e1300b1f32ccd refs/heads/qa-proc-239
11:13:36.728210 pkt-line.c:80           packet:          git< 7f6be184b370d9f948c6c20bbd62757f15ea5e33 refs/heads/qa-rad
11:13:36.728229 pkt-line.c:80           packet:          git< 49a2db6c42c17864ec8036c0fe8cc5a54e866d0a refs/heads/revert-1da2e89a
11:13:36.728248 pkt-line.c:80           packet:          git< d4517ef096e3229b1c59234d9cac52163ff069f6 refs/heads/revert-5af93ee7
11:13:36.728267 pkt-line.c:80           packet:          git< f6a7013cdd6e19234da7a9d91321ba2cb201dfa7 refs/heads/revert-ac92f0cb
11:13:36.728286 pkt-line.c:80           packet:          git< 3ad9e711a5aa485f7a21df6d8a2195992434d5d5 refs/heads/revert/adssl-48
11:13:36.728306 pkt-line.c:80           packet:          git< caf6b9ac16e91bd28177336e4e70001ad9d6c18b refs/heads/trunk
11:13:36.728325 pkt-line.c:80           packet:          git< 2945b32ebcc11f8dbb67ee323557af9a18f88a02 refs/heads/will/hunch
11:13:36.728342 pkt-line.c:80           packet:          git< 0000
11:13:36.730313 pkt-line.c:80           packet:          git> caf6b9ac16e91bd28177336e4e70001ad9d6c18b 6b52419701c154e16eca2a111234df77e49b4f65 refs/heads/trunk\0 report-status side-band-64k agent=git/2.21.0
11:13:36.730342 pkt-line.c:80           packet:          git> 0000
11:13:36.730461 pkt-line.c:80           packet:          git< 0094caf6b9ac16e91bd28177336e4e70001ad9d6c18b 6b52419701c154e16eca2a111234df77e49b4f65 refs/heads/trunk\0 report-status side-band-64k agent=git/2.21.00000
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 472 bytes | 472.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
11:13:52.359706 pkt-line.c:80           packet:          git< PACK ...
11:13:52.372780 pkt-line.c:80           packet:          git> 0000
11:13:53.210461 pkt-line.c:80           packet:     sideband< \1000eunpack ok0018ok refs/heads/trunk0000
11:13:53.210580 pkt-line.c:80           packet:     sideband< 0000
11:13:53.210621 pkt-line.c:80           packet:          git< unpack ok
11:13:53.210671 pkt-line.c:80           packet:          git< ok refs/heads/trunk
11:13:53.210703 pkt-line.c:80           packet:          git< 0000
11:13:53.210721 pkt-line.c:80           packet:          git> 0000
To https://code.corp.indeed.com/tkruse/proctor-data.git
   caf6b9ac16e..6b52419701c  trunk -> trunk