[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] Writing packs with bitmaps sometimes slower as without bitmaps?

Hi,
found time to proceed on this issue.

I can now reproduce the issue with the public gerrit repo. When pushing one new commit to a local clone of the gerrit repo (a repo having a lot of refs) then the push is 8 times slower if the sending repo has bitmaps. Most of the time is spent inÂorg.eclipse.jgit.revwalk.BitmapWalker.findObjectsWalk(). The script which execution log I append can be downloaded fromÂhttps://gist.github.com/chalstrick/864fecf5cc45c056e90225418d6b9c89


++ jgit --version
jgit version 5.1.8-SNAPSHOT
++ rm -fr gerrit.src.git gerrit.dst.git gerrit.client
++ git clone --bare --mirror https://gerrit.googlesource.com/gerrit gerrit.dst.git
...
++ cp -r gerrit.dst.git gerrit.dst.git.backup
++ git clone --bare --mirror gerrit.dst.git gerrit.src.git
Cloning into bare repository 'gerrit.src.git'...
done.
++ git clone gerrit.src.git gerrit.client
Cloning into 'gerrit.client'...
done.
++ cd gerrit.client
++ date
++ git add README.md
++ git commit -m 'modify README.md'
[master 7d26b6c82f8] modify README.md
Â1 file changed, 1 insertion(+)
++ git push origin
...
++ cd gerrit.src.git

##### The fast push (2s) when we don't have bitmaps

++ jgit push origin HEAD:refs/heads/master
Counting objects: Â Â Â 3
Finding sources: Â Â Â Â100% (3/3)
Getting sizes: Â Â Â Â Â100% (2/2)
Compressing objects: Â Â100% (4210/4210)
Writing objects: Â Â Â Â100% (3/3)
remote: Updating references: 100% (1/1)To /Users/d032780/git/repl_test/gerrit.dst.git
 Â78e6f24..7d26b6c ÂHEAD -> master

real  Â0m2.173s
user  Â0m5.084s
sys   0m0.525s
++ rm -fr gerrit.dst.git
++ cp -r gerrit.dst.git.backup gerrit.dst.git
++ cd gerrit.src.git
### Force creating bitmaps
++ git repack -a -d -b
Enumerating objects: 1053003, done.
Counting objects: 100% (1053003/1053003), done.
Delta compression using up to 8 threads
Compressing objects: 100% (436944/436944), done.
Writing objects: 100% (1053003/1053003), done.
Selecting bitmap commits: 238041, done.
Building bitmaps: 100% (352/352), done.
Total 1053003 (delta 467117), reused 1052949 (delta 467065)

##### The slow push (9s) when we have bitmaps

++ jgit push origin HEAD:refs/heads/master
Counting objects: Â Â Â 528225
Finding sources: Â Â Â Â100% (3/3)
Getting sizes: Â Â Â Â Â100% (2/2)
Compressing objects: Â Â100% (1142/1142)
Writing objects: Â Â Â Â100% (3/3)
remote: Updating references: 100% (1/1)To /Users/d032780/git/repl_test/gerrit.dst.git
 Â78e6f24..7d26b6c ÂHEAD -> master

real  Â0m9.070s
user  Â0m12.130s
sys   0m1.015s
>
ÂÂ

On Mon, May 13, 2019 at 2:43 PM Christian Halstrick <christian.halstrick@xxxxxxxxx> wrote:
Hi,
I was trying to reproduce it on a big public repo but up to know had
no success with that. That's a very time consuming task because I need
huge repos with a ton of refs. Tried with linux, no luck. Now I check
with chromium. I inform you when I managed that.

But by just inspecting the code and looking at the trace output I see
a difference between the bitmap aware code and the non-bitmap-aware
code in PackWriter#findObjectsToPack. In my repeatable case
BitmapWalker.findObjects(have, null, true)
runs for 15 minutes :-(

On Fri, May 10, 2019 at 10:53 PM Ivan Frade <ifrade@xxxxxxxxxx> wrote:
>
> Hi Christian,
>
>Â I have been working with bitmaps lately [1]. My changes doesn't affect PackWriter (they are not even committed yet!), but I am interested in everything bitmap-related.
>
>Â Did you gather any more information about this issue? Any chance to reproduce it in a test?
>
>Â Regards,
>
> Ivan
>
>Â [1] https://git.eclipse.org/r/c/140958/
>
>
>
> From: Christian Halstrick <christian.halstrick@xxxxxxxxx>
> Date: Wed, May 8, 2019 at 4:04 PM
> To: JGit Developers list
>
>> Hi,
>>
>> I am investigating a performance problem non-public gerrit servers.
>> The problem boils
>> down to slow performance of PackWriter when using bitmaps. When the
>> usage of bitmaps
>> is turned off then performance is at least 5 times better.
>>
>> I uploaded a the change https://git.eclipse.org/r/141842 to
>> demonstrate that. That change
>> adds some printfs to emit performance data for the PackWriter class.
>> Additionally a System
>> property PackWriterForceNoBitmap is introduced that when set to true
>> forces PackWriter not
>> to use bitmaps.
>>
>> Problem is the performance of PackWriter#findObjectsToPack. That
>> method delegates to
>> PackWriter#findObjectsToPackUsingBitmaps() which is in my case much
>> slower than using
>> the default code not using bitmaps. It looks like
>> findObjectsToPackUsingBitmaps() is first calculating all have objects,
>> then all want objects and then calculates the difference. In huge
>> repos calculating all have objects is consuming 900sec.
>> The non-bitmap code in findObjectsToPack() creates a walk where have
>> and want objects are both used and has to walk only over very few
>> objects which takes only 200sec.
>> In the end both algorithms (bitmap and non-bitmap aware code) find the
>> same result: only one commit with one new blob has to be sent.
>>
>> Is somebody aware of the fact that when working on packfiles of 2GB
>> size findObjectsToPackUsingBitmaps() is so much slower than
>> non-bitmap-aware code?
>>
>> Stats of the repo:
>> $ du -sh *.pack
>> 1.1MÂ Â pack-097c243a771df372d4e1098af0d89a3d25be8de6.pack
>>Â 36KÂ Â pack-0bc2fecc4d3c070b82285b70a6394b15f87a9e50.pack
>>Â 64KÂ Â pack-57c34e10b1cc0305784f48bda175064e2ad8fa1a.pack
>> 240KÂ Â pack-7cc06b124492dcab6fda3e797f3c16fe06929e2a.pack
>> 2.0GÂ Â pack-e7a9322fa39225c670cab245b9e012c1aa6f3a61.pack
>>
>>
>> Here are the traces:
>>
>> =================================================
>> added performance printfs, forced not to use bitmaps
>> =================================================
>> ...
>> Counting objects:Â Â Â Â1
>> Counting objects:Â Â Â Â4
>> TracePushPerf: findObjectsToPush(): code not using bitmaps runtime: 251776
>> TracePushPerf: findObjectsToPack() runtime: 251916
>>
>> Finding sources:Â Â Â Â Â25% (1/4)
>> Finding sources:Â Â Â Â Â50% (2/4)
>> Finding sources:Â Â Â Â Â75% (3/4)
>> Finding sources:Â Â Â Â 100% (4/4)
>> Finding sources:Â Â Â Â 100% (4/4)
>>
>> Getting sizes:Â Â Â Â Â Â33% (1/3)
>> Getting sizes:Â Â Â Â Â Â66% (2/3)
>> Getting sizes:Â Â Â Â Â 100% (3/3)
>> Getting sizes:Â Â Â Â Â 100% (3/3)
>>
>> Compressing objects:Â Â Â99% (8001/8060)
>> Compressing objects:Â Â 100% (8060/8060)
>> Compressing objects:Â Â 100% (8060/8060)
>>
>> Writing objects:Â Â Â Â Â25% (1/4)
>> Writing objects:Â Â Â Â Â50% (2/4)
>> Writing objects:Â Â Â Â Â75% (3/4)
>> Writing objects:Â Â Â Â 100% (4/4)
>> Writing objects:Â Â Â Â 100% (4/4)
>>
>> remote: Updating references: 100% (1/1)To
>> /Users/d032780/git/repl_test/hana.dst.git/
>>Â Â 766c519..cdcc784Â temp_d056507 -> temp_d056507
>>
>> =================================================
>> added performance printfs, using bitmaps
>> =================================================
>> Counting objects:Â Â Â Â1
>> Counting objects:Â Â Â Â41507
>> Counting objects:Â Â Â Â100967
>> Counting objects:Â Â Â Â165489
>> ...
>> Counting objects:Â Â Â Â3946346
>> Counting objects:Â Â Â Â3948655
>> TracePushPerf: findObjectsToPackUsingBitmaps() ms to find find haves: 929147
>>
>> Counting objects:Â Â Â Â4065807
>> TracePushPerf: findObjectsToPackUsingBitmaps()Â ms to find find want: 638
>> TracePushPerf: findObjectsToPackUsingBitmaps()Â ms to find find need: 8
>> TracePushPerf: findObjectsToPackUsingBitmaps()Â ms to add needed: 2
>> TracePushPerf: findObjectsToPackUsingBitmaps() runtime: 929795
>>
>> Counting objects:Â Â Â Â4103019
>> TracePushPerf: findObjectsToPack() runtime: 938584
>>
>> Finding sources:Â Â Â Â Â25% (1/4)
>> Finding sources:Â Â Â Â Â50% (2/4)
>> Finding sources:Â Â Â Â Â75% (3/4)
>> Finding sources:Â Â Â Â 100% (4/4)
>> Finding sources:Â Â Â Â 100% (4/4)
>>
>> Getting sizes:Â Â Â Â Â Â33% (1/3)
>> Getting sizes:Â Â Â Â Â Â66% (2/3)
>> Getting sizes:Â Â Â Â Â 100% (3/3)
>> Getting sizes:Â Â Â Â Â 100% (3/3)
>>
>> Compressing objects:Â Â Â99% (8001/8060)
>> Compressing objects:Â Â 100% (8060/8060)
>> Compressing objects:Â Â 100% (8060/8060)
>>
>> Writing objects:Â Â Â Â Â25% (1/4)
>> Writing objects:Â Â Â Â Â50% (2/4)
>> Writing objects:Â Â Â Â Â75% (3/4)
>> Writing objects:Â Â Â Â 100% (4/4)
>> Writing objects:Â Â Â Â 100% (4/4)
>>
>> remote: Updating references: 100% (1/1)To
>> /Users/d032780/git/repl_test/hana.dst.git/
>>Â Â 766c519..cdcc784Â temp_d056507 -> temp_d056507
>> _______________________________________________
>> jgit-dev mailing list
>> jgit-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://www.eclipse.org/mailman/listinfo/jgit-dev