Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] LFS File handles

Hey,

I tracked the issue back to https://git.eclipse.org/r/#/c/129501/ which seems to remove the in.close() call, which leaves the streams to the LFS storage open.

When I revert these commits for Gerrit:

 957b95c7fd Upgrade JGit to 5.1.1.201809181055-r
 db1da63283 Update Jetty to 9.4.12

And this one for the LFS plugin:

 04c943e Upgrade JGit to 5.1.1.201809181055-r

Things start working again, file handles are stable ~200-~500.

From here, someone else needs to take over, my knowledge and time come to an end :) But I'm sure David will find and fix it in a blink :)

Cheers,
Markus

-----Original Message-----
From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
Sent: Thursday, October 11, 2018 9:02 AM
To: Matthias Sohn <matthias.sohn@xxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: RE: LFS File handles

Hey,

Thanks again for the hint to the leak detector. I spent some more time with it. For some reason, it is quite hard to get it to trigger properly xD It seems that ulimit does not have any effect on the process in the docker container (I could open 2M files according to lsof, ulimit is 1M), and the threshold for the leak detector does not seem to work (it never triggers). I now set the ulimit below the actual file count in the LFS (2200, I set it to 2000), and now it fires. Must be something with LWP on linux and threads... :|

Whatever, this is the typical stack for each file:

#195 data/product/lfs/0d/78/0d788ceb1440b20fa5ebdbf627096846a1039d9ae74daa92dfc9fa9495e663a3 by thread:HTTP-112 on Thu Oct 11 06:57:39 UTC 2018
        at java.nio.channels.FileChannel.open(FileChannel.java:287)
        at java.nio.channels.FileChannel.open(FileChannel.java:335)
        at org.eclipse.jgit.lfs.server.fs.FileLfsRepository.getReadChannel(FileLfsRepository.java:151)
        at org.eclipse.jgit.lfs.server.fs.ObjectDownloadListener.<init>(ObjectDownloadListener.java:103)
        at com.googlesource.gerrit.plugins.lfs.fs.LfsFsContentServlet.doGet(LfsFsContentServlet.java:83)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:244)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

Seems to be related to JGit's LFS support then? I tried downgrading /only/ the LFS plugin's JGIT, that didn't seem to help. Next is to downgrade both LFS plugin and Gerrit's own JGit and see if that helps.

Cheers,
Markus

-----Original Message-----
From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
Sent: Wednesday, October 10, 2018 5:08 PM
To: Matthias Sohn <matthias.sohn@xxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: RE: LFS File handles

Thanks for the tip. Unfortunately it does not seem to get hold of the leak. It attaches successfully, but misses what happens. I played with thresholds vs. letting it run into hard limits, nothing is detected :|

Cheers,
Markus

-----Original Message-----
From: Matthias Sohn [mailto:matthias.sohn@xxxxxxxxx]
Sent: Wednesday, October 10, 2018 4:53 PM
To: Duft Markus <Markus.Duft@xxxxxxxxxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: Re: LFS File handles

You may try to use file-leak-detector [1] to track down the problem.
IIRC Chris used that successfully
in similar cases to find file handle leaks in JGit.

[1] https://github.com/kohsuke/file-leak-detector

-Matthias

On Wed, Oct 10, 2018 at 4:45 PM Duft Markus
<Markus.Duft@xxxxxxxxxxxxxxxx> wrote:
>
> I reverted db1da6328319c15903520a23e303dd91d2a4e270, and it does not help. I get less open file handles, but still ~170000 per clone.
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 4:40 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: RE: LFS File handles
>
>
>
> Ok… more info: it is not the LFS plugin J Further guesses would be the jetty upgrade (again). I rolled back to the previous Gerrit version for now.
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 4:08 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: RE: LFS File handles
>
>
>
> I tested this now, and it seems that a clone of a repo with relatively low LFS file count will increase the open file count unproportionally:
>
>
>
> mduft@fril0041 /tmp/wamas $ git lfs ls-files | wc -l
>
> 2230
>
>
>
> [root@git site]# lsof | grep lfs | wc -l
>
> 253368
>
> <cloned the repo here>
>
> [root@git site]# lsof | grep lfs | wc -l
>
> 521140
>
> [root@git site]#
>
>
>
> This is a test server with zero traffic except myself. Not good. K Trying to roll back now.
>
>
>
> Cheers,
>
> Markus
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 3:58 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: LFS File handles
>
>
>
> Hey,
>
>
>
> I’ve upgraded our gerrit installation to latest master and as well upgraded the LFS plugin to latest master. We’re now experiencing a whole lot of too many open files issues. It seems related to the LFS plugin:
>
>
>
> java      1 138 root 3821r      REG              253,6        4488  4721537 /usr/local/gerrit/site/data/product/lfs/9a/20/9a202f462d7ad8c27fbc7ebbc8d17cab02d19293c46418a8eb2087548801d405
>
> java      1 138 root 3822r      REG              253,6          88  4721677 /usr/local/gerrit/site/data/product/lfs/7c/ce/7cce17f0da539a28e96d5df3853daa638e0c244c390d2ae5c8f7f4cdc7247e80
>
> java      1 138 root 3823r      REG              253,6         825  6693745 /usr/local/gerrit/site/data/product/lfs/17/57/175797fde1ea61d136f25f8a109bf6b63587e7f7b42d51da5c752d360e66caa1
>
> java      1 138 root 3824r      REG              253,6      621454  4982722 /usr/local/gerrit/site/data/product/lfs/e9/5d/e95dcc99917a8eaedde432a60b932869889a77d30b875a7f64c01aff7d150a0f
>
> java      1 138 root 3825r      REG              253,6         698  6037124 /usr/local/gerrit/site/data/product/lfs/19/1c/191c6508c85c4249d016f69abeb73eefdf4692976e2c29f386c5a5c4cdb029fe
>
> java      1 138 root 3826r      REG              253,6        2363  4721342 /usr/local/gerrit/site/data/product/lfs/e9/3e/e93e4860dbdcf4f2e56a43b6d870d2a581984053c4d2a5a214dbee9bba2e21c6
>
> java      1 138 root 3827r      REG              253,6         250  4852489 /usr/local/gerrit/site/data/product/lfs/d9/78/d9785a8c792de86627a73279c647edb44d01eb423667a58cd864f6623c22f97e
>
> java      1 138 root 3828r      REG              253,6        1056  4721820 /usr/local/gerrit/site/data/product/lfs/f3/14/f31469ff9e59335484dd18f4b9f679c1d1e2bdaf33015ba432d01976a66ab9ad
>
> java      1 138 root 3829r      REG              253,6        9528  4589367 /usr/local/gerrit/site/data/product/lfs/ca/e0/cae0854a6bb5f5a01400f8ecf895907cbe2733daa48a2dc273e72d328fb098eb
>
> java      1 138 root 3830r      REG              253,6       18672  4589164 /usr/local/gerrit/site/data/product/lfs/8b/8a/8b8a19f671d8ac40d3ff83ab769f815d6cadb5b95532e8940f70240cbac68f82
>
> java      1 138 root 3831r      REG              253,6         488  4721373 /usr/local/gerrit/site/data/product/lfs/74/4c/744c7aba265ac3e9a0469abb61c9528ffbc6434a5f6a76aaea8a2c8baf234d4f
>
> java      1 138 root 3832r      REG              253,6         714  4852669 /usr/local/gerrit/site/data/product/lfs/e9/e1/e9e189bc03d32aa42e49a963b3279ba49ea57188e6fd9261c757b49a2ba46dfe
>
> java      1 138 root 3833r      REG              253,6          82  5251861 /usr/local/gerrit/site/data/product/lfs/66/92/669295c78c8c4a492b1fd265dd03816de462fb6aa5c8246ba1ed867fe2e42cc7
>
> java      1 138 root 3834r      REG              253,6         153  5116792 /usr/local/gerrit/site/data/product/lfs/af/af/afaf7cf17039497111e545fa0dfffe212d62667f27fbe3f59dd0390411bc9934
>
> java      1 138 root 3835r      REG              253,6        1400  1835368 /usr/local/gerrit/site/data/product/lfs/84/d0/84d0daad31520f2bbd24f331a9587eadccc7bb7cd280dee7f6eac58162254c9d
>
> java      1 138 root 3836r      REG              253,6         115  4984796 /usr/local/gerrit/site/data/product/lfs/43/6c/436c00e4f855a2fafa57d275ae0415634430858593c419c5b0301bc81f805138
>
> java      1 138 root 3837r      REG              253,6         244  1703966 /usr/local/gerrit/site/data/product/lfs/7d/18/7d1853205bc21208d7d7f0141d0f4e9c800f9ace89e02f199296bc49df118c63
>
> java      1 138 root 3838r      REG              253,6       18672  4589526 /usr/local/gerrit/site/data/product/lfs/38/da/38da45c14327cb3c96bb0ea1a7a07e859a2ae97c7922e36813af3fe559d73066
>
> java      1 138 root 3839r      REG              253,6      281298   141115 /usr/local/gerrit/site/data/product/lfs/1a/a9/1aa98cf694dc9c1abde450c386c0e87a19d2300fe6225b43d5ad59ff068018dc
>
> java      1 138 root 3840r      REG              253,6        1454  4589156 /usr/local/gerrit/site/data/product/lfs/84/67/8467fe4d83a9c6007887c4b2ee06b7f76a1787ce0b6528e5c1bd056460993003
>
> java      1 138 root 3841r      REG              253,6         159  4853185 /usr/local/gerrit/site/data/product/lfs/80/68/806825a90029c998211387e42e0b5e57a654af44c49ccbeacee54dd018ec3ba4
>
> java      1 138 root 3842r      REG              253,6         488  4588655 /usr/local/gerrit/site/data/product/lfs/75/c2/75c20a35d7433e17d02203c1254d2356dc4d2c63c9ef2d8654fda4eea401517b
>
> java      1 138 root 3843r      REG              253,6        2271  4852874 /usr/local/gerrit/site/data/product/lfs/80/a1/80a16ebc9c7101758f736f77528f9c44afc561fb756f87b0340a36dd04d104e1
>
> java      1 138 root 3844r      REG              253,6        8554  4588860 /usr/local/gerrit/site/data/product/lfs/2c/b4/2cb43f4ff31f4518cb1ce884df7d8caf140e1ca28d6341020cd67f0995d603bb
>
> java      1 138 root 3845r      REG              253,6        5469  4721323 /usr/local/gerrit/site/data/product/lfs/32/f2/32f2fbfca553e27040108c6060de4d50e38ab83e7e6bba6556cadc9420a51d0a
>
> java      1 138 root 3846r      REG              253,6        9245  4721733 /usr/local/gerrit/site/data/product/lfs/f7/38/f738756f2606f2c5e59bb99c4d1c28906386fb9c014f84bcd0b2df4d88c3c6dc
>
> java      1 138 root 3847r      REG              253,6       30169  1835391 /usr/local/gerrit/site/data/product/lfs/f0/49/f0493203958f2cae5914437cb29a4d9dbb3581ac0436f9098cb2274b97abce99
>
> java      1 138 root 3848r      REG              253,6         619  6430403 /usr/local/gerrit/site/data/product/lfs/17/09/170957220cea2a7157e5eb53c58e225ab3d75ebf2e3f3607166b95368176d55c
>
> java      1 138 root 3849r      REG              253,6         177  5248772 /usr/local/gerrit/site/data/product/lfs/42/58/42585643f1f4cf1a1c5f7ce7cb5ba7d1fb891cc69c5a7b7d0820127b4782fd5a
>
> java      1 138 root 3850r      REG              253,6         188  7219023 /usr/local/gerrit/site/data/product/lfs/a6/60/a660b1f46414d13ccfdd7b7e4778023fd3ebd5f656accfd711d89d986559ef3e
>
> java      1 138 root 3851r      REG              253,6         256   917951 /usr/local/gerrit/site/data/product/lfs/2d/8b/2d8b18b489fa85f40e73c12dfc96d7bc7ac075b6d586488dd6d5d9dc97e6d9e5
>
> java      1 138 root 3852r      REG              253,6     1639936  4588391 /usr/local/gerrit/site/data/product/lfs/23/b6/23b6d3ff787c3a5e962e182975a34191e2ba131b8d39d45b6a22b9a47f1c47e5
>
> java      1 138 root 3853r      REG              253,6        1140  4593682 /usr/local/gerrit/site/data/product/lfs/c0/32/c03280fbe24c509f08830ebf03bb7633807b482a02e546d4433ecabf92cef56e
>
> java      1 138 root 3854r      REG              253,6          94  5379916 /usr/local/gerrit/site/data/product/lfs/13/a2/13a2a3f79c433a5a999c4e3d2b219694b683ccf24b396bef61cb80b0a8081a01
>
> java      1 138 root 3855r      REG              253,6         736  7089050 /usr/local/gerrit/site/data/product/lfs/79/30/79305097d56596840f9ef16e7b0e08927c59871af66a7c63be4cc986bdef6ca6
>
> java      1 138 root 3856r      REG              253,6         184  4853975 /usr/local/gerrit/site/data/product/lfs/94/ad/94ad4798a111c967cc88fca1af4986879c4e87ad044045ffadf12ac4345bd401
>
> java      1 138 root 3857r      REG              253,6          85  5248830 /usr/local/gerrit/site/data/product/lfs/f8/aa/f8aad75fd7338c1f9dc6049f8670770fc7a38c14f951ced924cb907f4264ef68
>
> java      1 138 root 3858r      REG              253,6        8614  4852141 /usr/local/gerrit/site/data/product/lfs/9a/91/9a911050f762adc739edc1337997a0dcf89bcd880c16cf461afa515b0fb6be6f
>
> java      1 138 root 3859r      REG              253,6        9528  4589010 /usr/local/gerrit/site/data/product/lfs/bf/e3/bfe37ab720ee81ff3bc6c281c04fb5243bc1af3b1e1941264f9cf12f654ea091
>
> java      1 138 root 3860r      REG              253,6          98  5116809 /usr/local/gerrit/site/data/product/lfs/b3/49/b349866c8cab250148bc10bb771174f85226369a35c69f0915f6961a2dc9b73e
>
> java      1 138 root 3861r      REG              253,6         796  4721841 /usr/local/gerrit/site/data/product/lfs/ea/68/ea688403815848c9b93e7ca646a742612d560e0e9c24bf16d3c0dee8f79ca2fd
>
> java      1 138 root 3862r      REG              253,6       90521  4984825 /usr/local/gerrit/site/data/product/lfs/eb/29/eb2921e1f17aaeebaab106ad73d881d8c61d524bdbc570dd14f9a5a1ab2d44c9
>
> java      1 138 root 3863r      REG              253,6      580096  4588403 /usr/local/gerrit/site/data/product/lfs/0c/68/0c68291b035675486e1d871d04fea8d2dfad0d10842ef631096e291017c80bf2
>
> java      1 138 root 3864r      REG              253,6         759  4852912 /usr/local/gerrit/site/data/product/lfs/11/7e/117e400770d69b5e5bc7c44c158fc6f74ff14ea36229ff06d76a0a88dc84b186
>
> java      1 138 root 3865r      REG              253,6        9138  4854085 /usr/local/gerrit/site/data/product/lfs/94/e2/94e2f565221c25a09a2e58cc0aed4a64b78a1b27e23e54e435711f0c97942731
>
> java      1 138 root 3866r      REG              253,6        1582  4721783 /usr/local/gerrit/site/data/product/lfs/b6/52/b6525f3e76c34365e15d5ab82858cd1ffcde6cfb760f4428994eb139e33c37bf
>
> [root@git site]# lsof | wc -l
>
> 1094877
>
> [root@git site]#
>
>
>
> As you can see, there are more than a million open files J I’ll start investigating now, but suspect the JGit upgrade in the LFS plugin to be the cause (not sure yet). I will try to roll back only the LFS plugin to the previous version and see how it goes.
>
>
>
> Cheers,
>
> Markus
>
>
>
> --
>
> Mit freundlichen Grüßen / Best regards
>
>
>
> Markus Duft | Software Architect
>
> SSI SCHÄFER | SSI Schäfer IT Solutions GmbH | Friesachstraße 15 | 8114 Friesach bei Graz | Austria
>
> Phone +43 3127 200-575 | Fax +43 3127 200-22
>
> markus.duft@xxxxxxxxxxxxxxxx
>
> Website | Blog | YouTube | Facebook
>
>
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.

SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

--
--
To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
For more options, visit https://groups.google.com/d/optout.

SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

--
--
To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
For more options, visit https://groups.google.com/d/optout.

SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz

Back to the top