[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jgit-dev] LFS File handles
|
Hey,
I tracked the issue back to https://git.eclipse.org/r/#/c/129501/ which seems to remove the in.close() call, which leaves the streams to the LFS storage open.
When I revert these commits for Gerrit:
957b95c7fd Upgrade JGit to 5.1.1.201809181055-r
db1da63283 Update Jetty to 9.4.12
And this one for the LFS plugin:
04c943e Upgrade JGit to 5.1.1.201809181055-r
Things start working again, file handles are stable ~200-~500.
From here, someone else needs to take over, my knowledge and time come to an end :) But I'm sure David will find and fix it in a blink :)
Cheers,
Markus
-----Original Message-----
From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
Sent: Thursday, October 11, 2018 9:02 AM
To: Matthias Sohn <matthias.sohn@xxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: RE: LFS File handles
Hey,
Thanks again for the hint to the leak detector. I spent some more time with it. For some reason, it is quite hard to get it to trigger properly xD It seems that ulimit does not have any effect on the process in the docker container (I could open 2M files according to lsof, ulimit is 1M), and the threshold for the leak detector does not seem to work (it never triggers). I now set the ulimit below the actual file count in the LFS (2200, I set it to 2000), and now it fires. Must be something with LWP on linux and threads... :|
Whatever, this is the typical stack for each file:
#195 data/product/lfs/0d/78/0d788ceb1440b20fa5ebdbf627096846a1039d9ae74daa92dfc9fa9495e663a3 by thread:HTTP-112 on Thu Oct 11 06:57:39 UTC 2018
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.eclipse.jgit.lfs.server.fs.FileLfsRepository.getReadChannel(FileLfsRepository.java:151)
at org.eclipse.jgit.lfs.server.fs.ObjectDownloadListener.<init>(ObjectDownloadListener.java:103)
at com.googlesource.gerrit.plugins.lfs.fs.LfsFsContentServlet.doGet(LfsFsContentServlet.java:83)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:244)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
Seems to be related to JGit's LFS support then? I tried downgrading /only/ the LFS plugin's JGIT, that didn't seem to help. Next is to downgrade both LFS plugin and Gerrit's own JGit and see if that helps.
Cheers,
Markus
-----Original Message-----
From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
Sent: Wednesday, October 10, 2018 5:08 PM
To: Matthias Sohn <matthias.sohn@xxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: RE: LFS File handles
Thanks for the tip. Unfortunately it does not seem to get hold of the leak. It attaches successfully, but misses what happens. I played with thresholds vs. letting it run into hard limits, nothing is detected :|
Cheers,
Markus
-----Original Message-----
From: Matthias Sohn [mailto:matthias.sohn@xxxxxxxxx]
Sent: Wednesday, October 10, 2018 4:53 PM
To: Duft Markus <Markus.Duft@xxxxxxxxxxxxxxxx>
Cc: repo-discuss@xxxxxxxxxxxxxxxx
Subject: Re: LFS File handles
You may try to use file-leak-detector [1] to track down the problem.
IIRC Chris used that successfully
in similar cases to find file handle leaks in JGit.
[1] https://github.com/kohsuke/file-leak-detector
-Matthias
On Wed, Oct 10, 2018 at 4:45 PM Duft Markus
<Markus.Duft@xxxxxxxxxxxxxxxx> wrote:
>
> I reverted db1da6328319c15903520a23e303dd91d2a4e270, and it does not help. I get less open file handles, but still ~170000 per clone.
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 4:40 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: RE: LFS File handles
>
>
>
> Ok… more info: it is not the LFS plugin J Further guesses would be the jetty upgrade (again). I rolled back to the previous Gerrit version for now.
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 4:08 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: RE: LFS File handles
>
>
>
> I tested this now, and it seems that a clone of a repo with relatively low LFS file count will increase the open file count unproportionally:
>
>
>
> mduft@fril0041 /tmp/wamas $ git lfs ls-files | wc -l
>
> 2230
>
>
>
> [root@git site]# lsof | grep lfs | wc -l
>
> 253368
>
> <cloned the repo here>
>
> [root@git site]# lsof | grep lfs | wc -l
>
> 521140
>
> [root@git site]#
>
>
>
> This is a test server with zero traffic except myself. Not good. K Trying to roll back now.
>
>
>
> Cheers,
>
> Markus
>
>
>
> From: repo-discuss@xxxxxxxxxxxxxxxx [mailto:repo-discuss@xxxxxxxxxxxxxxxx] On Behalf Of Duft Markus
> Sent: Wednesday, October 10, 2018 3:58 PM
> To: Repo and Gerrit Discussion <repo-discuss@xxxxxxxxxxxxxxxx>
> Subject: LFS File handles
>
>
>
> Hey,
>
>
>
> I’ve upgraded our gerrit installation to latest master and as well upgraded the LFS plugin to latest master. We’re now experiencing a whole lot of too many open files issues. It seems related to the LFS plugin:
>
>
>
> java 1 138 root 3821r REG 253,6 4488 4721537 /usr/local/gerrit/site/data/product/lfs/9a/20/9a202f462d7ad8c27fbc7ebbc8d17cab02d19293c46418a8eb2087548801d405
>
> java 1 138 root 3822r REG 253,6 88 4721677 /usr/local/gerrit/site/data/product/lfs/7c/ce/7cce17f0da539a28e96d5df3853daa638e0c244c390d2ae5c8f7f4cdc7247e80
>
> java 1 138 root 3823r REG 253,6 825 6693745 /usr/local/gerrit/site/data/product/lfs/17/57/175797fde1ea61d136f25f8a109bf6b63587e7f7b42d51da5c752d360e66caa1
>
> java 1 138 root 3824r REG 253,6 621454 4982722 /usr/local/gerrit/site/data/product/lfs/e9/5d/e95dcc99917a8eaedde432a60b932869889a77d30b875a7f64c01aff7d150a0f
>
> java 1 138 root 3825r REG 253,6 698 6037124 /usr/local/gerrit/site/data/product/lfs/19/1c/191c6508c85c4249d016f69abeb73eefdf4692976e2c29f386c5a5c4cdb029fe
>
> java 1 138 root 3826r REG 253,6 2363 4721342 /usr/local/gerrit/site/data/product/lfs/e9/3e/e93e4860dbdcf4f2e56a43b6d870d2a581984053c4d2a5a214dbee9bba2e21c6
>
> java 1 138 root 3827r REG 253,6 250 4852489 /usr/local/gerrit/site/data/product/lfs/d9/78/d9785a8c792de86627a73279c647edb44d01eb423667a58cd864f6623c22f97e
>
> java 1 138 root 3828r REG 253,6 1056 4721820 /usr/local/gerrit/site/data/product/lfs/f3/14/f31469ff9e59335484dd18f4b9f679c1d1e2bdaf33015ba432d01976a66ab9ad
>
> java 1 138 root 3829r REG 253,6 9528 4589367 /usr/local/gerrit/site/data/product/lfs/ca/e0/cae0854a6bb5f5a01400f8ecf895907cbe2733daa48a2dc273e72d328fb098eb
>
> java 1 138 root 3830r REG 253,6 18672 4589164 /usr/local/gerrit/site/data/product/lfs/8b/8a/8b8a19f671d8ac40d3ff83ab769f815d6cadb5b95532e8940f70240cbac68f82
>
> java 1 138 root 3831r REG 253,6 488 4721373 /usr/local/gerrit/site/data/product/lfs/74/4c/744c7aba265ac3e9a0469abb61c9528ffbc6434a5f6a76aaea8a2c8baf234d4f
>
> java 1 138 root 3832r REG 253,6 714 4852669 /usr/local/gerrit/site/data/product/lfs/e9/e1/e9e189bc03d32aa42e49a963b3279ba49ea57188e6fd9261c757b49a2ba46dfe
>
> java 1 138 root 3833r REG 253,6 82 5251861 /usr/local/gerrit/site/data/product/lfs/66/92/669295c78c8c4a492b1fd265dd03816de462fb6aa5c8246ba1ed867fe2e42cc7
>
> java 1 138 root 3834r REG 253,6 153 5116792 /usr/local/gerrit/site/data/product/lfs/af/af/afaf7cf17039497111e545fa0dfffe212d62667f27fbe3f59dd0390411bc9934
>
> java 1 138 root 3835r REG 253,6 1400 1835368 /usr/local/gerrit/site/data/product/lfs/84/d0/84d0daad31520f2bbd24f331a9587eadccc7bb7cd280dee7f6eac58162254c9d
>
> java 1 138 root 3836r REG 253,6 115 4984796 /usr/local/gerrit/site/data/product/lfs/43/6c/436c00e4f855a2fafa57d275ae0415634430858593c419c5b0301bc81f805138
>
> java 1 138 root 3837r REG 253,6 244 1703966 /usr/local/gerrit/site/data/product/lfs/7d/18/7d1853205bc21208d7d7f0141d0f4e9c800f9ace89e02f199296bc49df118c63
>
> java 1 138 root 3838r REG 253,6 18672 4589526 /usr/local/gerrit/site/data/product/lfs/38/da/38da45c14327cb3c96bb0ea1a7a07e859a2ae97c7922e36813af3fe559d73066
>
> java 1 138 root 3839r REG 253,6 281298 141115 /usr/local/gerrit/site/data/product/lfs/1a/a9/1aa98cf694dc9c1abde450c386c0e87a19d2300fe6225b43d5ad59ff068018dc
>
> java 1 138 root 3840r REG 253,6 1454 4589156 /usr/local/gerrit/site/data/product/lfs/84/67/8467fe4d83a9c6007887c4b2ee06b7f76a1787ce0b6528e5c1bd056460993003
>
> java 1 138 root 3841r REG 253,6 159 4853185 /usr/local/gerrit/site/data/product/lfs/80/68/806825a90029c998211387e42e0b5e57a654af44c49ccbeacee54dd018ec3ba4
>
> java 1 138 root 3842r REG 253,6 488 4588655 /usr/local/gerrit/site/data/product/lfs/75/c2/75c20a35d7433e17d02203c1254d2356dc4d2c63c9ef2d8654fda4eea401517b
>
> java 1 138 root 3843r REG 253,6 2271 4852874 /usr/local/gerrit/site/data/product/lfs/80/a1/80a16ebc9c7101758f736f77528f9c44afc561fb756f87b0340a36dd04d104e1
>
> java 1 138 root 3844r REG 253,6 8554 4588860 /usr/local/gerrit/site/data/product/lfs/2c/b4/2cb43f4ff31f4518cb1ce884df7d8caf140e1ca28d6341020cd67f0995d603bb
>
> java 1 138 root 3845r REG 253,6 5469 4721323 /usr/local/gerrit/site/data/product/lfs/32/f2/32f2fbfca553e27040108c6060de4d50e38ab83e7e6bba6556cadc9420a51d0a
>
> java 1 138 root 3846r REG 253,6 9245 4721733 /usr/local/gerrit/site/data/product/lfs/f7/38/f738756f2606f2c5e59bb99c4d1c28906386fb9c014f84bcd0b2df4d88c3c6dc
>
> java 1 138 root 3847r REG 253,6 30169 1835391 /usr/local/gerrit/site/data/product/lfs/f0/49/f0493203958f2cae5914437cb29a4d9dbb3581ac0436f9098cb2274b97abce99
>
> java 1 138 root 3848r REG 253,6 619 6430403 /usr/local/gerrit/site/data/product/lfs/17/09/170957220cea2a7157e5eb53c58e225ab3d75ebf2e3f3607166b95368176d55c
>
> java 1 138 root 3849r REG 253,6 177 5248772 /usr/local/gerrit/site/data/product/lfs/42/58/42585643f1f4cf1a1c5f7ce7cb5ba7d1fb891cc69c5a7b7d0820127b4782fd5a
>
> java 1 138 root 3850r REG 253,6 188 7219023 /usr/local/gerrit/site/data/product/lfs/a6/60/a660b1f46414d13ccfdd7b7e4778023fd3ebd5f656accfd711d89d986559ef3e
>
> java 1 138 root 3851r REG 253,6 256 917951 /usr/local/gerrit/site/data/product/lfs/2d/8b/2d8b18b489fa85f40e73c12dfc96d7bc7ac075b6d586488dd6d5d9dc97e6d9e5
>
> java 1 138 root 3852r REG 253,6 1639936 4588391 /usr/local/gerrit/site/data/product/lfs/23/b6/23b6d3ff787c3a5e962e182975a34191e2ba131b8d39d45b6a22b9a47f1c47e5
>
> java 1 138 root 3853r REG 253,6 1140 4593682 /usr/local/gerrit/site/data/product/lfs/c0/32/c03280fbe24c509f08830ebf03bb7633807b482a02e546d4433ecabf92cef56e
>
> java 1 138 root 3854r REG 253,6 94 5379916 /usr/local/gerrit/site/data/product/lfs/13/a2/13a2a3f79c433a5a999c4e3d2b219694b683ccf24b396bef61cb80b0a8081a01
>
> java 1 138 root 3855r REG 253,6 736 7089050 /usr/local/gerrit/site/data/product/lfs/79/30/79305097d56596840f9ef16e7b0e08927c59871af66a7c63be4cc986bdef6ca6
>
> java 1 138 root 3856r REG 253,6 184 4853975 /usr/local/gerrit/site/data/product/lfs/94/ad/94ad4798a111c967cc88fca1af4986879c4e87ad044045ffadf12ac4345bd401
>
> java 1 138 root 3857r REG 253,6 85 5248830 /usr/local/gerrit/site/data/product/lfs/f8/aa/f8aad75fd7338c1f9dc6049f8670770fc7a38c14f951ced924cb907f4264ef68
>
> java 1 138 root 3858r REG 253,6 8614 4852141 /usr/local/gerrit/site/data/product/lfs/9a/91/9a911050f762adc739edc1337997a0dcf89bcd880c16cf461afa515b0fb6be6f
>
> java 1 138 root 3859r REG 253,6 9528 4589010 /usr/local/gerrit/site/data/product/lfs/bf/e3/bfe37ab720ee81ff3bc6c281c04fb5243bc1af3b1e1941264f9cf12f654ea091
>
> java 1 138 root 3860r REG 253,6 98 5116809 /usr/local/gerrit/site/data/product/lfs/b3/49/b349866c8cab250148bc10bb771174f85226369a35c69f0915f6961a2dc9b73e
>
> java 1 138 root 3861r REG 253,6 796 4721841 /usr/local/gerrit/site/data/product/lfs/ea/68/ea688403815848c9b93e7ca646a742612d560e0e9c24bf16d3c0dee8f79ca2fd
>
> java 1 138 root 3862r REG 253,6 90521 4984825 /usr/local/gerrit/site/data/product/lfs/eb/29/eb2921e1f17aaeebaab106ad73d881d8c61d524bdbc570dd14f9a5a1ab2d44c9
>
> java 1 138 root 3863r REG 253,6 580096 4588403 /usr/local/gerrit/site/data/product/lfs/0c/68/0c68291b035675486e1d871d04fea8d2dfad0d10842ef631096e291017c80bf2
>
> java 1 138 root 3864r REG 253,6 759 4852912 /usr/local/gerrit/site/data/product/lfs/11/7e/117e400770d69b5e5bc7c44c158fc6f74ff14ea36229ff06d76a0a88dc84b186
>
> java 1 138 root 3865r REG 253,6 9138 4854085 /usr/local/gerrit/site/data/product/lfs/94/e2/94e2f565221c25a09a2e58cc0aed4a64b78a1b27e23e54e435711f0c97942731
>
> java 1 138 root 3866r REG 253,6 1582 4721783 /usr/local/gerrit/site/data/product/lfs/b6/52/b6525f3e76c34365e15d5ab82858cd1ffcde6cfb760f4428994eb139e33c37bf
>
> [root@git site]# lsof | wc -l
>
> 1094877
>
> [root@git site]#
>
>
>
> As you can see, there are more than a million open files J I’ll start investigating now, but suspect the JGit upgrade in the LFS plugin to be the cause (not sure yet). I will try to roll back only the LFS plugin to the previous version and see how it goes.
>
>
>
> Cheers,
>
> Markus
>
>
>
> --
>
> Mit freundlichen Grüßen / Best regards
>
>
>
> Markus Duft | Software Architect
>
> SSI SCHÄFER | SSI Schäfer IT Solutions GmbH | Friesachstraße 15 | 8114 Friesach bei Graz | Austria
>
> Phone +43 3127 200-575 | Fax +43 3127 200-22
>
> markus.duft@xxxxxxxxxxxxxxxx
>
> Website | Blog | YouTube | Facebook
>
>
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
>
>
> SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
> Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
> Commercial Court: Landesgericht für Zivilrechtssachen Graz
>
> --
> --
> To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
> For more options, visit https://groups.google.com/d/optout.
SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz
--
--
To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
For more options, visit https://groups.google.com/d/optout.
SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz
--
--
To unsubscribe, email repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@xxxxxxxxxxxxxxxx.
For more options, visit https://groups.google.com/d/optout.
SSI Schäfer IT Solutions GmbH | Friesachstrasse 15 | 8114 Friesach | Austria
Registered Office: Friesach | Commercial Register: 49324 K | VAT no. ATU28654300
Commercial Court: Landesgericht für Zivilrechtssachen Graz
- Prev by Date:
[jgit-dev] [Announce] JGit releases 4.7.4, 4.9.5, 4.11.3, 5.1.0, 5.1.1 and EGit releases 5.1.0, 5.1.1
- Next by Date:
Re: [jgit-dev] LFS File handles
- Previous by thread:
[jgit-dev] [Announce] JGit releases 4.7.4, 4.9.5, 4.11.3, 5.1.0, 5.1.1 and EGit releases 5.1.0, 5.1.1
- Next by thread:
Re: [jgit-dev] LFS File handles
- Index(es):