[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
[jgit-dev] Concurrent run of repack and clone sometimes fails
|
Hi,
We are running several servers with jGit. We need to run
repack from time to time to keep the repos performant. I.e. after push
we test how many small packs are in the repo and when a threshold is
reached we run the repack.
After upgrading jGit version we've found
that if someone does the clone at the time repack is running the clone
sometimes (not always) fails because the repack removes .pack file used
by the clone. Server exception and client error attached.
I've
tracked down the cause and it seems to be introduced between jGit 5.2
(which we upgraded from) and 5.3 and being caused by this commit: Move
throw of PackInvalidException outside the catch -
https://github.com/eclipse/jgit/commit/afef866a44cd65fef292c174cad445b3fb526400
The
problem is that when the throw was inside of the try block the last
catch block catched the exception and called openFailed(false) method. It
is true that it called it with invalidate = false, which is wrong. The
real problem though is that with the throw outside of the try block the
openFail is not called at all and the fields activeWindows and
activeCopyRawData are not set to 0. Which affects the later called tests
like: if (++activeCopyRawData == 1 && activeWindows == 0).
The
fix for this is relatively simple keeping the throw outside of a
the try block and still having the invalid field set to true. See
the attached patch (against current master). I did exhaustive testing of
the change running concurrent clones and pushes indefinitely and with
the patch applied it never fails where without it takes relatively short
to get the error. However I'm not an expert in the jGit code and I may
well be wrong about how to fix this.
Please let me know If you
need me to do more, such as filing a bug and maybe provide a runable
test case (will have to be a jGit server though)
Thanks
Hrebejk
Attachment:
fix.patch
Description: Binary data
CLIENT
------
Cloning into 'gct'...
remote: Counting objects: 9615, done
remote: Finding sources: 100% (9615/9615)
remote: Getting sizes: 100% (6410/6410)
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
SERVER
------
2020-11-30 11:33:56,932 WARN [qtp1212116343-106] [/services@2062] Internal error during upload-pack from /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.pack
at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:719) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:516) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:603) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:386) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:614) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:414) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1724) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1701) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1689) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1674) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1227) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2333) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1079) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:836) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.http.server.UploadPackServlet.lambda$1(UploadPackServlet.java:184) ~[org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.http.server.UploadPackServlet.defaultUploadPackHandler(UploadPackServlet.java:207) ~[org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:201) [org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
...
Caused by: java.io.IOException: Unreadable pack index: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.idx
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:69) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:163) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.getReverseIdx(PackFile.java:1161) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.findEndOffset(PackFile.java:1131) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.representation(PackFile.java:1092) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:593) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:584) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.selectObjectRepresentation(WindowCursor.java:177) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1343) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1178) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
... 87 more
Caused by: java.io.FileNotFoundException: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.idx (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_171]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_171]
at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_171]
at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:163) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.getReverseIdx(PackFile.java:1161) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.findEndOffset(PackFile.java:1131) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.PackFile.representation(PackFile.java:1092) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:593) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:584) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.file.WindowCursor.selectObjectRepresentation(WindowCursor.java:177) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1343) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1178) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
... 87 more