Re: [jgit-dev] JGit GC unable to remove old pack files on Windows

[Denis Malyshkin <dmalyshkin@xxxxxxxxxx>]

Hello,

We found a similar bug.
But now it relates to the GC.deleteOrphans() method.
This method also unable to remove orphaned files on Windows if the files have R/O attribute set. The produced error is the next:
===============================================================================
2020-10-20 01:39:21,294-0400 ERROR  [o.e.j.i.storage.file.GC] E:\xxx\objects\pack\pack-d849246de612f3d29812703e482773ee5b1453c6.idx
java.nio.file.AccessDeniedException: E:\xxx\objects\pack\pack-d849246de612f3d29812703e482773ee5b1453c6.idx
    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
    at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at org.eclipse.jgit.internal.storage.file.GC.deleteOrphans(GC.java:981)
    at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:882)
    at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:267)
    at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:218)
[...]
===============================================================================

Thus if for any reasons some orphaned files remain in the 'pack' folder -- they will remain here infinitely and will flood the logs with the above error messages.


On 10.06.2020 3:56, Denis Malyshkin wrote:

Hello Matthias,

Thank you a lot.
I've tested with both JGit '5.8.0-SNAPSHOT' and '5.8.0.202005061305-m2' – and I confirm that the bug is not reproduced anymore.


On 09.06.2020 19:57, Matthias Sohn wrote:

On Tue, Jun 9, 2020 at 2:13 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:

Hello,

The simple example of code, mentioned in the previous email.

class Main {

    private static File REPO_DIR = new File("testRepo");
    private static final String REPO_URL = "https://github.com/denis-bigbrassband-com/testing.git";
    public static final String REF_SPEC_BARE = "+refs/heads/*:refs/heads/*";
    public static final String TAG_SPEC = "+refs/tags/*:refs/tags/*";

    public static void main(String[] args) {

        System.out.println("Start cloning...");
        CloneCommand cloneCommand = new CloneCommand();
        cloneCommand.setBare(true);
        cloneCommand.setCloneAllBranches(true);
        cloneCommand.setCloneSubmodules(true);
        cloneCommand.setURI(REPO_URL);
        cloneCommand.setDirectory(REPO_DIR);
        cloneCommand.setTimeout(60);

        try (Git git = cloneCommand.call()) {
            Repository repo = git.getRepository();
            System.out.println("Cloned. Press Enter key when the repository is changed...");
            System.in.read();

            System.out.println("Fetching...");
            try (Transport tn = Transport.open(repo, REPO_URL)) {
                tn.setTimeout(60);
                ProgressMonitor monitor = new TextProgressMonitor();
                tn.fetch(monitor, Arrays.asList(new RefSpec(REF_SPEC_BARE), new RefSpec(TAG_SPEC)));
            }

            System.out.println("Fetched. Starting GC...");
            GC gc = new GC((FileRepository) repo);
            PackConfig pConfig = new PackConfig(repo);
            pConfig.setBuildBitmaps(false);   // disable bitmap indexes
            gc.setPackConfig(pConfig);
            gc.setPackExpireAgeMillis(0);
            gc.gc();
            System.out.println("GC completed!");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}

On Tue, Jun 9, 2020 at 5:40 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:

Hello,

We found an issue with the JGit GC on Windows.
I've attached a simple example to expose this issue. Steps to reproduce:

1. Clone some repository using JGit.
2. Add commits to the repository on the git server side (for example, push commits from another computer).
3. Fetch the cloned repository using JGit.
4. Call the JGit GC on the cloned repository.
5. After the GC the cloned repository 'objects/pack' folder contains all packs -- both existed before the GC and created during the GC.
6. If you repeat steps 2-4 then new pack files appears on every iteration. And all old pack files remain in the repository 'objects/pack' folder.

It seems this is a bug.
The logic of the bug is the following:

1. The JGit clone & fetch operations mark the created pack files as 'read-only' in the ObjectDirectoryPackParser.parse method:
   
   
   
2. JGit GC operation also marks the created pack files as 'read-only' in the GC.writePack method:
   
   
   
3. JGit uses Files.delete() method in the GC.removeOldPack to remove old pack files:
   
   
   

   
   
4. This Files.delete() method unable to remove 'read-only' files on Windows (please see https://stackoverflow.com/questions/12139482/difference-between-filesdeletepath-and-filedelete), throws an exception, but this exception is silently ignored:
   
   
   
5. As a result all pack files remain in the repository, but nobody knows that something goes wrong:
   
   
   

It seems that the R/O attribute should be cleared before deleting old pack files.

this is https://bugs.eclipse.org/bugs/show_bug.cgi?id=408846 which was fixed recently.

Use the nightly build 5.8.0-SNAPSHOT until 5.8 is released next Wednesday.

-Matthias 

--

Best regards, Denis Malyshkin, Senior C++ Developer ISS Art, LLC - custom software development company Skype: dmalyshkin Phone: +73812909808

IMPORTANT: The contents of this email and any attachments are confidential and intended for the named recipient(s) only. If you have received this email by mistake, please notify the sender immediately and delete it from your system. You may not copy this email or disclose its contents to anyone. Thank you.