Re: [jgit-dev] JGit GC unable to remove old pack files on Windows

[Matthias Sohn <matthias.sohn@xxxxxxxxx>]

On Tue, Jun 9, 2020 at 2:13 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:

Hello,

The simple example of code, mentioned in the previous email.

class Main {

    private static File REPO_DIR = new File("testRepo");
    private static final String REPO_URL = "https://github.com/denis-bigbrassband-com/testing.git";
    public static final String REF_SPEC_BARE = "+refs/heads/*:refs/heads/*";
    public static final String TAG_SPEC = "+refs/tags/*:refs/tags/*";

    public static void main(String[] args) {

        System.out.println("Start cloning...");
        CloneCommand cloneCommand = new CloneCommand();
        cloneCommand.setBare(true);
        cloneCommand.setCloneAllBranches(true);
        cloneCommand.setCloneSubmodules(true);
        cloneCommand.setURI(REPO_URL);
        cloneCommand.setDirectory(REPO_DIR);
        cloneCommand.setTimeout(60);

        try (Git git = cloneCommand.call()) {
            Repository repo = git.getRepository();
            System.out.println("Cloned. Press Enter key when the repository is changed...");
            System.in.read();

            System.out.println("Fetching...");
            try (Transport tn = Transport.open(repo, REPO_URL)) {
                tn.setTimeout(60);
                ProgressMonitor monitor = new TextProgressMonitor();
                tn.fetch(monitor, Arrays.asList(new RefSpec(REF_SPEC_BARE), new RefSpec(TAG_SPEC)));
            }

            System.out.println("Fetched. Starting GC...");
            GC gc = new GC((FileRepository) repo);
            PackConfig pConfig = new PackConfig(repo);
            pConfig.setBuildBitmaps(false);   // disable bitmap indexes
            gc.setPackConfig(pConfig);
            gc.setPackExpireAgeMillis(0);
            gc.gc();
            System.out.println("GC completed!");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}

On Tue, Jun 9, 2020 at 5:40 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:

Hello,

We found an issue with the JGit GC on Windows.
I've attached a simple example to expose this issue. Steps to reproduce:

1. Clone some repository using JGit.
2. Add commits to the repository on the git server side (for example, push commits from another computer).
3. Fetch the cloned repository using JGit.
4. Call the JGit GC on the cloned repository.
5. After the GC the cloned repository 'objects/pack' folder contains all packs -- both existed before the GC and created during the GC.
6. If you repeat steps 2-4 then new pack files appears on every iteration. And all old pack files remain in the repository 'objects/pack' folder.

It seems this is a bug.
The logic of the bug is the following:

1. The JGit clone & fetch operations mark the created pack files as 'read-only' in the ObjectDirectoryPackParser.parse method:
   
   
   
2. JGit GC operation also marks the created pack files as 'read-only' in the GC.writePack method:
   
   
   
3. JGit uses Files.delete() method in the GC.removeOldPack to remove old pack files:
   
   
   

   
   
4. This Files.delete() method unable to remove 'read-only' files on Windows (please see https://stackoverflow.com/questions/12139482/difference-between-filesdeletepath-and-filedelete), throws an exception, but this exception is silently ignored:
   
   
   
5. As a result all pack files remain in the repository, but nobody knows that something goes wrong:
   
   
   

It seems that the R/O attribute should be cleared before deleting old pack files.

this is https://bugs.eclipse.org/bugs/show_bug.cgi?id=408846 which was fixed recently.

Use the nightly build 5.8.0-SNAPSHOT until 5.8 is released next Wednesday.

-Matthias