Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] JGit GC unable to remove old pack files on Windows

Hi Matthias,

Thank you.
I've tried the fix -- and it works well. Orphaned files have been removed, there are no errors in logs.


On 22.10.2020 13:26, Matthias Sohn wrote:
Can you try this fix ?

On Thu, Oct 22, 2020 at 12:36 AM Denis Malyshkin <dmalyshkin@xxxxxxxxxx> wrote:
Hello,

We found a similar bug.
But now it relates to the GC.deleteOrphans() method.
This method also unable to remove orphaned files on Windows if the files have R/O attribute set. The produced error is the next:
===============================================================================
2020-10-20 01:39:21,294-0400 ERROR  [o.e.j.i.storage.file.GC] E:\xxx\objects\pack\pack-d849246de612f3d29812703e482773ee5b1453c6.idx
java.nio.file.AccessDeniedException: E:\xxx\objects\pack\pack-d849246de612f3d29812703e482773ee5b1453c6.idx
    at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
    at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
    at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at org.eclipse.jgit.internal.storage.file.GC.deleteOrphans(GC.java:981)
    at org.eclipse.jgit.internal.storage.file.GC.repack(GC.java:882)
    at org.eclipse.jgit.internal.storage.file.GC.doGc(GC.java:267)
    at org.eclipse.jgit.internal.storage.file.GC.gc(GC.java:218)
[...]
===============================================================================

Thus if for any reasons some orphaned files remain in the 'pack' folder -- they will remain here infinitely and will flood the logs with the above error messages.


On 10.06.2020 3:56, Denis Malyshkin wrote:
Hello Matthias,

Thank you a lot.
I've tested with both JGit '5.8.0-SNAPSHOT' and '5.8.0.202005061305-m2' – and I confirm that the bug is not reproduced anymore.


On 09.06.2020 19:57, Matthias Sohn wrote:
On Tue, Jun 9, 2020 at 2:13 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:
Hello,

The simple example of code, mentioned in the previous email.

class Main {

    private static File REPO_DIR = new File("testRepo");
    private static final String REPO_URL = "https://github.com/denis-bigbrassband-com/testing.git";
    public static final String REF_SPEC_BARE = "+refs/heads/*:refs/heads/*";
    public static final String TAG_SPEC = "+refs/tags/*:refs/tags/*";

    public static void main(String[] args) {

        System.out.println("Start cloning...");
        CloneCommand cloneCommand = new CloneCommand();
        cloneCommand.setBare(true);
        cloneCommand.setCloneAllBranches(true);
        cloneCommand.setCloneSubmodules(true);
        cloneCommand.setURI(REPO_URL);
        cloneCommand.setDirectory(REPO_DIR);
        cloneCommand.setTimeout(60);

        try (Git git = cloneCommand.call()) {
            Repository repo = git.getRepository();
            System.out.println("Cloned. Press Enter key when the repository is changed...");
            System.in.read();

            System.out.println("Fetching...");
            try (Transport tn = Transport.open(repo, REPO_URL)) {
                tn.setTimeout(60);
                ProgressMonitor monitor = new TextProgressMonitor();
                tn.fetch(monitor, Arrays.asList(new RefSpec(REF_SPEC_BARE), new RefSpec(TAG_SPEC)));
            }

            System.out.println("Fetched. Starting GC...");
            GC gc = new GC((FileRepository) repo);
            PackConfig pConfig = new PackConfig(repo);
            pConfig.setBuildBitmaps(false);   // disable bitmap indexes
            gc.setPackConfig(pConfig);
            gc.setPackExpireAgeMillis(0);
            gc.gc();
            System.out.println("GC completed!");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

}




On Tue, Jun 9, 2020 at 5:40 PM Malyshkin, Denis <dmalyshkin@xxxxxxxxxx> wrote:
Hello,

We found an issue with the JGit GC on Windows.
I've attached a simple example to expose this issue. Steps to reproduce:
  1. Clone some repository using JGit.
  2. Add commits to the repository on the git server side (for example, push commits from another computer).
  3. Fetch the cloned repository using JGit.
  4. Call the JGit GC on the cloned repository.
  5. After the GC the cloned repository 'objects/pack' folder contains all packs -- both existed before the GC and created during the GC.
  6. If you repeat steps 2-4 then new pack files appears on every iteration. And all old pack files remain in the repository 'objects/pack' folder.
It seems this is a bug.
The logic of the bug is the following:
  1. The JGit clone & fetch operations mark the created pack files as 'read-only' in the ObjectDirectoryPackParser.parse method:
    изображение.png
  2. JGit GC operation also marks the created pack files as 'read-only' in the GC.writePack method:
    изображение.png
  3. JGit uses Files.delete() method in the GC.removeOldPack to remove old pack files:
    изображение.png
    изображение.png
  4. This Files.delete() method unable to remove 'read-only' files on Windows (please see https://stackoverflow.com/questions/12139482/difference-between-filesdeletepath-and-filedelete), throws an exception, but this exception is silently ignored:
    изображение.png
  5. As a result all pack files remain in the repository, but nobody knows that something goes wrong:
    изображение.png
It seems that the R/O attribute should be cleared before deleting old pack files.

this is https://bugs.eclipse.org/bugs/show_bug.cgi?id=408846 which was fixed recently.
Use the nightly build 5.8.0-SNAPSHOT until 5.8 is released next Wednesday.

-Matthias 


--

                    ISS Art website
Best regards,
Denis Malyshkin, Senior C++ Developer
ISS Art, LLC - custom software development company
Skype: dmalyshkin
Phone: +73812909808
 Read ISS Art Blog    Find ISS Art on Facebook    Join ISS Art on LinkedIn    Follow ISS Art on Twitter    Visit ISS Art on VK.com
IMPORTANT: The contents of this email and any attachments are confidential and intended for the named recipient(s) only. If you have received this email by mistake, please notify the sender immediately and delete it from your system. You may not copy this email or disclose its contents to anyone. Thank you.

Back to the top