Re: [jgit-dev] Problems with bitmaps and cloning and DFS back end

[Shawn Pearce <spearce@xxxxxxxxxxx>]

On Wed, Jun 19, 2013 at 12:02 PM, Alex Blewitt <alex.blewitt@xxxxxxxxx> wrote:
> On 19 Jun 2013, at 17:03, Shawn Pearce <spearce@xxxxxxxxxxx> wrote:
>
>> But my original idea for DFS backend was to encode the DfsPackDescription
>> data into the filename, and parse it back out when listing the packs
>> of the repository. Unfortunately this might not be possible if you
>> cannot rename a file, as some of the data arrives too late.
>
> Yes, I found that out too :-)
>
> However I found when the commitPack is called the info is there, so I write out a surrogate file with the data encoded in it.

Yes, our implementation does the same thing. The description data is
frozen just before commitPack() is invoked, so this is the point to
save it.

> The only real question is how much is needed; this is the first time I've needed anything at currently it works with only objectCount set.

Colby pointed out to me this morning that the object count is
necessary, and can be obtained from the PackIndex.

Getting it from there is not trivial. The DfsPackDescription coming
out of listPacks() needs the count set. The only way to get the object
count from the index is to load the entire index and call
getObjectCount(). Unfortunately the PackIndex you get from this
process won't be cached properly for the DfsPackFile to use it, so now
you have the index being loaded twice. Yuck.


I think a reasonable fix in JGit is to modify DfsCachedPack so that it
uses the DfsPackFile to get the object count, rather than the
DfsPackDescription. The count is only used once we have chosen to
reuse the pack, and reuse required us to look at the bitmap index,
which required us to load the PackIndex. So DfsPackFile can safely
delegate to the cached PackIndex and get a fast answer from memory.

> I imagine that fileSize and lastModified would be useful to enable the correct priority sorting to work

Yes, but this is overridable in your DfsPackDescription subclass by
replacing compareTo(DfsPackDescription).

> but I'm not sure whether the statistics are relevant outside of a DfsGarbageCollect result. I'm not sure about deltaCount.

The deltaCount can be 0. Its only used in a stats line shown to a
client when they do a clone and the bitmap file kicked in a reuse of
the entire pack. This can be safely 0, it just yields a possibly
confusing line.

> The question is what subset is strictly necessary? Perhaps unnecessary fields (if any) could be annotated with "transient".

Documenting this is a fantastic idea.