[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] [egit-dev] Expected performance profile of staging changes and committing then

On 11/06/2012 04:18 PM, Robert Munteanu wrote:
> Hi Markus,
> 
> On Tue, Nov 6, 2012 at 12:14 PM, Markus Duft <markus.duft@xxxxxxxxxx> wrote:
>> On 11/06/2012 10:14 AM, Tomasz Zarna wrote:
[snip]
>> [1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=393642
> 
> Thanks for the detailed information and for raising that bug.
> Hopefully someone can look into properly fixing this.

you might be interested in these changes i just pushed:

[1] https://git.eclipse.org/r/#/c/8535/
[2] https://git.eclipse.org/r/#/c/8536/

they belong together and bring staging/unstaging time down to a millisecond for me :D ofc this is still kind of a (dirty?) hack ;) however all unit tests run through with it and i did not observe any obvious problems with it. unit tests run a little slower since the Repository constructor now already causes an index read, but i don't think this will do harm in real-world scenarios as Repository is not a thing that is constructed all the time from scratch. Experts may now prove me wrong ;)

For still unknown reason events don't exactly behave as i'd expect them sometimes. it happened that the code did not detect changes from one to the other index state. i was not really able to reproduce this case, but still... i don't know exactly yet. in such a case the refresh button in the staging view would trigger a full refresh. Sometimes a full refresh is triggered too, which i don't fully understand either, but most of the time it's fast ;)

The initial construction of the IndexDiffCacheEntry still does a full scan, so does a RefChangedEvent (caused by every ref update (commit, rebase, ...)).

> 
> I know little about the internals of {j,e}git but I'd be more than
> happy to test any proposed fixes on my repository, add profiler traces
> and whatever else is needed.

Would be cool if you can build it yourself. if you can't, [3] is a pre-built snapshot with my change (and all (except one) changes from [4]).

[3] http://dev.gentoo.org/~mduft/wamas-egit.tgz
[4] https://git.eclipse.org/r/#/q/status:open+AND+owner:mduft%2540gentoo.org,n,z

(it's an update site archive, so you should be able to just install from it into your eclipse ;))

Markus

> 
> Robert
> 
> 
>>
>>>
>>> Cheers,
>>> Tomasz
>>>
>>> On Tue, Nov 6, 2012 at 8:15 AM, Markus Duft <markus.duft@xxxxxxxxxx <mailto:markus.duft@xxxxxxxxxx>> wrote:
>>>
>>>     On 11/05/2012 05:38 PM, Robert Munteanu wrote:
>>>     > Hi,
>>>     >
>>>     [snip]
>>>     > Some background information: my repository is pretty large, 17000
>>>     > files, 9000 directories, 1.2GB total size, 2.5 MB for .git/index. My
>>>     > computer is reasonably fast ( SSD, i7, Linux ) so I wouldn't look
>>>     > there for performance problems.
>>>
>>>     Hi!
>>>
>>>     We have a similar repo - ~30000 files, 1.1GB total, 5.5MB index. I use a linux workstation with 16 cores, 12G RAM and a spinning 10k rpm disk.
>>>
>>>     >
>>>     > I don't use the commit dialog since the performance is too slow [1]
>>>     > so I fell back to using the staging view.
>>>
>>>     Me too - i already contributed some changes to EGit vastly increasing the performance of the staging view, as it was too slow for me too ;)
>>>
>>>     >
>>>     > When I use the staging view I observe the following:
>>>     >
>>>     > 1. Staging changes involves at least two background jobs:
>>>     >
>>>     > - Git repository refresh
>>>     > - Re-indexing repository $NAME
>>>
>>>     Both jobs are triggered by the IndexChangedEvent i guess (or at least the re-indexing). This event is IMHO just too stupid, as it carries no information about /what/ changed, i just tells that /something/ in your 2.5MB index changed, thus EGit has to recalculate everything. This is an issue for me too, as you can imagine, as our index is even bigger :/
>>>
>>>     >
>>>     > 2. Committing changes involves at least three background jobs
>>>     >
>>>     > - Committing changes ( which is pretty fast )
>>>     > - Re-indexing repository $NAME
>>>     > - Building commit info
>>>
>>>     Same here, the re-indexing is triggered by the index change. I profiled those jobs a lot, and i guess for what they do, they are pretty fast. The problem is what they do - they are forced to do much to much, as the information about the actual changes is missing.
>>>
>>>     >
>>>     > Both these operations take a lot of time and I would expect them to be
>>>     > instant, since I'm changing 2 files out of 17.000 and egit knows which
>>>     > files have changed and doesn't need to recalculate the whole
>>>     > repository state.
>>>
>>>     This describes what we all would like it to be :D
>>>
>>>     Regards,
>>>     Markus
>>>
>>>     >
>>>     > Am I missing out on something? I've captured yourkit snapshots of both
>>>     > the 'stage' and 'commit' operations and I can provide them if needed.
>>>     > Before that I'd like to know whether this is expected and I should
>>>     > stop worrying and start drinking more tea or considered a performance
>>>     > bug and should be entered in bugzilla .
>>>     >
>>>     > Thanks,
>>>     >
>>>     > Robert
>>>     >
>>>     >
>>>     > [1]: 331273: Commit operations opens and reads almost all files
>>>     > https://bugs.eclipse.org/bugs/show_bug.cgi?id=331273
>>>     >
>>>     > --
>>>     > Sent from my (old) computer
>>>     > _______________________________________________
>>>     > egit-dev mailing list
>>>     > egit-dev@xxxxxxxxxxx <mailto:egit-dev@xxxxxxxxxxx>
>>>     > https://dev.eclipse.org/mailman/listinfo/egit-dev
>>>     >
>>>     _______________________________________________
>>>     egit-dev mailing list
>>>     egit-dev@xxxxxxxxxxx <mailto:egit-dev@xxxxxxxxxxx>
>>>     https://dev.eclipse.org/mailman/listinfo/egit-dev
>>>
>>>
>> _______________________________________________
>> egit-dev mailing list
>> egit-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/egit-dev
> 
> 
> 
> --
> Sent from my (old) computer
>