Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit » WorkDir dirty after merge
WorkDir dirty after merge [message #1397170] Wed, 09 July 2014 09:59 Go to next message
Rolf Meier is currently offline Rolf MeierFriend
Messages: 4
Registered: July 2014
Junior Member
Hey everyone

I'm posting here and not on the bugtracker since I'm unsure how to proceed (will open a proper ticket later if there's merit).

I've got the following situation:
When I merge two specific branches "t1" and "t2", the merge works without conflicts but leaves the working directory dirty.

About 11 files show up as modified immediately after the merge went through. The files are actually old versions neither contained in t1 nor t2 (nor the resulting merge) but probably in an ancestor commit along the merge path.

Even worse, when there is an actual conflict, those modified files will get displayed in the upper part of the "staged files" view along with the red marked conflicting files. Less knowledgeable users might happily drag these down into the staged area and commit them along with everything else, resulting in faulty merges and making it difficult to merge the new files back in from other branches without touching everything (bad idea).

This happens not just on my machine but for everyone who has a clone of the repository. I'm using EGit/JGit 3.3.0.201403021825-r but also tried 3.4.1.201406201815-r to the same effect.

Both Git for Windows and the official command-line Git on Linux work fine.

Did anyone ever see something similar? I'm willing to go to quite some length to resolve this since our company has been using EGit/JGit for some time and we are really quite happy with it.

Unfortunately, the repository is private so I can't upload it for anyone else to use for testing. We use a build server in the background which does automated merges and results in a really messy/complicated history. So I wouldn't be surprised if the problem is only triggered by our specific branch constellation and may not likely occur in normal projects.

Any ideas or suggestions would be most welcome. I will gladly provide more info or even debug the code when pointed in the right direction...

Best wishes
Rolf
Re: WorkDir dirty after merge [message #1397505 is a reply to message #1397170] Wed, 09 July 2014 20:48 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias SohnFriend
Messages: 612
Registered: July 2009
Senior Member
Do you have zig-zag merges between those 2 branches when this problem occurs ?

What's the result of "git merge-base -a t1 t2" [1] ? If this returns more than 1 base there
might be a problem in jgit's implementation of recursive merge.

If there is only 1 base:
Where the offending files changed between base and t1 or t2 ?
You can check this using something like

git log -p base..t1 -- <path of offending file>
git log -p base..t2 -- <path of offending file>

[1] https://www.kernel.org/pub/software/scm/git/docs/git-merge-base.html
Re: WorkDir dirty after merge [message #1397653 is a reply to message #1397505] Thu, 10 July 2014 01:51 Go to previous messageGo to next message
Rolf Meier is currently offline Rolf MeierFriend
Messages: 4
Registered: July 2014
Junior Member
Hello Matthias

Yes, a lot of zig-zaging going on. I'm sure you'll be horrified (remove spaces):
http :// i.pictr.com / 0551k4co3d.png

I've marked tbase1 and tbase2 that are output by "git merge-base -a t1 t2".

We work with a central repository model and topic branches. We only use merges since we don't trust users to handle rebases correctly. Our build server does automatic weekly merges from main branches "master" to "hotfix" to "update" to "release" and from there into individual topic branches depending on what they're based on.

This leads to a truly convoluted history cluttered by lots of automatic merges but makes life simple for normal users. Any changes to master will flow upwards into all topic branches by the end of the week. After reading up on recursive merges and multiple parents I realize how much work Git does in the background and wonder if the above model is sustainable... (I hope so)

I guess it might be a slight bug in JGit at that. The merge does work correctly, just taints the working directory. If I squash either t1 or t2 or both then the problem goes away which would probably be the simplest/quickest solution. However, I'd really hate to leave even a slight bug in JGit as users tend to have enough issues with trust.

I could set up a VNC session if anyone cared to have a look but I guess that's probably asking a bit too much. I might also try to reproduce the problem with a dummy repository somehow (doubtful that'll work) or debug it myself although I'm probably way in over my head.

As always, I'd appreciate any input (concerning the problem and/or our workflow).

Cheers
Rolf

Edit: Corrected the image. tbase2 was marked incorrectly.

[Updated on: Thu, 10 July 2014 11:27]

Report message to a moderator

Re: WorkDir dirty after merge [message #1397862 is a reply to message #1397653] Thu, 10 July 2014 09:03 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 113
Registered: July 2009
Senior Member
This looks like a bug very much worth to be investigated. I did a lot of work in JGit merge. Maybe JGit has problems during recursive merge when temporary files are created to clean up correctly. First, to be reproduce some aspects of this problem, I would
asked you to execute the following commands in your repo before you do the merge and send me the files in.dag and in.refsls. You can inspect the files, they contain ID's and the names of your refs.

git for-each-ref >in.refs
git log --all '--pretty=format:%H %ct %P' >in.dag


With JGits debug-rebuild-commit-graph command I can then reproduce the topology of your graph and see wheter JGit chooses right merge bases.

If that doesn't give insight maybe it would be good to produce a stripped down version of your repo (with filter-branch) which contains only a few files (especially those which are leftofers after the merge) but which is anonymous enough to share it.


Ciao
Chris
Re: WorkDir dirty after merge [message #1398061 is a reply to message #1397862] Thu, 10 July 2014 14:51 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 113
Registered: July 2009
Senior Member
I received your files and could reproduce you're projects version graph. Wow - that's one of the complicated graphs I saw. Don't your developers complain that they have to work in a git repository where the history contains mainly merge commits back and forth? I would not be able to develop code with git when I am not able to understand the version graph without a widescreen monitor Smile . Have you looked at http://nvie.com/posts/a-successful-git-branching-model/ or at the standard gerrit workflow?

Back to your problem: as I guessed JGit is picking the right common bases. So, that's not the problem. To reproduce the problem we really need a repo which contains the files which cause problems. With "git filter-branch --subdirectory-filter" [1] you can create repos which only contain content of a certain folder. Maybe you could try whether such a repo has the same problems merging t1 and t2.

[1] http://git-scm.com/docs/git-filter-branch



Ciao
Chris
Re: WorkDir dirty after merge [message #1398589 is a reply to message #1398061] Fri, 11 July 2014 09:31 Go to previous messageGo to next message
Rolf Meier is currently offline Rolf MeierFriend
Messages: 4
Registered: July 2014
Junior Member
I know the history is a complete mess. The graph confuses more than it helps.

When we introduced Git, it turned out that a clean history simply wasn't a high priority. We have about 30 to 40 topic branches at any time. We use a central repository for backup and syncing between users. When a branch is done, it gets tested by someone else, though not by looking at the code. We have a release cycle of 3 months, although hotfixes are automatically released, delayed by two weeks. When a release is being prepared, only tested branches get merged.

Developers are more comfortable knowing their long-running topic branches don't diverge too much from the code base so we previously did nightly automatic merges (recently switched to weekly). That's what blows up the history in this manner.

Using rebases instead of merges would greatly simplify everything but I'm not sure people can handle that properly. Besides, things have worked out great for us so far. Automatic merges can always be filtered out of the history if need be. Still, I'd love to hear about a better solution or get a few tips how to produce a cleaner history.

I'm not sure we can use Gerrit as it seems tailored to the needs of open-source projects. We have too few developers to code review and possibly reject anyone's work.

[Updated on: Fri, 11 July 2014 14:24]

Report message to a moderator

Re: WorkDir dirty after merge [message #1398987 is a reply to message #1398589] Fri, 11 July 2014 22:17 Go to previous messageGo to next message
Robin Rosenberg is currently offline Robin RosenbergFriend
Messages: 332
Registered: July 2009
Senior Member
Rolf Meier skrev 2014-07-11 11.31:
> I know the history is a complete mess. The graph confuses more than it helps.
>
> When we introduced Git, it turned out that a clean history simply wasn't a high priority. We have about 30 to 40 topic branches at any time. We use a central repository for
> backup and syncing between users. When a branch is done, it gets tested by someone else, though not by looking at the code. We have a release cycle of 3 months, although
> hotfixes are automatically released, delayed by two weeks. When a release is being prepared, only tested branches get merged.
>
> Developers are more comfortable knowing their long-running topic branches don't diverge too much from the code base so we previously did nightly automatic merges (recently
> switched to weekly). That's what blows up the history in this manner.
>
> Using rebases instead of merges would greatly simplify everything but I'm not sure people can handle that properly. Besides, things have worked out great for us so far.

If they can handle merge, they can probably handle rebase. Some education doesn't hurt, but basically it's the same problem.

> Automatic merges can always be filtered out of the history if need be. Still, I'd love to hear about a better solution or get a few tips how to produce a cleaner history.
>
> I'm not sure we can use Gerrit as it seems tailored to the needs of open-source projects. We have too few developers to code review and possibly reject anyone's work.

Devs can approve their own work (well you can set it up anyway you want). E.g. in EGit/JGit itself committers can submit their own changes, but we frown on it when it happens.

> Concerning the problem: I tried various ways to filter out all except a single subdirectory but it always lead to greatly simplified graphs where the problem is no longer
> reproducible. Is there any way to remove most files but leave the commit/merge structure intact?

No not now, but I think it's a sane idea.

> The problem is that the commit graph is so complicated that it's probably hard to understand where exactly JGit might do something wrong. In that sense, we might really be
> stress testing JGit in more ways than one.

Lots of merges do lead to complicated graphs, which is one reason for preferring rebase and fast-forward merging. Gitk, unlike JGit, has some tricks to try to visually
simplify the graph by omitting details. It doesn't make the graph less complicated though.

That aside we've had bugs in the graph layout closed only recently.

-- robin
Re: WorkDir dirty after merge [message #1402942 is a reply to message #1398589] Fri, 18 July 2014 14:47 Go to previous messageGo to next message
Christian Halstrick is currently offline Christian HalstrickFriend
Messages: 113
Registered: July 2009
Senior Member
Found the bug. Proposed a fix in https://git.eclipse.org/r/#/c/30128

Ciao
Chris
icon3.gif  Re: WorkDir dirty after merge [message #1402953 is a reply to message #1402942] Fri, 18 July 2014 15:55 Go to previous message
Rolf Meier is currently offline Rolf MeierFriend
Messages: 4
Registered: July 2014
Junior Member
Hello Chris

That solved it for me. The resulting worktree is now the same in JGit as in Git. Very Happy

Thank you so much for figuring this out. I guess the solution may appear simple in hindsight, but I'm sure it's that much harder when you have little to work with and need to debug analytically.

Cheers
Rolf
Previous Topic:Rebase interactive reordering commits: Wrong repository state: SAFE
Next Topic:Reword: Add change id for gerrit
Goto Forum:
  


Current Time: Thu Nov 27 23:38:39 GMT 2014

Powered by FUDForum. Page generated in 0.02753 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software