Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EGit » Re: Cloning job gets stuck
Re: Cloning job gets stuck [message #737975 is a reply to message #737921] Sat, 15 October 2011 16:41 Go to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
<html>
<head>
<meta content="text/html; charset=ISO-8859-15"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I tried it again and now it gets stuck at 27%:<br>
<br>
<img src="http://www.eclipse.org/forums/index.php?t=getfile&amp;id=4334" alt=""><br>
<br>
Do I have to repeat 7.6923 times? :P<br>
<br>
Cheers<br>
/Eike<br>
<br>
----<br>
<a class="moz-txt-link-freetext" href="http://www.esc-net.de">http://www.esc-net.de</a><br>
<a class="moz-txt-link-freetext" href="http://thegordian.blogspot.com">http://thegordian.blogspot.com</a><br>
<a class="moz-txt-link-freetext" href="http://twitter.com/eikestepper">http://twitter.com/eikestepper</a><br>
<br>
</body>
</html>
  • Attachment: fjihjhbc.png
    (Size: 4.71KB, Downloaded 116 times)
Re: Cloning job gets stuck [message #738017 is a reply to message #737975] Sat, 15 October 2011 18:34 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
Am 15.10.2011 18:41, schrieb Eike Stepper:
> Do I have to repeat 7.6923 times?
Jokes aside, repeating the same exercise always freezes at 27%.

One thing I've noticed when using a command line tool for cloning is that the Git objects that are received at 27% seem
to be quite large. The command line tool shows the total received bytes and I can see constant progress. but this
progress does not have a big impact on the number of received objects. Not enough to impact the percent value of the
reported progress.

To really be able to notice a freeze (a bug or network issue) as opposed to receiving large objects it would be helpful
to display the display the size of the received objects in the job's task message.

Please note that in my case it really freezes at 27% and I don't seem to have other network issues.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: Cloning job gets stuck [message #748715 is a reply to message #738017] Mon, 24 October 2011 21:24 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 594
Registered: July 2009
Senior Member
Are you trying to spam this forum ?
Why do you post 27 copies of the same question ?
Also your post contains quite a number of HTML tags I don't like to parse.

Which version of EGit are you using ?
Which repository are you trying to clone here ?
Do you have enough free heap when the clone stalls ? Check with VisualVM or some similar tool.
Re: Cloning job gets stuck [message #749483 is a reply to message #748715] Tue, 25 October 2011 08:42 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
Hi Matthias,

Comments below...


Am 24.10.2011 23:24, schrieb Matthias Sohn:
> Are you trying to spam this forum ?
No.

> Why do you post 27 copies of the same question ?
I didn't. I only see now that the ultra cool Eclipse forum software (which personally I never use) has gone mad again. I
just submitted https://bugs.eclipse.org/bugs/show_bug.cgi?id=361893

> Also your post contains quite a number of HTML tags I don't like to parse.
Again, only in the web forum (which I did not use). In fact I embedded some screenshots to make it easier for you to
understand my EGit problem.

>
> Which version of EGit are you using ?
1.1.0.201109151100-r

> Which repository are you trying to clone here ?
The image I embedded into my original posts would have clarified that. The problem occurs randomly (maybe in 1 of 4
cases, sometimes more often, sometimes less) with any of these repos: http://git.eclipse.org/c/cdo . All my
committershave struggled with this until they reverted to using a command line git tool.

> Do you have enough free heap when the clone stalls ?
I'm starting Eclipse with 2.5 gigabytes heap size and usually I have arund 2.0 gigabytes of them available (according to
the heap size trim).

> Check with VisualVM or some similar tool.
Sorry, that's beyond my capabilities and resources. The error log at least did not reveal a heap problem (or any other
problem).

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: Cloning job gets stuck [message #753344 is a reply to message #749483] Tue, 25 October 2011 20:45 Go to previous messageGo to next message
Robin Rosenberg is currently offline Robin Rosenberg
Messages: 331
Registered: July 2009
Senior Member
Eike Stepper skrev 2011-10-25 10.42:
> Am 24.10.2011 23:24, schrieb Matthias Sohn:

>> Check with VisualVM or some similar tool.
> Sorry, that's beyond my capabilities and resources. The error log at least did not reveal a heap problem (or any other problem).

It's not hard. You could get a stack dump using VisualVM (included with the JDK). Then
pick a vm a get a thread dump. If you have more than one report the one where you see
egit or jgit mentioned.

-- robin
Re: Cloning job gets stuck [message #753381 is a reply to message #749483] Wed, 26 October 2011 23:44 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 594
Registered: July 2009
Senior Member
I tried to reproduce this and cloned the cdo repository (size around 100MB) a couple of times and found it always takes around 18 min both with git and http protocol. Tried on Mac OS X 10.6.8 / Java 7 preview and using EGit recent nightly version.

I analyzed your repo and found if you would remove the biggest 33 pdf, ppt and jar files you could reduce the total repository size by 1/3. So considering removal of binaries from the repository using filter-branch may help to reduce clone effort for all contributors considerably.
Re: Cloning job gets stuck [message #753725 is a reply to message #753381] Fri, 28 October 2011 13:54 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
Am 27.10.2011 01:44, schrieb Matthias Sohn:
> I tried to reproduce this and cloned the cdo repository (size around 100MB) a couple of times and found it always
> takes around 18 min both with git and http protocol. Tried on Mac OS X 10.6.8 / Java 7 preview and using EGit recent
> nightly version.
Thank you for trying this! I'm also just testing a clone anew. Will report here when done...

>
> I analyzed your repo and found if you would remove the biggest 33 pdf, ppt and jar files you could reduce the total
> repository size by 1/3. So considering removal of binaries from the repository using filter-branch may help to reduce
> clone effort for all contributors considerably.
Hmm, I'd like to think about this hint. Can you tell me how to find these big files in history?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: Cloning job gets stuck [message #755155 is a reply to message #753725] Mon, 07 November 2011 14:50 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 594
Registered: July 2009
Senior Member
try the script [1]. I have a tweaked version I will send you directly.

[1] http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/
Re: Cloning job gets stuck [message #755157 is a reply to message #755155] Mon, 07 November 2011 14:59 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 594
Registered: July 2009
Senior Member
sligthyl tweaked version of the mentioned script:

#!/bin/bash
# Shows you the largest objects in your repo's pack file.
# usage:
#      $ git gc				# first run gc to move all data to packs
#      $ git-find-big-blobs 100		# find and list biggest 100 objects
# derived from
# http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/

# set the internal field spereator to line break, so that we can iterate easily over the verify-pack output
IFS=$'\n';

# list all objects including their size, sort by size, take top $1 biggest blobs
objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v chain | sort -k3nr | head -n $1`

echo "All sizes are in kB's. The pack column is the size of the object, compressed, inside the pack file."

output="size,pack,SHA,location"
for y in $objects
do
    # extract the size in bytes
    size=$((`echo $y | cut -f 5 -d ' '`/1024))
    # extract the compressed size in bytes
    compressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))
    # extract the SHA
    sha=`echo $y | cut -f 1 -d ' '`
    # find the objects location in the repository tree
    other=`git rev-list --all --objects | grep $sha`
    #lineBreak=`echo -e "\n"`
    output="${output}\n${size},${compressedSize},${other}"
done

echo -e $output | column -t -s ', '

Re: Cloning job gets stuck [message #755223 is a reply to message #755157] Mon, 07 November 2011 18:05 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
Hi Matthias,

Thanks a lot!! The script outputs an error:

estepper/bin/git-find-big-blobs: line 21: /1024: syntax error: operand expected (error token is "/1024")

nevertheless it outputs the 100 biggest objects. I'll discuss with my team if and how we can remove some of them. Thanks
again ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper



Am 07.11.2011 15:59, schrieb Matthias Sohn:
> sligthyl tweaked version of the mentioned script:
>
>
> #!/bin/bash
> # Shows you the largest objects in your repo's pack file.
> # usage:
> # $ git gc # first run gc to move all data to packs
> # $ git-find-big-blobs 100 # find and list biggest 100 objects
> # derived from
> # http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/
>
> # set the internal field spereator to line break, so that we can iterate easily over the verify-pack output
> IFS=$'\n';
>
> # list all objects including their size, sort by size, take top $1 biggest blobs
> objects=`git verify-pack -v .git/objects/pack/pack-*.idx | grep -v chain | sort -k3nr | head -n $1`
>
> echo "All sizes are in kB's. The pack column is the size of the object, compressed, inside the pack file."
>
> output="size,pack,SHA,location"
> for y in $objects
> do
> # extract the size in bytes
> size=$((`echo $y | cut -f 5 -d ' '`/1024))
> # extract the compressed size in bytes
> compressedSize=$((`echo $y | cut -f 6 -d ' '`/1024))
> # extract the SHA
> sha=`echo $y | cut -f 1 -d ' '`
> # find the objects location in the repository tree
> other=`git rev-list --all --objects | grep $sha`
> #lineBreak=`echo -e "\n"`
> output="${output}\n${size},${compressedSize},${other}"
> done
>
> echo -e $output | column -t -s ', '
>
>
Re: Cloning job gets stuck [message #755469 is a reply to message #755223] Tue, 08 November 2011 15:36 Go to previous messageGo to next message
Matthias Sohn is currently offline Matthias Sohn
Messages: 594
Registered: July 2009
Senior Member
I tested this script on Mac, looks like your platform doesn't like some of the cut options. Try "man cut" to find out what it doesn't like.
Re: Cloning job gets stuck [message #755825 is a reply to message #755469] Wed, 09 November 2011 17:13 Go to previous message
Eike Stepper is currently offline Eike Stepper
Messages: 5545
Registered: July 2009
Senior Member
Am 08.11.2011 16:36, schrieb Matthias Sohn:
> I tested this script on Mac, looks like your platform doesn't like some of the cut options. Try "man cut" to find out
> what it doesn't like.
Never mind, the results we got were okay and soon after M3 is out we'll remove some big files from the history. Thanks!

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Previous Topic:Interactive rebase a dead end?
Next Topic:Ubuntu 10.04 with Eclipse STS - Import fails
Goto Forum:
  


Current Time: Thu Oct 30 19:10:55 GMT 2014

Powered by FUDForum. Page generated in 0.02471 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software