Re: [eclipse-mirrors] permission denied in directories causes removal on

[carlos@xxxxxxxxxxxxxx (Carlos Carvalho)]

Denis Roy (denis.roy@xxxxxxxxxxx) wrote on 19 October 2009 14:12:
 >On 10/13/2009 07:04 PM, Carlos Carvalho wrote:
 >> rsync: opendir "/modeling/m2t/xpand/downloads/drops/0.8.0/N200910111926" (in eclipseMirror) failed: Permission denied (13)
 >> rsync: opendir "/modeling/tmf/xtext/downloads/drops/0.8.0/N200910111939" (in eclipseMirror) failed: Permission denied (13)
 >> rsync: opendir "/technology/linuxtools" (in eclipseMirror) failed: Permission denied (13)
 >>
 >> These mistakes caused some 200,000 files to be removed here... *sigh*
 >>
 >> This isn't the first time. The problem is that now it happened on a
 >> big directory
 >Carlos, you seem quite upset about this, and I'd like to understand 
 >why.

Because of the overhead it causes.

 >I've been at Eclipse.org for five years, and this problem has 
 >existed since... yet you're the first one to be passionately vocal about it.
 >
 >At this point, I'd like to make the world a perfect place, but I hardly 
 >see how it's worth it when you (our beloved mirror maintainer) can  just 
 >send Eclipse-related rsync errors to /dev/null.  I realize this is not 
 >ideal, and that we (Eclipse.org) will not have mirrors for those 
 >'broken' files, but since no one on the planet can download them anyway, 
 >the onus of fixing the permissions will quickly return to the project 
 >team who messed up the permissions in the first place...  So to 
 >reiterate, I'm just not understanding that the big deal is.

The problem is not that the files are unavailable, it's that they
re-appear. They're deleted and then have to be downloaded again. If
it's just a few ones nobody cares but in the event above about 330,000
(that's really 330 thousand) would have been removed. That's about 25%
of the whole repository. Pulling all that again takes long, and puts a
load on both machines, mainly on the master because it has to do the
same for many mirrors.

 >For what it's worth, a build directory that starts with N is likely a 
 >Nightly build, so if it doesn't appear on mirror sites, that's a bonus 
 >-- it saves your valuable disk space.

If they're really not important maybe you should revise your
recommendation. In the script you give as example you recommend that
mirrors pull everything. This puts a big load on the master, which
currently cannot handle it well. It takes from 25min to >60min to get
the file list of the whole repository, showing clearly that the
machine lacks ram and is at the limit of disk seeks.