Re: [eclipse-mirrors] Re: [carsten@xxxxxxxxx: [ftpsync-eclipse-ftp@ftp]

[Carlos Carvalho <carlos@xxxxxxxxxxxxxx>]

Eclipse Webmaster (Denis Roy) (webmaster@xxxxxxxxxxx) wrote on 31 May 2010 09:11:
 >1. You will occasionally get permissions errors on your RSYNC.  We have 
 >almost 1000 committers that can put files in the download area.  We set 
 >the umasks to allow for world-readable by default, but with 1000 
 >committers, something is bound to happen.  If you can ignore the 
 >permissions errors, I'd suggest you do so.

There are several issues here. First, permission errors can only be
ignored to a limited extent. There are two choices: not doing removals
or doing them anyway. The first choice will make the mirror repository
grow with files that are already removed upstream, so you'll have your
disk filled with junk (as is already happening). The second choice
will cause the removal of the entire forbidden area; the problem is
not just the removal, it's that it'll have to be downloaded all again
when the permissions are fixed. Very very inefficient...

I apologize to Carsten when I said the size is 324GB: we mirror from
osuosl, not from download.eclipse.org. Therefore the forbidden areas
are just dumped and we don't get them, so our repo is [much] smaller.

Second issue, your repository construction is wrong. You should not
allow committers direct access to it, exactly because it's impossible
to make sure permissions (and other things) will be correct. You
should pull commits from a staging area to the master archive. This is
pretty obvious and is what's done by the vast majority of software
distributions...

Third, your cluster is not suitable for mirroring. It is very good for
building but inefficient for mirroring because of the content
duplication in ram. The archive master should be a single machine with
at least 16GB of ram, or the repo should be split among the different
machines.

 >-> I will bring the issue up with our project teams once again, in hopes 
 >that some clean-up can occur.

Don't waste your and their time, it'll never be reliable.

 >If you have any specific questions, feel free to contact me directly at 
 >webmaster@xxxxxxxxxxx to keep list noise low.

These issues are relevant to all mirrors, so should be discussed in
the list.