Re: [eclipse-mirrors] Re: [carsten@xxxxxxxxx: [ftpsync-eclipse-ftp@ftp]

[Carlos Carvalho <carlos@xxxxxxxxxxxxxx>]

Denis Roy (denis.roy@xxxxxxxxxxx) wrote on 31 May 2010 11:11:
 >On 05/31/2010 10:03 AM, Carlos Carvalho wrote:
 >
 >    Eclipse Webmaster (Denis Roy) (webmaster@xxxxxxxxxxx) wrote on 31 May 2010 09:11:
 >     >1. You will occasionally get permissions errors on your RSYNC.  We have
 >     >almost 1000 committers that can put files in the download area.  We set
 >     >the umasks to allow for world-readable by default, but with 1000
 >     >committers, something is bound to happen.  If you can ignore the
 >     >permissions errors, I'd suggest you do so.
 >
 >    There are several issues here. First, permission errors can only be
 >    ignored to a limited extent.
 >
 >
 >Carlos,
 >
 >While not 100% ideal, many mirrors have been rsync > /dev/null with
 >us for many years. I don't see why others can't do that either, but
 >I'm sure you will tell me. If there is a major malfunction with our
 >mirror system, I will notice it quickly enough and fix it.

It's not just annoying error messages, it affects the mirror
repository. When there are permission errors in the source rsync
considers them as I/O errors, and by default skips file deletion. The
return code is 23. This means that if permissions stay wrong for
several days all would-be deletions will be skipped, filling the mirror
disk with junk.

The mirror admin can give the option --ignore-errors to rsync, and
it'll then do deletions anyway. However, since rsync got no files from
the forbidden dirs, it'll consider that the mirror shouldn't have
files either, and will wipe out those dirs; it won't just close the
permissions. Later, when the master admin fixes the permissions, the
mirror will have to download everything again. Since the closed trees
are often big, the overhead is significant, both to the mirrors and to
the master.

 >    Second issue, your repository construction is wrong. You should not
 >    allow committers direct access to it, exactly because it's impossible
 >    to make sure permissions (and other things) will be correct. You
 >    should pull commits from a staging area to the master archive. This is
 >    pretty obvious and is what's done by the vast majority of software
 >    distributions...
 >
 >
 >Please be cautious if you're comparing Eclipse.org to Fedora, or other
 >one-project shops where a single dedicated build team produces output for a
 >single (or very few) project(s).  Eclipse is closer in concept to Sourceforge
 >and Apache, where each project does its own little thing.

But you still have a single distribution tree. It applies to any
software distribution tree, irrespective of the internal organization.

 >When you say "you should pull commits", who is "you" exactly?

The master repository admin, which may or may not be the machine
admin. It is the role responsible for creating and maintaining the
directory tree in the master repository.

 >Of course, one could conceive a system where the downloads repository is
 >protected, and projects must indirectly submit content to be
 >replicated to it.

That's what I'm saying.

 >Perhaps that's what we need to do. But there are definite blockers
 >to enabling this, cost/benefit being one of them (inertia is costly
 >to overcome in large organizations). I've opened a bug against the
 >Eclipse Architecture council to assist in resolving the issue. But
 >there are many other issues we can resolve before exploring this
 >route.

You'll have to do it to fix the problem sooner or later. And it's
creating problems to your mirrors right now.

 >    Third, your cluster is not suitable for mirroring. It is very good for
 >    building but inefficient for mirroring because of the content
 >    duplication in ram. The archive master should be a single machine with
 >    at least 16GB of ram, or the repo should be split among the different
 >    machines.
 >
 >
 >Since February 2010, download.eclipse.org has been residing on a
 >single machine with 64G of RAM.

Excellent!

 >If you have not seen any difference in mirroring efficiency since,
 >then perhaps the inefficiencies are not as substantial as you claim.

They surely are, I didn't see it because I used osuosl. I've just
switched to download.eclipse.org.