Re: [eclipse-mirrors] Mirroring Problems

[carlos@xxxxxxxxxxxxxx (Carlos Carvalho)]

Eclipse Webmaster (Denis Roy) (webmaster@xxxxxxxxxxx) wrote on 15 September 2009 09:36:
 >On 09/15/2009 08:50 AM, Carlos Carvalho wrote:
 >> It's related. The machine needs more ram.
 >>    
 >'The machine' is a 5-node Itanium2 cluster, and each node has 8GB of RAM.

Nodes??!! Do you mean they're 5 independent machines, possibly sharing
a common disk storage? And each node serves the full repository??!! If
it's *really* like that it's a disaster for the mirror. You'd be much
better off switching off 4 nodes and putting all the ram in the
remaining one. Or you could split the repository in 5 fixed parts and
use a load balancer to assign each part to a fixed node. Note the
"fixed", twice.

 >On 09/15/2009 08:50 AM, Carlos Carvalho wrote:
 >> Files/directories without permission cannot be downloaded by users,
 >> therefore they're useless in a mirror.
 >Agreed, but I suggest you ignore permissions errors

In this case the mirrors cannot get the files. There are two options;
the first is to make rsync ignore the errors and proceed, like you
say. That's exactly what I do. Since the files in the locked dirs are
invisible rsync will remove them here. The other option is to tell
rsync to not delete anything in the whole transfer, aborting the job
due to the read errors. I will not do this.

 >since the files on download.eclipse.org are uploaded by hundreds of
 >people with varying *nix abilities.

Allowing them to write directly to the repository is bad. They should
write to a staging area and you move them periodically to the repo,
fixing permissions. This way you don't have to scan the whole
repository, just the much smaller changes, so that you can do it fast
and frequently, without loading the machine.

 >We have scripts to fix permissions, but they don't run in realtime.

That's lightly put. The problem has remained for more than 3 days.