Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] What's up with

s3sync.rb -v -r -p --delete ~/downloads/releases

The Access denied response seems to be because the file isn't there yet.  Judging by the name of the file, I think that's a fair statement.

I have essentially three issues with Amazon:

1. Upload speed is quite slow.  I can't seem to upload much faster than 3-4 megabits/sec

2. s3sync.rb wastes lots of time with retries, timeouts, dropped connections, server error 500's etc.

3. Since s3sync seems to rely on md5 checksums to determine if a file has changed, a sync of takes an insane amount of time even if there are no changes.


On 09/29/2009 03:03 PM, Markus Knauer wrote:
How is the download content mirrored to Amazon S3?
E.g. For our cloudfront offerings I am using s3sync for doing this with settings like this:

  s3sync.rb --recursive --public-read --progress --make-dirs ... bucket-name:...

To me it looks like there are too many "access denied" and maybe the public read access bit is not set. Maybe this is the root cause for the other failures?

Regards, Markus

2009/9/29 Eric Gwin <eric.gwin@xxxxxxxxxx>

I don't know if the main proxy is just at capacity or if there is a bigger issue. I also don't know if this will help any (but I'm hoping so).
I'm seeing similar results when just trying to download from:

using the Main Download Site (Canada):"">
which remaps to:


<?xml version="1.0" encoding="UTF-8" ?>
- <Error>
 <Message>Access Denied</Message>


Denis Roy wrote:
On 09/29/2009 11:14 AM, Thomas Hallgren wrote:
If this list is sorted, I think the top 40 files or so (down to the first zip) fit well under 50Mb so uploading them should be easy. Perhaps P2 meta-data should be prioritized for upload to cloudfront given the request frequency for those files?

p2 metadata seems to be scattered across our directory tree, making it difficult to strategically redirect content.  It's also easy to draw up a plan 5 days after the release.  I mean, would you have expected /tools/mylyn/update/e3.4/content.jar to be a top-3 downloaded file?

But all in all, as I said earlier, I think the process when we release needs to be improved. We can't open the taps until we are absolutely sure that the infrastructure is ready to take the hit.

Sorry, but you are stating the obvious.  I would not deliberately open the taps if I didn't think we were ready.  Really.  We did mirror tests.  But then we had one of those unforeseen problems (see bug 289408 comment 37 <>).  The problem is that there are many moving parts to these releases, and we seem unable to foresee those unforeseen problems.

I know this is frustrating.  I've spent the last five years trying to make these releases as painless as possible, but as long as our servers and your code keep evolving, there are always going to be some unexpected results.


cross-project-issues-dev mailing list
cross-project-issues-dev mailing list

Markus Knauer
###   phone: +49 721 664 733 0  (GMT +2)
###     fax: +49 721 664 733 29
###     web:

Innoopract Informationssysteme GmbH
Stephanienstrasse 20, 76133 Karlsruhe Germany
General Manager: Jochen Krause
Registered Office: Karlsruhe, Commercial Register Mannheim HRB 107883

_______________________________________________ cross-project-issues-dev mailing list cross-project-issues-dev@xxxxxxxxxxx

Back to the top