| Re: [eclipse.org-committers] Action required: download.eclipse.org mirror size |
If you're only getting a handful of downloads each day, I think it's safe to put those files on the archives.
Our bandwidth usage will always go up regardless -- and since mirrors handle about 95% of our download requests, we don't want to impose too much on them.
Thanks for everyone's quick action.
Denis
Hello Denis,
(please feel free to answer publicly if you think this is generally useful)
Can I get some guidance on whether something is worth leaving on the download and therefore mirrored or not? I assume there is a tradeoff of mirror storage vs EF bandwidth usage, or similar.
For example, for EPP each download file is ~100-400MB, for Neon there are 70 such files, 34 of which have < 1 download per day. With the most downloaded file (eclipse-jee-neon-3-win32-x86_64.zip) being almost 700 per day*.
So how many of these downloads per day make a file worth having mirrored? If the answer is < 1 download per day is not worth mirroring, then I could potentially transfer between 40 and 80 GB to archive for EPP.**
The above questions all assume that "Older releases must be moved to archive.eclipse.org." has a non-specific definition for "Older".
Thanks,
Jonah
* I ran query on https://dev.eclipse.org/committers/committertools/stats.php for last 30 days for /technology/epp/downloads/release/neon/3/ and divided numbers by 30 to calculate these numbers.** How much I save depends a lot on how much time I spend identifying which specific files to archive as some files (JEE specifically) in each release still gets downloaded a lot.
On Tue, 14 Apr 2020 at 08:37, Ed Willink <ed@xxxxxxxxxxxxx> wrote:
_______________________________________________Hi Denis
Unfortunately
https://bugs.eclipse.org/bugs/show_bug.cgi?id=560989
https://bugs.eclipse.org/bugs/show_bug.cgi?id=560941
Moving P2 repos to archive is unhelpful for a user who attempts a Maven build from the archive repo.
The first Bug above is a Tycho fix for the original OCL Bug report. Alternatively the EF could ignore a non-archive p2.mirrorsURL so that the archived repo behaves just as if the archiving had stripped the p2.mirrorsURL.
Regards
Ed Willink
On 13/04/2020 14:51, Denis Roy wrote:
Greetings,
As download.eclipse.org is mirrored worldwide, is it important to keep your download footprint to a minimum. Most of our mirror sites are also host to a number of OSS projects, and enterprise disk space is not cheap. Some strategies to note:
- Nightly, weekly, milestone and RC builds should be deleted
- Older releases must be moved to archive.eclipse.org. If moved into the same directory structure, links typically do not need to be updated, as our 404 Not Found handlers will look for the file in the archives before throwing the error.
- Please use this compiled Archive index (courtesy of the Oomph team) to find your project's mirror footprint:
https://download.eclipse.org/oomph/archive/eclipse/index.html
The risk of not cleaning up is significant. Mirrors, looking to liberate disk space, will simply purge the worst offenders, and with fewer mirror sites offering Eclipse bits, our servers must handle more downloads, which means unnecessary expense for the Eclipse Foundation.
Many thanks for your cooperation.
Denis
--
Denis Roy
Director, IT Services | Eclipse Foundation, Inc.
Eclipse Foundation: The Platform for Open Innovation and Collaboration
Twitter: @droy_eclipse
_______________________________________________ eclipse.org-committers mailing list eclipse.org-committers@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/eclipse.org-committers
eclipse.org-committers mailing list
eclipse.org-committers@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/eclipse.org-committers
Denis Roy
Director, IT Services | Eclipse Foundation, Inc.
Eclipse Foundation: The Platform for Open Innovation and Collaboration
Twitter: @droy_eclipse