Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] List of files on Eclipse Mirror servers

I would recommend serving content directly whether it is on primary storage or in archive. Sending a 301 redirect may or may not work, depending on a particular client. Arguably, using a redirect exposes aspects of internal storage architecture that don’t need exposing.

 

The download server could be implemented to function similarly to how memory systems work. A single interface, with different levels of storage. The client doesn’t know if they are hitting L1, L2 or whatever.

 

Hope this is helpful.

 

- Konstantin

 

 

From: cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] On Behalf Of Denis Roy
Sent: Tuesday, June 19, 2012 1:36 PM
To: cross-project-issues-dev@xxxxxxxxxxx
Subject: Re: [cross-project-issues-dev] List of files on Eclipse Mirror servers

 

On 06/19/2012 03:40 PM, Konstantin Komissarchik wrote:

Currently, only the mirror selection script is able to delegate to archive.eclipse.org, so auto-archiving would break those repositories.

That gives me an idea.

We currently use a small PHP script to handle 404's on download.eclipse.org.  If the request comes from Eclipse or a robot, we send a small 404 response.  If it's from a browser, we send the stunningly beautiful page you see here:  http://downlaod.eclipse.org/khasjkas

We could alter the script to return a 301redirect if the 404 file in question is indeed found on archive.eclipse.org.  This means in theory, you _could_ move files and p2 repos to the archives without any breakage at all.

Or even better, the PHP script could just send the file's contents down the wire with a 200 response, no redirects.

I would be concerned about the increased CPU load, since download.e.o currently handles about 6 million 404's a day.  But it would likely be a very elegant solution.

Denis



 

If there are resources available to work on this, it would be much better if download.eclipse.org served as central gateway to both the primary download server and the archive. Then one can implement all sort of archiving policies without involving projects or affecting end users referencing these URLs… You can auto-archive based on age, you can auto-archive based on demand, heck you can even pick a goal for how much to mirror and have a heuristic that shuffles between primary and archive to meet that goal.

 

- Konstantin

 

 

From: cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] On Behalf Of Wayne Beaton
Sent: Tuesday, June 19, 2012 12:15 PM
To: cross-project-issues-dev@xxxxxxxxxxx
Subject: Re: [cross-project-issues-dev] List of files on Eclipse Mirror servers

 

Does it make sense to consider moving files that are old--say three or more years old--to the archive server automatically?

Wayne

On 06/19/2012 10:33 AM, Denis Roy wrote:

To help you clean up older builds and files from the Eclipse download server, I've compiled a list of files that are actually sync'ed to our mirrors.

http://download.eclipse.org/technology/phoenix/download.eclipse.org-filelist.txt.gz

I each project to look at the file list, and either remove files that are no longer needed, or move older releases[1] to the archives.

Thanks,

Denis


[1] http://wiki.eclipse.org/IT_Infrastructure_Doc#Downloads

 


Back to the top