Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cbi-dev] [eclipse.org-committers] Fwd: [eclipse-mirrors] Eclipse mirror size ~1.4 TB
  • From: Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx>
  • Date: Wed, 17 Apr 2019 09:44:37 -0400
  • Autocrypt: addr=denis.roy@xxxxxxxxxxxxxxxxxxxxxx; prefer-encrypt=mutual; keydata= mQENBFSQVYcBCACphCllQd59H3sF23T1XUD8h6pNPvRwuKKpfcwTAL7qe98PbMBQBLuy3ocj 04PcvlRj8lCR9JmChFc9V5Bi61MTdCrshEbiyomw/ZWdfFbmc4/rlMWg00gTO94SP0zMl96x np0aNi8cGwD258NUVD1+tlvraukr6yixD25qCGQ313lOO71l1R8eJun/Kx7SEFXtMFJ7Er4r /QnfFioFFwWGiUXbGhioCZhggQWnq3U+LOZzGKp0SuWSQuu0RzQB57Fy8EN2bmX61EKGaFnr TR/8yk8EdifupYgDc6XCxndktFJYFse4v4XQSPNIIUwcBT/Ji732xdXiR4YnL0SNAVBDABEB AAG0NkRlbmlzIFJveSAoRWNsaXBzZSkgPGRlbmlzLnJveUBlY2xpcHNlLWZvdW5kYXRpb24u b3JnPokBOAQTAQIAIgUCWWjoCAIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQQI23 cA21ZyMHGgf/THQqUR43XqUhSytf3q0Wmq1KBCDUpLXQYfpe1uh464robj4iUsoCMDhlEzbU uy0p+de87aVv68ZGRPOlz0oIt80NZbHxLErhunjFGhJvnw/V5jf2lYThiWysDcQMUSIGERuy 3wIyVFVF4RDZB4jjvrPvQ/dy+x+mN38t76C8x5sp4oup19sy9u4EUzPpEInBNs8ADoKO9kOl ZbTOkIFSqC4aI0yABRMbur+mks+b9IaVVQytK4zlzTROuBNHnjx6kgfAV5L8ezQpgBkVLrLw HwB+Rmsnx/qbRXrRmESiiD6aVQQntkfj0zXK/XzMpm1+9qxA8TBj6nFXo/ivNqDx7LkBDQRU kFWHAQgAy/xnfKzcxiCyHtP2a3mbRW04xMwDoLWj+3khfudiHgQWNLDiN/oTAJag5cOrh17L ewRDuhEctJ71Nb+bkMNHvJvq5KC0oB1x54ocQGa493Q/9irIYaEbXVcVqyggmTO6lPvLcA9+ mdQZNvfFlnxS05z+c2CvNNqhgrFU4xS6MTuGEA6oduha0mbDodPKNBh8MkL7OjlYO/44H4l0 MsBYJ57QhQqeOVv1C8Sc2hstk8ZcCc3G2W29Beh0Li5WFYf7F8L0JtxuDSLGisWJL5cAZvdE Zrd/BNAOE02+fANl9iYkX+pG1rjYMdtm/UpLo38jovcyWhOc+TE7qab72wO8iQARAQABiQEf BBgBAgAJBQJUkFWHAhsMAAoJEECNt3ANtWcjxEMH/Rer0weIKA1jV72FNQpHaKuGuUmqW6ak qQHSRKZEECjTa9E1M/Mvdr21qE/zYR7tFD9zc2wqhxBDBD8azKQLZo2TSTnqyUPdr2Dy3KFS FoavCvaVzEt7c3sA/TxXWXQhxGJL56yj2VZaF+k7j/DtYl6nG62bS0rQbg/3/RllmXa51QIT V5ZPnTo6NFJ5K3lbn7EoQbVT2LdkPmabPqw6DT+ZjQlJS3Q8iI0wSoS6r9D7QQG/VaeKGjoK TRDRn/5YUnof4vjpao8jIRMJ+5RBlRto96jwt7hfNm842GsqT4ddMyYmTLGnYLUae9rl2GmN ahBMbQ9PB8Ta/DeCpIxpOyk=
  • Delivered-to: cbi-dev@xxxxxxxxxxx
  • List-archive: <https://www.eclipse.org/mailman/private/cbi-dev>
  • List-help: <mailto:cbi-dev-request@eclipse.org?subject=help>
  • List-subscribe: <https://www.eclipse.org/mailman/listinfo/cbi-dev>, <mailto:cbi-dev-request@eclipse.org?subject=subscribe>
  • List-unsubscribe: <https://www.eclipse.org/mailman/options/cbi-dev>, <mailto:cbi-dev-request@eclipse.org?subject=unsubscribe>
  • Openpgp: preference=signencrypt
  • User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1

Sam,

Adding the cbi-dev mailing list, as your question is a good one.

On 2019-04-15 6:13 p.m., Sam Davis wrote:

Given that there is no longer shell access to build.eclipse.org, what is the recommended way to do this cleanup? Create Jenkins build jobs that list directory contents in the console, decide what can be deleted, and then create another job to do the deletion?


We really need a file browsing mechanism, but in the meanwhile, your suggestion of using Jenkins jobs is how I recommend you do it.  It's clunky, but it's all we have right now.

Denis






Sam

--
Sam Davis
Senior Software Engineer, Tasktop
Committer, Eclipse Mylyn
http://tasktop.com


On Mon, Apr 15, 2019 at 7:41 AM Denis Roy <denis.roy@xxxxxxxxxxxxxxxxxxxxxx> wrote:

Folks,

The disk footprint of mirrored content on download.eclipse.org is up to 1.4TB. We need to do a better job (much better) of keeping it small.  I've filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=546434 to track this.


download.eclipse.org should only be used to store current release builds. Archived builds should be moved to archive.eclipse.org, and nightly and integration builds should be deleted once stale.

See the docs for more info:
https://wiki.eclipse.org/IT_Infrastructure_Doc#Downloads

Below is a table of projects that seem to be consuming more disk space than I'd expect.

_ALL PROJECTS SHOULD CLEAN THEIR AREA, NOT JUST THOSE LISTED BELOW_

14G	4diac	
6.6G	acceleo	
15G	app4mc	
57G	birt	
11G	dirigible	
20G	e4	e4/sdk/drops 2011 -> move to archive.eclipse.org
72G	eclipsescada	
12G	ecp	
9.7G	efxclipse	
9.5G	epsilon	
16G	gemoc	
16G	ice	
25G	jdtls	
9.8M	keyple	
13G	kura	
6.5G	mat	
4.8G	mmt	
160G	modeling	
37G	emf	
36G	gmp	
72G	mdt	
8.0G	tmf	
14G	n4js	
41G	orion	
98G	rcptt	
76G	eclipselink	
38G	sirius	
18G	staging	
398G	epp	
71G	cdt	
110G	orbit	
41G	ptp	
36G	tracecompass	
40G	webtools	Lots of old stuff that can be moved


-------- Forwarded Message --------
Subject: [eclipse-mirrors] Eclipse mirror size ~1.4 TB
Date: Sat, 13 Apr 2019 19:29:29 +0100
From: Gökdeniz Karadağ <gokdenizk@xxxxxxxxx>
Reply-To: Eclipse download mirror administrators discussion and announcements <eclipse-mirrors@xxxxxxxxxxx>
To: eclipse-mirrors@xxxxxxxxxxx


Hi Eclipse Mirror team,

Currently eclipse mirror is ~1.4 TB, surpassing both debian and ubuntu in size.

Is it possible to create a more compact rsync module containing most requested files?

For example OpenSuse has "hotstuff" modules for mirrors who want to provide value while conserving space and downstream sync bandwidth. (i.e. I don't like to download 375 versions of a file -over the life time of the mirror- for that single person over there using that project, they can go to the main mirror for that file)

--
Gokdeniz Karadag
_______________________________________________
eclipse.org-committers mailing list
eclipse.org-committers@xxxxxxxxxxx
https://www.eclipse.org/mailman/listinfo/eclipse.org-committers

IMPORTANT: Membership in this list is generated by processes internal to the Eclipse Foundation.  To be permanently removed from this list, you must contact emo@xxxxxxxxxxx to request removal.

Back to the top