Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [papyrus-rt-dev] Download server disk usage
A follow-up.

Looking into other folders there I found that the "builds" folder also contained old 0.7.x artifacts. I assumed we didn't need those either so I removed all of it.

We also have, rather confusingly, folders "rcp" and "rcps". The first contains this:

.

|-- nightly

|   `-- 0.8.0

|       `-- 20161212

|           `-- org.eclipse.papyrusrt.rcp.product-win32.win32.x86_64.zip

`-- releases

    `-- neon

        |-- 0.8.0

        |   |-- org.eclipse.papyrusrt.rcp.product-linux.gtk.x86_64.zip

        |   |-- org.eclipse.papyrusrt.rcp.product-macosx.cocoa.x86_64.tar.gz

        |   `-- org.eclipse.papyrusrt.rcp.product-win32.win32.x86_64.zip

        `-- 0.9.0

            |-- org.eclipse.papyrusrt.rcp.product-linux.gtk.x86_64.zip

            |-- org.eclipse.papyrusrt.rcp.product-macosx.cocoa.x86_64.tar.gz

            `-- org.eclipse.papyrusrt.rcp.product-win32.win32.x86_64.zip


And the second contains

.

`-- nightlies

    `-- 0.8.0

        `-- 20161212

            `-- org.eclipse.papyrusrt.rcp.product-win32.win32.x86_64.zip


The actual p2 update sites are under "updates" whose structure is as follows:

.

|-- integration

|   |-- neon

|   |   |-- 0.8.0

|   |   |-- 0.8.0-beta

|   |   |-- I201704060516

|   |   |-- I201704130516

|   |   |-- I201704200516

|   |   |-- I201704270515

|   |   |-- I201705110515

|   |   `-- I201705250515

|   `-- oxygen

|       `-- I201706220515

|-- milestones

|   `-- neon

|       `-- 0.8.0

|           `-- M201607271727

|-- nightly

|   |-- neon

|   |   `-- last

|   `-- oxygen

`-- releases

    |-- mars

    |   |-- 0.7.0

    |   |   |-- codegen

    |   |   |-- codegen-old

    |   |   |-- codegen-old-2

    |   |   |-- codegen-old-3

    |   |   |-- codegen-old-4

    |   |   |-- codegen-old-5

    |   |   `-- tooling

    |   |-- 0.7.0-old

    |   |   `-- tooling

    |   |-- 0.7.1

    |   |   |-- codegen

    |   |   `-- tooling

    |   |-- 0.7.1-old

    |   |   `-- tooling

    |   `-- 0.7.2

    |       |-- codegen

    |       `-- tooling

    `-- neon

        |-- 0.8.0

        |   |-- R201610241656

        |   |-- test

        |   `-- test_release

        `-- 0.9.0

            `-- R201703231548



We also have a top-level "0.7.0" folder which looks quite outdated.

Finally we also have a "download" folder with the following:

.

|-- 0.7.0

|   |-- N201508071412

|   |   `-- Papyrus-RT-Update-N201508071412.zip

|   |-- N201508071431

|   |   `-- Papyrus-RT-Update-N201508071431.zip

|   |-- N201508071616

|   |   `-- Papyrus-RT-Update-N201508071616.zip

|   `-- N201509010919

|       `-- Papyrus-RT-Update-N201509010919.zip

|-- 0.7.1

|   |-- N201510120954

|   |   `-- Papyrus-RT-Update-N201510120954.zip

|   `-- N201510121606

|       `-- Papyrus-RT-Update-N201510121606.zip

|-- 0.7.2

|   `-- N201512141100

|       `-- Papyrus-RT-Update-N201512141100.zip

|-- 0.8.0

|   |-- M201607280150

|   |-- M201607280214

|   |   `-- Papyrus-RT-Update-test.zip

|   |-- R201607280301

|   |   `-- Papyrus-RT-Update-test.zip

|   `-- R201610241656

`-- 0.9.0

    `-- R201703231548


Note that the last two "release" drops are empty.

I do not know who created all these.

So, from the above, which should I remove and which should I keep?

I assume we can remove all *-old* folders. Should we remove all 0.7.*? all 0.8.*? all 0.9.*? all test*?

Should I remove the top-level "rcps" which looks redundant? Do we need rcp/nightly?

--
Ernesto Posse
Zeligsoft




On Wed, Jul 5, 2017 at 11:32 AM Ernesto Posse <eposse@xxxxxxxxxxxxx> wrote:
I took a look and found that this archive folder used to be populated by the codegen build job (by the scripts/codegen/archive.sh script which I wrote to mimic the top-level archive.sh written by Celine). The codegen job used to have a different build than tooling, which is why it had separate scripts, but later the build was unified and the codegen build no longer invokes that script.

So these are indeed very old files and I have removed them all.

As for archiving nightlies, I'm not sure what we are doing exactly. Celine and Christian would have a better idea as they did the build reorganization. I see that the PapyrusRT-Master-* jobs have the archive post-build action enabled with "repository/**/*,*.zip" and for PapyrusRT-Master-Product it's "repository/**/*,*.zip ,**/products/*.zip , **/products/*.tar.gz". I'm not sure Furthermore, the build-after.sh script which is invoked by every job seems to move the generated p2 to ${WORKSPACE}/repository (component-wise or all) and the generated zipped p2 to ${WORKSPACE}. So in effect it looks like we have one archived copy of the nightly in each https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-Master-<Component>/ws/ and for the whole thing at https://hudson.eclipse.org/papyrus-rt/job/Papyrus-RT-Master-Product/ws. The scripts remove any previous archive found at this locations, so in effect we have exactly one nightly archived.

Hudson's help states that 

Archives the build artifacts (for example, distribution zip files or jar files) so that they can be downloaded later. Archived files will be accessible from the Hudson webpage. 
Normally, Hudson keeps artifacts for a build as long as a build log itself is kept, but if you don't need old artifacts and would rather save disk space, you can do so.

I don't know how long the "build log" is kept. 

The maximum number of build records to keep is set to 30 and the maximum number of builds to keep with artifacts is 3 for each PapyrusRT-Gerrit-* job, and 10 and 3 respectively for each PapyrusRT-Master-* job. It's not clear to me where exactly those last 3 artifacts are stored, though.

--
Ernesto Posse
Zeligsoft








On Wed, Jul 5, 2017 at 9:55 AM charles+zeligsoft.com <charles@xxxxxxxxxxxxx> wrote:
Ernesto: Could you please comment on this?


Regards,

Charles Rivet
Senior Product Manager, Papyrus-RT product leader

On 2017-07-05, at 08:49 , Peter Cigéhn <peter.cigehn@xxxxxxxxx> wrote:

Hi,

When looking a bit closer at the contents of http://download.eclipse.org/papyrus-rt/archives/?d it only seems to be the code-gen/run-time/textual stuff in those .zip-files anyway. Not sure who have any use of *only* those bits of Papyrus-RT. It also looks like the .zip-files mainly are between the period of 2015-05-13 up until 2016-09-14, so they are also pretty obsolete as well.

/Peter Cigéhn

On 5 July 2017 at 14:40, Remi Schnekenburger <rschnekenburger@xxxxxxxxxxxxxxxxx> wrote:
Nightlies are supposed to be volatile, they should not be kept long. It may be interesting to save the last one from a branch for "history", and some of last ones during development on a branch, but I do not see any use case for keeping such history on old branch. 
All the archive folder contains that kind of bundles...

Cheers,
Rémi 

2017-07-05 14:09 GMT+02:00 charles+zeligsoft.com <charles@xxxxxxxxxxxxx>:
There are currently 275 entries on that page…and if each of these represent one build per individual days, this means we are keeping ~9 months of data! Do we actually need to keep nightly builds for more than a week or two?


Regards,

Charles Rivet
Senior Product Manager, Papyrus-RT product leader

On 2017-07-05, at 03:05 , Remi Schnekenburger <rschnekenburger@xxxxxxxxxxxxxxxxx> wrote:

Hi team,

On the review materials for the release/graduation review, I got some message from Wayne about download server usage [1].
I had a look on the server, it seems indeed that some components are dropping some archives on the download server and do not have a task to remove them periodically [2]. So the disk usage is growing, with no specific added value (nightly bits). Could you have a look for example to the "archives" folder? There is probably some cleaning to perform.

Cheers,
Rémi


--
Remi Schnekenburger

Senior Software Architect / General Manager
EclipseSource Paris

Email: rschnekenburger@xxxxxxxxxxxxxxxxx
Web: http://eclipsesource.com/paris 
Phone: +33 1 85 41 08 65
Hangouts: rschnekenburger@xxxxxxxxxxxxxxxxx

EclipseSource France SAS
7 rue de la Croix Martre
91120 Palaiseau

General Manager: Remi Schnekenburger
Registered Office: 7 rue de la Croix Martre, 91120 Palaiseau, France
Commercial Register 824 977 516  R.C.S. EVRY
_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt-dev


_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt-dev




--
Remi Schnekenburger

Senior Software Architect / General Manager
EclipseSource Paris

Email: rschnekenburger@xxxxxxxxxxxxxxxxx
Web: http://eclipsesource.com/paris 
Phone: +33 1 85 41 08 65
Hangouts: rschnekenburger@xxxxxxxxxxxxxxxxx

EclipseSource France SAS
7 rue de la Croix Martre
91120 Palaiseau

General Manager: Remi Schnekenburger
Registered Office: 7 rue de la Croix Martre, 91120 Palaiseau, France
Commercial Register 824 977 516  R.C.S. EVRY

_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt-dev


_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt-dev

_______________________________________________
papyrus-rt-dev mailing list
papyrus-rt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/papyrus-rt-dev
--
Ernesto Posse
Zeligsoft
--
Ernesto Posse
Zeligsoft

Back to the top