Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [eclipse.org-committers] Build service update

Hi Denis

Thank you for your efforts in what is clearly a very challenging situation.

Unfortunately your comments are somewhat optimistic; only 10 JIPPs are less than three months stale. My 4 JIPPs are 6 months stale during which time two phases of infrastructure evolution had to be accommodated, so I need the updated jobs to avoid redevelopment.

Regards

Edward Willink

On 30/04/2025 03:10, Denis Roy via eclipse.org-committers wrote:

Folks,


I was overly optimistic about the recovery process of the storage cluster that is hosting Jenkins data (job configurations, etc). Recovery is likely, but not guaranteed, and the timeline is not certain.

Unfortunately, we’ve been relying on this technology to not fail in the manner it has, and the end result is that the availability of Jenkins backup data is not consistent for each hosted instance. While we may be able to recover some (and hopefully most), there are instances for which we don’t have a recent backup.


To avoid prolonging the outage any further, we have two options:


  1. We can provision a Jenkins instance with the last known version of the Jenkins data. This data is the last snapshot prior to migration to the storage cluster, and, depending on when your instance was migrated, may be weeks, or up to several months old.


  1. We can provision a new, bare Jenkins instance for your project. If your build is defined in pipelines alongside your code, recovery may be more straightforward.


You can also choose to wait for the storage cluster to recover, so that we can either continue its usage, or extract data from it to provision a new instance. We do not, however, have an ETA for this recovery, nor do we know with any level of certainty that it actually will recover.


We recognise that none of these options are ideal, and we will work with your project to restore service as best we can. We’ve compiled a list[1] of Jenkins instances as well as the date of the last-known-good data, to help with your decision. When you’re ready to proceed, please file an issue on HelpDesk[2] with Project Lead +1 and the team will assist as soon as possible. 


Once service is restored, we will perform a postmortem and assess the path forward and share that information with you. 


Regardless of what we find in the postmortem, we obviously missed in not ensuring backups of these jobs were captured regularly in case of a catastrophic failure such as this one. We sincerely apologize and thank you for your understanding.


[1] https://docs.google.com/spreadsheets/d/13zNUbwAHkYacC193HFwgZOd1fqn-N9cyzXf_lHke07s

[2] https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/new


--

Denis Roy

Director, IT Services | Eclipse Foundation

Eclipse Foundation: The Community for Open Innovation and Collaboration


_______________________________________________
eclipse.org-committers mailing list
eclipse.org-committers@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/eclipse.org-committers

Back to the top