Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[eclipse.org-committers] Build service update

Folks,


I was overly optimistic about the recovery process of the storage cluster that is hosting Jenkins data (job configurations, etc). Recovery is likely, but not guaranteed, and the timeline is not certain.

Unfortunately, we’ve been relying on this technology to not fail in the manner it has, and the end result is that the availability of Jenkins backup data is not consistent for each hosted instance. While we may be able to recover some (and hopefully most), there are instances for which we don’t have a recent backup.


To avoid prolonging the outage any further, we have two options:


  1. We can provision a Jenkins instance with the last known version of the Jenkins data. This data is the last snapshot prior to migration to the storage cluster, and, depending on when your instance was migrated, may be weeks, or up to several months old.


  1. We can provision a new, bare Jenkins instance for your project. If your build is defined in pipelines alongside your code, recovery may be more straightforward.


You can also choose to wait for the storage cluster to recover, so that we can either continue its usage, or extract data from it to provision a new instance. We do not, however, have an ETA for this recovery, nor do we know with any level of certainty that it actually will recover.


We recognise that none of these options are ideal, and we will work with your project to restore service as best we can. We’ve compiled a list[1] of Jenkins instances as well as the date of the last-known-good data, to help with your decision. When you’re ready to proceed, please file an issue on HelpDesk[2] with Project Lead +1 and the team will assist as soon as possible. 


Once service is restored, we will perform a postmortem and assess the path forward and share that information with you. 


Regardless of what we find in the postmortem, we obviously missed in not ensuring backups of these jobs were captured regularly in case of a catastrophic failure such as this one. We sincerely apologize and thank you for your understanding.


[1] https://docs.google.com/spreadsheets/d/13zNUbwAHkYacC193HFwgZOd1fqn-N9cyzXf_lHke07s

[2] https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/new


--

Denis Roy

Director, IT Services | Eclipse Foundation

Eclipse Foundation: The Community for Open Innovation and Collaboration

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature

Back to the top