| [eclipse.org-committers] Update: Current infrastructure issues and next steps |
Hi everyone,
I know the recent infrastructure issues have been causing real friction for many of you, and I want to sincerely apologize for the disruption to your work.
I want to be completely transparent with all of you about what is happening, how we are fixing it right now, and how we are preventing it in the future.
Our infrastructure has been under immense stress this week. This is primarily due to the release of Antigravity 2.0 during Google I/O, which caused an unprecedented, order-of-magnitude spike in traffic. We are also observing new system behaviors alongside this traffic that we are currently analyzing.
The good news is that the infrastructure kept delivering most of our services under this storm, and we have learned a lot. However, I recognize that doesn't solve the immediate bottlenecks you are facing.
To stabilize the situation and get things back on track, we are taking the following actions:
Denis has drastically increased our bandwidth, which should make things easier right away.
We are migrating OpenVSX to another platform to restore a manageable load on our own data center. Moving OpenVSX is absolutely the right move, though in hindsight, doing it two weeks earlier would have prevented this stress. It is a fact that you are always late when these kind of things happen. We have pinpointed the core bottlenecks, and this migration will resolve the underlying issue by next Friday.
Looking past this week, I want to reassure you that our infrastructure management is vastly different from what we experienced a year ago.
Denis has built a new team of highly talented Site Reliability Engineers with extensive experience in the domain. Together with Matt’s team, they are driving a major overhaul toward "Infrastructure as Code" and I should say "Everything as Code". You should see dramatic, concrete improvements in our reliability and agility in the coming months as these structural changes take full effect.
Finally, I want to give massive support to Denis, Chris, and their teams. They are working hard behind the scenes to keep things running and finalize the OpenVSX migration. If you cross paths with them in the coming days, please join me in sending some extra encouragement their way. They definitely deserve it.
Thank you for your patience as we navigate these growing pains. Please feel free to reach out if you have any questions.
Best,
--
Gaël Blondelle
Vice President Community Operations
Secretary | Eclipse Foundation AISBL
P: +33 (0) 6 73 39 21 85 | LinkedIn
Rond Point Schuman 11 - Brussels 1040 - Belgium