Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[] Database server outage -- download stats


About 24 hours ago our secondary database server suffered a segfault, which required a hard reset. Unfortunately, that led to a corrupted filesystem that required a fresh install of the underlying OS.

Not taking any chances with our data, we decided to wipe it clean and re-sync all the data from the master. This is the first time we've had to do this since 2007.

The server is now back up; however, for several hours a single database server was powering our site, and it could barely keep up with the demands placed on it, as we normally operate with three database servers (DB server #3 suffered a series of hardware failures last December and was retired).

The added stress of syncing 200GB of data made it impossible for me to sync the massive download stats content while the website was up. My only choices were to turn off for two hours, or truncate the download stats. I've chosen to truncate the download stats.

I apologize for terminating this data. Hindsight being 20/20 I should have provisioned a replacement third DB server sooner. This would have allowed us to re-sync from a slave, preserving the download stats and site performance. We'll work on that in the weeks to come.

If you have any questions or concerns, please don't hesitate to contact Matt or I at webmaster@xxxxxxxxxxx


Denis Roy

Back to the top