Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[] Quick update - 3.1 postmortem, upgrades and planned outage on the way

Greetings to all,

In this issue:

1. We survived 3.1... *phew*
2. Access the Bugzilla database from PHP!
3. New backend servers - outage on the way
4. CVS Proxy for firewalled users

1. We survived 3.1

Looking back at it, a few things caused the infrastructure to be generally unavailable for the 18 hours following the release of 3.1:

- Files were available for public download as soon as they were on the servers, and mirrors didn't have time to sync up. After a while, there were so many connections to that mirrors couldn't even get through.

- It took too long for me to identify that we were being attacked.

- We didn't have enough bandwidth to meet the demand. This could have been avoided by allowing more time for mirrors to sync up.

- We didn't have a bittorrent distribution channel set up to take advantage of P2P

The big day-saver was a WhiteBox server we're using as a disaster-recovery box. It's designed to be an offsite "plug-in-and-serve" box for critical services, and within minutes of connecting it up to a separate 100Mb line, was saved.

2. Access the Bugzilla database from PHP!

You can now access the Bugzilla database using PHP scripts. For
integrity purposes, the database is read-only, and you cannot read user
passwords or e-mail addresses.

This may be useful to pull your own bugzilla stats. Other database
access, such as user-databases and access to the downloads stats
database should be available soon after the new hardware is installed
(See 3.)

For more information on database access, please use your committer
account and log into the Committer Tools at

3. New backend servers - outage required

The three IBM p5 550 servers will be ready for production use soon. Two
redundant servers will be handling the backend requests, such as
Database, File Serving and user authentication. These servers are
equipped with 16 GB of RAM and better I/O devices than the current backend server (15 disks vs. 3), so bottlenecks related to replication are likely to disappear. Also, total storage capacity will go from 250GB to 1TB.

In order to replace the current backend server with the new servers, we
will have to shut down many services, including Bugzilla and CVS, for a
period of less than one hour.

This outage is scheduled to occur during Sunday, July 17/2005 at 6:00am
Eastern time.

4. CVS Proxy for pserver and SSH connections

As per bug 87918 (, we've configured our load balancer to allow connections on standard ports to be redirected to the CVS ports for those unable to connect to CVS because of firewall restrictions. Here are the details: port 80 (http) --> port 2401 (PSERVER) port 443 (https) --> port 22 (SSH)

The attached screenshot shows a sample configuration to Extssh.

We will be updating our CVS How-To pages shortly with this information.

As usual, your comments, suggestions and questions are welcome.




Eclipse WebMaster - webmaster@xxxxxxxxxxx
Questions? Consult the FAQ at
View my status at

JPEG image

Back to the top