Emergency Downtime: Mon, Nov 3, 2008

On Monday, November 3, 2008, we will have an emergency downtime from 3:00am to 9:00am EST (note time).

During this time we will be shutting down our infrastructure so that we can re-install the operating system on the file server.

We continue to have performance/connectivity problems with the file server. The vendor (who has been working the issue over the weekend) is suggesting that we do a fresh install of the OS so that we have clean slate. While we don’t know if this will correct the problems, our hope is that, at the very least, we will regain critical tracing capability that will then help us find and correct the problem.

Update 9:00am: Our systems are still down so that we can continue to debug the problem with our filesystem. We will bring our systems back online by 11:00am. Our hope is that we will have resolved the file server issue by then. However, in the event that we are unsuccessful, we will revert to our previous configuration/state that, while problematic, works more than 0% of the time.

Update 11:10am: We were not able to correct the problem with the file server and had to rollback to our previous state. We will announce another downtime soon.