May 2016

Date: Tuesday, May 3, 2016 9:30PM

Who is affected:
Users of CS Department Services

Problem:
We are currently having issues with the CS file server. We are working to restore service and will post updates here as we learn more. CS Staff is currently on-site at the data center. We are working with the vendor to track down the issue. We should have another update by 10:00 PM.

Update 10:00 PM:

Services are starting to get restored. We are now working to bring things back online. We will have another update at 10:30 PM

Update 10:30 PM:

We are still in the process of restoring services. We will post another update at 11:00 PM

Update 11:00 PM:

We had to reboot all the file server nodes and we have one node left to reboot. Once the nodes come back online we will need to check and maybe reboot some CS servers to restore all services back to normal. We will post another update at 11:30 PM.

Update 11:30 PM:

All file server nodes are back online and working as expected. We are in the process of checking on each CS server and rebooting if needed. We will post another update at 12:00 AM.

Update 12:00 AM:

We are still checking on the status of all the CS servers. Some services have already been restored. We will post another update at 12:30 AM.

Update 12:15 AM:

Most CS services have been restored. We have a few servers that are still coming up.

Update 12:30 AM:

All CS services have been restored. Please let us know if you experience any continued trouble.

At this time, CS Department services are experiencing a widespread outage. We are aware of the problem, and working to bring things back. More updates will be posted as we learn more.

Date: Thursday, May 5, 2016 (07:30-08:00)

Who is affected:
Users of CS Department Email Services (IMAP, POP, webmail, etc.)

What is happening:
One of the two CS mailbox servers supporting the IMAP, POP, and Webmail
services will be shutdown for replacement of a defective DIMM. As a result,
users whose accounts are hosted by this server will experience
disconnection and an inability to read e-mail for a brief time while the
server is offline.

Why is it happening:
Hardware monitoring has indicated that a DIMM in this system is in a
pre-failure state and likely to fail soon. So as to avoid an unplanned
failure that may threaten the stability of the host, we are scheduling this
outage to replace the failing DIMM preemptively.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime