May 2023

Good morning.

You may notice that CS systems have had a significant outage overnight. This was due to major power problems at the campus data center facility at Forrestal. All of our systems in the HPCRC were unexpectedly powered off and suffered various other power effects throughout the evening and night, and some did not recover on their own.

CS Staff members have been on site at the data center and also working remotely to recover systems this morning. At this time, all major systems are expected to be online again. Some ionic cluster nodes are still down, but are being brought back.

If you notice any persistent issues with CS systems, please let us know and we will do our best to address them. Thank you for your patience.

-CS Staff

Date: Wednesday, May 31, 2023 (08:00-10:00)

Who is affected:

All users of CS Department computing facilities and services, including cycles, ionic, web services, email, DNS, and wired networking.

What is happening:

During this window, the core switch handling CS Department network traffic at the HPCRC will be replaced. The actual outage time for any particular service or network access point should be only a few seconds, and the total outage window is likely to be shorter than announced. However, owing to the uncertain nature of technological change, outages may occur throughout this window and may be up to several minutes in length.

Why is it happening:

This upgrade will replace a 12-year-old core switch with a new, much faster device. This is the first stage of more upgrades upcoming this Summer, primarily focused on increasing network capacity for the ionic HPC cluster, the Department’s central storage cluster, and other related systems.

We will post updates to the status page: www.csstaff.org as necessary.

If this downtime will cause you undue hardship, please contact csstaff@cs.princeton.edu immediately, so we can discuss options to reduce any negative impact. Your patience is appreciated.

Sincerely,

CS Staff