Date: Tuesday, June 21, 2016 (05:00-08:00)

Who is affected:
All users of the CS department computing.

What is happening:
We are upgrading our storage operating system, which requires for CS
storage to be rebooted. All services that depend upon access to storage
will be unavailable, including – cycle servers, ionic cluster, web content,
home directories, CIFS, etc.

Why is it happening:
This is necessary in order to fix numerous bugs on our file system.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________

Update – 08:05 – An unexpected problem with the wrap-up up this morning\’s maintenance has resulted in a widespread outage of CS Department Services. We are working to restore service ASAP.

Update – 09:02 – We are still working to restore service ASAP.

Update – 09:20 – We are in the process of bringing services back online. We will have another update at 9:45.

Update – 09:48 – Most services are now back online. We should have everything restored by 10:00.

Update – 10:14 – A few services are still coming up. SMTP server is not up yet so sending email is not working. You can use webmail.cs.princeton.edu to send and receive email.

Update – 10:50 – SMTP service is now working.

Date: Tuesday, May 3, 2016 9:30PM

Who is affected:
Users of CS Department Services

Problem:
We are currently having issues with the CS file server. We are working to restore service and will post updates here as we learn more. CS Staff is currently on-site at the data center. We are working with the vendor to track down the issue. We should have another update by 10:00 PM.

Update 10:00 PM:

Services are starting to get restored. We are now working to bring things back online. We will have another update at 10:30 PM

Update 10:30 PM:

We are still in the process of restoring services. We will post another update at 11:00 PM

Update 11:00 PM:

We had to reboot all the file server nodes and we have one node left to reboot. Once the nodes come back online we will need to check and maybe reboot some CS servers to restore all services back to normal. We will post another update at 11:30 PM.

Update 11:30 PM:

All file server nodes are back online and working as expected. We are in the process of checking on each CS server and rebooting if needed. We will post another update at 12:00 AM.

Update 12:00 AM:

We are still checking on the status of all the CS servers. Some services have already been restored. We will post another update at 12:30 AM.

Update 12:15 AM:

Most CS services have been restored. We have a few servers that are still coming up.

Update 12:30 AM:

All CS services have been restored. Please let us know if you experience any continued trouble.

At this time, CS Department services are experiencing a widespread outage. We are aware of the problem, and working to bring things back. More updates will be posted as we learn more.

Date: Thursday, May 5, 2016 (07:30-08:00)

Who is affected:
Users of CS Department Email Services (IMAP, POP, webmail, etc.)

What is happening:
One of the two CS mailbox servers supporting the IMAP, POP, and Webmail
services will be shutdown for replacement of a defective DIMM. As a result,
users whose accounts are hosted by this server will experience
disconnection and an inability to read e-mail for a brief time while the
server is offline.

Why is it happening:
Hardware monitoring has indicated that a DIMM in this system is in a
pre-failure state and likely to fail soon. So as to avoid an unplanned
failure that may threaten the stability of the host, we are scheduling this
outage to replace the failing DIMM preemptively.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, March 15, 2016 (08:00-14:00)

Who is affected:
All users of the CS Department HPC Cluster (ionic.cs.princeton.edu).

What is happening:
On this day, the \”ionic\” cluster will be reinstalled and upgraded to the
latest OS distribution version, Springdale 7.2. This is a complex upgrade,
which is the reason for the unusually long outage window.

SPECIAL NOTE: As we are reloading the operating system, any crontabs on the
cluster will be lost. If you have crontabs that you wish to persist, you
will need to back up your crontabs before the downtime, and restore them
after.

Why is it happening:
This is part of normal maintenance of the publicly-accessible systems, and,
along with the patching of the cycles and penguins machines (announced
separately), will bring our primary public computational systems into
harmony running the same versions of Springdale Linux.

Please note that with this upgrade, some older versions of software, or
some packages which are no longer part of the distribution, may no longer
be available. We encourage you to verify your workflows after this upgrade
to ensure you are able to continue your work. CS Staff stands ready to
assist with any unforeseen trouble.

For most workflows, it should be possible to test your code on the cycles
machines in advance of the upgrade, as they are already running a version
of Springdale Linux that is close to the version to be installed during
this window. Some things will likely still differ after the upgrade, since
the cycles machines will also be updated, but the cycles machines are
receiving only minor updates, so breaking changes should be few or none.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, March 15, 2016 (08:00-09:00)

Who is affected:
All users of the CS Department public login servers (tux and opus, aka
\”penguins\”) and cycle servers (soak, wash, rinse, and spin, aka cycles\”)

What is happening:
During this window, these systems will be patched and updated from their
current Springdale 7.1 OS to the latest version, Springdale 7.2.

This is considered a minor update. Any resulting problems should be small
and relatively easily addressed. Since these machines are being patched
rather than reinstalled, cron jobs will remain intact.

Why is it happening:
This is part of normal maintenance of the publicly-accessible systems, and,
along with the upgrade of our HPC cluster (announced separately), will
bring our primary public computational systems into harmony running the
same versions of Springdale Linux.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Thursday, February 18, 2016 (09:00-15:00)

Who is affected:
CS email users who attempt to interact with the Proofpoint spam filtering
system

What is happening:
We will be updating the Proofpoint spam filtering software on our mail
servers.

While no email outage is expected, it is possible that you may receive some
extra spam during the upgrade as the filters go offline for updates. You
may also find that you are unable to interact with the Proofpoint systems
during this time to perform actions such as releasing quarantined messages.
All mail sending, delivery, and reading mechanisms are expected to remain
operational.

Note that the maintenance window does not start until 09:00, so as to
provide some time after the 08:00 digests are sent out during which you may
release messages from the morning digest.

Why is it happening:
This is a necessary part of the regular maintenance of our systems.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, December 22, 2015 (06:00-09:00)

Who is affected:
All users of CS Department email services

What is happening:
All incoming email services will be shutdown while the Zimbra server
software is upgraded.

During this time, IMAP, POP, and Webmail will all be unavailable, as will
any email forwarding or filtering. All incoming messages should be safely
queued on our MTAs for delivery after the downtime. IMAP and POP users
should not notice any changes after the downtime. Webmail users will see an
updated interface.

Why is it happening:
This is a reschedule of an upgrade originally scheduled for November 5. At
that time, problems with the upgrade procedure resulted in an inability to
complete the upgrade during the scheduled window. In the intervening time,
several behind-the-scenes updates have taken place and paved the way for a
successful upgrade on this pass.

This upgrade will bring our mail server software up-to-date with the
current release version from Zimbra. Among other things, it will address
several recent vulnerabilities announced in various portions of the
software.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Wednesday, November 25, 2015 (07:30-08:30)

Who is affected:
All users of CS Department public login (cycles/penguins/portal) and
compute (ionic) systems.

What is happening:
During this window, these machines will be rebooted in order to clear some
unkillable user processes which are interfering with some research work.

We will also take this opportunity to make configuration changes to reduce
the likelihood of recurrence for this and related issues.

Why is it happening:
As some user processes have entered an unkillable state, and as those
processes are preventing research work, they require a system reboot to
clear.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

As you may have noticed, this upgrade was not completed this morning due to issues with the upgrade process. Some critical parts of the system were successfully upgraded, which should allow us to upgrade more components in a less-disruptive way, but the process is not completed at this time.

More updates will follow at a future date. Thanks for your patience.

-CS Staff

—– Original Message —–
From: csstaff@CS.Princeton.EDU
To: downtime@lists.cs.princeton.edu
Sent: Tuesday, October 27, 2015 11:38:30 AM
Subject: [downtime] CS Email Downtime, Thursday, November 5, 2015, 06:00-09:00

Date: Thursday, November 5, 2015 (06:00-09:00)

Who is affected:
All users of CS Department email services

What is happening:
All incoming email services will be shutdown while the Zimbra server
software is upgraded.

During this time, IMAP, POP, and Webmail will all be unavailable, as will
any email forwarding or filtering. All incoming messages should be safely
queued on our MTAs for delivery after the downtime. IMAP and POP users
should not notice any changes after the downtime. Webmail users will see an
updated interface.

Why is it happening:
This upgrade will bring our mail server software up-to-date with the
current release version from Zimbra. Among other things, it will address
several recent vulnerabilities announced in various portions of the
software.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime