All

Date: Wednesday, July 27, 2016 (09:00-11:00)

Who is affected:
All users of the CS Department Courselab servers (courselab01 and
courselab02).

What is happening:
During this window, these systems will have their OSes reinstalled and
upgraded to the latest distribution version, Springdale 7.2.

Why is it happening:
This is part of normal maintenance of the publicly-accessible systems, and
will bring newer versions of installed tools and software.

Please note that with this upgrade, some older versions of software, or
some packages which are no longer part of the distribution, may no longer
be available. We encourage you to verify your workflows after this upgrade
to ensure you are able to continue your work. CS Staff stands ready to
assist with any unforeseen trouble.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: August 1, 2016

Who is affected:
All users of the CS Department public login hosts, tux.cs.princeton.edu and
opus.cs.princeton.edu (penguins.cs.princeton.edu).

What is happening:
On August 1, 2016, tux and opus will be turned off and retired. Future
logins should use cycles.cs.princeton.edu (soak, wash, rinse, or spin).

NOTE: As these servers are retiring, all crontabs on tux and opus will be
retired with them. If you need to maintain a crontab currently active on
one of these systems, you will need to relocate it to one of the cycles
machines before August 1.

The DNS names \”tux\”, \”opus\”, and \”penguins\” will all remain for a period of
at least six months, and will become aliases for \”cycles.cs.princeton.edu\”.
We encourage you to use the time to update your scripts, configurations, or
other references to the retiring names.

Why is it happening:
The purpose of tux and opus, for the last several years, has been to
provide a space for lightweight interactive work such as reading email or
organizing files. This was specifically intended to separate these
interactive activities from the more computationally intensive work done on
the cycles servers, so as to reduce the incidence of conflict.

In recent years, advances in kernel technology and our configuration
management systems have enabled us to provide a more stable and fair
environment in the cycles servers such that most users can maintain a
reasonable share of system resources, even while other users are doing
computationally intensive work. For this reason, the separation of the
penguins servers is no longer as useful as it once was, and the costs of
maintaining the distinct system configurations (as well as user confusion
resulting in computationally intensive work running on penguins) has risen
enough to outweigh the benefits.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this system retirement will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Date: Tuesday, June 21, 2016 (05:00-08:00)

Who is affected:
All users of the CS department computing.

What is happening:
We are upgrading our storage operating system, which requires for CS
storage to be rebooted. All services that depend upon access to storage
will be unavailable, including – cycle servers, ionic cluster, web content,
home directories, CIFS, etc.

Why is it happening:
This is necessary in order to fix numerous bugs on our file system.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________

Update – 08:05 – An unexpected problem with the wrap-up up this morning\’s maintenance has resulted in a widespread outage of CS Department Services. We are working to restore service ASAP.

Update – 09:02 – We are still working to restore service ASAP.

Update – 09:20 – We are in the process of bringing services back online. We will have another update at 9:45.

Update – 09:48 – Most services are now back online. We should have everything restored by 10:00.

Update – 10:14 – A few services are still coming up. SMTP server is not up yet so sending email is not working. You can use webmail.cs.princeton.edu to send and receive email.

Update – 10:50 – SMTP service is now working.

Date: Tuesday, May 3, 2016 9:30PM

Who is affected:
Users of CS Department Services

Problem:
We are currently having issues with the CS file server. We are working to restore service and will post updates here as we learn more. CS Staff is currently on-site at the data center. We are working with the vendor to track down the issue. We should have another update by 10:00 PM.

Update 10:00 PM:

Services are starting to get restored. We are now working to bring things back online. We will have another update at 10:30 PM

Update 10:30 PM:

We are still in the process of restoring services. We will post another update at 11:00 PM

Update 11:00 PM:

We had to reboot all the file server nodes and we have one node left to reboot. Once the nodes come back online we will need to check and maybe reboot some CS servers to restore all services back to normal. We will post another update at 11:30 PM.

Update 11:30 PM:

All file server nodes are back online and working as expected. We are in the process of checking on each CS server and rebooting if needed. We will post another update at 12:00 AM.

Update 12:00 AM:

We are still checking on the status of all the CS servers. Some services have already been restored. We will post another update at 12:30 AM.

Update 12:15 AM:

Most CS services have been restored. We have a few servers that are still coming up.

Update 12:30 AM:

All CS services have been restored. Please let us know if you experience any continued trouble.

At this time, CS Department services are experiencing a widespread outage. We are aware of the problem, and working to bring things back. More updates will be posted as we learn more.

Date: Thursday, May 5, 2016 (07:30-08:00)

Who is affected:
Users of CS Department Email Services (IMAP, POP, webmail, etc.)

What is happening:
One of the two CS mailbox servers supporting the IMAP, POP, and Webmail
services will be shutdown for replacement of a defective DIMM. As a result,
users whose accounts are hosted by this server will experience
disconnection and an inability to read e-mail for a brief time while the
server is offline.

Why is it happening:
Hardware monitoring has indicated that a DIMM in this system is in a
pre-failure state and likely to fail soon. So as to avoid an unplanned
failure that may threaten the stability of the host, we are scheduling this
outage to replace the failing DIMM preemptively.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, March 15, 2016 (08:00-14:00)

Who is affected:
All users of the CS Department HPC Cluster (ionic.cs.princeton.edu).

What is happening:
On this day, the \”ionic\” cluster will be reinstalled and upgraded to the
latest OS distribution version, Springdale 7.2. This is a complex upgrade,
which is the reason for the unusually long outage window.

SPECIAL NOTE: As we are reloading the operating system, any crontabs on the
cluster will be lost. If you have crontabs that you wish to persist, you
will need to back up your crontabs before the downtime, and restore them
after.

Why is it happening:
This is part of normal maintenance of the publicly-accessible systems, and,
along with the patching of the cycles and penguins machines (announced
separately), will bring our primary public computational systems into
harmony running the same versions of Springdale Linux.

Please note that with this upgrade, some older versions of software, or
some packages which are no longer part of the distribution, may no longer
be available. We encourage you to verify your workflows after this upgrade
to ensure you are able to continue your work. CS Staff stands ready to
assist with any unforeseen trouble.

For most workflows, it should be possible to test your code on the cycles
machines in advance of the upgrade, as they are already running a version
of Springdale Linux that is close to the version to be installed during
this window. Some things will likely still differ after the upgrade, since
the cycles machines will also be updated, but the cycles machines are
receiving only minor updates, so breaking changes should be few or none.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, March 15, 2016 (08:00-09:00)

Who is affected:
All users of the CS Department public login servers (tux and opus, aka
\”penguins\”) and cycle servers (soak, wash, rinse, and spin, aka cycles\”)

What is happening:
During this window, these systems will be patched and updated from their
current Springdale 7.1 OS to the latest version, Springdale 7.2.

This is considered a minor update. Any resulting problems should be small
and relatively easily addressed. Since these machines are being patched
rather than reinstalled, cron jobs will remain intact.

Why is it happening:
This is part of normal maintenance of the publicly-accessible systems, and,
along with the upgrade of our HPC cluster (announced separately), will
bring our primary public computational systems into harmony running the
same versions of Springdale Linux.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Thursday, February 18, 2016 (09:00-15:00)

Who is affected:
CS email users who attempt to interact with the Proofpoint spam filtering
system

What is happening:
We will be updating the Proofpoint spam filtering software on our mail
servers.

While no email outage is expected, it is possible that you may receive some
extra spam during the upgrade as the filters go offline for updates. You
may also find that you are unable to interact with the Proofpoint systems
during this time to perform actions such as releasing quarantined messages.
All mail sending, delivery, and reading mechanisms are expected to remain
operational.

Note that the maintenance window does not start until 09:00, so as to
provide some time after the 08:00 digests are sent out during which you may
release messages from the morning digest.

Why is it happening:
This is a necessary part of the regular maintenance of our systems.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

Date: Tuesday, December 22, 2015 (06:00-09:00)

Who is affected:
All users of CS Department email services

What is happening:
All incoming email services will be shutdown while the Zimbra server
software is upgraded.

During this time, IMAP, POP, and Webmail will all be unavailable, as will
any email forwarding or filtering. All incoming messages should be safely
queued on our MTAs for delivery after the downtime. IMAP and POP users
should not notice any changes after the downtime. Webmail users will see an
updated interface.

Why is it happening:
This is a reschedule of an upgrade originally scheduled for November 5. At
that time, problems with the upgrade procedure resulted in an inability to
complete the upgrade during the scheduled window. In the intervening time,
several behind-the-scenes updates have taken place and paved the way for a
successful upgrade on this pass.

This upgrade will bring our mail server software up-to-date with the
current release version from Zimbra. Among other things, it will address
several recent vulnerabilities announced in various portions of the
software.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime