[downtime] CS Storage Maintenance, Wednesday, March 23, 2022,

Tuesday, March 15, 2022 at 9:42am EDT by csstaff

Date: Wednesday, March 23, 2022 (09:00-15:00)

Who is affected:
All users of the CS department storage or computing facilities

What is happening:
We are upgrading our storage operating system, which requires the
components of the CS storage system to be rebooted. This upgrade is
expected to be mostly non-disruptive, and department services will continue
as usual, but sporadic moments of interruption may be noticeable at times.

All services that depend upon access to storage share the same minor risk
of performance hiccups for some periods during this window, including cycle
servers, ionic cluster, web content, home directories, CIFS, etc.

Why is it happening:
This upgrade is necessary in order to get our storage aligned with the
latest operating system, which is crucial for our upcoming storage hardware
upgrade.

While we do not anticipate any extended service outages, you may find that
there are momentary interruptions, and some connections (especially
CIFS/SMB connections) may need to be reestablished.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Storage Maintenance, Wednesday, February 23, 2022,

Tuesday, February 22, 2022 at 8:25am EST by csstaff

Date: Wednesday, February 23, 2022 (13:00-14:00)

Who is affected:
Users of CS Department Storage Systems

What is happening:
During this window, one node in our storage cluster will be taken offline
to replace a failing DIMM.

Why is it happening:
Proactive monitoring has noted a DIMM that is nearing failure, so a
replacement has been arranged.

Some minor performance effects may occur due to the offline node. No outage
is anticipated, but maintenance always carries the risk of unexpected
problems, which is the reason for this notice.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] Emergency CS Email Upgrades, TODAY Monday, February 7,

Monday, February 7, 2022 at 9:50am EST by csstaff

Date: TODAY Monday, February 7, 2022 (10:00-13:00)

Who is affected:
All users of CS Department Email Services

What is happening:
Patches will be applied to the CS email servers to mitigate a serious
zero-day vulnerability that was released over the weekend. During this
patching, email services should continue as normal, though you may notice
brief outages while services are restarted.

If your mail client with a “remembered” password suddenly prompts you for a
password, this may be a sign that you’ve been disconnected due to a service
restart. The best approach is usually to cancel the password prompt and
wait a minute or two before restarting your email client. This should
preserve the “remembered” password, which should work after the service
restarts.

Why is it happening:
A serious zero-day vulnerability in our mail services was released over the
weekend. Leaving this vulnerability unpatched could lead to account
compromises by malicious emails.

As a reminder, you should always be cautious with unexpected emails,
particularly with attachments or links included. This particular
vulnerability requires that email recipients click on links in malicious
messages in order to effect the compromise.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Network Maintenance, Wednesday, October 20, 2021,

Thursday, October 14, 2021 at 12:48pm EDT by csstaff

Date: Wednesday, October 20, 2021 (07:00-09:00)

Who is affected:
Users of CS Department Network Services

What is happening:
The CS Department’s network uplink to OIT’s network (and onward to the
internet) will undergo configuration changes to finalize the deployment of
redundant upstream peering to OIT’s new routers.

This work is expected to cause some very short outages of our uplink, but
most people or systems should not notice them. As technological changes
always present the possibility of unexpected results, though, this message
is notice in case of the unexpected.

Why is it happening:
This change will finalize the router peering with OIT’s new routers in the
Lewis Library and New South buildings, providing more redundancy than the
original design which uplinked the department only through Lewis Library
(and 87 Prospect prior to that).

This last step will move the actual routing of CS Department traffic
through the new routers.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this work will cause you undue hardship, or if you have questions or
concerns, please contact csstaff@cs.princeton.edu to discuss. Your patience
is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Network Maintenance, Wednesday, October 13, 2021,

Monday, October 11, 2021 at 3:23pm EDT by csstaff

Date: Wednesday, October 13, 2021 (07:00-08:30)

Who is affected:
Users of CS Department Network Services

What is happening:
The CS Department’s network uplink to OIT’s network (and onward to the
internet) will undergo configuration changes in preparation for deployment
of OIT’s Next Generation Network.

No noticeable outage is anticipated, but as technological changes always
present the possibility of unexpected results, this message is notice in
case of the unexpected.

Why is it happening:
This change will enable router peering with OIT’s new routers in the Lewis
Library and New South buildings, providing more redundancy than the
original design which uplinked the department only through Lewis Library
(and 87 Prospect prior to that).

This is the second step in a multi-step process of reconfiguring this
uplink.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this work will cause you undue hardship, or if you have questions or
concerns, please contact csstaff@cs.princeton.edu to discuss. Your patience
is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Database Downtime, Tuesday, October 12, 2021,

Wednesday, October 6, 2021 at 2:13pm EDT by csstaff

Date: Tuesday, October 12, 2021 (07:00-07:30)

Who is affected:
Users of CS Department web sites and other services including:

https://adm.cs.princeton.edu/
https://csguide.cs.princeton.edu/
https://fam.cs.princeton.edu/
https://iw.cs.princeton.edu/portal/
https://keymanager.cs.princeton.edu/
https://pac.cs.princeton.edu/portal/
https://ris.cs.princeton.edu/
https://tigerfile.cs.princeton.edu/
https://www.cs.princeton.edu/

What is happening:
During this window, our main internal database server will be rebooted.
This will result in a brief outage for the above-listed sites, but should
otherwise not be noticeable.

Other department services, including email, networking, and research group
and other hosted web sites, will continue uninterrupted.

Why is it happening:
The database server is suffering from memory errors, for which a BIOS
update fix is required.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Email Maintenance, Wednesday, September 29, 2021,

Tuesday, September 28, 2021 at 11:32am EDT by csstaff

Date: Wednesday, September 29, 2021 (05:00-06:00)

Who is affected:
Users of CS Department Email Services

What is happening:
CS Department email services will be momentarily unavailable while critical
updates take place.

Why is it happening:
On September 30, 2021, an important TLS Root CA Certificate (“DST Root X3”)
will expire. This expiration may cause communication disruption for some
older servers, including one server in our email infrastructure. This brief
outage will allow us to update the system to avoid unplanned disruption
after the certificate expiration.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this work will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.
Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Network Downtime, Thursday, August 19, 2021,

Friday, August 13, 2021 at 11:09am EDT by csstaff

Date: Thursday, August 19, 2021 (07:00-08:00)

Who is affected:
Users of CS Department Network Services, including e-mail, web servers, and
other remotely-accessed services

What is happening:
During this window, the CS Department’s border router (csgate) uplink to
the OIT network (and to the internet from there) will undergo a
reconfiguration. This reconfiguration will enable BGP for csgate and
disable RIP.

Actual outage time is expected to be less than a minute, but we are
scheduling a longer window to account for the unexpected.

Why is it happening:
As part of OIT’s Next Generation Network planning and deployment, the
campus network is evolving from RIPv2 routing to use BGP instead. This
switch to BGP for csgate is the first step in a multi-step process that
will end with the department uplink having better redundancy and better
performance. Stay tuned for those updates over the next few months!

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] CS Ionic / Cycles System Downtime, Tuesday, August 17,

Tuesday, August 3, 2021 at 12:49pm EDT by csstaff

Date: Tuesday, August 17, 2021 (07:00-12:00)

Who is affected:
All users of the CS Department Beowulf high performance computing cluster,
known as ionic.

All users of the CS Staff-managed public login systems, including the
cycles, courselab, and armlab systems.

What is happening:
During this window, OS patches will be applied to update our managed ionic
systems from Springdale 7.8 to 7.9. After patching, all systems will be
rebooted. In addition, cluster management and job scheduling system slurm
and its database will be upgraded. No data loss is anticipated.

Cycles and courselab systems will have their OSes reinstalled and upgraded
from Springdale 7.8 to 7.9.

Armlab systems will be rebooted.

SPECIAL NOTE: As we are reloading the Linux servers, all crontabs will be
deleted. If you have crontabs that you wish to persist, you will need to
back up your crontabs before the downtime, and restore them after.

Why is it happening:
This is part of regular maintenance to keep systems up-to-date.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime

[downtime] Emergency CS Storage Downtime, Wednesday, July 28, 2021,

Monday, July 26, 2021 at 12:34pm EDT by csstaff

Date: Wednesday, July 28, 2021 (09:30-11:30)

Who is affected:
All users of the CS department computing.

What is happening:
A single node of our file server cluster will be shutdown for the
replacement of a defective DIMM.

Why is it happening:
Hardware monitoring has indicated that a DIMM in one of the file server
cluster nodes is in a pre-failure mode and likely to fail soon. So as to
avoid an unplanned failure that may threaten the stability of the node, we
are scheduling this outage to replace the failing DIMM preemptively.

We do not anticipate any outages, but you may find that some connections
(especially CIFS/SMB) may need to be reestablished.

We will post updates to the status page: http://www.csstaff.org
as necessary.

If this downtime will cause you undue hardship, please contact
csstaff@cs.princeton.edu immediately, so we can discuss options to reduce
any negative impact. Your patience is appreciated.

Sincerely,
CS Staff
_______________________________________________
downtime mailing list
downtime@lists.cs.princeton.edu
https://lists.cs.princeton.edu/mailman/listinfo/downtime