This is a repost of an OIT message. This downtime is being performed by OIT but affects wireless networking in the CS Building.

Service(s) affected: Wireless networking in all Administrative and Academic Buildings on both Main Campus and Forrestal B Site 6/15/2013 from 06:00 to 10:00.
Date/time of outage: 06/15/2013 6:00 am
Duration of outage: 4 hours 0 minutes

There will be a wireless networking outage in all administrative and academic buldings on both Main Campus and Forrestal B Site on 6/15/2013 from 06:00 to 10:00 AM. This is needed for a system reconfiguration.

Who is affected:

• All users of the CS Department \”cycles\” or \”penguins\” systems (soak, wash, rinse, spin, opus or tux).

What is happening:

• On Tuesday morning, the cycles systems will be replaced with newer, faster hardware with more memory. The existing cycles machines will later be added to the ionic compute cluster. At the same time, opus and tux will have OS updates applied.
• SPECIAL NOTE: As we are replacing the hardware, the operating systems on the cycles systems will also be reinstalled, so all crontabs will be deleted. You will need to back up your crontabs before the downtime, and put them back after.

Why is it happening:

• As part of routine maintenance, and to provide enhanced capabilities, the hardware running our systems is periodically replaced, and OS updates are periodically applied.
• Further, OS updates are urgent at this time for all of our Linux systems as a result of recent kernel vulnerabilities for which we have seen active exploits occurring.

4:00pm : We are experiencing a problem with our infrastructure. E-mail, websites, cycle servers, and file systems are impacted. We are investigating.

4:25pm: The CS LDAP servers are currently having some issues, which are causing problems with email and the file system. We are working to address the problem and should have an update soon.

4:40pm: The LDAP Servers have been fixed. The file system and email should start working again in the next 15 minutes.

We are working to resolve an issue with DropBox. The site is currently down. We will post status updates here.

Update 11:39 AM – DropBox is now working again. Please let us know if you have any issues.

On Tuesday, March 19, 2013, we will have a scheduled downtime from 6:00am to 8:00am EDT.

Some of this work (bullets \”1\” below) was originally scheduled for March 5, 2013.

Who is affected:

1. Users of the public login machines (soak, wash, rinse, spin, tux, opus), certain CS websites (jobs, kiosk, msdnaa, search, wiki), database servers, runscript, CAS, lpdrelay (printing), labpc machines (Friend Center fishbowl lab) and the ionic cluster.
2. Dynamic websites for projects or groups with hostnames of the form project-or-group-name.cs.princeton.edu that are accessing executables (e.g., python, perl, java) on the shared /usr/local filesystem.

What is happening:

1. These hosts will receive a critical security update and be rebooted.
2. Websites of the form project-or-group-name.cs.princeton.edu are moving to a new server that no longer mounts the shared /usr/local filesystem.

Why is it happening:

1. There is a critical kernel security patch available for our Springdale hosts. It addresses a specific security vulnerability. As this is a kernel update, all machines must be rebooted. Actual expected downtime should only be a minute or two for each host to reboot.
2. The website migration is the next step to decommission the shared /usr/local filesystem. This work has already been done for the cycles and penguin machines and will result in decreased load on our central file server and simplified management of our web server infrastructure. Additional notes about the website migration:
   • In the week leading up to the downtime, we will reach out to the owners of as many affected websites as we can identify with instructions on what they need to do for the migration. We anticipate that very few sites will need modification and that those modifications will be minor.
   • After the downtime, all projects and groups should double check that their sites are operating as expected. If not, please notify CS Staff immediately and we can assist and/or temporarily move the site back to the old server.

On Saturday, March 16, 2013, we will have a scheduled downtime from 6:00am to 8:00am EDT.

Who is affected:

• All users of the CS wired network (including PlanetLab at 221 Nassau, CS hosts in CITP in Sherrerd, and the CS section of the data center at 151 Forrestal) as well as users of the OIT wireless network in the CS Building.

What is happening:

• We will be updating the firmware on the department\’s gateway router and firewall service module. During this time, there will be no OIT wireless connectivity in the CS Building (as the OIT access points use the CS wired infrastructure) and there will be no Internet connectivity between the CS network and the outside world.

Why is it happening:

• This update will address some ongoing network issues affecting a limited number of services.

OIT is reporting that the puwireless is not available to some users. For more details, see their outage message.

On Tuesday, March 5, 2013, we will have a scheduled downtime from 6:00am to 7:00am EST.

UPDATE: This downtime did not happen and the work has been postponed to March 19, 2013.

Who is affected:

• Users of the public login machines (soak, wash, rinse, spin, tux, opus), certain CS websites (jobs, kiosk, msdnaa, search, wiki), database servers, runscript, CAS, lpdrelay (printing), labpc machines (Friend Center fishbowl lab) and the ionic cluster.

What is happening:

• These hosts will receive a critical security update and be rebooted.

Why is it happening:

• There is a critical kernel security patch available for our Springdale hosts. It addresses a specific security vulnerability. As this is a kernel update, all machines must be rebooted. Actual expected downtime should only be a minute or two for each host to reboot.

The temperature in the central computing facility has risen very quickly to critical levels this morning. OIT and facilities are aware of the problem. We have already started to see equipment failures. To protect our infrastructure, we are very likely to begin shutting down equipment.

UPDATE: At 10:43am, OIT reports, \”All air-handlers at the HPCRC are currently stopped.\”

UPDATE 10:55am: The penguin machines (tux, opus) and cycles (wash, rinse, spin, soak) are being shutdown. Our primary DNS server automatically shutdown due to the high temperature.

UPDATE: At 10:56am, OIT reports, \”The air handlers are running again and temperatures are returning to normal.\” CS Staff is monitoring temperature sensors in our area.

UPDATE 11:13am: Primary DNS server has been restarted. We expect to restart penguins and cycles within 15 minutes. The switch for the ionic cluster failed. We have a spare in the CS building. It will be a while (no ETA yet; could be a full day) before it is configured, installed, and operational.

UPDATE 2:00pm: We may not be out of the woods yet. We are seeing elevated temperatures again and we are keeping a close eye on our systems. From OIT, \”HPCRC control system is having problems again. Facilities staff are still at HPCRC and are responding.\”

UPDATE 2:55pm: We have seen temperatures peak and then decline again. OIT now reports, \”HPCRC cooling is now operating normally.\”

UPDATE 4:20pm: Facilities believes that they have identified and corrected the root cause. As a precaution, facilities staff will stay at the HPCRC all night to be able to respond quickly in the event of another problem.

On Saturday, February 2, 2013, from 5:00am to 9:00am, the campus network will be offline for scheduled work.

See: http://helpdesk.princeton.edu/outages/view.plx?ID=3998

This outage includes all OIT networks including the wireless network in the CS building.

The CS network will effectively be disconnected from the rest of the campus and the Internet during this time. However, if you are on the CS network (e.g., on a wired connection in the CS building), you will be able to reach other systems on the CS Network (even if they are at the remote data center).