Patmos - Phoenix: Some Servers Unreachable – Incident details

Root Cause Analysis

On the evening of 11/19 at roughly 5:00 PM MST, we began receiving alerts as a building power changeover unexpectedly caused one of our UPS units to fall into bypass. A battery issue inside the UPS led to a sudden loss of power for the servers connected to it, resulting in hard shutdowns. Our team immediately moved into emergency response, including flying in additional technicians to help expedite service restoration.

By 7:00 PM MST 11/21, UPS functionality had been restored, but the power event affected each server differently, which meant recovery required individualized attention rather than a single fix for all systems. Progress continued steadily through the night; by the morning of 11/22, most servers were back online, with only a small number still offline. By the morning of 11/24, the vast majority of servers had been restored, with only a few systems still undergoing more extensive hardware repairs.

We are reviewing generator-transfer procedures and continuing routine maintenance and checks on the UPS and broader datacenter infrastructure to ensure safeguards are confirmed, strengthened, and any potential future issues are proactively detected. We are also updating our internal response playbooks based on lessons learned from this event.

The UPS is back online, and the load has been successfully transferred.

Our team continues working to restore all servers affected by the recent power event. Each server requires individual troubleshooting, so restoration is progressing steadily but not simultaneously.

Thank you for your patience as we work to bring all services fully online.

During scheduled power maintenance by the building owner yesterday at 4:25 PM MST, a UPS failure occurred, forcing the system into bypass mode. We are currently operating on generator power with the UPS in maintenance bypass.
Replacement parts have been expedited and will arrive Friday (November 21). Technicians are scheduled to be onsite at 6:00 PM MST Friday to complete repairs. We expect the UPS to be fully restored by late Friday evening or early Saturday morning.

Some servers experienced disruptions during the initial event. Our technicians are onsite working through affected systems individually and updating support tickets as work is completed.
All scheduled maintenance activities have been paused until the UPS is back online and fully operational.

Next update: Friday at 6:00 PM MST when technicians begin repairs.

This incident required manual reboots for many servers. Most are now back online and accessible, though a small portion are still being actively troubleshooted. We are continuing to investigate the root cause and will share a full root cause analysis once finalized.