β¦ but was long enough some UPS systems drained. Then when power came back it came back in a surge and fried two UPS systems and many server power supplies. Repair and recovery took most of the day. Our generator contractor is coming out today to diagnose and hopefully repair
Update on our outage last week. The power went out in our industrial park and for the first time in 12 years our backup generator failed to start. (It is regularly exercised and just started less than a week prior) The outage was short, approx 15m, β¦ π§΅
We are experiencing a variety of systems issues this morning. Working on bringing things back online. Current ETA for the majority of issues is Noon PST, 3 PM EST
Shifting a few racks around today, trying to retire some older racks that aren't deep enough for new equipment. Uptime discovered on one of the client machines we're moving:
14:24:23 up 3218 days, 3:17, 1 user, load average: 0.11, 0.15, 0.12
A shame to have to power it down
@RegattaCentral Apologies, I missed this the other day. According to our peer in Seattle as of yesterday the routers have been patched against a BGP issue and should be fixed going forward.
Root cause for the issues in Seattle appears to be a failure in the software portion of the BGP routing engine in a pair of routers. Reboots were fixing it temporarily and yesterday engineers applied patches recommended by the router vendor that should take care of it long term.
@LiveFire717 Yes, unfortunately. Weβve been told a set of software patches have been applied to the affected routers now. According to the engineers that should take care of the issue.
Issues in Seattle have continued to be an intermittent issue. The peer in Seattle is aware and working to resolve it. We do not have root cause from them at this time.
@GigLibraries A large portion of our traffic routes through Comcast in Seattle, they are down for many regionally and are investigating. We do not have a root cause and unfortunately their NOC has been difficult to reach. (approx 1h wait times)