Phillips Data, Inc.

[Phillips Data, Inc. @phillipsdata]

We will be restarting some network devices between 8-10pm pacific time today 9/28/24, that we believe will resolve some connectivity issues. We expect an outage duration of approximately 10 minutes.

[Phillips Data, Inc. @phillipsdata]

Commercial power has been restored as of 11:43am.

[Phillips Data, Inc. @phillipsdata]

The data center is currently on generator power due to commercial power loss, which seems to be affecting the general area. Edison is aware of the issue and has a crew on the way. Everything is otherwise operating normally.

[Phillips Data, Inc. @phillipsdata]

Engineers identified & took down a compromised client server this morning, the source of an outbound attack affecting some network devices.

[Phillips Data, Inc. @phillipsdata]

Power was restored @ 7:05 am DST this morning. Everything operating normally. Failure with UPS #4. Investigating cause, likely power surge.

[Phillips Data, Inc. @phillipsdata]

We are investigating a power issue affecting part of the data center, which may be related to a thunderstorm.

[Phillips Data, Inc. @phillipsdata]

Edison power has been restored and we are now on commercial power.

[Phillips Data, Inc. @phillipsdata]

We are experiencing a power outage with Edison and are currently on generator power.

[Phillips Data, Inc. @phillipsdata]

The issue in Tustin has been partially resolved about an hour ago. Some carriers remain down, but all inbound/outbound traffic is flowing.

[Phillips Data, Inc. @phillipsdata]

Network issue at Tustin DC. Techs investigating.

[Phillips Data, Inc. @phillipsdata]

Been back up for a while now. The SSB was swapped out in LA, and routing has been fixed. Even if LA goes down again we should stay up.

[Phillips Data, Inc. @phillipsdata]

Router failed again, Emergency Maintenance -- swapping out the SSB.

[Phillips Data, Inc. @phillipsdata]

Sorry routing instability not DNS

[Phillips Data, Inc. @phillipsdata]

Coming back online but some DNS instability remaining, working on it.

[Phillips Data, Inc. @phillipsdata]

BGP reset unexpectedly again, working on it..

[Phillips Data, Inc. @phillipsdata]

Traffic has returned to normal. A core router went down in LA, and traffic did not properly reroute. Techs investigating and will correct.

[Phillips Data, Inc. @phillipsdata]

@donaldsellers Sorry for the inconvenience, it looks like traffic is returning now. It was a routing issue out of LA, we are investigating.

[Phillips Data, Inc. @phillipsdata]

Routing issue @ Tustin facility, techs on it should be resolved soon.

[Phillips Data, Inc. @phillipsdata]

Service affecting maintenance Friday Nov 25, beginning 11pm pacific time in our Tustin data center. Expected duration: 15min

[Phillips Data, Inc. @phillipsdata]

Emergency maintenance in SLC data center, expected downtime 2 hours. @3am pacific time.