Seth Gallmeyer

It is not my job to illuminate the shortsightedness which you maintain on the back of partisan outrage.

[Seth Gallmeyer @SethGallmeyer]

Perhaps with the quickly approaching singularity of AI, people will stop equating talent to righteousness.

[Seth Gallmeyer @SethGallmeyer]

@CTM_Market @NamecheapCEO @Namecheap You're right; there is no fail-over if DNS goes down, the fancy load balancer won't do anything. It's difficult to anticipate every failure scenario. An entire DC going down for a day is a train-wreck.

[Seth Gallmeyer @SethGallmeyer]

@NamecheapCEO At least the Swedish chef is gone!

[Seth Gallmeyer @SethGallmeyer]

@NamecheapCEO Now I can sign in, but the VPS panel is offline. (Sad trombone)

[Seth Gallmeyer @SethGallmeyer]

@CTM_Market @NamecheapCEO @Namecheap Did you pay for distributed hosting?

[Seth Gallmeyer @SethGallmeyer]

@sunglassesface @TheBuzzWagon @NamecheapCEO @BobbyJayOTA @Namecheap They have offsite backups. I think you are confusing catastrophic recovery and fail-over. We aren't in a catastrophic recovery scenario & if you want fail-over you have to pay or set it up.

[Seth Gallmeyer @SethGallmeyer]

@the_doc_doctor @Namecheap I agree! LOL

[Seth Gallmeyer @SethGallmeyer]

@Taco_Tom @NamecheapCEO @Namecheap @wearehostingcom @LiquidWeb If you need distributed hosting it's up to you to set that up, or pay for the service. An entire DC going down is not something which can just fail over.

[Hillan Klein @NamecheapCEO]

A further update on the ongoing cooling incident at our PhoenixNAP data center:

Temperatures continue to fall. The cooling process is taking effect and temperatures are consistently trending in the right direction.

Based on the progress we are seeing, our teams have now begun the process of preparing our infrastructure to safely bring customer services back online. We are doing this in a careful, staged approach:
1. Restore all physical network devices.
2. Restore all virtual network devices.
3. Begin restoring customer services.

We expect the first two steps to take approximately one hour. Based on current conditions, we anticipate being able to begin bringing customer services back online at approximately 3:00–3:30pm ET.

Services will then be restored progressively as we ensure each part of our infrastructure can be brought back online safely.

Our teams remain on site and are working continuously to move through this process as quickly and safely as possible. We know every additional minute of this outage matters to our customers, and we are deeply sorry for the disruption this extraordinary situation has caused.

Thank you for your continued patience. We are making progress, and our entire focus is now on safely restoring all affected services as quickly as we can.

[Seth Gallmeyer @SethGallmeyer]

@nicholosberry @Namecheap https://t.co/g5iH0f8mnX

[Hillan Klein @NamecheapCEO]

We’re currently experiencing a significant and highly unusual infrastructure incident at our PhoenixNAP data center, where the facility has suffered a failure of its cooling systems.

To protect our customers’ infrastructure and prevent equipment from overheating or suffering longer-term damage, we made the difficult but necessary decision to take affected services offline while we work with the data center operator to resolve the issue.

The latest information we have is that PhoenixNAP has brought 2 of its 4 chillers back online, and we are seeing temperatures in the data center begin to drop. We anticipate that the third chiller will be back online in approximately 3 hours, at which point we believe we will be able to begin bringing our services back online. PhoenixNAP is also installing additional cooling units as we speak to help accelerate the cooling of the data center.

This is an extraordinary situation, and one we have never experienced before. I’m deeply sorry for the disruption it is causing our customers.

The impact is broad, currently affecting more than 5,000 servers and impacting https://t.co/tIzv2QtGPu, Shared/VPS/Dedicated Hosting, EasyWP, DNS, Private Email and hosting email, email forwarding, URL redirects, and some hosting operations. Websites and services may be unavailable, slow, or return errors until the infrastructure can safely be brought back online.

For email specifically, delivery may be delayed, but messages are not expected to be lost. Sending servers should retry once connectivity is restored.

Our teams on site are working around the clock alongside PhoenixNAP to restore cooling, safely bring infrastructure back online, and return services to normal as quickly as possible.

We know how much our customers rely on us to keep their businesses, websites, email, and domains online. We take that responsibility incredibly seriously, and we are truly sorry that we are letting you down today.

Our immediate priority is restoring service safely and as quickly as we can. We will continue to provide updates as we make progress.

Thank you for your patience and understanding.

[Seth Gallmeyer @SethGallmeyer]

@adaptif_lab @Namecheap https://t.co/g5iH0f8mnX

[Hillan Klein @NamecheapCEO]

We’re currently experiencing a significant and highly unusual infrastructure incident at our PhoenixNAP data center, where the facility has suffered a failure of its cooling systems.

To protect our customers’ infrastructure and prevent equipment from overheating or suffering longer-term damage, we made the difficult but necessary decision to take affected services offline while we work with the data center operator to resolve the issue.

The latest information we have is that PhoenixNAP has brought 2 of its 4 chillers back online, and we are seeing temperatures in the data center begin to drop. We anticipate that the third chiller will be back online in approximately 3 hours, at which point we believe we will be able to begin bringing our services back online. PhoenixNAP is also installing additional cooling units as we speak to help accelerate the cooling of the data center.

This is an extraordinary situation, and one we have never experienced before. I’m deeply sorry for the disruption it is causing our customers.

The impact is broad, currently affecting more than 5,000 servers and impacting https://t.co/tIzv2QtGPu, Shared/VPS/Dedicated Hosting, EasyWP, DNS, Private Email and hosting email, email forwarding, URL redirects, and some hosting operations. Websites and services may be unavailable, slow, or return errors until the infrastructure can safely be brought back online.

For email specifically, delivery may be delayed, but messages are not expected to be lost. Sending servers should retry once connectivity is restored.

Our teams on site are working around the clock alongside PhoenixNAP to restore cooling, safely bring infrastructure back online, and return services to normal as quickly as possible.

We know how much our customers rely on us to keep their businesses, websites, email, and domains online. We take that responsibility incredibly seriously, and we are truly sorry that we are letting you down today.

Our immediate priority is restoring service safely and as quickly as we can. We will continue to provide updates as we make progress.

Thank you for your patience and understanding.

[Seth Gallmeyer @SethGallmeyer]

@Namecheap @wearehostingcom Confirmation on some of my assumptions:
https://t.co/g5iH0f8mnX

[Hillan Klein @NamecheapCEO]

We’re currently experiencing a significant and highly unusual infrastructure incident at our PhoenixNAP data center, where the facility has suffered a failure of its cooling systems.

To protect our customers’ infrastructure and prevent equipment from overheating or suffering longer-term damage, we made the difficult but necessary decision to take affected services offline while we work with the data center operator to resolve the issue.

The latest information we have is that PhoenixNAP has brought 2 of its 4 chillers back online, and we are seeing temperatures in the data center begin to drop. We anticipate that the third chiller will be back online in approximately 3 hours, at which point we believe we will be able to begin bringing our services back online. PhoenixNAP is also installing additional cooling units as we speak to help accelerate the cooling of the data center.

This is an extraordinary situation, and one we have never experienced before. I’m deeply sorry for the disruption it is causing our customers.

The impact is broad, currently affecting more than 5,000 servers and impacting https://t.co/tIzv2QtGPu, Shared/VPS/Dedicated Hosting, EasyWP, DNS, Private Email and hosting email, email forwarding, URL redirects, and some hosting operations. Websites and services may be unavailable, slow, or return errors until the infrastructure can safely be brought back online.

For email specifically, delivery may be delayed, but messages are not expected to be lost. Sending servers should retry once connectivity is restored.

Our teams on site are working around the clock alongside PhoenixNAP to restore cooling, safely bring infrastructure back online, and return services to normal as quickly as possible.

We know how much our customers rely on us to keep their businesses, websites, email, and domains online. We take that responsibility incredibly seriously, and we are truly sorry that we are letting you down today.

Our immediate priority is restoring service safely and as quickly as we can. We will continue to provide updates as we make progress.

Thank you for your patience and understanding.

[Seth Gallmeyer @SethGallmeyer]

Phoenix data center power loss is effecting @Namecheap @pulsetic and @wearehostingcom. It appears to me they are shutting servers down due to the cooling issue. This appears to be the data center status:
https://t.co/X7wgFQF7Sz

[Seth Gallmeyer @SethGallmeyer]

@Namecheap @wearehostingcom Status is back up!
"The datacenter technicians have activated mitigation efforts to reduce rising space temperatures."
Yeah... saw that!

[Seth Gallmeyer @SethGallmeyer]

@Namecheap @wearehostingcom The data center status page has been rate limited by Cloudflare (LOL).

[Seth Gallmeyer @SethGallmeyer]

@Namecheap https://t.co/d6DOgRTHoc

[Seth Gallmeyer @SethGallmeyer]

Phoenix data center power loss is effecting @Namecheap @pulsetic and @wearehostingcom. It appears to me they are shutting servers down due to the cooling issue. This appears to be the data center status:
https://t.co/X7wgFQF7Sz

[Seth Gallmeyer @SethGallmeyer]

@Namecheap Well, it's not just you.
https://t.co/BZSQwDWnol

[Seth Gallmeyer @SethGallmeyer]

@jjglss @SunWeatherMan Van Allen belt micro-nova powered lightning.

[Seth Gallmeyer @SethGallmeyer]

@realBlondeMan @Real_JFK_Jr_ ...and Star Trek transporters are suicide machines.

[Seth Gallmeyer @SethGallmeyer]

@TaraBull808 Wow, sewing seeds of confusion and descent even before they hit us. Of course the sleeper cells are active! There is Middle Eastern music playing in the background. LOL