Breakdown of AWS outage in simple words
1. Sunday night, a DNS problem hit AWS - DynamoDB endpoint lost
2. This meant services couldn't find DynamoDB (a database that stores tons of data).
3. AWS fixed the DNS issue in about 3 hours.
4. But then EC2 (the system that creates virtual servers) broke because it needs DynamoDB to work.
5. Then the system that checks if network load balancers are healthy also failed.
6. This crashed Lambda, CloudWatch, SQS, and 75+ other services - everything that needed network connectivity.
7. This created a chain reaction - servers couldn't talk to each other, new servers couldn't start, everything got stuck
8. AWS had to intentionally slow down EC2 launches and Lambda functions to prevent total collapse.
9. Recovery took 15+ hours as they fixed each broken service while clearing massive backlogs of stuck requests.
This outage impacted: Snapchat, Roblox, Fortnite, McDonald's app, Ring doorbells, banks, and 1,000+ more websites.
This all happened in one AWS region (us-east-1).
This is why multi-region architecture isn't optional anymore.
Just did a zoom interview and at the end when they asked me if I had a question i asked them "What would be a reason you wouldn't hire me" and they were too stunned to speak. The moderator lady ended up saying none so if i get rejected I'll use this recording in a court of law😭
If you are at the stage where you can build a full monolithic system, containing frontend and backend, and you wanna upskill, please learn microservices with Docker, K8s, and a cloud platform (AWS,Azure, GCP).
You won’t believe the opportunities that come your way from there on
Sugar in your blood is called diabetes.
Sugar in your brain is called dementia.
Sugar in your teeth is called cavities.
Sugar in your liver is called fatty liver.
Sugar in your cells is oxidative stress.
Sugar on your skin is called aging.
Avoid sugar for a healthy life.
Are you a front-end developer?
Become a full-stack developer!
It means many more opportunities for you.
You can build a complete SaaS and become a startup founder.
An intern deletes the entire production DB
Wesend is a fast growing startup that provides email delivery service
Tim Cook, CEO of WeSend hires there first remote intern dev from India
Jatin, a Sabka College student gets hired for this remote job
.
.
.
.
Jatin is a regular OSS contributor, but mostly does readme commits.
He was hired by Tim to write some API usage documentation
Jatin wanted to do more..
He built a good network at WeSend and finally got some backend code to write
Tim is on a rocketship he thinks enterprise grade infra management is not a priority rn.
WeSend runs on a multi cluster Kubernetes setup
Jatin writes a migration script that will add a new column in the existing USER table
So, he copied the migration framework code from ChatGPT and fills in the ALTER query
And it worked like a charm. In LOCAL
Jatin pushes the code to Dev Env
and boom 💥
The migration starts affecting the PROD DB
All the users got affected and
X just lits up on fire 🔥
Tim calmly asks Jatin, "What just happened?"
Jatin, "I just ran the migration in Dev Env but don't know why the Prod tables got affected"
Fierce Tim
Runs to Steve
Steve Jobs the overtly smart programmer from UC Berkeley says,
"
I know about this,
I was provisioning Consul, a service mesh to control the communications of the services in the cluster
I was testing if the access control works.
So, I updated the environment variable to PROD
Expected scenario:
Prod Services should only be accessible from the Prod NameSpace in that specific cluster
Since Consul also allows multi cluster communication the requests were passing through directly.
I guess there's some issue in the envoy proxies or I need to redeploy the config.yaml "
Tim SCREAMS, " Repeat this in English" 😱
Scared Steve
"It was my mistake it didn't tell everyone to stop deploying. I am fixing this."
----------------------
Points to consider:
1. Kubernetes is a great tool to mess up things. It requires experience and broader understanding of org to operate at scale.
2. Human Communication is the backbone of Software Engineering and a big problem of Microservices Architecture
3. Service Mesh, Control Plane, Envoy Proxy, Multi Cluster K8S etc are simple tools in the right context. Go Read about them now.
4. Dig deeper into infrastructure and SRE this is just the tip of the tip of the iceberg.
5. Infra and Platform is a huge field with all these tools it has become important than ever to simplify this.
---
If you like these stories do consider to follow my account.
⚡️ I am Sagar, I simplify building software and career in tech with stories.