If you’ve seen “Railway breach” floating around but have no idea what Railway is or what actually went wrong, this is for you.
Railway is a platform where developers host their apps. They upload their code, and Railway runs it for them so they don’t have to manage servers themselves.Every app needs secrets, passwords, API keys, database logins. Developers store these on Railway, so the platform holds secrets for thousands of apps in one place. Here’s how the attack happened:
1️⃣ Railway has a “build system” that packages developers’ code so it can run.
2️⃣ Attackers found a flaw in it and used it to gain more access than they should have had.
3️⃣ From there, they got into the system that stores customers’ secrets and pulled them out. The secrets are encrypted, and at first the attackers couldn’t read them.
4️⃣ Then they got into Railway’s internal build and deployment environment and found more credentials. Using a combination of the stolen credentials, they decrypted some of the secrets.
So encryption didn’t save those secrets. Encryption only protects you if the keys that unlock it stay safe.
Here, the attackers got hold of Railway’s own internal credentials, so they could unlock some of the secrets the way Railway’s own systems would.
How many people were affected? Railway says at most 1.85% of its workspaces (about 100,000) may have had some secrets exposed. A workspace is just a customer’s space on Railway.
But Railway couldn’t tell exactly which secrets were unlocked, so it also contacted other customers it considered high risk. The number of people who had to take action is higher than that 1.85%.
Who was the attacker after? Railway says they were likely looking for AI provider keys, which let apps use paid AI models on the owner’s account.
Railway paused new builds, got things running again, and told everyone who signed up before October 21 to rotate their keys (replace them with new ones so any stolen copy stops working). Running apps stayed online, and Railway says no user accounts were logged into. It hasn’t yet ruled out that source code was accessed.
I’m not a security expert, just explaining what I’ve read from Railway’s own report and online. Devs, where do you keep your secrets? Env variables, a vault, something else?
Same lesson as the PHP bug the fewer secrets a system can reach, and the easier they are to rotate, the smaller the damage.
I’m not a security expert, just explaining what I’ve read from Railway’s own report and online.Devs, where do you keep your secrets? Env variables, a vault, something else?
And did you have to rotate keys this week? 👇
Most of my posts barely reach anyone.
Even the ones where I’m trying to explain something clearly. Still writing them anyway.
Looking to connect with people who post useful tech/AI stuff.
If that sounds like you. Let’s connect.