we've found and have been working on disclosure for a remote code execution in a widely-deployed compression library.
it's both an impactful issue and an example that shows how complicated it can be to coordinate patching, since it's integrated into a lot of downstream software:
The small 1.85% detail potentially reveals a lot about railway and is interesting to nitpick:
"Railway’s system is built so that if one set of keys is compromised, they should not compromise the entire system. As a result, we can validate that at most 1.85% of all workspaces had some environment variables potentially compromised"
- assuming 1.85% was one worker group's key that's like 100k workspaces on one worker key, ~54 worker groups total. why chunk the workers there? if you can split them into 54 groups why not 540 or 5400, what was stopping railway from going lower
- Ok they're using a multi tenant system but why were the worker, build, & ci credentials/keys that could decrypt customer secrets long lived and not scoped per customer workspace? You can have a service that decrypts those secrets for the worker but not for any other (potentially malicious) requester instead of storing that workers key in exploitable memory.
- I get that they were planning to move to single tenant but it's not unknown knowledge on how to make multi tenant builds secure
The interesting question becomes: why were these higher order consequences ignored or not thought about? Railway is a killer engineering team that has recently supercharged itself with ai & agents.
If these weren't conscious engineering decisions, then higher order decision making by Ai agents is still not there and cannot currently be relied on, especially at enterprise scale where things Will go wrong
@mmhornbacher You could probably setup internal networking with a vault + your apps but you would still have to have secret0 inside a railway env to auth your apps with that vault. I don't think you can have an ideal system when using things like railway compared to proper cloud iam
This is a great lesson to all the normies that you do not store important things in environment variables even though you were taught to.
You can store maybe feature flags, staging or enviornment domains
But not important things, there is a reason why Hashicorp Vault is as big as it is
They did drain crypto keys that were in Railway.
Seems like an okay sweeper script (read: not something homegrown for this one op)
Attacker:
0xd9773325282874d18cb83f1fe50f14c39750b2ee
~$371,000
Oct 21 19:36 UTC
500+ txns
200+ distict addresses
19+ EVM chains