Time for a big systems advice thread!
In distributed systems there's no magic "push everything to prod at once" button. Every service gets pushed independently and nodes within a service get updated incrementally. If you mess up forwards/backwards compatibility you can fail irrecoverably.
So how to avoid this?
1/5: Decouple data and code changes. Never push out a release that changes how data is stored at the same time as the code that uses this new data. If there's a bug and you need to roll back to the old version of your code it won't be able to handle the new data in the new format. Instead push out a release that first changes the data in a way that’s compatible with both the old and new code (e.g., optional fields etc), when that’s stable push out the new code that uses it, then when that’s stable you can change the data to remove backwards compatibility. This is known as a “migration” in the database world and yes it’s annoying, but yes you need to do it.
These challenges are very real. Like, very real!
But the answer isn't to wait for the tools to get better, because the problem isn't the tool, it is how we work.
This is not everything, but one aspect is that you gotta be absolutely ruthless about throwing away stuff.
Sunk cost fallacy was always bad. But now it got 10x worse with brains that trigger "wow, that demo is so far along, we might as well ship it" where they should trigger "This was 30 minutes of work. That investment should play absolutely zero role in the decision whether to proceed"
Whatever your take on this, writing code has always been tremendously easy compared to designing/maintaining large systems. The best engineers in the world are the best because of this stuff.
Currently LLMs will not save you from bad architectural decisions. You need to constrain them to do the right thing.
Non-technical types think engineers just “translate” specs into code but that is not how great products are built. PM/design/engineering must work together throughout the process. Throwing stuff over the wall doesn’t work. The best ideas don’t come from the top down, they come after building and experimenting and understanding what’s technically possible. Simply replacing engineers with AIs that do only what you tell them is not a recipe for success.