To be a little less vague, I suspect that we're likely (not certain, but likely) to be entering into a period of unprecedented software degradation, and we're going to be seeing an increasing frequency of outages like this across many high profile products.
But IMO the cause is actually not just the-one-thing-that-everyone-is-always-talking-about, it's a number of things that have all been bubbling away at just below critical levels for a long time. Some of the things off the top of my head:
- Poorly designed / optimised software has been getting a free ride on hardware improvements pretty much since the invention of the computer. That chapter is now coming to an end, and will only be worsened by the enormous industry-wide pivot to producing & innovating on AI specific hardware, rather than general purpose CPUs etc.
- The ZIRP era created a temporary suspension of reality in our industry, and now that it's ended we need to deal with the hangover. Companies that spent years making no profit, paying extravagant compensation to employees / shareholders and giving away server time for free are now pivoting into extraction mode, which is putting further pressure on their low quality software. QA is being laid off, hardware budgets are being reduced, timelines for shipping features are becoming more aggressive, etc.
- The enormous amount of free money incentivised too many new people to join the industry too quickly. This has led to an abundance of poor quality education programs (bootcamps, uncertified colleges etc) and an influx of people into the industry who frankly aren't interested in programming. If you compared the average person in the industry now to 20 years ago, I suspect the difference in motivations would be stark. I'm not saying it's these people's fault necessarily, it's simply an inevitable result of the absurd compensation / performance expectations ratio that our industry has enjoyed for the last 15+ years. Working for a tech company has also become socially prestigious, which further adds to the problem.
- Because computer programming was once an incredibly niche area of interest, many of our fundamental systems are built on trust. We're now starting to see that if systems like open source, public supply chain, discussion spaces, education etc become flooded with bad actors, we have no real mechanisms to deal with them.
- Our hiring / recruitment pipeline has totally misaligned incentives. Even before the AI resume / AI HR-filtering arms race disaster that we're experiencing now, the widespread adoption of the leetcode style interviews IMO selected for a very narrow personality type, and filtered candidates that would have made great contributions to the industry long term.
- The pivot from purchasing long term stable releases of software, to paying a subscription for constantly updating software has done huge damage to software quality as a whole. Companies have lost their incentive to get their software "right" because they can just "fix it later", and for the consumer - you can't just go back to the version of github that still works because the new one has problems.
This was all happening well before AI entered the picture. I won't belabor the point because there has been endless discussion about it. But to me personally, there are two additional and deeply worrying problems with AI code generation.
- It's undeniable at this point that it negatively affects the people who use it. It stops juniors from getting better, and it burns seniors out and makes them hate their jobs. Like it or not, humans are still the core of this industry, and I don't see this ending well.
- It's completely unfit for purpose in the most important, high-stakes situations. One of the reasons that we excuse all the small errors it makes, is because it's low effort to type "do it again and fix this bug". That kind of thing doesn't fly when you only get one attempt because a mistake results in data loss or an outage. The damage is done.
All the above has led to a silent exodus of many of our most experienced and impactful people. There are so many amazing programmers who made enough through stock options / compensation that they didn't need to work anymore, and were only doing it because they enjoyed it. Many of these people have just quit the industry and switched to doing hobby projects in the last 5 years. These are the types of people who have the experience and foresight to prevent the types of outages that we're seeing at github today.
It's very easy to assume that the proverbial straw that broke the camel's back is entirely to blame here. But I think it's a reckoning that has been on the horizon for a very long time.
Had an interview with a “crypto” recruiter. We talked for about 40 minutes, and then they asked me to look at some code.
Their first instruction was to clone the repo. I didn’t. They seemed surprised, so I told them I wanted a moment to check whether it was safe first.
I ran a quick analysis with Claude.
Turns out the code had a backdoor. It would copy my environment variables and send them to a remote server.
The recruiter went speechless and ended the call pretty quickly.
Be careful who you talk to. Scammers are real.
New blog post: A Decade of Slug
This talks about the evolution of the Slug font rendering algorithm, and it includes an exciting announcement: The patent has been dedicated to the public domain.
https://t.co/xWEz0q2c4N
Bummer! Last live stream was a disaster, but you only learn from failures.
I learned that encoding 4 4K streams destroys the GPU 😅
So I tweaked everything and now it doesn't give me warnings.
Let's go with camera control pt.2
https://t.co/1TmGd0giRT
I've finally been able to get my windowed rendering, time to implement some camera movement and have fun with matrices in Vulkan!
https://t.co/WTXjtlQF6X
Going live with a new improvement: windowed rendering. A quality of life option to make it easier iterate through builds and testing.
https://t.co/yZ73E0LnC9
Check out these examples of how it looks in a mixed reality and immersive app! If you have any questions, we're here to help. Feel free to ask here or join us on Discord for more in-depth discussions. We're excited to see what you'll create!
-Atom and the LIV team