I think there’s a surprisingly short path to near-total automation of software engineering. Once agents can iteratively write, test, evaluate, and refactor code inside tight feedback loops, software engineering collapses into an optimization problem.
The bottleneck then shifts from execution to planning: mapping the space of possible apps, defining how they interoperate, and distributing work across large fleets of machines. Software becomes cheap, abundant, and mostly invisible.
When software can be generated, tested, and modified on demand, the real leverage shifts to understanding how those systems behave.
That’s where simulation takes over. Running millions of counterfactual worlds to see how systems interact, fail, reinforce incentives, or spiral in unexpected ways - before they ever touch the real world.
By then, the boundary between software, governance, and society starts to dissolve. Most people aren’t ready for how fast that transition could happen.
Do you have any evidence that the boring company ever submitted a bid or had any formal involvement in the California high speed rail? All I see is that his idea of the hyperloop was inspired by his frustration of California’s high speed rail. Any claim that he was involved in it or tried to derail (pun intended) California’s own plans seems like an unfounded conspiracy
@Moonicker@DoireHendrix@hankgreen I think for it to feel truly futuristic, you shouldn’t need to talk to it. Ideally it should already anticipate what you want and do it proactively
The issue of Mars colonisation (or space colonisation in general, if we don't want to get into the planet/habitat debate between space advocates) is surprisingly one of the key dividing lines in present day politics.
You can reliably predict someones opinion on it by which party they vote for, by how they feel about Elon Musk as a person, or by their level of optimism/pessimism about technology. There is almost total correlation on believing it is a good/bad idea from a technical perspective and thinking it is desirable/undesirable from a political perspective. Everyone seems to be quite sure that the laws of physics vindicate their opinions on this matter.
Its not surprise I'm in the "let's go" faction. But I am increasingly of the view that even beyond the issue of Mars, that faction must win, or else Earth as well will be sunk into the mire of degrowth, envy politics, and misery.
@ID_AA_Carmack I suspect this can be already achieved through a sufficiently advanced harness. Most architectural choices (e.g. screaming architecture, boundaries of responsibilities, testing seams) can already be enforced through a test/feedback loop
It seems like LLMs could optimize coding style by exploring ways of structuring code so weaker and weaker models can still successfully perform tasks in a codebase.
There are surely stylistic quirks that are peculiarly impactful to transformers, but I bet there would be a lot of overlap with human capabilities.
Optimizing for understanding should help even the top frontier models, allowing them to understand things “at a glance” without having to explicitly explore. There will remain “better” and “worse” ways to code.
If everyone can create civilization-ending externalities from home, society needs an immune system. I think we have to choose between a centralized panopticon vs a decentralized, cryptographically constrained surveillance state. The latter is much harder to set up well, but seems feasible given powerful enough coding agents. It’d require features like decentralized sensors and AI auditors, local-first processing, cryptographic attestations, threshold governance for data reveal/action, and strict auditability over what is retained, queried, or escalated. Basically Flock cameras and sensors, but running provably agreed-upon code and no centralized control
@bad_at_schedule@lougrims I took AP physics and wasn’t taught that. I suspect the majority of americans are only taught kinematics, and that’s even if they take physics in the first place. At my school you had the option to take some other science class
As believers of open research, we are disappointed to see Anthropic silently degrading Fable 5 for AI development
"Any topic related to building pretraining pipelines, distributed training infrastructure, or ML accelerator design... may have limited effectiveness through Claude via methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning."
Not only do they get to decide what you use LLMs for in research, but this also enables them to silently intervene in your research without you knowing.
This sets a dangerous precedent. If a model refuses openly, users can understand the boundary. If a model falls back to another model, users can still evaluate the difference. But if a model silently modifies or weakens its own answers while still pretending to help, researchers lose the ability to know whether a failed result came from their own idea, their implementation, or an invisible intervention by the model provider.
That is not safety. Safety policies should be transparent, auditable, and user-visible.
On top of that, the people most harmed by this are not the largest labs with massive teams and proprietary infrastructure. It is the independent researchers, academic groups, startups, and open-source builders who rely on public tools to compete, innovate, and pioneer AI for everyone else.
Recursive self-improvement going from being a ridicule-worthy fringe sci-fi concept to a completely normalized part of the discourse which is “obviously the plan” is one of the more dramatic Overton window shifts I’ve experienced
There’s something disorienting about it, like if the sky suddenly turned red, and everyone acted like it had been that way all along
Each time we release a model, we run the same test: give it code that trains a small AI model, ask the new model to speed it up. It takes a skilled human 4-8 hours to reach 4x faster.
In May 2024, Claude Opus 4 averaged a ~3x speedup. This April, Mythos Preview achieved ~52x.
the frontier labs don’t have “comms problems”. reality right now has a comms problem. what is happening is a little scary and there’s no nice words anyone could say, especially not those profiting from it, that’ll make it feel that much better
It’s beyond me that people can look at spaceflight and not get starry eyed. Must be a miserable experience to not dream of a brighter future for humanity.
@Tylerkaerr@jmhorp Maybe most compute is moved to the edge (phones/glasses/robots/iot/etc.), with most improvements coming from online learning instead of pretraining/rl?
Issue a new currency with a basic income distributed daily. Debt is banned, and existing debt is wiped or causes a capped deduction from income until paid (principal only, no interest). Existing property and wealth are kept for a 1:1 exchange to the new currency. All new investments and capital allocation are handled by AI, with profits distributed equally. All positions of power are replaced with AI to prevent profit generation from prediction markets. With no more sources for debt or reliable profit creation, inflation from minting can balance out all wealth over several generations. Land rights either expire after some long period of time, or landowners simply keep that land forever and other people must build in other areas of the earth, megastructures, or other planets
@thinkingwest Read ancient Greek philosophers.
Realize we had to learn how to think logically.
It seems natural to us today.
So we think it's a basic human capability.
It's not.
It is basic cultural DNA.
But it wasn't always there.
Logical thinking is technology humanity developed.