@thorstenball i think the overhang is due to two things: models being trained on past engineering patterns, and human attitudes making progress a funeral at a time.
most of software is essential complexity, and is only in now with agents slowly being rewritten without all the baggage
With cheap re-implementation from coding agents, instead of vertically hitching a ride on the genealogy of some adopted framework, software ideas are becoming more horizontally transmitted, and potentially thereby becoming more virulent.
AI safety really comes down to a reductio of rationality. The question becomes if we also want to take out humanity in the process of establishing this reductio.
Why do we insist on factoring through what is true, instead of simply suffering the consequences of making wrong decisions? Perhaps it is because we want to collect rent for being right?
Why are our conscious lives so full of suffering? Because the role of consciousness is PR, and PR gets involved mostly when shit goes wrong and our reputations need defending.
When they say AI will make academics obsolete, this is a good thing!
Claude understands my position on naive binary forecasting (and betting markets) better than 100% of academics in Decision "Science"/Psychology, all verbalistic.
"Why does Taleb despise @PTetlock"?
The unreasonable effectiveness of mathematics in the natural sciences, comes from games which play well together being jointly more fruitful than their separate sum, and from science being also a game.
Theory builders define games, the objective being fruitfulness. Problem solvers play them, the objective is to win. Both involve meta-game coordination dynamics.
Math is seen to be this body of unavoidable, necessary facts only because what math is used in practice has long been negotiated and polished. Even with problem solving at the frontier, the frame has already been decided by past choices in just what math we should care about.
Saying mathematics is about "understanding" still trivialises how un-pregiven and actively-negotiated such understandings are. LLMs can decide to understand something a certain way only within some culture. Whether that culture also involves human participation is the question.
@almostlikethat win what though? you can’t win unless the game is already defined, and there used to be healthier disagreement as to which game was even worth playing in the first place
I think much difference in attitudes to LLMs can be explained by how propositional one’s understanding (and meta-understanding) is. Those who operate non-propositionally recognise that most of life isn’t factored through propositions.
Why have the rationalist nerds and capitalist billionaires been so prescient about the future? They are simply the most ardent participants of the game, and they have played in such a way as to advance the continued playing of the game.
At the same time, rationality is a self-reinforcing game which is eating up more and more of the world. It is a mega Schelling point more and more of us are coordinating about. Could be the universe was already long being tiled, by propositions.
Funny how we seem to think the more fundamental something is the more necessary, when the compressibility interpretation of randomness suggests quite the opposite.
Even human-level political-social intelligence among agents would scare the non-AI pilled general public far more any amount of nerdish superintelligence ever would.