Summary: universal explainers will use universal computers to solve all possible known/unknown problems.Universal computers are never going to be existential risk for universal explainers.
Terence Tao told me something that is both clarifying and unsettling about large language models.
The mathematics underlying today’s LLMs is not especially exotic.
At its core, training and inference mostly involve linear algebra, matrix multiplication, and some calculus.
This is material a competent undergraduate could learn. In that sense, there is very little mystery about how these systems are constructed or how they run.
And yet the real mystery begins there.
What we do not understand well is why these models perform so impressively on certain tasks while failing unexpectedly on others. Even more striking, we lack reliable principles that allow us to predict this behavior in advance.
Progress in the field remains largely empirical. Researchers scale models, change datasets, run experiments, and observe what emerges.
Part of the difficulty lies in the nature of the data itself.
Pure randomness is mathematically tractable.
Perfectly structured systems are also tractable.
But natural language, like most real-world phenomena, lives in an intermediate regime. And we humans hate that liminal space!
It is neither noise nor order but a mixture of both. The mathematics for this middle ground remains comparatively underdeveloped.
So we find ourselves in a peculiar position. We understand the machinery, yet we cannot reliably explain its capabilities. We can describe the mechanisms that produce these systems, but we cannot predict when new abilities will appear or how performance will vary across tasks.
That tension, between relatively simple mathematical tools and highly unpredictable behavior, is the central puzzle of modern AI.
(Video link in comments)
LLMs dominate high-temp exploration & high-K decompression, but they’re terrible at the low-temperature, low-Kolmogorov-complexity, hard-to-vary verification layer that actually makes explanations good.
When Copernicus proposed heliocentrism in 1543, it was actually less accurate than Ptolemy's geocentric model - a system refined over 1,400 years with epicycles precisely tuned to match observed planetary positions.
It took another 70 years before Kepler, working from Tycho Brahe's unprecedentedly precise observations, replaced Copernicus’s circles with ellipses - finally making heliocentrism empirically superior.
Terence Tao's point is that science needs a high temperature setting. If we only fund and follow what's most state of the art today, we kill the ideas that might need decades of work to surpass some overall plateau.