Terence Tao: The math behind today’s LLMs is actually simple.
Training and running them mostly uses linear algebra, matrix multiplication, and a bit of calculus, material an undergraduate can handle. We understand how to build and operate these models.
The real mystery is why they work so well on some tasks and fail on others, and why we cannot predict that in advance. We lack good rules for forecasting performance across tasks, so progress is largely empirical.
A key reason is the nature of real-world data. Pure noise is well understood, perfectly structured data is well understood, but natural text sits in between, partly structured and partly random. Mathematics for that middle regime is thin, similar to how physics struggles at meso-scales between atoms and continua.
Because of this gap, we can describe the mechanisms but cannot yet explain capability jumps or give reliable task-level predictions. That mismatch, simple machinery versus hard-to-predict behavior, is the core puzzle.
----
Video from Prof @Briankeating YT Channel (Link in comment)
Jack London in his study. He consistently wrote 1,000 words a day.
On becoming a better writer, he wrote:
“Spell it in capital letters. WORK. WORK all the time. Find out about this earth; this universe…And by all this I mean WORK for a philosophy of life….The three great things are: GOOD HEALTH; WORK; and a PHILOSOPHY OF LIFE. I may add, nay, must add, a fourth—SINCERITY. Without this, the other three are without avail; and with it you may cleave to greatness and sit among the giants.”
“I lived so carefully, thinking someone was watching. But the stage was empty, the audience never came.”
- Osamu Dazai
Osamu Dazai’s haunting reflection explores the tragedy of spending life performing for an audience that may not even exist. By portraying existence as an empty theater, he reveals how easily people can become consumed by expectations, judgment, and the belief that others are constantly watching.
Discovering that the seats were empty all along is both unsettling and freeing. It suggests that much of the pressure to be accepted was self-created, revealing the cost of living for others instead of oneself—and how easily authenticity can be lost while trying to please an imaginary crowd.
Meet the raspy cricket, also known as the leaf-rolling cricket, a fascinating insect from the Gryllacrididae family.
What makes it truly unique?
It can produce silk, not from its body like a spider, but from its mouth.
Fields Medalist Terence Tao:
"AI lets us create scientific output much faster, but it could come at the cost of nurturing the next generation of scientists"
If AIs replace grad students, they may produce grad-student-level papers — but we risk losing the next generation of scientists
Bertrand Russell survived a plane crash into the sea near Trondheim in 1948, swimming clear at the age of 76, and afterwards liked to say that smoking had saved his life, since he had been seated in the smoking section.
“Wittgenstein, and also the group of philosophers in Vienna called the Vienna Circle, condemned what they called ‘metaphysics.’ And metaphysics, in their view, consisted at least partly in any attempt to describe the world in non-scientific terms.
They thought that this was the only world there is, and that science was a matter of forming theories about it which were verified by observation. Any assumption of a superior world, a world inhabited by God or anything of that sort, was nonsensical.
[Russell] was always concerned to justify science [...]. In [his] last book, he sets out the set of assumptions that require to be made if belief in scientific theories is to be justified and makes it quite clear that these assumptions, as it were, had to be taken on faith.”
—A.J. Ayer on Bertrand Russell’s philosophy