Patrick

AI × LLMs × math
figuring out how machines think

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

Here's the clearest answer to why ChatGPT doesn't ramble nonsense. Out of roughly 100,000 possible next words, the model only lets a handful through, and OpenAI's own docs tell developers to touch this dial or temperature, never both.

If a model sampled from every word, the 10% of probability sitting in the junk pile would show up in more than 99% of 50-word answers. Top-p sampling sorts the words, keeps adding them until the total hits p, and throws the rest away.

5 minutes. bookmark & watch today. one rule keeps a chatbot from going off the rails

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

Here's the clearest answer anyone has given to why ChatGPT makes things up. OpenAI showed that one of its older models got 75 out of 100 answers wrong and still scored higher on accuracy than a newer one that got only 26 wrong

On most tests a wrong answer and "I don't know" both score zero, so even a model that's only 30% sure should always guess. Add a penalty for wrong answers and the math flips: suddenly saying "I don't know" is the smart move

11 minutes. bookmark & watch today. the scoreboard taught AI to bluff, just like it taught us in school

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

Watch this once and you'll understand Golden Gate Claude better than most people who build AI. Anthropic made its own chatbot believe it was the Golden Gate Bridge, and for a day anyone could talk to it. Ask it how to spend $10 and it suggested paying the bridge toll.

Inside the model, every idea is a direction in a huge space of numbers, and there are far more ideas than numbers, so they're packed in almost on top of each other. Researchers found the one direction that means "Golden Gate Bridge" and turned it up to 10 times its normal maximum, with no retraining and no tricky prompt.

11 minutes. bookmark & watch today. one turned-up direction, and a chatbot became a bridge

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

The single best explanation on the internet of why AI is starting to eat itself. Researchers let an AI learn from its own writing nine times in a row, and a story about medieval church towers turned into a list of jackrabbits.

It works like a photocopy of a photocopy - every copy loses a little detail, and the rare, unusual things disappear first.
That’s why real human writing is getting expensive: Google reportedly pays Reddit about $60 million a year just for people’s posts.

11 minutes. Bookmark & watch today. The simple math that decides how fast AI forgets the world.

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

Every layer of GPT-2 and GPT-3 passes its numbers through one function: x times capital Phi of x. Three symbols. It replaced the simplest switch in deep learning - if a number is negative, make it zero - with a coin flip weighted by the bell curve.

The old switch, ReLU, has a flaw. Below zero its slope is exactly zero, so a neuron that slides negative can stop learning for good. GELU keeps each input with a probability that rises smoothly with its size: Phi of 2 is about 0.98, Phi of minus a half about 0.31. Multiply by x and big positives pass almost untouched, big negatives fade toward nothing, and the region near zero stays smooth, so the gradient doesn’t die. Phi has no simple closed form, so GPT-2’s code ships a tanh shortcut with the constant 0.044715.

10 minutes. Bookmark & watch today. The bell-curve switch firing inside every transformer layer.

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

Google built an AI that writes a whole paragraph at once instead of word by word, and by Google's own numbers it turns out close to 1,500 tokens a second. That's a short essay in about a second.

It's called Gemini Diffusion, shown at Google I/O in May 2025, and it borrows the trick behind image generators: start with a row of blanks and fill them all in parallel. The catch hides in the math. Ask it to fill every blank in one step, and a model that doesn't know whether you're flying to New York or Los Angeles will half the time send you somewhere that doesn't exist, like New Angeles, because each blank picks its word without seeing its neighbour.

The fix works like a crossword. Lock in the words you're sure of first, then let them settle the hard ones. Twelve words in four steps means a third of the trips through the network that ChatGPT-style models need, and the number of steps becomes a dial that trades speed for quality.

10 minutes. Bookmark & watch today. Next time a company brags about tokens per second, you'll know the one question to ask.

[Patrick @corsaldi]

the most expensive autocomplete in history: $500 billion so chatgpt, claude and gemini can guess your next word

here’s every formula they run on ↓ https://t.co/4MjEuJzJBg

[Patrick @corsaldi]

OpenAI researchers forgot to stop a training run, and that mistake turned into one of the strangest results in AI

In 2022 they were training a tiny model, just two layers and about 400,000 weights, on simple math. Within about a thousand steps it had memorized every answer it was shown, and it still failed every new problem. By the usual rules the experiment was over. Then some runs were left going for days instead of hours. Around step 100,000 the model suddenly started solving problems it had never seen. The team, which included Igor Babuschkin, who later helped Elon Musk start xAI, borrowed a word from a sci-fi novel to name it: grokking.

Nobody told the model to look for a rule. It kept memorizing until weight decay, a small tax on every weight, made the lookup table too expensive to keep. When researchers later opened up a similar model, they found it doing arithmetic with sines and cosines, turning numbers into points on a circle. One GPT-4 training run reportedly cost more than $100 million. At that price, knowing whether your model is about to "get it" or whether you should pull the plug is worth a fortune.

11 minutes. Bookmark & watch today. The circle trick hiding inside the weights is the best part

[Eron @0xEronn]

https://t.co/fQgIxVyHZJ

[Patrick @corsaldi]

@0xorenx does the 11 km a day show up directly though? the receiver solves for its own clock bias, so i thought the bigger problem is satellites drifting out of sync with ground time

[Patrick @corsaldi]

in 1951 Claude Shannon published a guessing game from Bell Labs that predicted ChatGPT 70 years early

the score behind it is H = −Σ p·log₂ p, entropy measured in bits per letter

the guesser, reportedly his wife Betty, named the next letter of a book and got 69% on the first try

swap Betty for a neural network that read the internet and you get every chatbot alive

the loss your model trains on equals that entropy plus a penalty for believing wrong probabilities

in 2023 DeepMind showed Chinchilla 70B squeezes Wikipedia to about 8% while gzip only gets 32%

predicting well and compressing well turn out to be the exact same skill

11 minutes, bookmark it if you want to read the perplexity number on any model card ↓

[Eron @0xEronn]

https://t.co/fQgIxVyHZJ

[Patrick @corsaldi]

@0xzivo the taper is a sinh curve and its steepness scales with √(λσ²/η), so at the same risk aversion a more volatile stock gets sold off faster

[Patrick @corsaldi]

OpenAI Made a Model 135x Smaller Than GPT-3 Beat It With Human Feedback Using Less Than 2% of the Compute

in 2022 openai showed that people preferred answers from instructgpt, a 1.3 billion parameter model, over GPT-3 with 175 billion, a model about 135 times bigger

no new GPU cluster and no new nvidia chip, openai just changed how the model learned from people, instead of scoring every answer from 1 to 10, labelers picked which answer was better

those choices trained a reward model that learns to predict what humans prefer, and that reward model then steers the language model, openai reported this RL stage used less than 2% of the compute it took to train GPT-3

the math goes back to 1952, when ralph bradley and milton terry built a model for the odds that one item beats another, every answer gets a hidden strength and its chance of winning depends on how it stacks up against the other one

that became a core piece of RLHF, the step that turned text predictors into assistants that follow instructions, and the same pairwise idea now powers Chatbot Arena, whose company lmarena hit a reported $1.7 billion valuation in 2025

you can try a mini version yourself, give one prompt to a model, generate three answers, compare them head to head and write down which one people keep picking

bookmark this and run your own preference test, because the technique that helped reshape ChatGPT was built on a surprisingly simple question, which answer is actually better? ↓

[Eron @0xEronn]

https://t.co/fQgIxVyHZJ

[Patrick @corsaldi]

Google DeepMind Beat Its 280B Model With One 4x Smaller AI and Changed the Economics of Training

google deepmind discovered that training a smaller AI on more data could beat a model four times its size, challenging the strategy behind billions of dollars in AI infrastructure spending at Google, OpenAI, Meta, and Anthropic

in 2022, deepmind compared gopher, a 280 billion parameter model, with chinchilla, a 70 billion parameter model trained using the same compute budget but significantly more data

chinchilla won, proving that throwing more parameters at a model can waste enormous amounts of compute when you do not give it enough training data

deepmind trained more than 400 models to derive a better scaling strategy, and the result changed how researchers think about the relationship between model size, training tokens, and compute

the video breaks down the loss formula symbol by symbol, showing how an error floor and two scaling terms determine how much intelligence you can extract from a training budget

then comes the 6ND rule of thumb, a simple way to estimate training compute and check whether the numbers behind an AI model card make sense

but Meta's Llama 3 shows why the cheapest training strategy is not always the best business strategy, because a model must also be affordable to serve at scale

every extra parameter can increase inference costs, and when millions of users send prompts every day, that decision can turn into a massive operating expense

the real competition between Google DeepMind, OpenAI, Meta, and Anthropic is not simply about who can build the biggest model, it is about who can squeeze the most capability out of every dollar spent on GPUs, electricity, and training data

a model four times smaller can be more capable under the right training conditions, and that changes the economics of the entire AI race

watch this before you trust another billion-dollar AI announcement based on parameter count alone ↓

[Eron @0xEronn]

https://t.co/fQgIxVyHZJ

[Patrick @corsaldi]

An IBM computer spent 1,200 hours on one math proof and no human has checked it all

Can every map be colored with four colors so that no two neighbors ever match

Alfred Kempe’s 1879 paper claimed a proof and it stood 11 years until someone found the hole

Mathematicians fought over it for a century and then fought over whether a computer proof even counts

The video shows why five colors are easy with one Euler rule and four are brutally hard

Now a startup cofounded by Robinhood’s Vlad Tenev is worth $1.45 billion betting AI can write proofs

Its agent Gauss did in three weeks what Terence Tao’s team spent 18 months on

Save it and next time AI hands you math ask for a proof a machine can check

[Eron @0xEronn]

https://t.co/fQgIxVyHZJ