Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and self-refinement.
00:00 Introduction and inference-time scaling recap
05:02 Loading the pretrained LLM
08:00 Comparing and scoring model answers
10:18 Building a rule-based scorer
17:53 Token probabilities and sequence likelihood
26:47 Computing token probabilities in PyTorch
30:12 Token indexing and shifted targets
37:27 Log probabilities and numerical stability
45:57 Scoring answers with average log probabilities
56:24 How self-refinement works
59:07 Generating critiques and revised answers
1:01:00 Implementing the self-refinement loop
1:05:57 MATH-500 evaluation results
1:07:35 Takeaways and next steps
I accidentally discovered how to compress a semester of learning into 48 hours.
A grad student at MIT showed me his NotebookLM setup. I thought he was just organized. Then I watched him pass a qualifying exam on a subject he'd never studied before.
Here's exactly what he did:
First: he didn't upload a textbook.
He uploaded 6 textbooks, 15 research papers, and every lecture transcript he could find on the subject.
Then he asked NotebookLM one question:
"What are the 5 core mental models that every expert in this field shares?"
Not "summarize this." Not "explain this topic."
Mental models. The stuff that takes professors years to develop.
But the next part is what broke my brain.
He followed up with:
"Now show me the 3 places where experts in this field fundamentally disagree, and what each side's strongest argument is."
In 20 minutes he had a map of the entire intellectual landscape of the field:
the debates, the consensus, the open questions.
Most students spend a full semester just figuring out what those debates even are.
Then he did something I've never seen before.
He asked:
"Generate 10 questions that would expose whether someone deeply understands this subject versus someone who just memorized facts."
He spent the next 6 hours answering those questions using the source material. Every wrong answer triggered a follow-up:
"Explain why this is wrong and what I'm missing."
By hour 48, he could hold a conversation with his thesis advisor without getting destroyed.
The tool didn't change. The questions did.
Most people treat NotebookLM like a fancy highlighter.
These students are using it like a private tutor who has read everything ever written on the subject.
The difference between a semester and 48 hours isn't the amount of content.
It's knowing which questions to ask.
Math for vault and rebase token (like SUSDS and aToken) calculate the same redeemable amount with different mechanisms.
But in code, precision loss causes these 2 ways of calculations to diverge.
Notes
https://t.co/kDVMqOz21D
Code
https://t.co/fY2LvFwjrl
With AI Engineering skills, you actively shape the build: You influence what gets built, and drive the build loop. Here're key skills to do this. https://t.co/sysOYdzuZY
A lot of important progress on Frames (EIP-8141) has been quietly happening over the last few months. Highly recommend reading this, also the updated EIP https://t.co/jYqeS55j6P
https://t.co/CPYONKnWZc
Announcing the new https://t.co/rbgeRe2tOn
Sign in with Ethereum (ERC-4361) is the standard that powers the bridge between web2 and web3 for thousands of crypto (and non crypto) apps.
Our goal is to make sovereign cryptography a default option for authentication on the web.
We should own our finances, and we should own our web accounts. SIWE makes that possible, while ensuring users have great UX when doing so.
Already used by millions, including the likes of OpenRouter, Polymarket, MetaMask, Privy, OpenSea, and many many more.
The libraries we develop and support enable apps and wallets to seamlessly integrate the SIWE standard. If you have any questions or need help integrating, please reach out to any of us:
@caveman_onchain ([email protected]), @jalilwahdat ([email protected]), @yougogirl_eth ([email protected]).
Looking forward to working together on improving Ethereum UX.