Systems, geometry, and the structure of learning.
I don't optimize mistakes; I replace them with better principles.
Everything else is implementation detail.
Astra의 experimental feature인 노트에 기반한 무한 메모리 관리 기능은 사실 CoMeT이라는 이름으로 완전히 똑같은 알고리즘을 내가 이미 한참 전에 냈었다. (Lossless Memory보다도 며칠 더 이전에 코드까지 공개했었다.) 이제 다들 따라하고 있네 ㅎㅎ 꼬우면 빅테크 오라는 건가.
제대로 된 리뷰어 만나기 힘든 연구 시장에 환멸이 나서 더 이상 논문을 안 내고 있는데, 그래도 아카이브에라도 올려놨어야 하나 싶은걸.
Getting ready to publish my complete guide to RL for LLMs tomorrow morning. Although the post contains many of my own thoughts / learnings, it is also a synthesis of so many great resources that have been published over the years:
- The RLHF Book (https://t.co/n3aUxqWtgS) by @natolambert
- Reinforcement Learning by Richard S. Sutton and Andrew G. Barto
- Spinning Up in Deep RL (https://t.co/EYcOOllvIy) from OpenAI
- Build an LLM (https://t.co/6HfXuofVwq) and Reasoning Model (https://t.co/l5uS247A04) from Scratch by @rasbt
- Various notes (https://t.co/hpsyAKOnRv) and papers (TRPO, PPO, etc.) from John Schulman
- Policy Gradient Algorithms (https://t.co/caNLakSjo5) by Lilian Weng
- A Vision Researcher’s Guide to RL (https://t.co/25awrjj5Bd) by @YugeTen
- From REINFORCE to Dr. GRPO (https://t.co/M23e3Fl3VZ) by @qingfeng_lan
- Async GRPO in the Wild (https://t.co/A5qeYMwcy4) by @yumo_xu
- Open RL infrastructure like TRL (https://t.co/TGrrnJ574t) and OpenInstruct (https://t.co/L9pObiUb1c)
I highly recommend reading all of them. They’ve truly helped me to learn so much.
A better analogy would be to say that your brain suddenly gets placed in the pencil for it to moved around when you need to do the exam.
These are entirely different things you are comparing. The harness is much more similar to the body than it is to the classroom and environment.
We need to normalize measuring and judging models against a standardized test harness
"Oh but model X performs best in their own proprietary harness"
I could not care less. When I take exams, I go to the standardized classroom, get the standardized pencil and exam sheet, and have to solve it under 2 hours
This system arose because we have a LOT of people to test
Guess what? We now have a LOT of models, and they are multiplying by the day
"Oh but model X performs substantially better in ARC-AGI-3 with a custom harness"
I don't care... Then imbue model X with enough knowledge so that it can reconstruct that harness on the spot
The main harness could be mini-swe-agent, terminus 2, vanilla pi or something along those lines
It needs to be simple, and stay roughly the same over time
There is already too much complexity in the benchmarking space right now, and I feel like not enough people are putting their feet down to cut away some part of it
Codex wrote cleaner code and did a better job of checking the broader picture than Claude. It also got stuck in A-B-A back-and-forth loops noticeably less often.
One piece of feedback: despite Codex having an excellent memory system, it still wasn’t very good at detecting when it had fallen into longer cyclical improvement loops like A-B-C-D-A.
I hope this gets better in the next version.
That may make AGI difficult to reach.
But reaching perfect AI does not necessarily require a completely new paradigm. With sufficient data, scale, and ideal training, today's probabilistic learning paradigm could be enough in theory.
The limitation is not that AI is probabilistic.
It is that humanity has not yet reached the threshold of what probabilistic models can do.
You've probably heard this before: "AI only makes up plausible-sounding things." But is that really true?
Before answering that, let's first talk about probabilistic models.
A commonly forgotten point is that mathematical probability and statistical probability are different.
Given enough data, the model's estimate of plausibility can converge toward actual truth.
The real limitations are practical, not inherent to probability itself.
We never have enough data, unlimited compute, an infinitely large model, or a perfect training method.
In deployment, we also make compromises for cost, speed, safety, and usability. That is why real AI systems are imperfect.
Such a model would have learned every piece of knowledge the world could contain. With enough data, it could determine almost perfectly what is true and what is false.
The belief that even an ideal AI must hallucinate comes from misunderstanding what "plausible" means.
"Plausible" simply means "highly likely to be correct."
Suppose every piece of knowledge that could exist—past, present, and future—has been turned into data.
Now suppose we have a model large enough to contain all of it, and that the model has been trained perfectly.
This is impossible in practice, but coherent in theory.
Generative models, including language models, are not fundamentally different. They are trained so that what they generate has as high a probability of being correct as possible. This is where the claim comes from that AI merely produces plausible-looking statements.
The claim is simple: AI only produces what appears statistically likely. To challenge it, let's imagine an ideal AI model.
Most AI systems are enormous probability machines. They learn inductively from vast amounts of data, so they can be viewed as statistical probability machines.
So what probability does AI predict? In broad terms, it estimates, on a scale from 0 to 1, how likely a piece of data is to be correct. Closer to 1 means more likely to be correct; closer to 0 means less likely.
Mathematical probability is deductive: the probability of an event can be defined exactly.
Statistical probability is inductive: it estimates the probability of an event from repeated observations.
The two are connected by the law of large numbers. As the number of trials approaches infinity, statistical probability converges to mathematical probability.
Now let's start from there.