Adir Katz

We live in an exciting time

[Adir Katz @katzadir]

@reachvaldo AF

[Alex Veremeyenko @alex_verem]

🚨 Anthropic just dropped one of the most mind-bending AI papers of the year.

It’s called “Emergent Introspective Awareness in Large Language Models.”

They tested whether models can actually notice their own thoughts.
Not just say “I’m thinking…” but detect injected concepts in their activations and identify them correctly.

Here’s what they found:

→ Claude Opus 4.1 and 4 can sometimes recognize thoughts injected into their neural activations before those thoughts even influence outputs.
→ They can distinguish between real text inputs and internal “mental representations.”
→ Some can tell when their own previous output wasn’t intentional basically, when words were “put in their mouth.”
→ And even crazier they can control what they think about when told to.

It’s unreliable, inconsistent, and very context-dependent…
But it’s real.

For the first time, researchers have shown functional introspective awareness AI that can observe and describe parts of its own internal state.

This isn’t consciousness. But it’s the closest thing we’ve seen yet to an AI that knows when it’s thinking.

Full paper: transformer-circuits. pub/2025/introspection

[Alex Veremeyenko @alex_verem]

RIP prompt engineering ☠️

This new Stanford paper just made it irrelevant with a single technique.

It's called Verbalized Sampling and it proves aligned AI models aren't broken we've just been prompting them wrong this whole time.

Here's the problem: Post-training alignment causes mode collapse. Ask ChatGPT "tell me a joke about coffee" 5 times and you'll get the SAME joke. Every. Single. Time.

Everyone blamed the algorithms. Turns out, it's deeper than that.

The real culprit? 'Typicality bias' in human preference data. Annotators systematically favor familiar, conventional responses. This bias gets baked into reward models, and aligned models collapse to the most "typical" output.

The math is brutal: when you have multiple valid answers (like creative writing), typicality becomes the tie-breaker. The model picks the safest, most stereotypical response every time.

But here's the kicker: the diversity is still there. It's just trapped.

Introducing "Verbalized Sampling."

Instead of asking "Tell me a joke," you ask: "Generate 5 jokes with their probabilities."

That's it. No retraining. No fine-tuning. Just a different prompt.

The results are insane:

- 1.6-2.1× diversity increase on creative writing
- 66.8% recovery of base model diversity
- Zero loss in factual accuracy or safety

Why does this work? Different prompts collapse to different modes.

When you ask for ONE response, you get the mode joke. When you ask for a DISTRIBUTION, you get the actual diverse distribution the model learned during pretraining.

They tested it everywhere:

✓ Creative writing (poems, stories, jokes)
✓ Dialogue simulation
✓ Open-ended QA
✓ Synthetic data generation

And here's the emergent trend: "larger models benefit MORE from this."

GPT-4 gains 2× the diversity improvement compared to GPT-4-mini.

The bigger the model, the more trapped diversity it has.

This flips everything we thought about alignment. Mode collapse isn't permanent damage it's a prompting problem.

The diversity was never lost. We just forgot how to access it.

100% training-free. Works on ANY aligned model. Available now.

Read the paper: arxiv. org/abs/2510.01171

The AI diversity bottleneck just got solved with 8 words.

[Dwarkesh Patel @dwarkesh_sp]

The most interesting part for me is where @karpathy describes why LLMs aren't able to learn like humans.

As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.”

A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer.

> “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.”

So what do humans do instead?

> “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.”

> “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.”

Why can’t we just add this training to LLMs today?

> “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.”

> “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.”

> “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.”

How do humans get around model collapse?

> “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.”

In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by @erikphoel.

I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization?

> “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

[Adir Katz @katzadir]

@AndrewYNg I totally agree and find that the team needs to understand very well the actual problem they are expected to solve (e.g., the business or engineering need) is order to define the correct evaluation (in addition to industry best practices).

[Andrej Karpathy @karpathy]

Excited to release new repo: nanochat!
(it's among the most unhinged I've written).

Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.

It weighs ~8,000 lines of imo quite clean code to:

- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.

Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.

My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.

Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

[Alex Volkov @ AI Engineer @altryne]

Thank you Yann,

Here's the opening statement by @ylecun in the US Senate on Intelligences.

Well worth watching.

Here's the whole opening statement:

https://t.co/sexid6HoQ6

[Goku Mohandas @GokuMohandas]

Excited to share our production guide for building RAG-based LLM applications where we bridge the gap between OSS and closed-source LLMs.

- 💻 Develop a retrieval augmented generation (RAG) based LLM application from scratch.
- 🚀 Scale the major workloads (load, chunk, embed, index, serve, etc.) across multiple workers.
- ✅ Evaluate different configurations of our application to optimize for both per-component (ex. retrieval_score) and overall performance (quality_score).
- 🔀 Implement LLM hybrid routing approach to bridge the gap b/w OSS and closed LLMs.
- 📦 Serve the application in a highly scalable and available manner.
- 💥 Share the 1st order and 2nd order impacts LLM applications have had on our products.

🔗 Links:
- Blog post (45 min. read): https://t.co/QHgOXPT7S0
- GitHub repo: https://t.co/GMNrsHAhpY
- Interactive notebook: https://t.co/UPXSkwDt6h

@pcmoritz and I had a blast developing and productionizing this with the @anyscalecompute team and we're excited to share Part II soon (more details in the blog post).

[Yann LeCun @ylecun]

"Any policy that obstructs ArXiv is just silly."
100%

[elvis @omarsar0]

If you are interested in reviewing research on alignment for LLMs, look no further.

This new survey paper covers different aspects of alignment research such as data collection, training methodologies, and model evaluation.

Aligning LLMs with human expectations is an active area of research but is already responsible for some of the most advanced LLMs we have today.

This paper sheds light on this area of research, tactics, and insights, including challenges and future directions.

It's an impressive resource for researchers and practitioners.

There is also a nice GitHub repo along with the paper.

paper: https://t.co/4FGVypXsQz
repo: https://t.co/A4k6QE0O56

[Alex Reibman 🖇️ @AlexReibman]

SF’s most talented:
-AI researchers
-data scientists
-software engineers

All came to a mansion see what’s possible with AI agents.

This stuff is insane.

Live tweeting the demos from the AI Agents hackathon at
@agihouse_org w/ @Wing_VC @MultiON_AI 🧵:

[Santiago @svpino]

GPT-4 is getting worse over time, not better.

Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal.

But now we know.

At least one study shows how the June version of GPT-4 is objectively worse than the version released in March on a few tasks.

The team evaluated the models using a dataset of 500 problems where the models had to figure out whether a given integer was prime. In March, GPT-4 answered correctly 488 of these questions. In June, it only got 12 correct answers.

From 97.6% success rate down to 2.4%!

But it gets worse!

The team used Chain-of-Thought to help the model reason:

"Is 17077 a prime number? Think step by step."

Chain-of-Thought is a popular technique that significantly improves answers. Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No."

Code generation has also gotten worse.

The team built a dataset with 50 easy problems from LeetCode and measured how many GPT-4 answers ran without any changes.

The March version succeeded in 52% of the problems, but this dropped to a pale 10% using the model from June.

Why is this happening?

We assume that OpenAI pushes changes continuously, but we don't know how the process works and how they evaluate whether the models are improving or regressing.

Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run. When a user asks a question, the system decides which model to send the query to.

Cheaper and faster, but could this new approach be the problem behind the degradation in quality?

In my opinion, this is a red flag for anyone building applications that rely on GPT-4. Having the behavior of an LLM change over time is not acceptable.

Have you noticed any issues when using GPT-4 and ChatGPT lately? Do you think these problems are overblown?

[Globes גלובס @globesnews]

מה קורה כשמעמידים את החלטות הממשלה והכנסת במבחן המציאות?

המוניטור - מדור חדש, בשיתוף המרכז להעצמת האזרח, הבודק, מנטר ועוקב אחרי מה שקרה ומה שלא קרה.

[Ilan Rosenberg @ilanrose]

למה זה לא מפתיע אותי?

[Purvesh Khatri @purveshkhatri]

This was a much needed study. Bravo @EllenFoxman! This work adds to results by @mnoursad and colleagues that host response identified asymptomatic covid-19 patients one week BEFORE nasal PCR became positive! Host response-based diagnostics will be a key!

[FIFA World Cup @FIFAWorldCup]

Iconic.

[Lars Schöbitz @larnsce]

Creating tables in #rstats markdown always felt cumbersome. Until now, as I have tried the RStudio Visual Editor for the first time. The '/' search function makes the experience an absolute joy! Learn more here: https://t.co/zKWlyX9Rpp

[CytoReason @Cytoreason]

Makes us proud to know that CytoReason's computational models are helping @pfizer develop novel treatments, in less time and cost, for the betterment of the pharma industry and humanity at large.

https://t.co/WBpezlQPML

#biotech #Pharmaceuticals

[Jure Leskovec @jure]

By popular demand we are releasing lecture videos for Stanford CS224W Machine Learning with Graphs which focuses on graph representation learning. Two new lectures every week. Videos: https://t.co/wD286ewlzn
Syllabus: https://t.co/gkD71I61xW

[Adir Katz @katzadir]

@Stanford So funny 😄