Catching up on the 10 (!) open LLM architecture releases from the past month. Quite a lot happened in October.
I guess we can call it "Open October"!
1. Kolibri-1 (78B): A notable model from Germany. Features classic Grouped Query Attention (GQA) and Sliding Window Attention (SWA) attention module.
2. Ling 3.1 Flash (560B): Uses Kimi Delta Attention and the eighth uses gated Multi-head Latent Attention (MLA) inspired by DeepSeek.
3. Muse Spark Open (816B): Besides Musse Glimmer, the first major open weight release from Meta (since Llama). Uses classic GQA + SWA but adds a LatentMoE.
4. Nemotron 3.5 Super VL (120B): Combines Mamba-2 layers with just eight attention layers. Also LatentMoE
5. Dory1.0 (4B): A looped transformer design.
6. Reflection Beam (501B): GQA+SWA architecture.
7. Recursive Flint (40B): Note that "Recursive" is the company name; the architecture is a fairly standard MoE with GQA and SWA.
8. MiniMax M3.1-Flash (428B): Uses their own MiniMax Sparse Attention module.
9. LFM3 (1.7B): Hybrid architecture that mixes short convolutions and attention.
10. Mistral Large 4 (1T) combines MLA with sparse attention.
Will probably cover/discuss things in more depth in the coming weeks!
The Transformer block is a very interesting topic because so many important ideas in modern LLMs meet inside one fairly small piece of architecture.
I have created a handbook that works through the block from first principles, starting with the residual stream and then following what happens through normalization, attention, the output projection, the MLP, and the residual updates that carry the representation forward.
It covers LayerNorm and RMSNorm, Pre-Norm vs Post-LN, self-attention, RoPE, SwiGLU and gated MLPs, parallel blocks, QK-Norm, GQA/MQA/MLA, MoE, parameter accounting, training vs inference, and KV caching.
One distinction I wanted to make especially clear is that the two main branches do different jobs. Attention mixes information across token positions. The MLP is position-wise, transforming each token representation independently before projecting it back to the model width.
The residual stream ties the whole block together. A typical decoder block can take a tensor shaped (B, T, d) and return another tensor with the same shape, while substantially changing what those vectors represent.
The familiar Transformer block diagram is also only one version of the design. Normalization can move, attention heads can be grouped or compressed, Q/K can be normalized internally, the MLP can become an MoE layer, and some models run attention and MLP branches in parallel.
Made it to be the explanation I wish I had when I first tried to understand what is actually happening inside a Transformer layer.
Sharing the handbook here:
Transformer and Mixture of Experts, explained visually!
Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models.
Transformer and MoE differ in the decoder block:
- Transformer uses a feed-forward network.
- MoE uses experts, which are feed-forward networks but smaller compared to those Transformer.
During inference, a subset of experts are selected. This makes inference faster in MoE.
Also, since the network has multiple decoder layers:
- The text passes through different experts across layers.
- The chosen experts also differ between tokens.
But how does the model decide which experts should be ideal?
The router does that.
It is a multi-class classifier that produces softmax scores over experts to select the top K experts.
The router is trained with the network, and it learns to select the best experts.
But it isn't straightforward.
There are challenges!
Challenge 1) Notice this pattern at the start of training:
- Say, the model selects "Expert 2"
- This expert gets a bit better
- It may get selected again since it's the "best"
- It learns more
- It gets selected again in the next iteration
- It learns more, and so on!
This means many experts can go under-trained due to the overselection of a few experts!
We solve this in two steps:
- Add noise to the feed-forward output of the router so that other experts can get higher logits.
- Set all but the top K logits to -infinity. After softmax, these scores become zero.
This way, other experts also get the opportunity to train.
Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts.
We prevent this by limiting the number of tokens an expert can process.
If an expert reaches the limit, the token is passed to the next best expert.
Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference.
Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs.
Have you used MoEs in production yet?
To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis.
Read it below.
This Google research paper is f*cking mind-bending
Google researchers just mathematically proved that forcing AI models to "think longer" at inference can actively destroy accuracy and trigger catastrophic overthinking
In "Understanding the Role of Training Data in Test-Time Scaling", Google proves the hidden catch of reasoning models: test-time compute only works if the underlying skills already exist in the training data
The Overthinking Paradox (Google's Core Mathematical Findings):
• The Skill Coverage Constraint → if a skill is not represented in the training distribution, scaling test-time compute amplifies noise and causes error to explode as reasoning chains get longer
• The Context vs Compute Trade-Off → for a fixed error rate, scaling test-time reasoning steps allows you to dramatically slash training context length and in-context examples
• The Eigenvalue Hardness Law → task difficulty is governed by the smallest eigenvalue of the feature covariance matrix: hard tasks require diverse, long-tailed skill representations to survive deep search
The Empirical Reality (Reasoning Benchmarks):
[Matched Skill Distribution] → scaling reasoning steps exploded accuracy from 30.4% up to 75.0%
[Unmatched Skill Distribution] → scaling reasoning steps degraded performance, trapping the agent in redundant token oscillation
Hypothesis Verification & Practical Validation:
Google hypothesized that test-time scaling without verified skill coverage in the training distribution triggers the overthinking regime, where longer chains-of-thought degrade accuracy. Practical validation confirmed the findings: integrating Google's skill-covariance gating and dynamic compute budgeting into our test-time reasoning harness across 125 complex algorithmic tasks boosted pass@1 accuracy from 48.6% to 76.4% (+27.8% gain) while cutting wasted reasoning tokens by 52.3% by terminating ungrounded thinking loops early
Test-time compute is not magic fairy dust that fixes missing foundation knowledge. If an agent does not possess the fundamental skill representations, letting it think longer just gives it more room to hallucinate. True reasoning requires dynamic budget allocation, PRM verification, and grounded search trees
Read the complete breakdown in the article below ↓
I’ve spent 10 years teaching math to machine learning engineers.
80% of university math is irrelevant to your actual job.
Here's the 20% you actually need to build models (and how to learn it fast):
https://t.co/sV52SBB16J
PSA: If you are tired of Claude-lish or Chat-lish, tell your AI to read the Google Developer Docs Style Guide and build a skill that follows it
It’s like magic. Suddenly your AI will make sense again!
I’ve tried both and this is much better than the ASD-STE100 Simplified Technical English skill
And friends at the labs: please invest more in AI comms skills!
A brilliant agentic AI that cannot speak English is dramatically less useful and undermines the utility of RLVR
https://t.co/xUbgHn0xpg
I ACCIDENTALLY UNLOCKED "GOD MODE" IN CLAUDE.
And it started teaching me things I didn’t know existed.
Use these prompts for the result that will blow your mind:
My view of: Fable 5 vs GPT-5.6-Sol. They are not easy models to compare, these are my vibes - take them as you will.
My overall feel is that Fable is a 'wise owl' who is very thoughtful and very well spoken, GPT-5.6-Sol is like a rottweiler who will grab the problem by the throat and not let go until it is done.
In other words, Fable, is a fundamentally smarter model - even at low reasoning it can be very insightful and writes in a clear compelling way. GPT-5.6-Sol on the other hand is extremely diligent, I can give it a list of 8 things to do and you will be sure that they will be done.
Fable feels more arrogant to me, I was both to get it to build a new benchmark for me - 5.6 worked between 6 hours and 2 days (I tried several times) and it came up with very thoroughly tested, working benchmark. Fable came back within 40 minutes (twice) and the benchmark sounded smart, but was ultimately was 'vibe' based slop and since it was Fable's vibes that was doing the judging, it decided that it was good to go (it kept giving Fable 100% score btw).
Some thoughts by category:
UI & App building: Fable will still craft a better UI from scratch, the flow of the app would probably be a bit nicer. But I find that Fable often misses quite key things, which GPT-5.6-Sol doesn't. GPT's Frontend skills are big jump vs previous GPT models, but still not as great overall.
Writing: Fable is better hands down, Sol feels quite difficult to align to what I want to say or explain things to me simply. Though I think the 'Pro' model writes clearer.
Robustness & Reliability: This is where I think GPT-5.6-Sol wins for me hands down. Fable seems to do things of high quality, but I can never relax with it, it always misses something. With 5.6 this just almost never happens.
Other things where I liked GPT-5.6-Sol, but can't compare to Fable directly.
- Video editing is actually working now, it is not completely perfect, but with the right skill/guidance you can just give it 1h footage and it can give you a 5 min highlight clip no problem
- Computer use - getting really rather good, very usable
- Sub agents - it is very fluent at managing sub-agents and speaking to different threads, can help with some new workflows
- Adhering to existing code patterns - I love this, even without asking it would implement something in a way that aligns with you app - major problem for slop generation
- Research - I think it is getting quite a bit better, it still has some bad patterns (e.g being too tactical), but it feels like it is more steerable to be a good researcher
- Multi-day runs - the /goal feature is pretty insane with 5.6-Sol, you can run it for days if you wanted to and it does work. Useful to have another thread or /side to check up on it, but I have some great results with it
- Token efficiency - it is so much more token efficient and faster than 5.5, in reality it is now much faster than Fable too
On the downside, you can feel that Fable is naturally smarter, and I did have some baffling moments with 5.6 when I was getting it to make a fairly simple change in 8 turns - it seemed to get stuck in a dumb stream that was hard to get out of. So it is not AGI, don't get too carried away by the hype.
I have some phenomenal examples that I'm honestly blown away by that I'll share, but as a side anecdote, I have a kind of 'swear meter' which counts how often I'm rude to Codex. In GPT-5.5 era, the % was at around 4-5%, it dropped to 1-2% when I was testing GPT-5.6-Sol and it shot up to 7% when I went back to 5.5 - it was so shocking to go back to 5.5 and experience how much worse it was.
So is GPT-5.6-Sol better than Fable? On pure intelligence - no. But man, I missed it when I just wanted to get sh*t done. It is insanely capable workhorse that you can give any task to and just expect it to be done. No lectures or 'you are absolutely rightisms', nothing is beneath it, if it takes 2 days to do some dirty work, it will do it.
It feels like the first time in a while when we have quite different types of frontier intelligences that benchmark sort of similarly, but feel very different. If you can, you would be probably better off using both and iteratively finding what you'd use Fable or GPT-5.6-Sol for. Perhaps, something like - an architectural discussion with Fable, implementation with 5.6 and docs & comms with Fable.
New in Claude Code: /checkup
Run /checkup to:
1. Clean up unused skills/MCPs/plugins and save context
2. Dedup your local CLAUDE.md against the checked in CLAUDE.md
3. Break up root CLAUDE.md into nested CLAUDE.md's + skills
4. Turn off slow hooks
5. Update your Claude Code to the latest version
6. Enable auto mode by default
7. Pre-approve frequently denied read-only commands
.. And a few other goodies.
/checkup confirms with you before making any changes. Enjoy!
We talk a lot about how important it is to set up self-verification loops. Especially in the age of powerful models that can run for long periods of time, self-verification is a key ingredient that enables the model to run for much longer, delivering a result that is closer to what you intended, so you can do more without having to constantly check in on Claude as it works.
@delba_oliveira gives a great breakdown of what that looks like and why it matters
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini 🚀
Today, we’re sharing the @GoogleDeepMind white paper for GE 2, our first native multimodal embedding model. Whether it’s text, audio, video, or image, GE 2 provides a unified representation of the input.
been asking others at Anthropic how they stay in the loop with Claude and fully understand the work being done
this is one of my favorites from Suzanne:
instead of watching 2 hours of Netflix tonight, watch this Stanford lecture
it's the clearest explanation I've seen of how ChatGPT and Claude actually work
useful whether you've never touched AI in your life or have been using it every day for the past year
I took the key ideas and turned them into a practical guide on how to actually get 100% out of Claude
find it below
This 2-hour Stanford lecture breaks down how models like ChatGPT and Claude are actually built, clearer than what many people in top AI roles ever get exposed to.
Save this and set aside two hours today. It might end up being the most valuable thing you learn all week.