Do Transformers always win?
In this post, we walk through an experiment in which we build both Transformers and LSTMs from scratch to find out. Implementation is based on former repo of @pbloemesquire.
Experiment Code:
https://t.co/In2g8XgPrK
Post:
https://t.co/DMhUvKok8a
I was inspired by this so I wanted to see if Claude Code can get into my Lutron home automation system.
- it found my Lutron controllers on the local wifi network
- checked for open ports, connected, got some metadata and identified the devices and their firmware
- searched the internet, found the pdf for my system
- instructed me on what button to press to pair and get the certificates
- it connected to the system and found all the home devices (lights, shades, HVAC temperature control, motion sensors etc.)
- it turned on and off my kitchen lights to check that things are working (lol!)
I am now vibe coding the home automation master command center, the potential is 🔥.And I'm throwing away the crappy, janky, slow Lutron iOS app I've been using so far. Insanely fun :D :D
Just forked the transformer from scratch repo from @pbloemesquire for my next project, I want to do some LLM benchmarking on top of it to compare it with a network of LSTM blocks with same amount of parameters. Let's see how it goes.
Check the blog post of them, if you haven't already, to understand how transformers work under the hood
https://t.co/EBdsL6OtHY
I really didn't expect another major open-weight LLM release this December, but here we go: NVIDIA released their new Nemotron 3 series this week.
It comes in 3 sizes:
1. Nano (30B-A3B),
2. Super (100B),
3. and Ultra (500B).
Architecture-wise, the models are a Mixture-of-Experts (MoE) Mamba-Transformer hybrid architecture. As of this morning (Dec 19), only the Nano model has been released as an open-weight model, so this post will focus on that one (shown in my drawing below).
Nemotron 3 Nano (30B-A3B) is a 52-layer hybrid Mamba-Transformer model that interleaves Mamba-2 sequence-modeling blocks with sparse Mixture-of-Experts (MoE) feed-forward layers, and uses self-attention only in a small subset of layers.
There’s a lot going on in the figure above, but in short, the architecture is organized into 13 macro blocks with repeated Mamba-2 → MoE sub-blocks, plus a few Grouped-Query Attention layers. In total, if we multiply the macro- and sub-blocks, there are 52 layers in this architecture.
Regarding the MoE modules, each MoE layer contains 128 experts but activates only 1 shared and 6 routed experts per token.
The Mamba-2 layers would take a whole article itself to explain (perhaps a topic for another time). But for now, conceptually, you can think of them as similar to the Gated DeltaNet approach that Qwen3-Next and Kimi-Linear use, which I covered in my Beyond Standard LLMs article.
The similarity between Gated DeltaNet and Mamba-2 layers is that both replace standard attention with a gated-state-space update. The idea behind this state-space-style module is that it maintains a running hidden state and mixes new inputs via learned gates. In contrast to attention, it scales linearly instead of quadratically with the input sequence length.
What’s actually quite exciting about this architecture is its really good performance compared to pure transformer architectures of similar size (like Qwen3-30B-A3B-Thinking-2507 and GPT-OSS-20B-A4B), while achieving much higher tokens-per-second throughput.
Overall, this is an interesting direction, even more extreme than Qwen3-Next and Kimi-Linear in its use of only a few attention layers. However, one of the strengths of the transformer architecture is its performance at a (really) large scale. I am curious to see how the larger Nemotron 3 Super and especially Ultra will compare to the likes of DeepSeek V3.2.
Introducing “The Kite’s Journey” — Dream Lab LA’s first short brought to life with Ray3 Modify.
When his best friend moves across the world, a boy sends a message via kite through impossible landscapes — proving distance can't break true connection.
Ray3 Modify is now live on Dream Machine by @LumaLabsAI
Lead Content Artist: Jacob Russo
I was checking the WMT 2025 results (largest conference on Machine Translation) the fact that Hunyuan-MT 7b translation model from tencent outperforms models with 100x more parameters is very impressive.
Model on hugging face:
https://t.co/6KEDwrSYl9
🔉 Introducing SAM Audio, the first unified model that isolates any sound from complex audio mixtures using text, visual, or span prompts.
We’re sharing SAM Audio with the community, along with a perception encoder model, benchmarks and research papers, to empower others to explore new forms of expression and build applications that were previously out of reach.
🔗 Learn more: https://t.co/FPnfv66UCP
This interesting week started with DeepSeek V3.2!
I just wrote up a technical tour of the predecessors and components that led up to this:
🔗 https://t.co/JSAd9cx2s6
- Multi-Head Latent Attention
- RLVR
- Sparse Attention
- Self-Verification
- GRPO Updates
Flux.1-dev has been the second most liked model on Hugging Face just after Deepseek R1 so super excited to see the release of Flux.2-dev by @bfl_ml today!
Download the weights or try the model (thanks to @fal) on @huggingface: https://t.co/kdmVlvdLZh
Read the blogpost: https://t.co/SFk5Pfm1kc
Let’s go!
I have met many students and young researchers lately who claim to be working on World Models or Embodied AI but do not even know the basics of 3D Vision or linked rigid body motions. When did we start to give students the illusion that they can *do* things right without *learning* anything right?
We’re launching a new frontier physics eval on Artificial Analysis where no model achieves greater than 9%: CritPt (Complex Research using Integrated Thinking - Physics Test)
Developed by 60+ researchers from 30+ institutions across the world including the Argonne National Laboratory and University of Illinois Urbana-Champaign, some of whom have previously worked on leading benchmarks such as SciCode and SWE-Bench, this evaluation tests language models’ reasoning abilities on novel, frontier physics problems suitable for a post-graduate researcher.
We’ve worked with the CritPt developers to launch their new benchmark, and are especially excited about several key elements differentiating this from other reasoning tests:
➤ True frontier evaluation: This benchmark tests models on physics research suitable for graduate-level researchers, with questions and answers written and tested by experts (e.g., postdocs and physics professors) in their subfields
➤ Hard for even frontier models: On release, the highest-scoring model was Google’s new Gemini 3 Pro Preview, with an accuracy of 9.1% (without tool use allowed). Many models fail to solve a single problem even given 5 attempts
➤ Diverse question set: The evaluation test set includes 70 total end-to-end research problem ‘challenges’ covering 11 physics subdomains: condensed matter, quantum physics, AMO, astrophysics, high energy, mathematical physics, statistical physics, nuclear physics, nonlinear dynamics, fluid dynamics, and biophysics
➤ Reflective of research assistant capabilities: Each challenge is designed to be feasible for a capable junior PhD student as a standalone project, but unseen in publicly-available materials. This means most problems require deep understanding and reasoning in frontier physics beyond the capabilities of today’s language models, but all are feasible to solve and independently verified
Congratulations to @MiniHui_zhu, @MinyangTian1, @haopeng_uiuc, and the broader CritPt team on this exciting new evaluation!
See below for further discussion of this eval, analysis, and where to learn more
The secret behind Gemini 3?
Simple: Improving pre-training & post-training 🤯
Pre-training: Contra the popular belief that scaling is over—which we discussed in our NeurIPS '25 talk with @ilyasut and @quocleix—the team delivered a drastic jump. The delta between 2.5 and 3.0 is as big as we've ever seen. No walls in sight!
Post-training: Still a total greenfield. There's lots of room for algorithmic progress and improvement, and 3.0 hasn't been an exception, thanks to our stellar team.
Congratulations to the whole team 💙💙💙
> Unfortunately, K2-Thinking's INT4 QAT doesn't have the "godlike technology" some people on social media are hoping for.
> We found that a relatively basic QAT solution can easily achieve lossless performance relative to the baseline, which is why we haven't spent too much time detailing the relevant technical solutions on our technical blog.
> As for why we chose INT4 instead of more "advanced" formats like MXFP4/NVFP4, it's indeed, as many have mentioned, to better support non-Blackwell architecture hardware.
Kimi K2 Thinking feels like a big milestone for open-source AI.
The first time in a while that open-source gets ahead of proprietary APIs on their big area of focus (agents).
Fun to see that it's happening at a time when the proprietary APIs have the most money/attention they've ever had (narrative violation to "they will win because they have more money/compute").
The paradigm shift continues!
This unsupervised or self-supervised idea is one of the reasons that we have such a strong LLMs in first place. The fact that the final finetuning could be unsupervised as well is just amazing.
We've trained an unsupervised language model that can generate coherent paragraphs and perform rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training: https://t.co/sY30aQM7hU
After ~4 years building SOTA models & datasets, we're sharing everything we learned in ⚡The Smol Training Playbook
We cover the full LLM cycle: designing ablations, choosing an architecture, curating data, post-training, and building solid infrastructure.
We'll help you navigate the messy training reality that LLM papers don't cover. Chapter highlights in the 🧵