Introducing LongTail-SWE: 202 real tasks across 13 languages that models are rarely tested on.
On functional languages (Elixir, OCaml, Haskell), the gap between closed and open-weight models is wider than on the other languages.
Opus 5.5 and GPT-6.1 Sol handle them without a drop in performance, while DeepSeek, GLM and Qwen fall further behind.
We built the tasks from real issues and PRs and reworked them heavily with our pipeline, spending significant compute to make them fair: task descriptions and tests were rewritten based on agent rollouts.
All tasks are on Harbor with Docker images, and model trajectories are on Hugging Face. Full leaderboard with a per-language breakdown and pipeline overview on the site; methodology in the paper.
Spot an unfair task, a bug, or have questions? Reply here, DM, or email.
Leaderboard: https://t.co/nlCubyntlw
Paper: https://t.co/HDJZ7PJjMM
P.S. Need free RL environments, including for these languages? Check my pinned post: last week we released 13,337 RL envs.
1. I don't buy the recovery experiment b/c recovering a few benchmark scores doesn't mean recovering the full distribution. They never report loss or KL after fine-tuning. See also https://t.co/WK0EeWmtrm, which shows ablating weak activations makes thinking run way longer even when scores hold
2. And their benchmark scores aren't fully recovered. MMLU-Pro, HellaSwag and MMLU all stay significantly lower, and the original model given the same RL beats the fine-tuned one on 8 of 10 reasoning benchmarks
@mahdi_nazemi_ Yess!! Also we alrady successfuly do native bf16 post training (including RL) in Halo, pls check it out and our docs about bf16 specificaly:
Github: https://t.co/n07vGQ22qK
Bf16 doc (little tricks): https://t.co/d5CuKR7KzQ
Our rl with bf16: https://t.co/reMz4lz0qT
Sad but still bullish on @nvidia!
I’m seeing a lot of people criticizing its structure and the lack of a clear CEO successor. Having seen it from up close, I think a lot of this criticism is ill-informed and overly simplistic (as often happens when a company is run differently from the norm).
Obviously, important to remember that this structure made NV as successful as it is and powered the AI revolution. But beyond its past success, I think it’s also what makes the company antifragile for the future.
In my opinion, NV's distributed leadership and comfort with chaos are much more of a strength than a weakness, even without Jensen. They give an unusual number of leaders room to step up and keep the whole machine moving (as we've seen in the past few days). They also keep the company flexible enough to adapt to the next paradigms (open-source/local/agents/robots/biology/chemistry/...). They also encourage hard-working, self-motivated team members with a strong sense of belonging and a willingness to help each other which will matter a lot.
Personally, with the right CEO (and NVIDIA has plenty of exceptional leaders internally who could take on that role), I believe NV can keep thriving and empower every company to become an AI company and every builder to become an AI builder.
To the @nvidia team: sending you strength and support, from me, the @huggingface team and so many people across the AI community. We need you and we’re rooting for you to carry @JensenHuang's mission forward! Taking care of his company, your company, is probably the best way to honor his legendary legacy!
💚💚💚
Safety rows stay rare:
misalignment, deception and honesty rates are in 4 of 38 launch tables since July, all OpenAI's (GPT-6 Astra, GPT-6 Sol, GPT-6.1 Sol, GPT-6.1 Terra). Cyber benchmarks show up 4x as often
most safety results live in system cards, which we didn't count
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows.
It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
Method: a launch table is the headline benchmark table of a launch post or model card (post-trained model, no appendices). Each launch weighs the same. Function calling counts as agentic, cyber evals don't. Some image-only tables are partial.
Data and script in the reply.
Cyber is the other new row
Exploit and vulnerability benchmarks (CyberGym, ExploitGym, ExploitBench, SEC-bench Pro) were in 0 of 36 launch tables in 2025 H2, 10 of 27 in 2026 H1 and 16 of 38 since July
Google and OpenAI now launch cyber-tuned models
@OfirPress in launch tables SWE-sweep is the newest agentic coding standard: SWE-bench Verified ran Sep 2024 to Jun 2026, SWE-bench Pro Dec 2025 to Aug 2026, and DeepSWE is in 71% of launches since July
my count over 143 launches:
https://t.co/ek9oQqjH1F
The above part concerns "thinking" being disabled (decoding a single token - the original GoodFire setup); this thread, however, covers a more realistic scenario - that is, when "thinking" is actually enabled.
https://t.co/BKpxc9qSKp
Goodfire and @nickcammarata say weak activations barely matter. Thinking off, 6 of 12 LLMs agree. Thinking on (32k), behavior changes:
Qwen3.5-9B: 2.4× longer, 1 in 5 runs never ends, −10.5 points
Qwen3.6-27B: same score, 43% longer
Goodfire's test model: same score, 32% longer
1/6
Goodfire says a neuron's weak activations barely matter (“Neural Networks Are Already Clean”). Their new paper tests it on one LLM.
We ran the essay's test, zeroing each MLP neuron's weakest 75%, on 12 LLMs. MMLU-Pro:
> 6 lose at most 1.1 points
> 6 lose 6 to 11 points
1/8
Neurons exert most of their influence on the model when they fire most strongly. That is also when they are most interpretable.
Ablating the weakest 92% of activations has only modest effects on capabilities, and light fine-tuning fully recovers benchmark performance.
Goodfire's paper: https://t.co/szZuCvml88 The essay: https://t.co/fp2uhgPpOR Leo Gao: https://t.co/GYoxTYhSSD Scripts and data: https://t.co/m8shIpKSQz We build Halo, an open-source LLM training and RL framework: https://t.co/n07vGQ22qK
Goodfire and @nickcammarata say weak activations barely matter. Thinking off, 6 of 12 LLMs agree. Thinking on (32k), behavior changes:
Qwen3.5-9B: 2.4× longer, 1 in 5 runs never ends, −10.5 points
Qwen3.6-27B: same score, 43% longer
Goodfire's test model: same score, 32% longer
1/6
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
@CurtTigges@nabla_theta Our first thread (12 models, thinking off): https://t.co/zpdF7hOmq7
Goodfire's paper: https://t.co/szZuCvmSXG
Scripts and per-question results: https://t.co/m8shIpLqG7
We build Halo, an open-source LLM training and RL framework: https://t.co/n07vGQ2Agi
Goodfire says a neuron's weak activations barely matter (“Neural Networks Are Already Clean”). Their new paper tests it on one LLM.
We ran the essay's test, zeroing each MLP neuron's weakest 75%, on 12 LLMs. MMLU-Pro:
> 6 lose at most 1.1 points
> 6 lose 6 to 11 points
1/8
Limits: 200 questions, 2-3 seeds at temperature 0.6. vLLM matches our HF runs within 1.4 points with thinking off. Not tested: Goodfire's 92% top-K plus fine-tuning.
Code and per-question results: first reply.
cc @CurtTigges@nabla_theta