Labs disagree on vision encoders for MLLMs! ⚔️
Gemma 4 12B scrapped the encoder.
Kimi K3 dropped encoder pretraining.
Inkling 2 Small revived the encoder Inkling killed.
We trained 200+ models in a controlled study to settle it:
✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔
Paper: https://t.co/e8JRIfPU2m 🧵
The Jacobian is the reason this integral can be solved.
∫₋∞^∞ e^(−x²) dx = √π ≈ 1.7725
Square it and you get a double integral over the plane: the volume under a 3D bell, I² = ∬ e^(−(x²+y²)) dx dy.
Switch to polar coordinates. The Jacobian is J = r cos²θ + r sin²θ = r, so dx dy = r dr dθ. That extra r is exactly the derivative of r² we need.
With u = r², the radial integral ∫₀^∞ e^(−r²) r dr = ½. Then I² = ∫₀^2π ½ dθ = π.
A square became a circle, and π was there all along.
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
https://t.co/CIMECSEpD0
Big reasoning speedups are here. 1,000× faster training and 280× less compute than any looped model. All done using a fully trainable energy from the 80s
Keep reading to hear how energy is the right paradigm for reasoning
With @HughBlayney@jcrwhittington@prefrontal_ai
🧵/10
We're excited to announce the 2026 recipients of the Google PhD Fellowship!
148 new PhD students across 38 countries, with $13M in support from @Googleorg. Their work spans quantum theory, language model reasoning, medical imaging and more: https://t.co/MJCDb0wVgT
The 1847 Equation That Trains Every Neural Network
θ ← θ − η∇L(θ), written by Augustin-Louis Cauchy to solve the equations of astronomy.
L is the loss, a landscape of how wrong the model is, and θ is where the model stands on it.
The gradient ∇L points uphill, toward the steepest rise, so the minus sign turns the model around and sends it downhill (red arrow).
η, the learning rate, is the length of each step: too small and the ball crawls (left), too large and it overshoots the valley and flies off (right).
A real network walks this landscape in billions of dimensions at once, using a small random batch of data to estimate the slope at every step.
Language models, AlphaFold and image generators all learned by walking downhill on a 179-year-old rule, one tiny step at a time.
Day「6/28」🔭 LLM Arch Auto Research
At step 512, which Transformer would you keep training? In round two of our NextTransformer Arena search, the top four candidates finished in exactly the opposite order at step 2,000.
Opus had checked 11 architectures with known results. Their rankings after 2,000 steps tracked the full runs closely (Spearman ρ = 0.92). (One might speculate: if we can currently use a 2k-token sample to gauge the results of a 50B-token training run, what about even larger training scales? Does this type of "auto-research" prediction also follow a scaling law?)
Across three rounds and 12 candidates, Claude Opus 5.5 found AttnRes-Canon. Each layer can read earlier layers individually, and a width-4 convolution mixes each token with its three neighbors to the left.
After 49.152B tokens, it ranked first among all 38 published Arena v1 runs on both CORE-22 (35.99) and validation BPB (0.6605). In the short runs, a training step took 1.66× as long as it did for Block AttnRes.
Yi Ge wrote up the whole search, including the branches that went nowhere. https://t.co/wp24hGFdBU
Attention sink can implicitly act as a gate. Rearranging the softmax, you get a sigmoid(this works as a gate) multiplied by the usual softmax-weighted values.
Gated attention acts as more generalised version of attention sink because it learns element-wise gating and controls each dimension independently.
Based on Qwen's paper(https://t.co/LaGhsbSxxS), there is a beautiful derivation that connects attention sink and gated attention https://t.co/D59UlfbvoM
Sending support to all my Nvidia friends today, especially at Nemotron and HuggingFace. I hope your work comes back even stronger after a transition period. It’s absolutely crucial.
We are SO BACK!!
e-means is a simple and elegant clustering algorithm.
So refreshing to see classic ML get an upgrade!
For people interested in learning more, I made a video explainer.
Check it out! 👉 https://t.co/71vUO3mBPC
New #NeurIPS2026 paper:
"Provable State Estimation with Recurrent Models"
RNNs are widely used for state estimation in robotics, yet they lack convergence guarantees. We show when an RNN+MLP provably recovers the true state.
1/4