Yan

[Volodymyr Kuleshov 🇺🇦 @volokuleshov]

Congratulations to Google on open-sourcing Gemma Diffusion!

I want to give a shout-out to a group of really talented Cornell students who developed in the lab a lot of the new ideas that we see in this model:

@mariannearr -- Block diffusion is what enables Gemma Diffusion to generate arbitrary length sequences and support KV caching.

@mariannearr @SchiffYair -- Efficient encoder-decoder diffusion (E2D2) extends block diffusion and is part of what makes Gemma really fast, speeding up inference by running a smaller decoder model.

@SchiffYair @ssahoo_ @Guanghan__Wang -- Uniform diffusion LMs (UDLMs) are the family of discrete diffusion models that underlie Gemma and define its noise process and training objective. This work builds on our earlier simplified losses in MDLMs.

@ssahoo_ -- Uniform diffusion supports built-in error correction and is especially effective with distilled fast samplers like the ones introduced in Duo.

This is a great overview of Gemma Diffusion: https://t.co/MXLfgPPNc4

Check out the students' papers below:

[Nick Levine @status_effects]

New work with @AlecRad and @DavidDuvenaud:

Have you ever dreamed of talking to someone from the past? Introducing talkie, a 13B model trained only on pre-1931 text.

Vintage models should help us to understand how LMs generalize (e.g., can we teach talkie to code?). Thread:

[Yu Su @ysu_nlp]

Introducing @NeoCognition, the agent lab for specialized intelligence.

Everyone needs experts, but human expertise does not scale.

Backed by $40M seed funding, we build self-learning agents that specialize across domains to make expertise abundant.

[Andrew Gordon Wilson @andrewgwils]

@getreddy4this "Frontier" is largely a misnomer for what's happening now. There is a misalignment of strengths. Great scientists are great because they can ask the important questions that no one else is asking. There are many many more people who can proficiently execute an incremental agenda.

[pdawg @prathamgrv]

I made a Claude Code skill that turns any arxiv paper into working code.

Every line traces back to the paper section it came from & any implementation detail the paper skips will be flagged, and not assumed.

open sourcing it -
https://t.co/sSio4JfpIo

[Matt Harrison @__mharrison__]

For my friends who are still using UV and might be a little weary about recent compromises to PyPi packages, stick this in your pyproject.toml.

You can let all of those pip users find and report the compromises...

[JJ @JosephJacks_]

Frankly disappointed. Three iterations deep and the paper itself concedes pure SSMs can’t do retrieval and hybrids (SSM + attention) are the future.. So Mamba is converging on being a better compression sublayer inside someone else’s architecture — not a replacement. The “inference-first” framing is doing heavy rhetorical lifting over what are solid but incremental control theory refinements (complex transitions, trapezoidal discretization, MIMO) that don’t touch the core constraint: fixed state = lossy history. ~5% decode speedup over Mamba-2 SISO. Kernel engineering is genuinely good. But this isn’t the trajectory you want if the original pitch was obsoleting transformers.

[Andrej Karpathy @karpathy]

@nummanali tmux grids are awesome, but i feel a need to have a proper "agent command center" IDE for teams of them, which I could maximize per monitor. E.g. I want to see/hide toggle them, see if any are idle, pop open related tools (e.g. terminal), stats (usage), etc.

[Tri Dao @tri_dao]

The FA4 paper is finally out after a year of work. On Blackwell GPUs, attention now goes about as fast as matmul even though the bottlenecks are so different! Tensor cores are now crazy fast that attn fwd is bottlenecked by exponential, and attn bwd is bottlenecked by shared memory bandwidth. 

Some fun stuff in the redesigned algorithm to overcome these bottlenecks: exponential emulation with polynomials, new online softmax to avoid 90% of softmax rescaling, 2CTA MMA instructions that allow two thread blocks to share operands to reduce smem traffic.

[Bo Wang @BoWang87]

Prof. Donald Knuth opened his new paper with "Shock! Shock!"

Claude Opus 4.6 had just solved an open problem he'd been working on for weeks — a graph decomposition conjecture from The Art of Computer Programming.

He named the paper "Claude's Cycles."

31 explorations. ~1 hour. Knuth read the output, wrote the formal proof, and closed with: "It seems I'll have to revise my opinions about generative AI one of these days."

The man who wrote the bible of computer science just said that. In a paper named after an AI.

Paper: https://t.co/juSOmK9vOt

[Yacine Mahdid @yacinelearning]

hey folks tomorrow at 12-14h EST we're going to interview the first authors of the Maximum Likelihood Reinforcement Learning paper

I've spent 2 weeks peering into RL algorightms like a mad man in an effort to sample the weirdest questions

drop yours below I'll ask them too

[Dimitris Papailiopoulos @DimitrisPapail]

512 parameters: a new top scorer for 10-digit addition with transformers!

Who can beat it?

[You Jiacheng @YouJiacheng]

Interesting. After some struggles, a HC-like architecture change finally works on modded-nanogpt -- it reduces ~3% steps (1490→1450).

[Elon Musk @elonmusk]

From this goal of Grok, all things flow:

Rigorous truth-seeking

Appreciation of beauty

Fostering humanity

Discovering all physics

Inventing all useful technologies

Consciousness to the stars

Love

[Grigory Sapunov @che_shr_cat]

1/
Standard VAEs use arbitrary KL penalties to regularize latents. We just guess the scaling factor and hope for the best. But what if we replace that arbitrary KL divergence with a continuous diffusion prior?

The compute efficiency frontier just moved by @GoogleDeepMind . 🧵

[Nicholas Boffi @nmboffi]

We just brought flow maps to language modeling for one-step sequence generation 💥

Discrete diffusion is not necessary -- continuous flows over one-hot encodings achieve SoTA performance and ≥8.3× faster generation 🔥

We believe this is a major step forward for discrete generative modeling and language modeling alike. 🚀

Full thread from first author @chandavidlee: https://t.co/7HIBNbQdFO

[Idan Beck @idanbeck]

They hard coded the variance - meaning the VAE encoder is only predicting the mean latent distribution, then they use a scaled identity covariance for the reparam trick - and bingo bango, no more instability and you can train everything e2e

Salimans strikes again!

[Sander Dieleman @sedielem]

*nods vigorously*
Strongly agree with Ivan's take on drifting models -- I love the fresh perspective, but the hype, not so much. Let's not get ahead of ourselves.

(Of course, I _would_ say that, given how invested I am in diffusion models🙃)

[nathan chen @nathancgy4]

random rabbit holes with unique design tastes are a gift