Kazz

math, physics, AI/ML

[Kazz @kazzorr_]

@huffinpuff123 @HealthRanger Chicken hawks love their soldiers killed so they can feel patriotic - the worst kind of traitors; a cancer in the nation.

[Kazz @kazzorr_]

@wbusse2 @FmrRepMTG "monumental battle"
Lmao dipshit.

[Lorenz Richter @ICML 2026 @lorenz_richter]

Tomorrow (Thu, 10:30am, #3501), we present Tensor Train Diffusion at ICML: a diffusion-based sampler that replaces neural networks with tensor trains. For low-rank densities, this can dramatically reduce training time, while even improving performance; see https://t.co/zwinbqCc07

[Goodfire @GoodfireAI]

If models think in shapes, our tools should too.

Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)

[elie @eliebakouch]

computed the similarity (CKA) on the J-lens geometry of every layer inside and across 38 open models. the patterns are weirdly universal: same depth layout, same organization at the same relative depth, even between unrelated families like llama and olmo

https://t.co/C6188kLOXg

[Yujia Zheng @YujiaZheng9]

J-space is really something we have been exploring since 2022. Glad to see it continues to work well at scale!

Some of the related work along this direction:

- How to recover the latent process using Jacobians (Identifiability of nonlinear ICA): https://t.co/uZkCxCMyrq

- How to handle dependent latents and assumption violations (again, through Jacobians): https://t.co/JNFPThjvkW

- For general latent variable models, what remains recoverable with guarantees, and why Jacobians are universally helpful? (We could generalize SAEs to the general nonlinear case, with Jacobians!): https://t.co/Mmh9lgglA0

Feels like we're still only beginning to uncover what Jacobian structure can tell us about representations.

[Dmitry Rybin @DmitryRybin1]

pre-training scaling laws => LLMs generalize on test data

I convinced myself that this implication can be derived without much cheating in math. I suspect this was known to many experts, but I never seen it written down explicitly.

[Volodymyr Kuleshov 🇺🇦 @volokuleshov]

New blog post: How to Build a Diffusion Language Model.

Diffusion LLMs went from open problem to reality in 2 years (Mercury, Gemma Diffusion, Nemotron Diffusion). With my Cornell group, we wrote up the research advances that make them work.

We use today's OSS large diffusion models as examples and show how they're built from core techniques:
• masked diffusion (MDLM)
• iterative refinement (UDLM, ReMDM)
• variable-length generation (block diffusion, encoder-decoder architectures)
• controllable generation (D-CFG, D-CBG)
• fast samplers (Duo)
• RL post-training (d1, d2)

This post covers research led by a talented group of Cornell PhD students including @mariannearr @SchiffYair @Guanghan__Wang

The content is adapted from talks I gave this year on dLLMs. It covers most of the main ingredients behind open-source diffusion language models today.

Link: https://t.co/lpu3BCLwiE

[Aaron Havens @aaronjhavens]

Excited to be at ICML to present our Spotlight paper: Flow Sampling!

We propose a simple fixed-point objective for learning diffusion samplers, built on the flow matching marginal construction. More details to come soon!

Paper: https://t.co/ftdv9F7Xno

[François Porcher @foporcher]

1/ My first PhD paper is out! 🎓

Title: Flow Matching in Feature Space for Stochastic World Modeling

tldr: we build stochastic world models directly in high-dimensional DINOv3 feature space, instead of relying on low-dimensional VAE latents.

[wafer @wafer_ai]

🚨 OpenAI is launching GPT-5.6 Sol on Cerebras at up to 750 tps.

Here's How kernels work on Cerebras Chips

cerebras built a chip with 900,000 cores on a single silicon wafer.

CSL (Cerebras Software Language) is a Zig-inspired DSL that gives you direct control over the Wafer-Scale Engine. the SDK is publicly documented and the programming model is genuinely the most alien one we've ever looked at.

in CUDA you write one thread's perspective and launch millions. in CSL, there are no threads, no warps, no shared memory, and no kernel launches. you write code for individual Processing Elements: 900,000 independent cores arranged in a 2D mesh on a single silicon wafer. each PE has its own 48 KB of private SRAM, its own program counter, and a 5-port router connecting it to its 4 neighbors. that's actually it. no DRAM. no HBM. no cache hierarchy. 48 KB is your entire world per PE, where code and data must both fit.

the programming model is dataflow.

data moves between PEs as 32-bit messages called wavelets, traveling along virtual channels called colors. when a wavelet arrives at a PE on a specific color, it activates a task (a chunk of code bound to that color at compile time). tasks run to completion, then hardware picks the next activated task. tasks cannot call each other. they can only be activated.

so instead of "launch N threads," you think of it like: "place code on PEs, define routes, let data flow."

the memory model is also very different from GPUs.

on an H100 you get 80 GB of HBM shared across all SMs. on the WSE-3, memory is 48 KB per PE, and there are 900,000 of them! this gives you 44 GB total on-chip SRAM at 21 PB/s aggregate bandwidth (vs 3 TB/s on H100). every access is single-cycle. no coalescing needed. no bank conflicts. no cache misses. but also no way to access another PE's memory. all inter-PE communication is explicit wavelet routing through the fabric.

take a distributed GEMV for example. you would write a layout file that physically routes wavelets across the mesh. two PEs sit side by side. the left PE computes a partial result and routes it eastward. the right PE receives those wavelets from the west and accumulates. routing is defined at compile time. both operations are asynchronous and activate a task when they finish. you're physically routing data across silicon at 1 clock cycle per hop.

cerebras is an extremely technically interesting & alien beast. reports 95-210x speedups over H100 on stencil computations. 3,000 tokens/sec inference on gpt-oss-120B. $10B+ deal with OpenAI for 750 MW of inference infrastructure. very exciting times for alternative accelerators!

deep dive 1/6 by @gpuemi

[XY Han @XYHan_]

Most misleading part clarified: The “J-space” is not a separate, hidden space. It is an alternative *coordinate system* for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations.

[Machine Learning Street Talk @MLStreetTalk]

This paper looks great, note that heavyweights @kenneth0stanley, Bert Chan (Lenia) and @SakanaAILabs have had their hands all over it!

[Mathias Lechner @mlech26l]

Small language models (<10B) often struggle with doom loops (=model stuck repeating itself) in long reasoning traces.
We show that this can be fixed with specialized preference optimization

[Physical Review E @PhysRevE]

With a generative model based on nonequilibrium visible latent dynamics, researchers show that reversible dynamics limit exploration, while nonequilibrium latent cycles enhance sampling efficiency and likelihood.

Read the paper: https://t.co/5RdhcKVNP9

[Phillip Isola @phillip_isola]

At ICML, I'll present our Neural Thickets work today:

2:30–4:15 PM KST
Hall A #2602

Come by if you have questions or want to chat!

[prathosh A P @prathoshap]

Link for the course - https://t.co/PclKsiqcPJ

Link to the notes - https://t.co/qRmpAPAeeJ

[Jack Morris @jxmnop]

my paper won an award at icml 😁

some thoughts:
• this work was rejected from NeurIPS. i cleaned it up a small amount and it got great reviews from ICML! don't give up
• ICML received 24k submissions and only gives out 7 awards, which is crazy. feeling grateful
• i distinctly remember sitting at my desk two winters ago wondering if i would ever finish this project. most of all this is the product of sitting down and forcing myself to keep working for several months straight. the results emerged from running the experiments over and over and fixing a long sequence of tiny details. eventually, the curves looked like that 👇
• also happy that the insights in this paper are becoming more widely accepted: 3.3 bits/param, thinking about capacity "LLM as flashdrive" mentality
• the method here is used successfully for selecting midtraining data at least one frontier lab, which is cool!
• i am grateful to my collaborators, but Meta is no longer a great place for academic research imo and this almost never got published for a number of reasons. i shall not elaborate further
• for future work, i think analyzing the implications of on-policy algorithms on capacity, as well as LoRA and things like it, are fruitful potential research directions
• sadly i'm not in Korea but am following the conference online from california and happy to chat!

a nice end to one phase of my research career :)

[Jack Morris @jxmnop]

new paper from our work at Meta!

**GPT-style language models memorize 3.6 bits per param**

we compute capacity by measuring total bits memorized, using some theory from Shannon (1953)

shockingly, the memorization-datasize curves look like this:
___________
/
/

(🧵)

[Lars Holdijk @HoldijkLars]

Our book "Generative AI and Stochastic Thermodynamics: A Tale of Free Energies" is out this month. With @sirui_lu97 and @wellingmax I'm posting about topics it covers. Yesterday: heat and work in variational EM.

Today: the variational free energy, and why the ELBO is one.