1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share:
Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion
We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.
Say what you want about LLMs - and how smart you are without them. I can say without hesitation that they’ve made me smarter. Being able to interrogate an idea, ask anything, and go as deep or sideways as I want has changed how I think. Bottom line: I’m learning more, and better, than at any other point in my professional life.
Each word you say is planned by a small patch of brain above your left ear, about a quarter of a second before you say it. It sits in a temporal lobe region textbooks describe as only for listening. Our paper today in @ScienceMagazine: https://t.co/EXOgl3YsNI
@UCSF@NeurosurgUCSF
We introduce Posterior Matching Distillation (PMD), a new member of distribution-based distillation for few-step generation. PMD trains a stochastic student whose samples reproduce the teacher's exponentially tilted means, each obtained from a single frozen-teacher evaluation at an input shifted along the tilt direction.
We prove that matching these tilted means over a neighborhood uniquely determines the teacher’s full posterior over clean images. The result is exceptional diversity across seeds for the same prompt.
w/ Julius Berner (@julberner), Arash Vahdat (@ArashVahdat), and Pinar Yanardag (@PINguAR).
🧵 1/5
I do not want to deny the conceptual novelty of these alleged proofs, I have no idea.
I just do not see much conceptual novelty coming from AIs in any field I am competent in, which makes me a bit skeptical that the situation is so much different in math.
It might turn out that a lot of math that was assumed to be difficult for deeply conceptual reasons was just difficult because it would have taken a disproportionate amount of work to get it done
@GBatz97 You can also call Stockfish's chess moves 'brilliant'.
That doesn't mean that they are the result of high intelligence.
Fine intelligence plus combinatirial search can do the trick.