(1/6) Can Autoregressive Models (ARMs) go beyond fixed or random generation orders and learn the optimal order to generate new samples?
We believe so! We are excited to present our paper, “Learning-Order Autoregressive Models with Application to Molecular Graph Generation,” at ICML 2025 in Vancouver on July 18th. Feel free to drop by!
In our new paper, we introduce Learning-Order Autoregressive Models (LO-ARMs), which are trained to find a consistent, context-dependent generation order on their own. Our model matches and surpasses state-of-the-art results on two major molecular graph generation tasks.
Read our paper here: https://t.co/hbXWjSdVkQ
Joint work with @thjashin, Nicolas Heess, @ArthurGretton, Michalis K. Titsias
More in 🧵👇
Excellent talk by @thjashin:
- What separates autoregression and diffusion?
- Bridging the gap between continuous and masked discrete diffusion via loss reweighting
- Insertion-based sequence generation
Very clear and easy to follow! (via @mblondel_ml)
https://t.co/YNmzhQpUyz
AR models append tokens only at the end. But what if generation could insert tokens anywhere?
It's strictly more expressive, but much harder to train with standard max likelihood. We introduce a scalable variational learning method for insertion-based generative models.
https://t.co/SquBMOJsIK
Qiang Liu, Chris Oates, and I are writing a monograph on Probabilistic Inference and Learning with Stein’s Method, and we’d love to get your feedback on the first draft
📢 Job alert
We are looking for a Postdoctoral Fellow to work with @ArthurGretton on creating statistically efficient causal and interaction models with the aim of elucidating cellular interactions.
⏰Deadline 27-Aug-2025
ℹ️ https://t.co/4B4UIu1uh3
(1/6) Can Autoregressive Models (ARMs) go beyond fixed or random generation orders and learn the optimal order to generate new samples?
We believe so! We are excited to present our paper, “Learning-Order Autoregressive Models with Application to Molecular Graph Generation,” at ICML 2025 in Vancouver on July 18th. Feel free to drop by!
In our new paper, we introduce Learning-Order Autoregressive Models (LO-ARMs), which are trained to find a consistent, context-dependent generation order on their own. Our model matches and surpasses state-of-the-art results on two major molecular graph generation tasks.
Read our paper here: https://t.co/hbXWjSdVkQ
Joint work with @thjashin, Nicolas Heess, @ArthurGretton, Michalis K. Titsias
More in 🧵👇
(6/6) Finally, can LO-ARMs generate "better" molecules?
Yes, they can. On both the QM9 and ZINC250k datasets, LO-ARMs match or exceed state-of-the-art (SOTA) results.
Performance was evaluated across key metrics for distribution similarity and drug-likeness.
(5/6) Can LO-ARMs learn consistent and meaningful generation orders on their own?
Yes — and the strategy/generation order they discover is surprisingly intuitive.
On two major molecular generation tasks (QM9 and ZINC250k), our models learned a consistent, two-phase process:
1. Build the Skeleton: First, generate the chemical bonds.
2. Infill the Atoms: Then, add the atoms to the skeleton.
We find that 99% of the new samples follow this generation order.
Here’s an example of this in action for generating a ZINC250k molecule. We start with a blank slate where all tokens are masked, and in each step, the model adds one new piece — either an atom or a bond, until the molecule is complete.
@ZoubinGhahrama1@geoffreyhinton Just took the Gatsby courses last year (now they are split to two modules approximate inference and unsupervised learning), and the courses are incredibly helpful. I even read your matlab code!😁
📢 We have an opportunity for students to join our PhD programme in Theoretical Neuroscience and Machine Learning this September.
Application deadline is 27 May 2025.
Information & how to apply 👉 https://t.co/nbdHOqwGSA