Sirui Xie

Research Scientist @GoogleDeepMind. Prev: PhD @UCLA. BEng @HKUST. Views are my own.

[Sirui Xie @SiruiXie]

TailRL has sparked an interesting discussion about moving beyond expected-return RL.

It connects closely to a question we explored in Generative Actor Critic (GAC), posted Dec 2025:

Should policy improvement use a fixed training objective—or can it remain a test-time choice? 🧵

[Valentin De Bortoli @ValentinDeBort1]

🔥Welcome 🔺Simplex Diffusion Models🔻!

Standard discrete diffusion models suffer from "information collapse" because they discard uncertainty at intermediate steps by sampling categorical tokens. (1/5)

[David Ruhe @djjruhe]

📄 Paper: https://t.co/R8bsqCSs2h

Huge thanks to our amazing intern @k01047339268, and to co-authors @ArnaudDoucet1 , @AndrewC_ML , @ValentinDeBort1 , and Thomas Mensink!

[Google DeepMind @GoogleDeepMind]

Introducing Gemini 4 Argon – our new frontier model.

It’s built for complex workflows across coding, enterprise knowledge work, and cybersecurity defense – rolling out today to a set of trusted testers through our Fairwind Program.

[Sirui Xie @SiruiXie]

Really nice perspective on why preserving the upper tail matters for RL.

It connects closely to a question we explored in Generative Actor Critic: should that upper-tail preference be built into the training objective, or remain a test-time inference choice?

A few thoughts on the connection: https://t.co/HB2FAL2VUw

[Sirui Xie @SiruiXie]

TailRL has sparked an interesting discussion about moving beyond expected-return RL.

It connects closely to a question we explored in Generative Actor Critic (GAC), posted Dec 2025:

Should policy improvement use a fixed training objective—or can it remain a test-time choice? 🧵

[Sirui Xie @SiruiXie]

7/ It’s exciting to see this broader direction getting more attention.

TailRL shows what can be gained from designing a better objective over the return distribution.

GAC’s bet is complementary: when possible, preserve the flexibility to ask different decision-making questions of the same learned model.

GAC
https://t.co/powQLNrYyc

TailRL
https://t.co/bVIwDJTRnZ

[Sirui Xie @SiruiXie]

6/ This makes the relationship between TailRL and GAC especially interesting.

TailRL: choose a better training objective when we care about the upper tail.

GAC: learn a generative model that lets the decision objective remain a test-time choice — including whether and how to target the upper tail.

This points to a deeper question:

When should an RL system commit to a notion of “better” — during training, or when a decision is actually made?

[Sirui Xie @SiruiXie]

5/ This is where the trajectory dimension matters.

A return distribution tells us what outcomes may occur. But control also requires producing the behavior that realizes the outcomes we prefer.

By modeling p(trajectory, return), GAC connects desired outcomes directly to those behaviors.

Changing what we want can therefore change which behavior we infer, not merely how we evaluate it.

[Sirui Xie @SiruiXie]

3/ GAC takes a different route.

We learn the joint generative model

p(trajectory, return)

and separate learning that model from choosing how to improve behavior.

Decision making becomes an inference query over the learned joint distribution, rather than being tied to one policy-improvement objective during training.

[Sirui Xie @SiruiXie]

4/ This lets the same learned GAC model support different test-time queries — the idea illustrated in Fig. 1 of our paper.

For exploitation, we use expected return.

For exploration, we shift toward the upper tail and infer trajectories associated with those outcomes.

More generally, different desired outcome distributions can define different queries without relearning the underlying model.

[Sirui Xie @SiruiXie]

2/ TailRL gives an interesting new answer on the objective side.

It optimizes over the upper tail of the return distribution, with a clean connection to a mixture of Best-of-k objectives.

The result is a policy trained to preserve probability on rare high-reward outcomes across different inference-time sampling budgets.

So TailRL asks:

What objective should we train the policy with when upper-tail coverage matters?

[Sirui Xie @SiruiXie]

TailRL has sparked an interesting discussion about moving beyond expected-return RL.

It connects closely to a question we explored in Generative Actor Critic (GAC), posted Dec 2025:

Should policy improvement use a fixed training objective—or can it remain a test-time choice? 🧵

[Sirui Xie @SiruiXie]

1/ Distributional RL already showed the value of learning more than expected return.

But learning a return distribution and having flexible policy improvement are different things.

We still need to decide how that distribution should determine behavior — through expected return or some other, potentially tail-oriented, objective.

Once chosen, that objective shapes the policy we learn.

[Sirui Xie @SiruiXie]

7/ It’s exciting to see this broader direction getting more attention.

TailRL shows what can be gained from designing a better objective over the return distribution.

GAC’s bet is complementary: when possible, preserve the flexibility to ask different decision-making questions of the same learned model.

GAC
https://t.co/powQLNrYyc

TailRL
https://t.co/bVIwDJTRnZ

[Sirui Xie @SiruiXie]

6/ This makes the relationship between TailRL and GAC especially interesting.

TailRL: choose a better training objective when we care about the upper tail.

GAC: learn a generative model that lets the decision objective remain a test-time choice — including whether and how to target the upper tail.

This points to a deeper question:

When should an RL system commit to a notion of “better” — during training, or when a decision is actually made?

[Michael C. Mozer @mc_mozer]

What if a foundation model could tell us how to modify its architecture to boost inference and reasoning instantaneously—without retraining? What if that tweak incurred near zero latency cost during generation and supported indefinite state tracking? https://t.co/ye8P6Em40F

[Kevin Patrick Murphy @sirbayes]

New paper: "Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs".

Our system (BLF) matches human superforecasters on ForecastBench, and beats all the top methods (GPT-5, Cassi, Grok 4.20, and Foresight-32B).

🧵