Alexander Swerdlow

CS PhD Student @Stanford
Prev: @physical_int @CarnegieMellon @UCLA

[Alexander Swerdlow @alexjswerdlow]

@IMordatch Thanks! Better prompting for robot policies really unlocks a lot!

[Alexander Swerdlow @alexjswerdlow]

In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores.

So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data.

We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)

[Alexander Swerdlow @alexjswerdlow]

@aneeshers tysm!!

[Alexander Swerdlow @alexjswerdlow]

@dwait_bhatt Glad you liked it!

[Alexander Swerdlow @alexjswerdlow]

Thanks a lot!

I wouldn't say it's always better in the current form, mainly due to the added complexity, such as the return discretization and how this interacts with computing the likelihood (e.g., over each bin).

We didn't do any systematic failure analysis in the paper, but we did find some cases in which the prediction collapsed to a single bin (hence using target smoothing in some cases), which is a common problem for this sort of categorical/binned value learning.

Overall, the main reasons to pick traditional value learning over our generative value learning approach are similar to the distributional vs. non-distributional RL tradeoffs (added compute & hyperparameters).

[Alexander Swerdlow @alexjswerdlow]

@off_jangir Thanks so much Yash!

[Alexander Swerdlow @alexjswerdlow]

arXiv: https://t.co/VhUEKgAO3A
Website: https://t.co/DvolrkRW5N
Code: https://t.co/crj9G87arF

This was a great collaboration with @perryadong and @chelseabfinn!

[Alexander Swerdlow @alexjswerdlow]

There are a lot of open questions here, but we're particularly excited to see how this framework can take advantage of pretrained policies. We’ve seen evidence that models with return/advantage conditioning (e.g., π*0.6) enable meaningful policy improvement, and being able to take advantage of this - without changing the architecture or core objective - is a natural next step.

[Alexander Swerdlow @alexjswerdlow]

@moo_jin_kim @StanfordAILab @OpenAI @chelseabfinn @percyliang Congrats Moo Jin!!🥳

[Lars Ankile @larsankile]

I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently.

In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed.

On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.

[Yuejiang Liu @liu_yuejiang]

Excited to share that I’ll join @NUSComputing as an Assistant Professor in 2027

🏛️ I’ll build LEMA Lab: https://t.co/qzKA7dpA00, study the principles of embodied intelligence, & empower every lab member to thrive

📢 Recruiting 3-6 PhD students in the next application cycles

[Armaan Abraham @armaanabraham]

In RL, what if we could learn from any experience from any policy in a way that is reliable and scalable?

This would be helpful in domains like robotics where new data is expensive.

We introduce Long-horizon Q-learning (LQL) to tackle this https://t.co/1Ckb5ZePyo.

[Delta @DeltaCommunityX]

Today, we’re thrilled to share that our friends at @TrajectoryLabs have officially launched from stealth!!!

We first met Ronak a few days before Delta was founded in February 2025, and it’s been an incredible journey seeing him grow from his time at Windsurf, to DeepMind, and now to Trajectory.

The Trajectory team has seen Delta’s full trajectory, and now we’re incredibly excited for the world to see theirs!! 🚀

[Chelsea Finn @chelseabfinn]

FASTER makes top diffusion RL algos (eg IDQL, EXPO) computationally cheaper while retaining performance

Key idea: a denoising critic that operates on noise, enabling best-of-N filtering before diffusing

Paper + code: https://t.co/ylVWZxu4L9

[Perry Dong @perryadong]

Top RL algorithms today are getting powerful — but they can be prohibitively expensive, relying on test-time scaling techniques such as best-of-N sampling

We propose FASTER, a method that maintains the performance gains while eliminating the computational costs

(1/6)