Max Sobol Mark

PhD student at @CarnegieMellon. Prev: @GoogleDeepMind @Stanford

[Max Sobol Mark @maxsobolmark]

Excited to share Big Picture Policies (BPP), from my internship at Google DeepMind! 🤖
Many tasks we expect robots to do need memory, but the best policies today are memoryless. We study how to fix this, unlocking long-horizon searching behaviors and solving 6-step tasks. 🧵👇

[Aviral Kumar @aviral_kumar2]

Check our work led by @maxsobolmark on making robot policies use memory or history well👇

We detect a minimal set of frames to represent history. Training with these *key* frames, i.e., focusing on the "big picture" is effective for learning from history.

Some perspectives 🧵⬇️

[Chris Paxton @chris_j_paxton]

I still believe memory is one of the most important challenges we need to address for true physical intelligence.

[Dhruv Shah @shahdhruv_]

We study the challenges of training imitation learning policies with memory and propose a simple solution: condition on "key history frames"!

This allows policies to learn long-horizon search and multi-step behaviors over several minutes: https://t.co/H94CPxFyl4

See Max's 🧵👇

[Jacky Liang @jackyliang42]

Super cool work from @maxsobolmark!
Robot policies need memory for many tasks, but naively adding history to observations degrades performance. In this work we explain why this is happening and propose a method that effectively leverages history. See more details in the thread!

[Max Sobol Mark @maxsobolmark]

Joint work with @jackyliang42, @JMarakiii, @ChuyuanFu, @debidatta, @aviral_kumar2, and @shahdhruv_. Thanks to the GDM Robotics team!
Paper: https://t.co/EZta0uC92N
Website: https://t.co/5A2jfO6YAJ

[Max Sobol Mark @maxsobolmark]

Excited to share Big Picture Policies (BPP), from my internship at Google DeepMind! 🤖
Many tasks we expect robots to do need memory, but the best policies today are memoryless. We study how to fix this, unlocking long-horizon searching behaviors and solving 6-step tasks. 🧵👇

[Max Sobol Mark @maxsobolmark]

What's next? BPP could be used to train generalist history-conditioned robot policies by scaling up keyframe labeling. It can also unlock faster robot ICL, e.g. quickly learning from grasping mistakes by only conditioning on "key segments".

[Fahim Tajwar @FahimTajwar10]

Are we done with new RL algorithms? Turns out we might have been optimizing the wrong objective.

Introducing MaxRL, a framework to bring maximum likelihood optimization to RL settings.

Paper + code + project website: https://t.co/j9BCBF7K3R

🧵 1/n

[Dhruv Shah @shahdhruv_]

Excited to release Gemini Robotics On-Device and bunch of goodies today
🍬 on-device VLA that you can run on a GPU
🍬 open-source MuJoCo sim (& benchmark) for bimanual dexterity
🍬 broadening access to these models to academics and developers

https://t.co/4X8qJwD4QN

[Fahim Tajwar @FahimTajwar10]

RL with verifiable reward has shown impressive results in improving LLM reasoning, but what can we do when we do not have ground truth answers?

Introducing Self-Rewarding Training (SRT): where language models provide their own reward for RL training!

🧵 1/n

[Max Sobol Mark @maxsobolmark]

I'll be presenting Policy-Agnostic RL: Fine-Tuning of Any Policy Class and Backbone at the Robot Learning (Sunday) and GenBot (Monday) workshops as Orals at #ICLR2025!

Happy to chat or meet!

[Max Sobol Mark @maxsobolmark]

Robotics foundation models struggle with new tasks. Can a single RL method efficiently improve any policy, including large generalists (7B OpenVLA) and diffusion policies, in the real world?

Our new paper PA-RL does just that!
https://t.co/bdHbCw3286
https://t.co/EvEI8ZXmr9
🧵👇

[Yuxiao Qu @QuYuxiao]

🚨 NEW PAPER: "Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning"!

🤔 With all these long-reasoning LLMs, what are we actually optimizing for? Length penalties? Token budgets? We needed a better way to think about it!

Website: https://t.co/G5JTmryx0d

🧵[1/9]

[Max Sobol Mark @maxsobolmark]

@edward_s_hu We're looking into this! Local optimization might require more gradient steps to work on higher dims, but it shouldn't be a bottleneck.

[Max Sobol Mark @maxsobolmark]

Robotics foundation models struggle with new tasks. Can a single RL method efficiently improve any policy, including large generalists (7B OpenVLA) and diffusion policies, in the real world?

Our new paper PA-RL does just that!
https://t.co/bdHbCw3286
https://t.co/EvEI8ZXmr9
🧵👇

[Aviral Kumar @aviral_kumar2]

How can we fine-tune generalist policies autonomously w/ RL (value functions)?

@maxsobolmark's new paper on Policy-agnostic RL provides a single way to fine-tune generalist VLAs w/ any backbone, output, size (we fine-tune 7B OpenVLA on real robot)

https://t.co/1VKM9Wj0Ae🧵⬇️

[Max Sobol Mark @maxsobolmark]

This was my first PhD project at @aviral_kumar2's lab in CMU! I'm grateful to have collaborated with @TianGao_19, @georgiagsampaio, @mohansrirama, @archit_sharma97, and @chelseabfinn.

Check out our paper for more details and results, and our website for videos!

[Max Sobol Mark @maxsobolmark]

PA-RL bypasses these problems by decomposing policy improvement into two steps: Action Optimization (improve actions directly not model parameters); and Policy Distillation (learn to produce optimized actions with Supervised Learning, something most policies handle well)