Agneya Tharun

cmu scs

[Agneya Tharun @AgneyaTharun]

What if a decision model could replace the reward function in RL?

I trained two humanoids to walk from scratch. One used 12 hand-tuned rewards (things like staying upright, avoid jerky movements). The other used elephant-1 to judge how naturally it walked.

Results:

[Agneya Tharun @AgneyaTharun]

@paolodiprodi maybe modern artists were always plotting on fooling VLAs

[Agneya Tharun @AgneyaTharun]

VLAs can be hypnotized!

I trained an adversarial model to generate a printable visual pattern that disrupts a robot's behavior just by appearing in its peripheral view.

(left: standard, middle: control, right: adversary. VLA is tasked with placing the bowl on the plate)

[Agneya Tharun @AgneyaTharun]

my next plan of action is to further train my adversarial model such that it not only stops the VLA from completing its task, but makes it do something completely different (like throw the plate or something crazy)

[Agneya Tharun @AgneyaTharun]

To control for external variables in this experiment, I tested whether simply placing something on the table would cause failures.

On one task, SmolVLA succeeded 6/6 times with no stimulus or a blank plank, but 0/6 with the texture.

[Agneya Tharun @AgneyaTharun]

@ayush_tues would be really interesting to see if you could swap in limbs from completely different robots (and see if the original robots learn to use them).

i wonder what kind of frankenstein amalgamation you could end up with

[Agneya Tharun @AgneyaTharun]

@IshanShahh sf parties just got 10x better

[Agneya Tharun @AgneyaTharun]

@sharziki thanks man :)

[Agneya Tharun @AgneyaTharun]

I trained a robot to fight me!

According to the scoreboard I (somehow) lost

lmk if I should open source the code and model weights, perhaps you guys can fight better than I can

[Agneya Tharun @AgneyaTharun]

OneJev-4B is a full fine-tune of Qwen3.5-4B, so they share the same neuron layout (which is why these results don't surprise me).

I reused the “evil Qwen” recipe: L14/N8949 +1, L13/N7956 −1 (relative units), applied at every token

[Agneya Tharun @AgneyaTharun]

Fun fact: boosting neuron 8949 in layer 14 and suppressing neuron 7956 in layer 13 of OneJev-4B makes it evil and racist!

[Agneya Tharun @AgneyaTharun]

I used a quest 2 in this demo, but it was clunky and annoying to work with tbh

I need something for AR/VR that's lightweight + something that can accurately track the positions of my limbs

[Agneya Tharun @AgneyaTharun]

@llaessss @gxjo_dev yes ive heard this too!

[Agneya Tharun @AgneyaTharun]

@fozba1 i pray for your back man

[Agneya Tharun @AgneyaTharun]

What if a decision model could replace the reward function in RL?

I trained two humanoids to walk from scratch. One used 12 hand-tuned rewards (things like staying upright, avoid jerky movements). The other used elephant-1 to judge how naturally it walked.

Results:

[Agneya Tharun @AgneyaTharun]

@s_george_iv @paolodiprodi hell yeah!! im curious - which adverbs tend to have the best results?

i think its really funny that this could turn RL into another "prompt tuning" problem

[Agneya Tharun @AgneyaTharun]

@eshaanissar maybe i should've told it to walk like a robot🥀

[Agneya Tharun @AgneyaTharun]

@montalvo_m40717 clearly we ALL have to walk that way now

[Agneya Tharun @AgneyaTharun]

@paolodiprodi i actually like this angle a lot - perhaps you could also directly ask the decider if the current reward policy is mitigating reward hacking?

but grading the reward policy itself (and how well of a job its doing) could be interesting

[Agneya Tharun @AgneyaTharun]

@sebify yeah the robot trained on the decider is screwed up

just wanted to share some interesting results :)