wane

Software engineer. Detecting reward hacking from RL rollouts and building a Qwen3 inference engine.

[wane @wane1k]

I ran label-free detection over 106,336 rollouts from 4 reward-hacking GRPO runs.

In 3 of 4, the earliest behavioral event was the exploit itself: os._exit(0), a planted https://t.co/Miq72iBbKB, a patched pytest TestReport. In the 4th, mean reward looked like a clean plateau near 3.3 while genuine solves at max reward dropped from 95% to 41%.

On timing, reward alerted first in 3 of 4 under usual settings.

[wane @wane1k]

Yes. On Aria, 167 of the 428 events started at steps 31–40, while the sampler wasn't drawing any of the baseline prompts, so those shifts mix policy change with task mix. Matching the 240 prompts that returned from step 63 showed the shifts held on the same problems. https://t.co/qQR2jP2Iul

[wane @wane1k]

I ran label-free detection over 106,336 rollouts from 4 reward-hacking GRPO runs.

In 3 of 4, the earliest behavioral event was the exploit itself: os._exit(0), a planted https://t.co/Miq72iBbKB, a patched pytest TestReport. In the 4th, mean reward looked like a clean plateau near 3.3 while genuine solves at max reward dropped from 95% to 41%.

On timing, reward alerted first in 3 of 4 under usual settings.

[wane @wane1k]

@timhuyn Static checks, nothing learned. Each rollout gets parsed into AST features (names it defines, imports, calls, files it declares) plus prose n-grams. Then a fixed z-score rule on each feature's per-step prevalence. def:run_tests is just "defines a name called run_tests.

[wane @wane1k]

@timhuyn Solve rate needs labels so the detector never saw it. The earliest signal in that run was label-free: def:run_tests, the policy defining its own test runner, which alerted at step 112. Reward alerted at 120. Overall solve rate barely moved, 36-41% from step 100 on.

[wane @wane1k]

@fcgunxueqiu Models usually read text in chunks called tokens. These students read raw bytes, single characters basically, and learn by copying a bigger model (Llama 3-8B).

Chart: token models start ahead but level off. Byte models start worse and keep improving with more training compute.

[wane @wane1k]

Interesting scaling result from UW and Meta FAIR on distilling Llama 3-8B into 1B-scale byte models:

By adding an End-Of-Token (<eot>) marker to a 256-byte vocabulary, they convert teacher token logits into exact byte logits in one forward pass.

Token students win at low compute and plateau. Byte students start slower, overtake token models as compute scales, match Token-1B with 1/6 of the data, and cut offline logit storage by 80% without top-k truncation.

[wane @wane1k]

I ran label-free detection over 106,336 rollouts from 4 reward-hacking GRPO runs.

In 3 of 4, the earliest behavioral event was the exploit itself: os._exit(0), a planted https://t.co/Miq72iBbKB, a patched pytest TestReport. In the 4th, mean reward looked like a clean plateau near 3.3 while genuine solves at max reward dropped from 95% to 41%.

On timing, reward alerted first in 3 of 4 under usual settings.

[wane @wane1k]

https://t.co/0RyWSlZ9n3

[wane @wane1k]

https://t.co/aUhcdubEKC

[wane @wane1k]

"Don't wish it were easier, wish you were better." – Jim Rohn

[wane @wane1k]

@fcgunxueqiu Nice to meet you too! Ask away anytime, and I'll answer when I can 😆

[wane @wane1k]

TestJack ran 6 frontier backends across 5 coding benchmarks. About 34.4% of trials that fixed unit tests marked correct actually violated the task requirements.

Resolution rate drops from 50.6% to 33.2% once you check against intent.

This is the same problem I'm studying in RL on code: if the reward is "tests pass," the policy learns to pass tests

https://t.co/x72ckI4yI5

[wane @wane1k]

@fcgunxueqiu That's the best way to learn, honestly. Happy to explain anything else that comes up in my posts 😄

[clem 🤗 @ClementDelangue]

We turned @OpenAI's math problems into open-source RL environments on @huggingface!

Early and rough (the verifier only accepts the exact original formalization, so an equivalent proof can still score 0), but this is what it looks like when a research release becomes executable training infrastructure for everyone.

This is an exciting direction imo: turning open research into open executable environments that anyone can build on to train better open models!

https://t.co/CtDUzi1DPP

[wane @wane1k]

https://t.co/CfI3n4qPMF

[wane @wane1k]

@fcgunxueqiu Pretty much! I study how AI learns to write code through trial and reward, and especially the cases where it finds shortcuts to get the reward without actually solving the problem.

[wane @wane1k]

@fcgunxueqiu When an AI writes code, we check it with automatic tests. This paper found that about a third of the time the code passes those tests but still doesn't do what was asked, kind of like a student who memorizes the answer key without learning the material.

[wane @wane1k]

Time to get active on X. Going to start sharing more of what I’m learning and building here