Shaowen Wang

Phd in Tsinghua University.

[Kaifeng Lyu @vfleaking]

Check out our new paper: One Period Is Often Enough for RoPE!

[Guancheng Du @ducatiqwq]

RoPE encodes positional information by rotating query and key vectors based on their positions, with each RoPE channel contributing a cosine function of the relative position.

But is this periodicity really beneficial?

Introducing our new paper — WavePrune: One Period Is Often Enough for RoPE!

[Shaowen Wang @wangsw5653]

@alexkrstern the unlooped depth 18 is narrrower so that the per token flops is kept

[Shaowen Wang @wangsw5653]

Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched?

We introduce SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers.

The answer is yes. And the advantage grows with scale. 🧵 1/8
Paper: https://t.co/CElIjvFqJS

[Ziniu Li @ZiniuLi]

What if you had 100, 1,000, or even 100,000 GPUs to scale LLM RL—what would you scale first? 🤔

One of the most fundamental knobs is batch size. But how should it grow with available compute—and where should it stop?

This may sound like hyperparameter tuning. At scale, however, it becomes a question of sample efficiency, hardware utilization, and ultimately the GPU-hours required to reach a target capability.

Answering it requires a principled understanding of RL scaling. So what is the right batch-size scaling law for LLM RL?

I’m excited to share my recent research paper on this topic.

#LLM #RL #Scaling

[Benhao Huang @huskydogewoof]

The looped-model four-quadrant chart

[Shaowen Wang @wangsw5653]

@stalkermustang We'd like to bring attention to our paper, which scales looped transformers up to 54B. Looping indeed scales better if you match the parameters.
https://t.co/CElIjvFqJS

[alphaXiv @askalphaxiv]

“HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?”

Most agent benchmarks fix the harness and only evaluate the model.

So this paper explores whether LLMs can build and improve the harness itself, including the execution loop, tools, context, recovery, and verification.

They showed that they can already create competitive harnesses from scratch, but self-improvement is still brittle, with gains often failing to generalize across unseen tasks or different executor models.

https://t.co/CcIPATPnLP

[Ge Zhang @GeZhang86038849]

0/n
# Toward Human-Like RSI

Blog / project: https://t.co/FjwP5Pt6ZL

Recursive self-improvement (RSI) is often pictured as a model repeatedly rewriting its own weights against a fixed objective. Humans improve differently: we decide what “better” means when the goal is vague, test ourselves, judge what happened, and change both what we know and the tools we work with.

This three-paper series turns those parts of improvement into controlled, auditable research questions:
ASPIRE studies how agents decide *what* to learn from a vague goal;
S³Gym studies how they test, judge, and learn from noisy self-generated experience;
HarnessDev studies how they build and evolve the external systems that shape their behavior.

These benchmarks use hidden evaluation as a proxy for whether a change transfers beyond the agent's own local signals - they do not claim to demonstrate open-ended RSI.

[Shaowen Wang @wangsw5653]

@_ryantlee Thanks for pointing this out! We’ll add your work as a row in Table 1 in our next revision.

[Shaowen Wang @wangsw5653]

@hayden_prairie @huskydogewoof Parcae Sec. 5.2 holds total training FLOPs fixed by reducing training tokens as recurrence increases, while, as you note, per-token/inference FLOPs and KV increase. We agree the “FLOPs” header is ambiguous, so we’ll revise it to “FLOPs/token” and clarify the entry.

[Shaowen Wang @wangsw5653]

@hayden_prairie @huskydogewoof Thanks for raising this point! As stated explicitly in our abstract and consistently defined throughout the paper, our matched budgets are per-token FLOPs, total non-embedding parameters, and KV cache. This is what Table 1 was intended to summarize.

[Shaowen Wang @wangsw5653]

Looking ahead, we hope this fair-comparison framework and the SMELT recipe can offer a useful setting and baseline for the looped-model community, and a reference point for exploring more looped architecture variants.

[Shaowen Wang @wangsw5653]

Across pretraining data, visit 2 weakens the attention sink throughout the repeated block, reversing the Baseline's depth trend.

Dyck makes this concrete: visit 1 mostly attends to the BOS sink, while visit 2 shifts attention to answer-bearing demonstrations. 8/8