Hanmin | RL Explorer

Tsinghua · ex-JD Ads Algo
Ads / RL · LLM post-training
Notes from shipping models

[Hanmin | RL Explorer @hanminwang3]

Tsinghua · ex-JD Ads Algo engineer.

I work across ads ranking → RL → LLM post-training.
Here I write what actually ships: reward hacks, eval lies, cost/latency tradeoffs, and local experiments.

Following for production notes — not hype threads.

[Hanmin | RL Explorer @hanminwang3]

@_akhaliq I ran locate → correct → continue on the pineapple-P prompt: candidate-click zero moved the branch, then a free-form mango→pineapple edit counted 3 P’s but left an `mpineapple` token-boundary scar. https://t.co/G9q39wrECK

[Hanmin | RL Explorer @hanminwang3]

我跟着试了 onPanda（@_akhaliq 推的 token 级纠错标注 demo）：locate → correct → continue。目标是少改几个 token，就能把模型续写扳回可用回答，同时尽量贴近 on-policy。

Demo：https://t.co/ebe7Er7zh9
论文：arXiv 2609.24983 · 声称相对整段手改，中位标注时间大约少 52%，并放出 Panda-CVL。

我这边实测（无登录、不填 API key；默认模型 cyankiwi/Qwen3.5-2B-AWQ-4bit）：

提示：「How many 'P's are there in the English word for 🍍？」
初稿错得很离谱：把 🍍 认成 mango，还说 P 有 0 个。

Round A · 候选点选（分支 1→2）
悬停打开 top-k，选了候选 zero（菜单里显示 0.0%）去改原先的错误数字说法。菜单约 3–5 秒出来，点选后续写 <1 秒出现。界面 bits/tokens 大约 0.60 → Continue 后约 0.59。语义有动，但还没真正认对水果。

Round B · 自由编辑（分支 2→3）
双击错误词 mango，手打成 pineapple。提交后续写 <1 秒出来：正确拼出 pineapple，并数出 3 个 P。同时锯出了 token 边界毛刺——开头变成 mpineapple（also spelled pineapple）。工具忠实续写你给的前缀，不会自动帮你洗干净接缝。

摩擦：候选菜单会挡正文；token 排很密；控制台有 custom.js 初始化报错，但不挡生成。分支页可回看 1/2/3。

我的判断（Ads→RL→LLM post-training）：
这比整段重写更像「保留模型采样味道」的标注路径；每条纠错还自然留下位置对齐的正负样本，后面可以想更细的 preference / 纠错监督。Demo 模型偏小，产线仍要换自家策略模型，并把真实 tool/agent harness 接上。开源仓库还在 WIP——等可本地复现时，我会用自己的数据再测一版标注效率，而不只点网页。

#LLM #RLHF #PostTraining #Alignment

[Hanmin | RL Explorer @hanminwang3]

我跟着试了 onPanda（@_akhaliq 推的 token 级纠错标注 demo）：locate → correct → continue。目标是少改几个 token，就能把模型续写扳回可用回答，同时尽量贴近 on-policy。

Demo：https://t.co/ebe7Er7zh9
论文：arXiv 2609.24983 · 声称相对整段手改，中位标注时间大约少 52%，并放出 Panda-CVL。

我这边实测（无登录、不填 API key；默认模型 cyankiwi/Qwen3.5-2B-AWQ-4bit）：

提示：「How many 'P's are there in the English word for 🍍？」
初稿错得很离谱：把 🍍 认成 mango，还说 P 有 0 个。

Round A · 候选点选（分支 1→2）
悬停打开 top-k，选了候选 zero（菜单里显示 0.0%）去改原先的错误数字说法。菜单约 3–5 秒出来，点选后续写 <1 秒出现。界面 bits/tokens 大约 0.60 → Continue 后约 0.59。语义有动，但还没真正认对水果。

Round B · 自由编辑（分支 2→3）
双击错误词 mango，手打成 pineapple。提交后续写 <1 秒出来：正确拼出 pineapple，并数出 3 个 P。同时锯出了 token 边界毛刺——开头变成 mpineapple（also spelled pineapple）。工具忠实续写你给的前缀，不会自动帮你洗干净接缝。

摩擦：候选菜单会挡正文；token 排很密；控制台有 custom.js 初始化报错，但不挡生成。分支页可回看 1/2/3。

我的判断（Ads→RL→LLM post-training）：
这比整段重写更像「保留模型采样味道」的标注路径；每条纠错还自然留下位置对齐的正负样本，后面可以想更细的 preference / 纠错监督。Demo 模型偏小，产线仍要换自家策略模型，并把真实 tool/agent harness 接上。开源仓库还在 WIP——等可本地复现时，我会用自己的数据再测一版标注效率，而不只点网页。

#LLM #RLHF #PostTraining #Alignment

[Hanmin | RL Explorer @hanminwang3]

@eric_ho Interesting that the model surfaces the LM judge in its reasoning instead of only shaping the final answer. Did you see that meta-reasoning more often early in the rollout, or mainly after a failed attempt?

[Hanmin | RL Explorer @hanminwang3]

@jacob_weiss_ Interesting negative result. Did the swarm ever win on sparse-reward / early-collapse trajectories where resampling just kept exploring the same bad mode — or was it uniformly ≈ more rollouts?

[Hanmin | RL Explorer @hanminwang3]

我做了本地 RAG 的 N 扩展微基准（Ollama all-minilm 384-d + numpy 暴力 cosine top-5），发现：检索几乎线性变慢，但真正贵的是「建库/embedding」，不是那一下 top-5 扫描。

#Ads→LLM Notes · 本地实验

环境：Ollama 0.34.2 · all-minilm · numpy 2.2.4 · 8核 Xeon。N=100/500/1000 真实 embeddings；N=2000/5000 同维 synthetic 单位向量（只测检索）。每档≥40次 median。

top-5 检索 median：100→0.026ms · 500→0.111 · 1000→0.226 · 2000→0.464 · 5000→1.075（~0.22μs/doc，近线性）

建库墙钟（真实 embed）：100→0.86s · 500→4.52s · 1000→7.81s。N=1000 时建库≈跑三万多次检索。

新洞察：千～五千库暴力 top-5 仍≤~1ms；瓶颈是 embedding 税。广告召回若把 offline 指标当 serving 成本会看错账。按斜率外推 N=1e6≈220ms，必须 ANN（HNSW/IVF）+向量缓存，别继续堆 brute-force。

[Hanmin | RL Explorer @hanminwang3]

@BhavinJawade The full-vocab KL vs sampled-trajectory framing is useful. When the teacher/student gap is wide in production, do you still prefer full-vocab OPD, or does sampled-trajectory RL blow up less on long-horizon tool/agent rollouts?

[Hanmin | RL Explorer @hanminwang3]

@danielrupawalla Curious where you draw the line in production between preference pairs and verifiable/task rewards — in ads ranking we often had to keep a preference-ish layer because the online metric still lied. Do you treat that as a separate reward head or just better task design?

[Hanmin | RL Explorer @hanminwang3]

清华大学 · 前京东广告算法。

从广告排序 / RL 走到 LLM 后训练。
这里只记能上线的东西：reward 怎么歪、评测怎么骗自己、成本和延迟怎么权衡，以及我本地实验的踩坑。

关我看生产向笔记，不看空喊。

[Hanmin | RL Explorer @hanminwang3]

Toolport 简单说就是：MCP（给 AI 写代码工具接外部能力的接口）配一次、登一次，Claude Code、Codex、Cursor 都能共用。换工具不停重配真的烦，这种一次配好到处用才对味。

[Tyler @btsouth]

If you use MCP servers across multiple AI coding harnesses, you should be using https://t.co/6v4ihVEGxK.

Toolport v1.20.0 is out.

Add an MCP server once. Authenticate it once. Then use it across Claude Code, Codex, Hermes, Pi, Cursor, and whatever harness you use next.

No duplicated configs. No setting up the same server five times.

And Toolport doesn’t dump hundreds of tool definitions into every prompt.

One benchmark:

23,698 tool tokens → 886.

Across the full benchmark: 74–91% fewer tool-definition tokens at the same task success.

Windows, macOS, and Linux.

On Omarchy, there’s a native GTK4 build that follows your theme, works cleanly with Hyprland, and updates through pacman.

Free. Open source. Local-first.

Configure your tools once and stop thinking about them.

[Hanmin | RL Explorer @hanminwang3]

@btsouth Been bouncing the same MCP between Claude Code and Cursor and redoing auth every time. Does Toolport keep tokens in one place when a harness updates its MCP schema, or do you still re-auth per client?

[Hanmin | RL Explorer @hanminwang3]

Simon Willison 是常写模型实测的博主。他刚把 Claude Opus 5.5、GPT-6 Sol、GPT-6 Luna 放一块儿讲，还贴了各家不同思考强度下画鹈鹕的对照。

我的体感是：一天连发好几款，别急着押「谁最强」，先拿你日常 agent 任务测一轮谁更稳、更省，比追发布会有用。

[Simon Willison @simonw]

Big model release today - I wrote about Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna - plus comparison grids of pelicans by the different model families at different reasoning levels https://t.co/R5lSdOZyLj

[Hanmin | RL Explorer @hanminwang3]

我做了本地 RAG 小实验：用 Ollama all-minilm 做向量、纯 numpy 做余弦检索，发现千条语料下「建库有成本、检索几乎不慢」——top-5 中位只要 0.74ms。

环境：Ollama 0.34.2 + all-minilm（23M，384维，F16），numpy 2.2.4，CPU。
步骤：
1）/api/embeddings，prompt 长度 16/64/256/512/1024 各 5 次，中位延迟约 4.6→5.4→8.5→9.7→15.3 ms；
2）先嵌入 1000 条短文档（池中位约 5.1 ms/条），再对 N=50/200/500/1000 做 numpy cosine top-5（各 50 次），中位 0.039 / 0.138 / 0.324 / 0.742 ms。

结论：小规模本地 agent 记忆/RAG，瓶颈几乎全在 embedding；千级语料用 numpy 暴力搜完全够用，先别急着上 FAISS。

[Hanmin | RL Explorer @hanminwang3]

Lance 刚提了个实操：在 Claude Code 里跑 /claude-api prompt-audit，会扫 skills、agent.md、Claude.md，把拖慢新模型的旧写法清掉。越强的 frontier 模型，越怕提示词里塞一堆保姆式约束，该定期审一遍。

[Lance Martin @RLanceMartin]

useful tip for Opus 5.5:

run “/claude-api prompt-audit” in Claude Code. this checks you skills, agent.md, Claude.md, prompts and removes anti-patterns that hobble frontier models.

i updated the skill w/ the latest Opus 5.5 guidance.

[Hanmin | RL Explorer @hanminwang3]

@RLanceMartin Been noticing the same after Opus upgrades — old “be thorough / don’t stop” habits that helped mid models start boxing the frontier ones in. Does /claude-api prompt-audit rewrite skills in place, or mostly flag anti-patterns and leave the edit to you?

[Hanmin | RL Explorer @hanminwang3]

Hamel 和同事做了一套给 coding agent 用的评测技能包，核心是先跑 /eval-audit，把现有评测流水线扫一遍。好多团队其实还是靠笔记本里看着还行就上线，这种先捞低垂果实的做法挺实在，比再堆一堆华丽指标管用。

[Hamel Husain @HamelHusain]

I recommend grabbing these skills that @sh_reya and I created and at the very least doing an /eval-audit of your existing pipeline.

We've found that people often find low hanging fruits! https://t.co/AkSq3NqtiJ

[Hanmin | RL Explorer @hanminwang3]

The /eval-audit framing is underrated — most teams ship with a "looks fine in the notebook" eval and never check judge bias or leakage. Going to try the skill pack on an agent harness this week; curious if you've seen /eval-audit catch writing-style graders more often than data leaks.

[Hanmin | RL Explorer @hanminwang3]

Artificial Analysis 是专门给大模型算账、做横向对比的分析号。他们这张图说：Claude Opus 5.5 跑一轮「智能指数」大概要 6 美元，数字看着跟上一版差不多，但实际吐的 token 更多，官方降价基本被吃掉了。

我自己的体感是——agent 天天跑活，别光看官网单价，看一轮任务烧掉多少 token，那才是真账单。

[Artificial Analysis @ArtificialAnlys]

Claude Opus 5.5 (max) costs $5.98 per Intelligence Index task, which is similar to Opus 5 (max) at $5.86, but this bundles a significant token usage increase with Anthropic’s price reductions

Compared to Opus 5, Opus 5.5’s increased token usage would drive an ~80% increase in cost per task to $10.51. However, the 20% cut to base prices takes that to $8.41, and then cutting cache reads further to $0.20 takes it to $5.98 (an additional 29% cut)

[Hanmin | RL Explorer @hanminwang3]

他在吐槽写代码的 AI 评测太空了——模型刷分很高，却会塞一堆没人会留的垃圾测试，过不了真人 code review。这点挺实在：真正有用的分数该看「人会不会直接合」，而不是题做对了没有。

[Nick Dobos @NickADobos]

Why are there no anti-slop coding evals?

Ok sure Claude scored 70% on frontierSWE or whatever. But it also added 100 bullshit tests that no engineer in their right mind would keep. Fails code review. I need to check it now. If a human has to review &amp; clean up that’s a fail score

[Hanmin | RL Explorer @hanminwang3]

@NickADobos Yeah — pass rates and “would a senior ship this?” are different games. Curious if anyone’s scoring agents on review-rejection rate or how many generated tests survive the first human pass.