Po🎏

Staff SWE | Builder
Prev: SWE @EthStorage @QuarkChain

[Po🎏 @pochenai]

@shubh6200 @philipkiely real nice interactive tutorial.
would be better if it's opensourced at github

[Po🎏 @pochenai]

🫡🫡

[Jeff Dean @JeffDean]

Oh look! My longtime friend and colleague @Sanjay_Ghemawat is now on Twitter! Give him a follow!

[Po🎏 @pochenai]

盛老板(SGLang创始人，母公司RadixArk)的���赞访谈:
- LLM方向正在从训练(可用性)->推理(更低的服务成本)->RL(模型定制化)方向转变，推理在逐渐成熟，RL则是兴起 的方向
- 我也挺好奇RadixArk要做的下一代AI是怎样的，没听到剧透有点发痒😂

believing is seeing

[Benhao Huang @huskydogewoof]

对话盛颖：xAI，Infra的浪漫，SGLang，开源，平权与“甄嬛传” https://t.co/d7EuwzgFno 来自 @YouTube

This talk is amazingly enjoyable!

[Po🎏 @pochenai]

顺便写了个简单的chatbot和gateway的grafana metric监控玩玩
0.5B模型相应速度还挺快，8G显卡的GPU都有100+token/s的吞吐（不过复杂问题还是解决不了，比如鸡兔同笼😂)

[Po🎏 @pochenai]

折腾了 3 小时,终于在一张 8G 显卡上把 SGLang 全链路跑通了: 2 个 Qwen2.5-0.5B-Instruct worker + sgl-model-gateway 做前置路由。

小显存也能玩多实例 + gateway 架构~

[Po🎏 @pochenai]

wait for the full receipe. gonna try it

[harrrshall @HarshalsinghCN]

introducing BarunLM (35m) 🤗

i pretrained the world's best small model under 100m parameters.

a 35m parameter language model that outperforms models over 6x larger, including LiquidAI's lfm2.5-230m, while training on a single H200 gpu.

the gains come from a better architecture. we took inspiration from deepseek, kimi, and other frontier open source models.

BarunLM combines hybrid local and global attention, grouped query attention to reduce memory, partial rope, gated attention, bounded swiglu, and residual selectors to improve how information flows through the network.

after pretraining, we further improved the model with rlvr (reinforcement learning with verifiable rewards) using deterministic verifiers and rloo, allowing the model to reinforce correct reasoning without relying on human preference labels.

[Po🎏 @pochenai]

不到半小时就深入浅出介绍了inference infra的前沿，怎么最大化利用GPU性能:
- SGLang的跨请求/机器KV cache复用的调度，降低显存利用
- Continous batching 批处理请求
- Disaggregated Prefill and decode: Prefill和decode分离
- 量化/投机采样
- Miles在RL中推理和训练更好结合的技术(这个还有待了解，理解不深)

邦华大佬有个采访也很赞: https://t.co/9szjSqwT7T

期待下期盛老板的访谈，想听听她如何预见AI infra的爆发，并做出了��么牛的开源技术

[Po🎏 @pochenai]

不到半小时就深入浅出介绍了inference infra的前沿，怎么最大化利用GPU性能:
- SGLang的跨请求/机器KV cache复用的调度，降低显存利用
- Continous batching 批处理请求
- Disaggregated Prefill and decode: Prefill和decode分离
- 量化/投机采样
- Miles在RL中推理和训练更好结合的技术(这个还有待了解，理解不深)

邦华大佬有个采访也很赞: https://t.co/9szjSqwT7T

期待下期盛老板的访谈，想听听她如何预见AI infra的爆发，并做出了这么牛的开源技术

[Po🎏 @pochenai]

One 8GB GPU, <100 lines of core code, one day — enough to go from zero to running the full SFT → DPO → GRPO post-training pipeline.

Along the way, a repro of RL's Razor (@pulkitology et al.): on the same new task, SFT degrades the model's general ability by 10–21%, on-policy RL only 0.3% — because RL updates close to the model's own distribution, learning without forgetting.

Then <$5 of a rented 48GB GPU reproduces a DeepSeek-R1 aha moment on a 3B model — in the spirit of TinyZero by @jiayi_pirate.

Full write-up and code👇
https://t.co/NsCeWHUTnh

[Po🎏 @pochenai]

一张 8GB 显卡、不到 100 行核心代码、一天时间，就能把 LLM 后训练 SFT → DPO → GRPO 从上手到跑通全走一遍。

顺手复现 RL's Razor：同样学会一个新任务，SFT 让模型通用能力掉 10–21%，on-policy RL 只掉 0.3%——因为 RL 贴着模型自身分布更新，学得会、又忘得少。

再花不到 $5 租张 48G 卡，还能在 3B 模型上跑出 DeepSeek-R1 的 aha moment。

全文 + 代码👇
https://t.co/OEZjf5TEJm

[Po🎏 @pochenai]

@aminkarbasi Cool. Curious how long it takes for the training on 8 H100 GPUs

[Po🎏 @pochenai]

@Xudong07452910 👍

[Po🎏 @pochenai]

TL;DR
LLM推理强度(reasoning effort)的调节旋钮主要是response length，主��有几种调节手段（不过各大厂商其实细节都没怎么公布，主要是作者猜的）:
- SFT中混合no-thinking/不同长度thinking的数据
- 把response effort设置到reward function对应的参数中

自动调节reasoning effort还是圣杯啊

不过有意思的问题是，感觉其实除了response length，我感觉也可以建一个critic model来预估还有多少juice可榨，然后把这个评价指标和reason effort关联起来？

[Sebastian Raschka @rasbt]

How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less?

I put together a “little” article explaining how these effort levels are implemented at inference time and during training.

[Po🎏 @pochenai]

Beyond response length, it may be useful to define a critic model that estimates how much remaining value the current reasoning state has in reaching the correct answer, and then use that reasoning effort to decide when to stop. For example, one could compare the critic values over the most recent (k) steps with those over the preceding (k) steps as a relative measure of reasoning progress.

[Po🎏 @pochenai]

最��一直在想 CoT 为什么 work。一开始直观想到的是两个常见解释：上下文更长了、能提取出更多知识，或者多做了几次 forward pass（算力更多）。

想了想，第一个是错的，第二个方��对但没说到点上。真正说得通的是计算复杂度这条线。
- 先说上下文：固定 L 层的 Transformer，一次前向传播就只有 L 层深度。而且 prefill 是并行的——prompt 再长也只是加宽度，不加深度。塞再多东西，还是锁死在 TC⁰（电路复杂度类）里。其实早期CoT(Chain-of-Thought Prompting Elicits Reasoning in Large Language Models)指的就是这个意思，但是不work

- 再说 forward pass：方向是对的，但光「多做几次」不够。Let's Think Dot by Dot:Hidden Computation in Transformer Language Models这篇paper发现，无意义的「......」填充符也能提升某些任务，但论文明确说填充符再多也留在 TC⁰ 内——同样是多次 forward pass，一个突破了、一个没有。

差别在于写回的 token 里��了什么。CoT 把真实算出的中间结果写回输入流，下一步从第 0 层重新进入 → 深度预算重置 → T×L → 这才真正跳出 TC⁰。

不过复杂度只决定了天花板。还得靠 RL 用「答案对不对」这个信号，把模型推到真正会用这块草稿纸的地方——复杂度定天花板，RL 决定你够不够得到这个天花板。

[Po🎏 @pochenai]

@XShude 相同的体感+不能退款。。

[Po🎏 @pochenai]

@flyq_crypto @Kimi_Moonshot 确实claude code cli的体验是独一档

[Po🎏 @pochenai]

都在吹kimi v3, 买了个年度会员，但是发现kimi code的相对于claude vscode插件使用体感还是差距很大
- 配置auto edit/thinking强度之类的不直观
- 然后没干一会儿任务就把上下文占了好多

然后用了不到10分钟，想申请退款，还不给退钱，只能在线维权...

@Kimi_Moonshot