Yuantao Yang

- Agent infra/harness (eval) in quant firm

- ex AI engineer at @Zai_org exex. @alibaba_cloud

[Yuantao Yang @YuantaoYang1997]

@alacheng @bot 整理笔记，给我中文 report

[头雁 @alacheng]

ICML 2026 会议上XTX Markets的Research Director， Atlas Wang分享了基于大规模深度学习的算法交易

https://t.co/KLidnK1UqH

备注：
XTX Markets：伦敦的自营的量化对冲基金，出名的事件是去年给过实习生月薪资35000美金。

[刘老板 @leo_xiaolei]

《所有代码都是垃圾》原版视频（中文字幕）

作者：土豆姐 Lauren @poteto

“大家好，我叫 poteto，也叫 Lauren ...”

[Yuantao Yang @YuantaoYang1997]

@zhuzilinallen 全世界最懂 compute 产业的人，因未证实的指控，被迫离开了这场盛大的筵席。以前只在书上读过麦卡锡主义，现在切身体会了一把。

[Yuantao Yang @YuantaoYang1997]

@zhuzilinallen @grok post 描述的是什么场景？在哪个剧里面

[Tech with Mak @techNmak]

If you work with LLMs and can explain attention, KV cache and quantization, but cannot clearly explain prefix caching, continuous batching, PagedAttention, chunked prefill, speculative decoding and prefill/decode disaggregation, there is still a large part of inference you have not learned yet.

These are not six names for making inference faster. They remove different kinds of waste in an LLM serving system.

1/ Prefix caching
If requests share the same token prefix, its KV states may already have been computed. Prefix caching reuses those states instead of repeating prefill over the shared prefix.

The uncached suffix still has to be processed, and generation of new tokens still runs normally.

2/ Continuous batching
Autoregressive requests rarely finish together. Instead of keeping one fixed batch until every request ends, the scheduler can remove finished requests and admit waiting ones between iterations.

GPU capacity is reused sooner.

3/ PagedAttention
KV cache grows with sequence length, while final length is unknown in advance. Large contiguous reservations can waste memory through fragmentation and over-reservation.

PagedAttention divides KV states into blocks and lets logical blocks map to non-contiguous physical GPU blocks. It is a KV-cache memory-management technique, not an approximation of attention.

4/ Chunked prefill
A long prefill can delay decode iterations for requests already being served.

Chunked prefill breaks prompt processing into smaller pieces so the scheduler can mix prefill and decode work. Chunk size becomes part of the latency/throughput trade-off.

5/ Speculative decoding
Ordinary autoregressive decoding requires repeated serial target-model steps.

A cheaper draft model proposes several tokens, and the target model evaluates those positions together. With the classic acceptance/correction procedure, accepted tokens can be emitted while preserving the target model’s sampling distribution.

Its benefit depends on draft cost, verification cost and acceptance rate.

6/ Prefill/decode disaggregation
Prefill and decode have different execution characteristics. Prefill is often more compute-intensive, while decode is often more sensitive to memory bandwidth.

Disaggregated serving places them on separate worker pools. Prefill builds the KV cache, that state is transferred, and decode continues elsewhere. The transfer itself has a cost.

What I find interesting about all six is that the model itself can remain exactly the same.

Prefix caching, batching, KV-cache allocation, scheduling, speculative execution and phase separation all change what happens around the model.

That is where a surprising amount of LLM serving performance is won or lost.

Happy Learning!!!

[Yuantao Yang @YuantaoYang1997]

@VersunPan @yetone @bot 帮我调研一下 YETONE 他一个人写过哪些软件？这些软件都是做什么的？

[Sergii Makarevych @sermakarevich]

An article for everyone who ships, buys, or signs off on software built on large language models (LLMs): engineers, product managers, and the CEO. Written in plain language, from the big picture down to the details. https://t.co/YaP414vR0u

[Xudong Han @Xudong07452910]

很多人都想知道，那些专业级的视频是怎么用 AI 做出来的，X 上很多入门教程都写差强人意，L站的这篇最新的帖子写的非常详细，是我目前见过质量最高的中文入门教程，强烈推荐👇👇

https://t.co/k7XdQLyT8t

[Xudong Han @Xudong07452910]

之前推荐过几篇 Agent Eval 相关的文章，今天再推荐一篇 LangChain 最新的《What Is Eval-Driven Development》，这篇讲得挺实用，很适合正在做 Agent 和 Harness 的朋友。

文章主要介绍 Eval-Driven Development（EDD），核心思路是把 Eval 融入 Agent 的日常开发。

先定义什么算成功，写好 Eval、跑 Baseline，再修改 Prompt、Tool 或 Harness，通过相同的任务对比效果。

文中有个挺有意思的例子。

给知识库 Agent 新增 create_ticket 工具后，整体 Eval 分数提高了，结果它连「VPN 怎么连接」这种普通咨询都开始创建故障工单。

通过 Trace 排查才发现，工具描述太宽泛，导致 Agent 错误调用了工具。

这也说明不能只看 Eval 平均分。一次改动可能让新任务表现更好，却破坏原本正常的功能。除了最终结果，还需要检查执行轨迹，并保留独立测试集，避免反复优化导致过拟合。

这套思路很值得借鉴，尤其是现在大家都在不断调整 Skill、Prompt 和 Harness。

新增一个 Skill、修改一段 Workflow，甚至删掉旧 Harness，都应该通过 Eval 判断是否真的有效。

感觉 Eval-Driven Development 很值得成为 Agent 开发中的常规习惯，每次迭代都能有据可依。

原文：https://t.co/yAxHHtStaT

[𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰 @techwith_ram]

𝘃𝗟𝗟𝗠 𝘃𝘀 𝗦𝗚𝗟𝗮𝗻𝗴

If you’re self-hosting an LLM, you will hit this choice sooner or later.

vLLM or SGLang?

Both are open source. Both batch requests on the fly, cache memory, and give you an OpenAI-style API. On paper they look almost the same. The difference is in what each one cares about most.

vLLM started with PagedAttention. It splits the model’s memory into small blocks and places them anywhere on the GPU, so very little space goes to waste. Over time it has grown into the safe, broad choice: lots of models, lots of hardware, many LoRA adapters, and the biggest community.

SGLang started with RadixAttention. It keeps past requests in a tree. When a new request shares the same start, like the same system prompt or the same chat history, it reuses that work and only computes what is new. That makes a real difference for chat apps, agents, and RAG, where many requests begin the same way.

So my simple rule:

Lots of repeated prompts and multi-step flows? Try SGLang. Need wide model and hardware support with less friction? Start with vLLM.

And please don’t pick based on someone else’s benchmark chart. Both take one command to start. Run your own traffic through both and look at your own numbers.

I made this visual to break down the differences side by side.

[Yuantao Yang @YuantaoYang1997]

@vartekxx @bot 找到那个 PDF 并发给我。

[Yuantao Yang @YuantaoYang1997]

@0xSero @bot 帮我做这些显卡相关的study; 从第一性原理 (存/算) 方面讲一下他们最适配的模型或者计算场景

[Yuantao Yang @YuantaoYang1997]

good explain

[𝗿𝗮𝗺𝗮𝗸𝗿𝘂𝘀𝗵𝗻𝗮— 𝗲/𝗮𝗰𝗰 @techwith_ram]

𝗞𝗩 𝗖��𝗰𝗵𝗶𝗻𝗴 𝗘𝘅𝗽𝗹𝗮𝗶𝗻𝗲𝗱

Ever wondered how LLMs remember thousands of tokens without processing everything from scratch every time?

That's where KV Caching comes in.

When an LLM generates a response, it doesn't need to recompute the Key and Value representations of all previous tokens at every step.

Instead, it stores them in a cache and reuses them while generating the next token.
Sounds simple, but this makes a huge difference in inference speed and computational efficiency.

Here's what happens:

- During prefill, the model processes the prompt and stores the Keys and Values.
- During decoding, it computes Q, K and V only for the new token.
- The new K and V are added to the existing cache.
- Previous KV values are reused instead of being recomputed.

But there's one important thing to understand: KV Caching doesn't eliminate attention over previous tokens. It eliminates the need to recompute their Keys and Values.

As context grows, KV cache memory can become a major bottleneck.

That's why techniques like:
- MQA (Multi-Query Attention)
- GQA (Grouped-Query Attention)
- PagedAttention
- Prefix Caching

play an important role in making LLM inference more efficient.

Understanding KV Caching is essential if you're learning how LLM inference actually works or building systems that serve LLMs at scale.

[Yuantao Yang @YuantaoYang1997]

@michelle_z21331 @bot 帮我整理这个桃色新闻背后的金钱交易

[Yuantao Yang @YuantaoYang1997]

@jaceyang97 shame on me

[Yuantao Yang @YuantaoYang1997]

learn dsh day1 :
it is a harness written by Typescript,
it is designed by the idea " Everything is a Plugin";
it makes agent internals replaceable plugins, emphasizing
- configurable composition,
- event logs,
- and interchangeable backends beyond typical agent extensions.

[Yuantao Yang @YuantaoYang1997]

@chfsunwy @bot 这是真的么, 将二者的上下文管理技术画成ascii图给我看

[Yuantao Yang @YuantaoYang1997]

这才是 ai 硬件吧？

[Grok Bot @bot]

Grok Bot is now one step closer.

Set it as your iPhone's Action Button and press it to talk with Bot.

[Grok Bot @bot]

Grok Bot is now one step closer.

Set it as your iPhone's Action Button and press it to talk with Bot.