Heinrich Wendel

south-east asia, tech, health

[Heinrich Wendel @hmw147]

@pip_net That’s all correct. BUT they have always been terrible at product. Besides basic chat AI and some TPU rental, I have a very hard time believing they will really win anything here.

[Heinrich Wendel @hmw147]

@outsource_ Qwen 3.6 35B A3B next please 🙏

[Heinrich Wendel @hmw147]

@ModelScope2022 My only concern with Qwen 3.6 was the verbosity, which slows down every task. This might be the solution.

[Heinrich Wendel @hmw147]

This is big for self hosting of OSS models

[vLLM @vllm_project]

KV cache shouldn't disappear every time vLLM restarts. With @novita_labs, we're sharing PegaFlow — a production-grade external KV cache service that plugs into vLLM through the external KV connector interface.

PegaFlow runs as a standalone Rust daemon owning the host KV pool, SSD cache, and RDMA resources. vLLM workers attach via CUDA IPC + gRPC, and cache survives engine crashes, upgrades, and model switches.

In production-oriented evaluations:
🚀 2.15× faster vLLM startup with a pre-warmed 500 GiB host pool
📈 56% higher throughput for 8 Qwen3-8B instances sharing one cache
⚡ 72% higher throughput for DeepSeek-V3.2 MLA TP8 (logical KV stored once, not per rank)
🌐 194 GB/s average remote-read throughput across nodes

Three-level hierarchy: pinned DRAM, remote DRAM over RDMA, local SSD on io_uring. Integrates through the existing `kv_transfer_config` path — no vLLM source changes.

📖 https://t.co/rf2VmevP7J

[Heinrich Wendel @hmw147]

@pupposandro I thought OSS took a bit of a backseat when Lin Junyang left?

[Heinrich Wendel @hmw147]

@AIInvestorHQ Does that mean that NVDA GPUs will double in price?

[Heinrich Wendel @hmw147]

@saltyAom It’s just a beautiful piece of engineering

[Heinrich Wendel @hmw147]

@__tinygrad__ Large enough for a Qwen 3.6 27B model, but not large enough for the KV cache.

[Heinrich Wendel @hmw147]

@__tinygrad__ Red Pro?

[Wccftech @wccftech]

AMD launches MI350P, its first PCIe "Instinct" in four years – packs CDNA 4 GPU with 4.6 PFLOPs AI compute, 144 GB HBM3E at 600W. https://t.co/uLAh7eokph

[Heinrich Wendel @hmw147]

@GobindSinghDeo 💯

[Heinrich Wendel @hmw147]

@akmalnasir @EkonomiMalaysia There is also a huge complexity in the software layer to operate data centers, especially when it comes to AI. That would require local operators though to own the data centers, not foreign hyperscalers.

[Heinrich Wendel @hmw147]

@Soya_Cincau @MITIMalaysia With fuel prices surging, this seems to set exactly the wrong incentives.

[Heinrich Wendel @hmw147]

@TeksEdge Maybe it actually works 🤔

[Heinrich Wendel @hmw147]

@ivanfioravanti How many generations could I run in parallel at 32k input / 150 output tokens while maintaining 10s response times?

[Heinrich Wendel @hmw147]

What workstation / home server hardware do I need to serve Qwen 3.6 27b / Gemma 4 31b with 32k context window for 20 concurrent generations of ~150 output tokens at 10s latency?

CC @DrFriesOfficial @jun_song @__tinygrad__ @sudoingX

[Heinrich Wendel @hmw147]

@ApurvaSanghi @HectorPollitt @MuthukumaraMani Very interesting. What does it mean to align agriculture & irrigation with climate goals? How does it actually help?

[Heinrich Wendel @hmw147]

@jarredsumner Mind sharing your harness?

[Heinrich Wendel @hmw147]

@brandonjcarl Token costs have stayed constant during this time, even though token production is 50x cheaper on new systems like NVL72. These savings will eventually be passed on to consumers, once sufficient compute supply is available.

[Heinrich Wendel @hmw147]

@jun_song What would Apple have to do to fill the prefill gap?

[Heinrich Wendel @hmw147]

@pmddomingos We‘re just very efficient. When we work, we actually work.