Oli Wilkins

Senior AI engineer in fintech. CUDA enjoyer. LLM infra on K8s. Kool Nix and Neovim (btw) ricer 😎.

[Oli Wilkins @okwilkins]

@lauriewired It is a shame! This past year I've had the pleasure of going deeper into GPU architecture.

The joy that comes from finally understanding WHY things are the way they are in my field, ML/AI, can not be matched!

[Oli Wilkins @okwilkins]

@maxxfuu Get after it! Things will start to click soon after this chapter!

[Oli Wilkins @okwilkins]

@kalomaze The world before @charliermarsh

[Oli Wilkins @okwilkins]

@maxxfuu The vLLM blog post on this is a great read too! It very nicely explains how most of its core features function.

https://t.co/bkXZfF5WIr

[Oli Wilkins @okwilkins]

Great stuff from @MainzOnX again! The concepts here are really key to understanding WHY things like speculative decode or batching sizes for LLM inference are so important for performance.

[Adam Mainz @MainzOnX]

https://t.co/NsC1oQAA1l

[Oli Wilkins @okwilkins]

@MainzOnX Cool! Looking forward to it, they are a great way on brushing the basics again

[Oli Wilkins @okwilkins]

@eliebakouch Why are Thinking Machines breathing the same air as OpenAI/Anthropic. They probably shouldn't be in their office, if you ask me.

[Oli Wilkins @okwilkins]

PSA: If you’re running agentic workloads and seeing much lower prefix cache hit rates than expected, check whether GPU memory pressure is causing your cache to thrash.

Long shared prefixes common in agent loops get evicted quickly when the KV pool is too small. @vllm_project’s native CPU offloading can help by spilling blocks to host memory, which often beats recomputing heavy prefills. It’s not free but it’s worth trying when memory is the bottleneck on prefill-heavy agent traffic.

[Oli Wilkins @okwilkins]

@MainzOnX Hey you promised... and England did lose...

[Oli Wilkins @okwilkins]

Microsoft Teams Copilot could replace @OpenRouter, @fal and @modal with their offerings at the point, we truly are living in the future! 😎

[Oli Wilkins @okwilkins]

@xeophon Just like how Grok 3 was supposed to be open sourced.

https://t.co/hfDQHBbi5z

[Elon Musk @elonmusk]

The @xAI Grok 2.5 model, which was our best model last year, is now open source.

Grok 3 will be made open source in about 6 months.

https://t.co/TXM0wyJKOh

[Oli Wilkins @okwilkins]

@kalomaze Would love for open data to be normalised also. It then can truly be called open source. @allen_ai leads the way here!

[Oli Wilkins @okwilkins]

@thdxr If my Grandmother had wheels she would have been a bike

[Oli Wilkins @okwilkins]

Congrats on the day-0 release! The difference between DFlash and MTP is crazy here!

[Modal @modal]

Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.

Running on Modal Auto Endpoints with SGLang today.

[Oli Wilkins @okwilkins]

@MainzOnX Ok if England lose, it better make me feel a bit better!

[Oli Wilkins @okwilkins]

@_EldarKurtic @RedHat_AI @NVIDIAAI Awesome thanks and congrats on this release!

Been keen to see the speed benefits of DFlash over EAGLE and MTP, because as you stated before, it should give a FLOPS speed up.

[Oli Wilkins @okwilkins]

@_EldarKurtic @RedHat_AI @NVIDIAAI I should add as well that the previous (deleted) post was showing the example on the right running at 20-30 tok/s faster than the left, at around ~210 tok/s for DFlash.