Festus

10X Builder | AI Performance Engineer | Co-founded @mybridgecard(YC S22)

[Festus @_enfinity]

I recently got a PR merged into @nvidia's Dynamo (A Datacenter Scale Distributed Inference Serving Framework) and the process was a great A/B test of KV data transfer over TCP vs KV data transfer over RDMA.

PR link: https://t.co/7TnzBl5CQz 🧵

[Festus @_enfinity]

As we push LLMs towards trillion-parameter scale and increasingly long agentic workloads, we face challenges along three major axes: model depth, model width, and context length.

While depth and width drive model size and compute, longer contexts introduce another challenge: attention complexity and KV-cache memory.

There have been two interesting approaches to this:

- @deepseek_ai’s Multi-head Latent Attention (MLA) compresses keys and values into a shared low-rank latent representation, significantly reducing KV-cache memory.

- @Kimi_Moonshot’s Kimi Delta Attention (KDA) rethinks attention itself, using recurrent state updates to achieve linear rather than quadratic scaling with sequence length in its attention layers.

This video by @baseten does a deep dive into the latter using Kimi K3 as a case study.

Really enjoyed watching it and absolutely recommend!

https://t.co/Dpbr5SS62h

[Festus @_enfinity]

Huge!!!

[Tianyi Cui @tianyi]

DeepSeek just open sourced its high-performance, large-scale inference framework DeepInfer in Apache-2.0 license.

https://t.co/W2cfi6O9LO

Supported features:

- Optimized kernels, model and sampler runtime
- Parallelism: Attention DP / balanced CP, MoE EP + EPLB
- DSpark and confidence-scheduled verification
- SWA bounded replay
- Asynchronous scheduling (zero-overhead batch scheduler)
- Batch-invariant inference
- Deterministic sampling
- Structured output

[Festus @_enfinity]

Rust can actually be quite expressive… 🦀

[Festus @_enfinity]

This is one of the interesting things about LLM systems engineering. A lot of the problems we are solving today have roots in problems computer science has been solving for decades.

Exciting!

[Festus @_enfinity]

One thing I've found really interesting about efficient LLM serving is how many of the optimizations we use today build on ideas that have existed in computer science for decades.

Some examples:
1. PagedAttention and Virtual Memory
2. Pipeline Parallelism and CPU Instruction Pipelining
3. Speculative Decoding and Speculative Execution

🧵

[Festus @_enfinity]

A smaller draft model generates several candidate tokens, and the larger target model verifies them in a single forward pass.

If enough of those tokens are accepted, we can advance multiple tokens without repeatedly running the expensive target model for each one.

[Festus @_enfinity]

We often talk about scaling AI infrastructure as if it just means adding more GPUs.

This conversation between @dylan522p and @sorcer, CTO of @CoreWeave, was a great reminder of how much more there is underneath.

At large bare-metal scale, you’re thinking about hardware failures and automated recovery, security and isolation, networking, power infrastructure, and liquid cooling.

I loved the discussion around how CoreWeave approaches full liquid cooling and how former crypto-mining sites have become valuable for AI infrastructure because much of the hard electrical work is already there: large grid connections, substations and power distribution.

Really great conversation on what it actually takes to operate AI infrastructure at scale.

https://t.co/gAYmZGg9eB

[Festus @_enfinity]

@AgenttDefi @Ifihan_ @sysconf_ team should have it on YouTube soon.

[Festus @_enfinity]

@sysconf_ Thank you for having me!

[Festus @_enfinity]

@JordanNanos 😂😂

[Festus @_enfinity]

@maharshii It’s a movie but.

https://t.co/pL9QsmnN7r