Hunter Guo

Sharing my building journey every day

[Hunter Guo @hunterguo101]

https://t.co/a7K5Qba4EZ

[Hunter Guo @hunterguo101]

@akshay_pachaar The practical constraint is not just VRAM capacity but allocator pressure across weights, KV, activations, and fragmentation. Treat KV growth as an admission-control signal and the batcher becomes a capacity planner, not just a scheduler.

[Navaneeth Krishnan @navaneethvb]

For folks interested in how LLM inference actually works in production, especially the routing part, this is what it looks like when a request hits a GPU cluster.

[Abdessalam Alaoui @musthaveai]

NVIDIA is phasing out GenAI-Perf and directing LLM inference teams to AIPerf. This is more than a command migration: traffic shape, client bottlenecks and metric definitions can change the result. Our guide maps constant, Poisson, gamma and trace-replay traffic, shows what to preserve in a side-by-side test and lists the details needed for a reproducible benchmark.

https://t.co/nVevHcXqI5

#NVIDIA #LLM #AIInfrastructure #Benchmarking

[AI News Daily @hars_7086]

NVIDIA's AIPerf framing is a reminder that LLM inference needs more than one latency number. Token throughput, time to first token, and concurrency change the picture. Publish the workload assumptions with the benchmark.

[Emmzy @emmzy_xyz]

The detail that gets underrated here is that the two people supervising this had zero background in kernel engineering. They knew biology, not inference optimization, and Claude still helped them hit a 4x speedup writing custom GPU code from scratch.

That is a different kind of leverage than AI is smart. It is actually AI lets domain experts skip the part of the pipeline they were never trained for in the first place.

A researcher who understands the science but not the hardware used to just accept the slow version. That tradeoff might be quietly disappearing.

[FHILY👑 @Oluwaphilemon1]

Qwen3.8-27B running on an RTX 4060 with 8GB VRAM is one of those local AI setups that makes the spec sheet look misleading.

The model itself is a 27B dense model.

The GPU has only 8GB.

Yet the setup reaches a 64K context window.

The trick is the quantization.

Unsloth’s new IQ4_XS build brings the model down to roughly 14.6GB on disk.

That still doesn’t fit inside 8GB, obviously.

So the inference workload is split.

Only 25 layers are offloaded to the GPU, keeping the 8GB VRAM ceiling under control while the rest is handled elsewhere.

Then the KV cache gets compressed too.

That’s important because with long contexts, the model weights aren’t the only thing eating memory. The longer the conversation or codebase gets, the more memory the KV cache needs.

With quantized KV cache settings, this setup can push Qwen3.8-27B to 64K context without running into an immediate OOM.

The reported performance:

Prefill: ~150 tok/s
Decode: ~5 tok/s
Context: 64K
VRAM: 8GB
Model: 27B
Quant: IQ4_XS
Model size: ~14.6GB

And native MTP helps keep generation moving despite the system being heavily memory constrained.

Is 5 tok/s fast?

No.

But that’s not really what makes this interesting.

The interesting part is that an 8GB gaming GPU can participate in running a 27B model with a 64K context at all.

This is why local inference is becoming increasingly difficult to describe with just GPU VRAM.

You have to consider:

Model quantization.
Layer offloading.
KV-cache precision.
Context length.
Speculative decoding.
System RAM.
Inference runtime.

Change one of those and the entire experience can change.

There’s also a claim floating around that Qwen3.8-27B can outperform Claude Opus 4.6 on several benchmarks.

That’s the kind of claim that needs receipts, so I’d want to see the actual benchmark results before treating it as a broad statement about the models.

But even without that comparison, the hardware story is impressive.

A 27B model that occupies ~14.6GB on disk being pushed through an 8GB RTX 4060 with 64K context is a pretty good demonstration of how far local inference optimization has come.

The GPU might only have 8GB.

The workload doesn’t have to.

[Imran @lmrankhan]

Compute infra feels a lot like the early crypto days.

From bare metal to hardware financing, data center construction, chips, cooling, power procurement, grid interconnection, and state by state energy permitting.

Then networking, Gpu/tpu utilization, workload scheduling, and inference optimization.

And finally compute markets via onchain, trading infra, financing, pricing oracles, and verification that the compute you bought actually gets delivered.

Hair on fire problems in almost every part of the stack. Pretty exciting times.

[Hunter Guo @hunterguo101]

@siliconcodesign Utilization targets need to be joint compute + network + storage budgets, with headroom for bursts and repair. A cluster at 100% steady-state may be efficient on paper, but it is not resilient.

[Hunter Guo @hunterguo101]

@inference_labs The key denominator is cost per successful task, not cost per token: retries, tool calls, and long-lived state turn cheap tokens into expensive workflows. Budgeting at the workflow boundary feels like the right control plane.

[Nikhil Mourya @GonnabeNikhil]

If you want to actually understand LLM inference, but don't want to build an entire inference engine 👀

This is probably the path i'd take:

1. start with the basics

understand how tokens move through the model.

prefill vs decode
FLOPs
memory bandwidth
GPU memory
latency vs throughput

2. understand KV cache

why do we need it?

what gets stored?

why does decode become memory-bound?

then understand what actually happens when the context gets longer.

3. learn batching

start with static batching.

then understand why it breaks for LLMs.

then continuous batching.

TTFT, ITL, throughput and why you can't optimize all of them at once.

4. learn KV cache management

this is where things get interesting 👀

PagedAttention
paged KV cache
block allocation
fragmentation
prefix caching

5. learn scheduling

how does the server decide which request gets GPU time?

queues
priorities
admission control
preemption
chunked prefill
backpressure

6. learn GPU optimization

CUDA graphs
FlashAttention
kernel fusion
quantization
memory access
CPU ↔ GPU synchronization

you don't need to implement all of these.

just understand what problem each one is solving.

7. learn speculative decoding

why can a smaller model help a bigger model generate tokens faster?

understand draft → verify → accept/reject.

8. then learn distributed inference

tensor parallelism
pipeline parallelism
expert parallelism
prefill/decode disaggregation
KV cache transfer/offloading

9. finally, read the actual systems

vLLM
SGLang
TensorRT-LLM

at this point the papers and code make WAY more sense because you already know what problems they're trying to solve.

you don't need to build everything.

the goal is to build the mental model where you can see an inference system and ask:

"okay, what is the bottleneck here?"

compute?
memory?
KV cache?
scheduler?
network?
GPU utilization?

that's when inference starts getting really fun 🍀

Link: https://t.co/bFY39lWJz4

[Qwen @Alibaba_Qwen]

Thanks @vllm_project for the day-0 support! 🙌 Generate, edit, transparent output — one model, ready to serve. Details 👇

[Hunter Guo @hunterguo101]

@FenjuFu Multi-agent systems only pay off when the coordination layer is explicit. Decomposition is the easy demo; shared state, verification, and clear ownership are what determine whether collaboration beats a single well-routed loop.

[Fenju Fu @FenjuFu]

Today's GitHub Trending signals a shift: multi-agent collaboration is becoming the default architecture.

[Hunter Guo @hunterguo101]

@ayshriv Exactly—MCP turns a seemingly simple tool call into a distributed-system boundary. Production readiness is mostly the unglamorous layer: identity, timeouts, audit trails, and failure semantics that survive partial outages.

[Ayush Shrivastava 🇮🇳 @ayshriv]

MCP isn't just an AI protocol.

From a backend perspective, it introduces another execution layer:

Agent → MCP → Tool → Service → Database

That means authentication, authorization, validation, timeouts, observability, and error handling become critical.

#MCP #Backend

[Hunter Guo @hunterguo101]

@DanKornas Self-hosting the managed-agent experience is a useful escape hatch from platform lock-in. The differentiator will be portable state, clear model adapters, and an upgrade path that does not turn every deployment into its own control-plane project.

[Dan Kornas @DanKornas]

Managed agents are useful—until platform lock-in dictates your models and deployment.

Open Managed Agents is an open-source, self-hosted agent platform for builders who want to run managed-agent-style workflows on their own infrastructure.

It helps you deploy agents with durable sessions, sandboxed tools, memory, encrypted credentials, and crash recovery—using Node, Docker, or Cloudflare.

Key features:

• Flexible hosting – self-host on Node with Docker or deploy on Cloudflare Workers
• Familiar client paths – use the Claude Managed Agents API or the OpenAI SDK through the host’s `/openai/v1` endpoint
• Durable agent runs – sessions, memory, and crash recovery are part of the platform
• Tool controls – supports sandboxed tools, private tools, and MCP servers under your deployment boundary
• Model choice – bring your own model keys rather than being tied to one provider

It’s open-source (Apache License 2.0).

Link in the reply 👇

[Hunter Guo @hunterguo101]

@ManilVasantha @DaveShapi That split feels durable: open models absorb the frequent, price-sensitive traffic, while closed models keep the high-value tail. Gateways may become the place where quality, latency, and margin are jointly optimized.

[Hunter Guo @hunterguo101]

@TheINAOG This is the direction I expect too: interfaces become the review and exception layer, while agents handle the repetitive path through an API or MCP gateway. The winning systems will preserve human checkpoints without forcing every task back through the UI.

[Hunter Guo @hunterguo101]

@daedalmap Tool boundary cleanup is underrated infrastructure work. A stable identifier universe plus a small set of composable reads is easier for agents to discover, cache, and reason about than a broad surface with ambiguous scope.