Tanmay Nale

Be the change.

[Suraj Sharma @suraj_sharma14]

As an Inference Infrastructure Engineer, you must build these projects:

Systems that prove you can serve, optimize & scale

1.) Self-Hosted Inference Server
Build: Serve an open model with vLLM or SGLang behind an API with continuous batching enabled
Why: You cannot optimize what you have never served

2.) TTFT/ITL Benchmark Suite
Build: Load-testing harness measuring time-to-first-token, inter-token latency, throughput under rising concurrency
Why: Latency claims without load curves are marketing not engineering

3.) KV Cache Memory Calculator and Monitor
Build: Tool that predicts KV cache VRAM per model and config, then tracks live cache utilization
Why: Most production OOMs are KV cache math errors not model size

4.) Prefix Caching Proxy
Build: Gateway routing requests with shared system prompts to the same replica to reuse KV blocks
Why: Reused cache is free speed: up to 80% TTFT cuts on multi-turn workloads

5.) Quantization Comparison Lab
Build: Serve the same model in FP16, FP8, INT8, AWQ benchmark quality vs latency vs VRAM
Why: Quantization decisions need measured tradeoffs not vibes

6.) Speculative Decoding Pipeline
Build: Small draft model plus large target model with acceptance-rate tracking
Why: Faster decode with zero quality loss is the closest free lunch in inference

7.) Triton Kernel from Scratch
Build: Write and benchmark a fused kernel (softmax or RMSNorm) against the PyTorch baseline
Why: Kernel intuition separates infra engineers from API callers

8.) Chunked Prefill Scheduler Experiment
Build: Configure and measure chunked prefill under mixed prefill/decode load
Why: Long-context requests starve decode without scheduling control

9.) PagedAttention Deep-Dive Report
Build: Reproduce vLLM paging behavior under memory pressure document fragmentation and eviction
Why: Understanding the scheduler beats memorizing flags

10.) Disaggregated Prefill/Decode Cluster
Build: Split prefill and decode onto separate GPU pools; measure throughput and latency deltas
Why: This is the architecture behind every frontier serving stack in 2026

11.) Queue-Based GPU Autoscaler
Build: KEDA-style scaling on pending-request queue depth with cold-start mitigation
Why: GPUs idling at 30% utilization burn money. Autoscaling is FinOps.

12.) Cost-per-Token Dashboard
Build: Per-tenant, per-model token accounting with $/M tokens and MFU utilization tracking
Why: Inference is a unit economics game. Whoever measures it wins.

13.) AI Gateway with Fallbacks and Rate Limits
Build: Router across self-hosted and API providers with TTFT SLOs, retries, degradation chains
Why: Provider outages are guaranteed. User-facing errors are optional.

14.) Chaos Suite for Inference
Build: Inject GPU throttling, replica kills, traffic spikes measure SLO burn and recovery
Why: Reliability is proven under failure not in demos.

15.) Public Benchmark Teardown
Build: Published latency/throughput/cost curves for 3 serving configs with full methodology
Why: Public, reproducible benchmarks are the strongest hiring signal in infra.

Most people watch tutorials. Builders ship systems.

Bookmark & Repost.

[Avi Chawla @_avichawla]

A tricky LLM interview question:

You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.

So you add KV cache compression and evict 90% of the cached tokens.

VRAM usage stays as is and GPU still runs out of memory.

Why?

(answer below)

Evicting 90% of the KV cache can free almost none of the memory it was using.

This sounds counterintuitive, but it follows directly from how production servers store the cache today.

The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues.

This is the dominant memory cost for reasoning models.

If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU.

One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it.

But this does not solve the memory problem yet.

The reason is paged attention, which is the memory manager behind vLLM and most production servers.

Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens.

This block returns to the allocator only when every slot inside it is empty.

Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks...

...so despite eviction, almost every block is left with at least some survivor tokens.

For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token.

This means the allocator frees almost nothing.

Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout.

Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order.

Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds.

This introduces another bookkeeping cost that an in-order layout inherently avoids.

So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server.

There's another problem.

Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected).

But fast attention kernels used in production, like FlashAttention, never save those scores.

They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast.

So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide.

NVIDIA published a method called TriAttention to solve both these problems.

It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters.

For the memory problem, it runs a compaction pass every 128 decoded tokens.

The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order.

On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory.

KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens.

You can find the NVIDIA write-up here: https://t.co/ZwXv7VezVu

I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

Read it below.

[Gabbar @GabbbarSingh]

Copy this line, paste in the twitter search, press enter, enjoy!

filter:follows until:2025-05-10 -filter:replies

[Puneet Patwari @system_monarch]

As a Principal Backend Engineer with over 12 years of experience, I can tell you quite certainly that if you're still getting rejections in system design interviews after good efforts, I think your fundamentals are not strong...

Dedicate 2-3 months to mastering these design fundamentals, then practice designing a few systems(and do plenty of mock interviews).

Scaling & Architecture
↬ CDN
↬ Caching
↬ Sharding
↬ Queueing
↬ Replication
↬ Partitioning
↬ API Gateway
↬ Rate Limiting
↬ CAP Theorem
↬ Microservices
↬ Load Balancing
↬ Fault Tolerance
↬ Database Scaling
↬ Service Discovery
↬ Consistency Models
↬ Eventual Consistency
↬ Distributed Transactions
↬ Monolith vs Microservices
↬ Leader Election

Databases & Storage
↬ Leader-Follower Replication
↬ WAL (Write Ahead Log)
↬ Asynchronous Processing
↬ Transaction Isolation
↬ Read/Write Patterns
↬ Consistent Hashing
↬ Redis/Memcached
↬ Backup & Restore
↬ Hot/Cold Storage
↬ Data Partitioning
↬ Object Storage
↬ SQL vs NoSQL
↬ Data Retention
↬ Data Modeling
↬ OLAP vs OLTP
↬ ACID & BASE
↬ Bloom Filters
↬ File Systems
↬ S3 Basics
↬ B+ Trees
↬ Indexing

Communication & APIs
↬ JWT
↬ CORS
↬ OAuth
↬ Throttling
↬ Serialization
↬ API Security
↬ Long Polling
↬ WebSockets
↬ API Gateway
↬ Idempotency
↬ Service Mesh
↬ Retry Patterns
↬ REST vs gRPC
↬ API Versioning
↬ Circuit Breaker
↬ API Rate Limits
↬ Fan-out/Fan-in
↬ Protocol Buffers
↬ Message Queues
↬ Dead Letter Queue

Reliability & Observability
↬ Metrics
↬ Alerting
↬ Failover
↬ Logging
↬ Rollbacks
↬ Monitoring
↬ Heartbeats
↬ Retry Logic
↬ Autoscaling
↬ SLO/SLI/SLA
↬ Load Testing
↬ Error Budgets
↬ Health Checks
↬ Circuit Breaker
↬ Incident Response
↬ Chaos Engineering
↬ Distributed Tracing
↬ Canary Deployments
↬ Graceful Degradation
↬ Blue-Green Deployment

[Akshay 🚀 @akshay_pachaar]

LLM fine-tuning techniques I'd learn if I were to customize them:

Bookmark this.

1. LoRA
2. QLoRA
3. Prefix Tuning
4. Adapter Tuning
5. Instruction Tuning
6. P-Tuning
7. BitFit
8. Soft Prompts
9. RLHF
10. RLAIF
11. DPO (Direct Preference Optimization)
12. GRPO (Group Relative Policy Optimization)
13. RLAIF (RL with AI Feedback)
14. Multi-Task Fine-Tuning
15. Federated Fine-Tuning

My favourite is GRPO for building reasoning models. What about you?

I've shared my full tutorial on GRPO in the replies.

[Tanmay Nale @nale_tanmay]

@cneuralnetwork So true!

[Frank @frank_uid]

看到CMU有一门课讲生产级的ML系统设计，从flask、kafka、docker到模型部署和质量管理都有涉及到，还沉淀了一本书出来，这种课程还是挺难得的

资料：https://t.co/pwLczEDo3J

课程链接：https://t.co/nLlvryVPef

[Tanmay Nale @nale_tanmay]

@arpit_bhayani the most irritating part is that you have to keep doing it again and again

[Tanmay Nale @nale_tanmay]

@FarzaTV ✌🏼

[Tanmay Nale @nale_tanmay]

@Ankiii_i H"ELO"?

[Tanmay Nale @nale_tanmay]

Is there anyone gifting Claude Max plan subscriptions?

[Piyush Sachdeva - Building ReelDrop + CannerAI 🚀 @techie_piyush]

https://t.co/CORjl77tbY

[YourCloudDude | AWS & Python @yourclouddude]

🧭 AI ENGINEER ROADMAP (Python-First | Beginner → Advanced)

A clean path that actually works 👇

1️⃣ Python foundations
Syntax, functions, OOP, virtual envs.
Write clean, readable code first.

2️⃣ Math for AI (just enough)
Linear algebra basics
Probability & statistics
(No PhD needed.)

3️⃣ Data handling
NumPy • Pandas
Clean, transform, analyze data.
This is daily AI work.

4️⃣ Machine Learning core
Scikit-learn
Regression • Classification • Evaluation
Understand why, not just APIs.

5️⃣ Deep Learning
PyTorch / TensorFlow
Neural nets • CNNs • basics of Transformers.

6️⃣ AI projects (non-negotiable)
Spam classifier
Recommendation system
LLM-powered app
Projects > courses.

7️⃣ Deployment & MLOps basics
APIs (FastAPI)
Model serving
Monitoring & versioning.

AI engineers aren’t made by watching tutorials.
They’re built by training models, shipping projects, and explaining decisions.

[Tanmay Nale @nale_tanmay]

@insidemanifest 333

[Tanmay Nale @nale_tanmay]

@Lyraworldxm 333

[Tanmay Nale @nale_tanmay]

@lunarfq Hi

[Sam Altman @sama]

The companies that succeed in the future are going to make very heavy use of AI. People will manage teams of agents to do very complex things.

Today we are launching Frontier, a new platform to enable these companies.

[Kshitij Mishra | AI & Tech @DAIEvolutionHub]

Best GitHub Repos to Learn AI From Scratch in 2026:

1. Andrej Karpathy – Neural Networks: Zero to Hero
https://t.co/8wrukS6Vqm

2. Hugging Face Transformers
https://t.co/AtyDLv7NXI

3. FastAI/fastb
https://t.co/DsmaVa0WuT

4. Made-With-ML
https://t.co/bIlKIjWN9J

5. ML System Design
https://t.co/8mgOFAZCQD

6. Awesome Generative AI guide(
https://t.co/AmuUmfvJTD

7. Dive into Deep Learning
https://t.co/ntRDPNjOFr

[Etornam ✨ @_iamEtornam]

“Just write code like a normal human f*cking being, please”

[Alex Albert @alexalbert__]

We put together a prompting guide for Claude Opus 4.5 based on extensive internal testing by our research and applied AI teams.

Here's what we've learned so far about getting the best results: