Avneesh Balyan

[Avneesh Balyan @Ballu]

1966-2014 Haryana Election results and Government heads (CM).
Made with https://t.co/lX0Jh34C2U
#Dta #Visiualization #DataVisualization #Haryana #Elections #PoliticalJunkie #HaryanaAssemblyPolls #HaryanaAssemblyElections2019

[h100envy @h100envy]

Ex-vLLM core contributor explained how to make LLM inference 10x cheaper in 34 minutes - better than $3000 inference optimization bootcamps.

request comes in -> check LMCache -> hit? load KV cache from CPU/SSD/remote -> skip prefill -> serve.

That loop is why Bloomberg and other production stacks now push 300 terabytes of KV cache per week.

LMCache + vLLM + CPU/SSD/remote storage + zero-copy CUDA kernels - that's the stack.

Watch and save it, then wire the KV-offload into your inference stack.

[Movez @0xMovez]

Andrew Ng just dropped a 3-hour course on how to become an AI Engineer in 2026:

• 00:00 - How to build agentic AI systems
• 04:25 - Future of AI engineering
• 23:38 - AI Prompting full course
• 2:52:17 - Creating an app with AI in 30 minutes

This 3-hour watch could replace 10 AI engineering courses on the internet.

Watch it today, then read how to run a self-improving system in the article below.

[Tim Urban @waitbutwhy]

Some different ways to try to wrap your head around history. Famous people by birth decade, going back to 580 BC: (thread)

[Data Science Dojo @DataScienceDojo]

"RAG is dead," they said.

After explaining a system that indexes documents, embeds queries, and retrieves relevant chunks before generating an answer. Respectfully, that is still RAG.

👇 Phase 1 - Indexing (happens once, ahead of time):
- Your documents (PDFs, docs, websites) get broken into chunks
- Each chunk gets embedded into vector space
- Chunks are stored in a vector database for fast retrieval

👇 Phase 2 - Answering (happens every time someone asks a question):
- The user's question gets embedded into that same vector space
- The system retrieves the chunks closest to the question
- Those chunks get passed to the model, which generates an answer grounded in them

The part that trips people up building their first RAG pipeline: the question and the documents have to land in the same vector space, or retrieval quality falls apart no matter how good your embedding model is.

What actually changed isn't the mechanism, it's the packaging; agents, tool calls, and multi-step reasoning now sit on top of the same retrieve-then-generate loop.

RAG is one piece of the agentic stack. Our Agentic AI Bootcamp covers this alongside multi-agent systems, MCP, and the frameworks that tie it together. Next cohort starts July 14. Link in the comments

#RAG #VectorDatabase #LLMEngineering #AIEngineering #AgenticAI #RetrievalAugmentedGeneration

[Avi Chawla @_avichawla]

A tricky LLM interview question:

You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.

So you add KV cache compression and evict 90% of the cached tokens.

VRAM usage stays as is and GPU still runs out of memory.

Why?

(answer below)

Evicting 90% of the KV cache can free almost none of the memory it was using.

This sounds counterintuitive, but it follows directly from how production servers store the cache today.

The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues.

This is the dominant memory cost for reasoning models.

If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU.

One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it.

But this does not solve the memory problem yet.

The reason is paged attention, which is the memory manager behind vLLM and most production servers.

Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens.

This block returns to the allocator only when every slot inside it is empty.

Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks...

...so despite eviction, almost every block is left with at least some survivor tokens.

For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token.

This means the allocator frees almost nothing.

Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout.

Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order.

Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds.

This introduces another bookkeeping cost that an in-order layout inherently avoids.

So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server.

There's another problem.

Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected).

But fast attention kernels used in production, like FlashAttention, never save those scores.

They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast.

So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide.

NVIDIA published a method called TriAttention to solve both these problems.

It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters.

For the memory problem, it runs a compaction pass every 128 decoded tokens.

The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order.

On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory.

KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens.

You can find the NVIDIA write-up here: https://t.co/ZwXv7VezVu

I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

Read it below.

[h100envy @h100envy]

Anthropic just dropped a 33-page blueprint for building effective AI agents. Zero theory, just production architecture patterns used by Claude, Coinbase, Stripe, and Intercom.

Every system follows one cycle: Perceive -> Decide -> Act -> Evaluate -> Repeat.

Here are the 5 core patterns to know:

Single Agent: One model in a loop. Solves 80% of problems, don't over-engineer it.

Sequential: Step-by-step handoffs. Predictable and easy to audit.

Parallel: Tasks split across agents at once, then merged. Built for speed.

Hierarchical: A supervisor agent managing a team of specialists.

Evaluator-Optimizer: A 2-agent loop (generator + critic) refining quality over 2-4 cycles.

The Bottom Line: Multi-agent architectures outperform single models by 90.2% on complex tasks. Just match your complexity to the value.

Read the manual, then check out the "Loop engineering" article below.

[Femke Plantinga @femke_plantinga]

Skip transformer math to build AI agents in 2026.

You just need these 6 (+1) core architectural pillars.

𝟭. 𝗠𝗼𝗱𝗲𝗹 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹 (𝗠𝗖𝗣)
Think "USB-C for AI." One universal standard that lets any agent plug into external tools and data — instead of hand-building an integration for every tool. Anthropic introduced it; the industry adopted it fast.

𝟮. 𝗔𝗴𝗲𝗻𝘁 𝗟𝗼𝗼𝗽𝘀
The engine behind every agent. A cycle of: perceive → think → act → observe → repeat. The agent keeps looping until the task is done, or it decides it's stuck. No loop, no autonomy.

𝟯. 𝗦𝗸𝗶𝗹𝗹𝘀
The agent's job description. MCP handles the connection and tools expose the API, a Skill is the higher-level logic that orchestrates them into a finished outcome.

𝟰. 𝗦𝗶𝗻𝗴𝗹𝗲 𝘃𝘀 𝗠𝘂𝗹𝘁𝗶-𝗔𝗴𝗲𝗻𝘁 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲
Two ends of one spectrum. Single-agent: one LLM runs the whole pipeline. Multi-agent: specialized agents split the work, one retrieves, one validates, one writes, trading simplicity for scale.

𝟱. 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚
RAG with a brain. The agent can route queries to specialized knowledge sources, validate retrieved context, and make dynamic decisions about what information to use.

𝟲. 𝗔𝗴𝗲𝗻𝘁 𝗠𝗲𝗺𝗼𝗿𝘆
Short-term lives in the context window; long-term is pulled on demand from external stores (knowledge bases or vector databases). It's what keeps agents coherent across interactions, and lets them learn from past ones.

𝟳. 𝗛𝘂𝗺𝗮𝗻-𝗶𝗻-𝘁𝗵𝗲-𝗟𝗼𝗼𝗽 (𝗛𝗜𝗧𝗟)
The ultimate guardrail. Autonomous loops are powerful, but pure autonomy is dangerous for high-stakes tasks. HITL inserts human checkpoints for approval or correction before critical actions run.

Which term would you add? 🤔

[Raytar @Raytar]

Met a guy making $1.6 million a year.

Three days ago he was at a Meta conference. Told me he saw the best AI talk of his life.

Boris Cherny was on stage. Showed how the Anthropic team actually uses Claude day to day.

Boris deleted his IDE eight months ago. Now he codes from his phone.

I watched it last night. Had to pause it twice.

Not because it was hard. Because I realized I've been using Claude like a toy.

He sent me the recording. It was never published.

Posting it below.

[Simon Kuestenmacher @simongerman600]

Statistical fallacies that you will want to avoid. Source: https://t.co/gVRcszOQkH

[Robert Armstrong @rbrtrmstrng]

Wall Street consensus estimates for hyperscaler free cash flow. Provides a good snapshot of where the market's head is at, I think:

[Paul Iusztin @pauliusztin_]

I spent months optimizing GraphRAG retrieval.

But it turned out I was optimizing the wrong thing....

The biggest knowledge graph problems usually occur during ingestion (even though most conversations focus on retrieval).

Every new document creates a risk of graph corruption.

This is why I now think about knowledge graph ingestion as a 5-step pipeline:

1/ Extraction

Convert raw text into entities and relationships.

For example: Person → WORKS_AT → Organization

The goal is to extract what your ontology cares about.

2/ Resolution

This step standardizes names.

For example:

NYC → New York City
P Morgan → JPMorgan Chase
Jon Smith → John Smith

Most importantly, nothing has been merged yet.

3/ Embedding

Next, embed the entity's full context (not just the name).

Think:

Type
Attributes
Metadata
Relevant content

Because identity lives in context.

4/ Deduplication

Many systems fail here.

Because:

Apple the company ≠ Apple the fruit
Paris, France ≠ Paris, Texas
Two people can share the same name

Resolution answers naming.
Deduplication answers identity.

Those are two completely different jobs.

5/ Routing

Finally, the system decides:

Merge
Human review
Create new node

And the best systems follow a simple rule:

Evidence strength = permission strength.

Weak evidence → new node
Strong evidence → merge
Uncertain evidence → human review

Because false merges are expensive.

A duplicate node is annoying.

But a corrupted graph can silently poison retrieval quality for months.

My biggest takeaway?

Knowledge graph quality isn't determined by your retrieval strategy...

It's determined by the pipeline that creates the graph in the first place.

Get these five steps right... and retrieval becomes much easier.

P.S. I break down the full pipeline, entity resolution, deduplication thresholds, review queues, and production architecture in Decoding AI Magazine

Check it out here: https://t.co/iYSkSC0tJX

[Shraddha Bharuka @BharukaShraddha]

90% of AI engineers are using these 4 concepts interchangeably.

They're not.

And that's exactly why so many AI agents become slow, expensive, and impossible to debug.

The 4 concepts:

• Skills
• Subagents
• MCP
• Hooks

Each solves a completely different problem.

Here's the simplest way to think about them:

📚 SKILLS = What the agent KNOWS

Need reusable expertise?

A coding standard.
A file format.
A workflow.

Use a Skill.

Load it only when needed.

Not every task deserves permanent context.

🧠 SUBAGENTS = Where the agent THINKS

Need deep research?
Parallel analysis?
Messy exploration?

Use a Subagent.

Give the task its own workspace.

Keep the main conversation clean.

🔌 MCP = What the agent can REACH

Need access to:

• APIs
• Databases
• SaaS tools
• Internal systems

Use MCP.

If the agent must interact with something outside itself, MCP is usually the answer.

⚡ HOOKS = What the agent MUST OBEY

Need:

• Validation
• Security checks
• Formatting rules
• Logging

Use Hooks.

Don't trust the model to remember.

Enforce it.

The mental model:

Skills → Knowledge

Subagents → Thinking

MCP → Access

Hooks → Rules

Most people build AI systems like this:

"Can MCP solve it?"

The better question is:

"Does this even need MCP?"

Hot take:

A huge percentage of MCP servers should have been Skills.

People create integrations when all they needed was reusable knowledge.

The result?

More latency.
More auth headaches.
More maintenance.

Better AI systems aren't built by adding more pieces.

They're built by knowing which piece NOT to add.

What's your rule for deciding between a Skill and an MCP?

[Ashutosh Maheshwari @asmah2107]

The reading list that taught me how to think about agentic architecture.

Bookmark this.

1. Brewer's CAP Theorem (2000) — trade-off thinking
2. Netflix Hystrix docs — circuit breaker pattern
3. Martin Fowler: Saga Pattern — distributed rollback
4. The Twelve-Factor App — stateless service design
5. AWS Well-Architected Framework — blast radius thinking
6. "Thinking in Systems" — Donella Meadows
7. Designing Data-Intensive Applications — Kleppmann
8. Google SRE Book Ch.13 — cascading failures
9. OWASP LLM Top 10 (2025) — agent attack surfaces
10. Anthropic: Building Effective Agents (2024)
11. LangGraph docs — stateful agent patterns
12. Microsoft AutoGen paper — multi-agent orchestration
13. Gartner: Agentic AI Hype Cycle (2025)
14. EU AI Act Article 14 — human oversight requirements

Classic distributed systems stuff.
Applied to the next layer of the stack.

Follow for annotated breakdowns →
@asmah2107

[Fei-Fei Li @drfeifei]

https://t.co/Kt50ttQRMJ

[a16z @a16z]

World Labs CEO Dr. Fei-Fei Li: "The world is not made of words."

"Language models have given machines an extraordinary command of concepts, vocabulary, and reasoning, but the physical world, virtual or real, runs on a different substrate."

"Where language models learn the statistical structure of text, world models learn the statistical structure of space and time: how light falls on a surface, how a garden looks from an angle no camera has captured, how objects respond to force and follow the laws of physics."

"Language gave machines a way to talk about that world. World models are how machines will finally come to understand, imagine, reason and interact with it."

Full piece: https://t.co/C9qOJg5wuc

[Tech Fusionist @techyoutbe]

Phase 00: Setup & Tooling (12 lessons)
Phase 01: Math Foundations (22 lessons)
Phase 02: ML Fundamentals (18 lessons)
Phase 03: Deep Learning Core (13 lessons)
Phase 04: Computer Vision (28 lessons)
Phase 05: NLP (29 lessons)
Phase 06: Speech & Audio (17 lessons)
Phase 07: Transformers Deep Dive (14 lessons)
Phase 08: Generative AI (14 lessons)
Phase 09: Reinforcement Learning (12 lessons)
Phase 10: LLMs from Scratch (22 lessons)
Phase 11: LLM Engineering (15 lessons)
Phase 12: Multimodal AI (25 lessons)
Phase 13: Tools & Protocols (23 lessons)
Phase 14: Agent Engineering (42 lessons)
Phase 15: Autonomous Systems (22 lessons)
Phase 16: Multi-Agent & Swarms (25 lessons)
Phase 17: Infrastructure & Production (28 lessons)
Phase 18: Ethics, Safety & Alignment (30 lessons)
Phase 19: Capstone Projects (85 lessons)

Link in Comment ✍️✍️
build by @ghumare64

[Daily Dose of Data Science @DailyDoseOfDS_]

Naive RAG vs. Agentic RAG, explained visually:

Naive RAG has well-known failure modes:

- It retrieves once and generates once. If the context isn't relevant, it can't search again.
- It treats every query the same. A simple lookup and a complex multi-hop reasoning task go through the identical retrieve-then-generate path.
- There's no verification. The system blindly trusts whatever the retriever returns.

Agentic RAG introduces decision-making loops at each stage to fix this.

Steps 1-2) A query rewriting agent reformulates the raw query. This goes beyond fixing typos, like optimizing it for retrieval by making vague terms precise, decomposing complex queries into sub-queries, and expanding abbreviations.

Steps 3-5) A routing agent decides if the query even needs external context. If not, retrieval is skipped. If yes, a source selector picks the best backend for this specific query type.

Steps 6-7) The source selector routes to the most appropriate source: vector DB for semantic search, web search for real-time info, or structured APIs for tabular data. The retrieved context and rewritten query are combined into the prompt.

Steps 8-9) The LLM generates an initial response.

Steps 10-12) A validation agent (known as Corrective RAG) checks whether the response is relevant, grounded, and complete. If it passes, it's returned. If not, the system loops back to Step 1 with a reformulated query.

This continues for some iterations until we get a satisfactory response or the system admits it cannot answer.

The reason this works is that each agent acts as a quality gate. The rewriter ensures retrieval precision. The router ensures the right source is queried. The validator ensures the output is grounded. Individual failures get caught and corrected rather than silently propagated.

That said, the diagram below shows one of many blueprints of an Agentic RAG system. Production systems increasingly combine Corrective RAG, Adaptive RAG, Self-RAG, and hybrid search (vector + lexical with reranking) based on latency budgets and accuracy requirements.

👉 Over to you: What does your Agentic RAG setup look like?

[Thariq @trq212]

https://t.co/R6exTuF7P8

[Matt Van Horn @mvanhorn]

https://t.co/95lFnAyw0e

[McKinsey & Company @McKinsey]

Agentic orchestration layers allow AI agents, enterprise systems, and data connections to work together across functions.

As cognitive bottlenecks shrink, that allows decision-making to speed up, coordination to improve and new operating models to form. https://t.co/0to4HC26cu