PS

[Amit Shekhar @amitiitbhu]

Transformer architecture explained step by step - the full architecture, every attention variant, positional embeddings, and every layer inside a Transformer.

Learn here: https://t.co/Hkkgyd0a4I

[Mohit Goyal (Harness arc) @ByteMohit]

As an AI Infrastructure Engineer

Please learn:
- GPU architecture, VRAM fundamentals, CUDA & memory hierarchy
- Quantization (INT8/FP8/4-bit), batching & continuous batching
- vLLM, TensorRT-LLM, SGLang, llama.cpp & inference optimization
- KV caching, speculative decoding, prefix caching & token throughput
- Distributed training (DDP, FSDP, DeepSpeed, ZeRO)
- Model serving (Triton, vLLM, KServe, Ray Serve, SGLang)
- Kubernetes, Docker & GPU orchestration
- NCCL, InfiniBand & high-speed networking
- Multi-GPU & multi-node inference
- LoRA, QLoRA, PEFT & fine-tuning pipelines
- Vector databases, embeddings & RAG pipelines
- Prompt caching, semantic caching & cost optimization
- Observability (OpenTelemetry, Prometheus, Grafana, Langfuse, Phoenix)
- LLM evaluation, benchmarking & A/B testing
- Model routing & fallback strategies
- MCP, AI agents & workflow orchestration
- Data pipelines, Kafka & streaming inference
- Security, guardrails & prompt injection mitigation
- CI/CD for ML (MLOps), MLflow & model registries
- Linux, networking & storage fundamentals

[Avi Chawla @_avichawla]

Prompt, context, harness & loop engineering, clearly explained!

An agent is a while loop with four layers of engineering wrapped around it:

- Prompt engineering
- Context engineering
- Harness engineering
- Loop engineering

Each one wraps the last, and the model sits in the middle, so none of them compete with the others. Instead, they just zoom one level further out.

> Prompt engineering:

This defines the input the model sees on one call, often composed of a role, instructions, examples, and an output format.

The techniques here alter the internal computation and reasoning the model goes through due to the wording it sees:

- Chain-of-thought makes it work in steps before answering
- Few-shot examples define the format and the edge cases
- A JSON schema or XML tags make the output parseable by code
- Self-consistency samples a few chains and takes the majority

> Context engineering:

It's everything the model sees on a turn, not just the prompt. That includes the query, retrieved docs, memory, prior turns, and tool outputs from earlier steps.

The window is finite and fills up fast, so the engineering work is to rank inputs and cut everything that isn't pulling weight.

You do this by:

- Retrieving only the chunks relevant to the query, then reranking them
- Keeping key facts out of the middle, where accuracy drops
- Summarizing old turns, evict stale outputs, push big blobs to files

> Harness engineering:

It's the code around the model that defines the tools, parses the calls, retries on failure, and can route work to sub-agents so one handles retrieval and another handles code.

A verifier then grades the result by running tests, validating a schema, etc.

Prompt and context involve getting one call right. The harness involves everything that has to happen around that call for it to run in a real system.

> Loop engineering:

In the usual setup, you manage the outer loop, i.e, you write a prompt, read the turns the agent runs, write the next prompt, and repeat, while catching failures.

This layer hands that job to the agent itself. It kicks off on a schedule or an event, and runs many turns with no prompt in between.

A loop inherently doesn't know when it's finished. An agent can report that it's done and halt while the tests still fail. So the stop can't be the agent's word, but rather it has to be a real signal, like:

- A turn and token cap to stop stuck runs
- A no-progress detector to catch repeated calls
- A completion check to verify the goal with a separate model or a deterministic test

By this layer, you're operating on the whole run, so the engineering moves from writing each prompt to setting the goal and the stop conditions up front and letting it run.

If you want to dive deeper into loop engineering, my co-founder wrote a full breakdown of that outer loop.

It goes from the basic while loop to a run that finishes on its own, with the code behind each part, and the parts that are hard to get right, like knowing when to stop, context rot over a long run, and keeping the checker separate from the maker.

Read it below.

[Abhishek Singh @0xlelouch_]

As an AI Engineer.

Please learn:

- LLM fundamentals: tokens, context windows, embeddings, attention, temperature and sampling
- Prompt engineering, structured outputs, function calling and tool use
- RAG properly: chunking, embeddings, reranking, retrieval quality and evaluation
- Vector DBs, hybrid search, metadata filters and freshness of data
- Fine-tuning basics: LoRA, SFT, preference tuning and when not to fine-tune
- Agents, workflows, memory, retries, state machines and human approval steps
- Evals for LLM apps: hallucination checks, groundedness, latency, cost and task success rate
- Guardrails, prompt injection, data leakage and permission boundaries
- Model routing, prompt caching, token optimization and cost per successful task
- AI observability: traces for every LLM call, tool call, retrieval step and final answer
- Production APIs, rate limits, fallbacks, async jobs and handling model failures
- Building with users instead of making another generic chatbot wrapper

Most AI engineers can call an LLM API.

Very few can build an AI product that gives reliable answers, uses the right data, stays within budget, handles failures, and actually improves a real business workflow.

This is what production AI teams actually care about.

[Param @ParamSiddh]

As an AI Infrastructure Engineer.

Please learn:

- GPU/VRAM fundamentals, quantization & batching
- vLLM / TensorRT-LLM / inference optimization
- KV caching, speculative decoding & token throughput
- Distributed training basics (DDP/FSDP/DeepSpeed)
- Model serving & autoscaling
- Vector DB retrieval pipelines
- Prompt caching & cost optimization
- Observability for LLM apps

This is what production AI teams actually care about.

[Piyush Trades @piyush_trades]

So Jio and Airtel made a combined profit of ₹68,000 crore in a year, and now they are planning to increase prices again by 10-15%?

Indian telecom is truly a daylight robbery. The govt needs to stop taking magnesium glycinate, instead wake up, and end the duopoly.

[Sanju Choudhary @saaaanjjjuuu]

Jio and Airtel are planning to increase their recharge plans prices by up to 15%-20%.

Airtel already removed the ₹799 plan, the service is getting worse and the prices are increasing time to time.

There are only three major operators left we need more competition in the market.

[Bong Political Doctor 🧡 @bong_politics]

I am requesting @SuvenduWB sir to look into the matter of payment under Annapurna Yojana

Many eligible poor & lower middle class women did not get it yesterday and still waiting for it

Many confusion are circulated by Some SDOs regarding eligibility criteria

Plz solve this matter ASAP

As a well wisher of Bengal govt , I am requesting this to you so that Benefit of this scheme reaches to the Poor women 🙏

[Amit Shekhar @amitiitbhu]

Learn AI Agents step by step, from scratch - from function calling to agent loops to multi-agent systems, orchestration, and evaluation.

Start here: https://t.co/5Reb0zL4Ay

[tetsuo @tetsuoai]

Attention is a lookup. Each token builds a query, compares it against every key in the sequence, and pulls value vectors weighted by the match. Stack that 96 layers deep and you get a frontier model.

Video covers the full pipeline: Q/K/V, attention scores, encoder blocks.

[Ahmad @TheAhmadOsman]

Local AI hardware = capacity X bandwidth X software stack

- Capacity tells you what fits
- Bandwidth tells you how hard the box can breathe
- The software stack tells you how much of the spec sheet you can actually cash out.

Hardware by Memory Bandwidth
- Mac Studio M3 Ultra: up to 512GB @ 819 GB/s
- RTX PRO 6000 Blackwell: 96GB @ 1792 GB/s
- RTX 5090: 32GB @ 1792 GB/s
- RTX 4090: 24GB @ 1008 GB/s
- RX 7900 XTX: 24GB @ 960 GB/s
- Radeon PRO W7900: 48GB @ 864 GB/s
- AMD Radeon AI PRO R9700: 32GB @ 640 GB/s
- Intel Arc Pro B65: 32GB @ ~608 GB/s
- Tenstorrent Wormhole n300: 24GB @ 576 GB/s
- Tenstorrent Blackhole p150: 32GB @ 512 GB/s + 800G
- MacBook Pro M5 Max: 460-614 GB/s
- MacBook Pro M5 Pro: 307 GB/s
- DGX Spark: 128GB @ 273 GB/s (coherent + CUDA)
- Mac mini M4 Pro: 273 GB/s
- Ryzen AI Max / Strix Halo: ~256 GB/s (~96GB usable GPU)
- MacBook Air M5: 153 GB/s
- Snapdragon X2 Elite: 152-228 GB/s
- Intel Lunar Lake: 136 GB/s
- Snapdragon X Elite: 135 GB/s
- Mac mini M4: 120 GB/s
- Arc Pro B60: 24GB @ ~456 GB/s

Verdict

- GPUs are still the bandwidth kings

- Apple wins: stupid amounts of memory, don't want to shard across GPUs
- Apple loses: when raw tokens/sec & concurrency matter more

- DGX Spark: coherent memory + NVIDIA stack

- Strix Halo / Ryzen AI Max: first real x86 unified-memory contender

- Tenstorrent: fully OSS stack, excited to see this mature

Fitting != serving

Even if it fits, you still pay for
- bandwidth during decode
- KV cache growth
- dequantization
- batching + concurrency
- scheduler quality
- framework overhead

The only mental model that matters:

1. What must fit?
2. What bandwidth tier do I need?
3. What software stack can actually deliver it?

In short:
- NVIDIA -> fastest raw speed
- Apple Studio M3 Ultra -> biggest one-box memory
- Strix Halo -> first real x86 unified
- DGX Spark -> coherent NVIDIA dev appliance
- AMD / Intel Arc -> rising alternatives
- Tenstorrent -> fully opensource stack

Do ask: "which bottleneck am I buying?"

Not: "which hardware is best?"

[Avi Chawla @_avichawla]

Prefill & decode in LLM inference.

Have you ever noticed that the first token from an LLM always takes a moment to appear? But the subsequent tokens stream out smoothly?

That pause isn't a network lag, but rather it's a structural property of how LLMs fundamentally work.

Inference happens in two phases that share the same model and the same code path, but the workload looks completely different in each, with different bottlenecks.

> Prefill stage starts when you submit a prompt.

The model processes every input token in one parallel pass, computing Q, K, and V for all of them at once.

Attention runs as a matrix multiplication, and the GPU chips run at high utilization, doing fast math.

Prefill is compute-bound, and the metric that captures it is time-to-first-token (TTFT).

> Decode stage starts once the first token is out.

To generate the next one, the model only computes Q, K, and V for that single new token, because everything before it is already cached.

So the model loops one token per forward pass, multiplying a single query against the cached keys instead of a full matrix. This makes the inference fast due to the tiny computation.

But the GPU still has to load every weight and every cached entry from memory to do that tiny computation, so the bottleneck flips and compute sits idle while memory bandwidth becomes the limiting factor.

Decode is memory-bound, and the metric that captures it is inter-token latency (ITL).

GPU utilization peaks during prefill and drops sharply during decode because memory, not compute, is the bottleneck in the second phase.

Throwing more compute at a slow-streaming model often does nothing because the fix for memory-bound workloads is faster memory or a smaller cache, not more FLOPs.

Long contexts feel disproportionately slow because the KV cache grows with every token, and every decode step has to read all of it.

But maintaining the cache is an important optimization since it makes decoding viable.

- Without KV cache, every new token would force a recomputation of attention over the entire growing sequence.
- With KV cache, the cache is built once during prefill, then grows by exactly one entry per decode step, with existing entries reused rather than recomputed.

The cache lives in GPU memory and grows linearly with sequence length, so a 13B model roughly requires 1 MB per token, which means a 4K context consumes 4 GB of VRAM on the cache alone.

The entire field is now optimizing around this constraint with quantized caches, sliding windows, grouped-query attention, and PagedAttention, while DeepSeek's V4 series goes further and redesigns attention itself so the cache stays small from the start.

The practical takeaway is that when someone says their model feels slow, the first question is whether it's slow to start or slow to stream.

Slow to start means prefill and a compute bottleneck, while slow to stream means decode and a memory bottleneck.

The article below is a first-principles guide to LLM inference that walks through everything between your prompt and the streamed response, covering tokenization, embeddings, attention, the prefill and decode split, KV caching, and quantization.

It will give you a complete mental model of how inference actually works under the hood.

Read it below.

[Ashutosh Maheshwari @asmah2107]

Inference optimizations I’d study if I wanted sub-second LLM responses:

Bookmark this.

1.KV-Caching
2.Speculative Decoding
3.FlashAttention
4.PagedAttention
5.Batch Inference
6.Early Exit Decoding
7.Parallel Decoding
8.Mixed Precision Inference
9.Quantized Kernels
10.Tensor Parallelism
11.Pipeline Parallelism
12.Sequence Parallelism
13.Graph Optimization (ONNX, TensorRT)
14.Dynamic Batching
15.Memory Offloading
16.Streaming Generation

[Avi Chawla @_avichawla]

A tricky LLM interview question:

You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.

So you add KV cache compression and evict 90% of the cached tokens.

VRAM usage stays as is and GPU still runs out of memory.

Why?

(answer below)

Evicting 90% of the KV cache can free almost none of the memory it was using.

This sounds counterintuitive, but it follows directly from how production servers store the cache today.

The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues.

This is the dominant memory cost for reasoning models.

If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU.

One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it.

But this does not solve the memory problem yet.

The reason is paged attention, which is the memory manager behind vLLM and most production servers.

Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens.

This block returns to the allocator only when every slot inside it is empty.

Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks...

...so despite eviction, almost every block is left with at least some survivor tokens.

For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token.

This means the allocator frees almost nothing.

Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout.

Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order.

Attention can still compute the right answer from that, but only if every slot now carries a separate note recording which position it actually holds.

This introduces another bookkeeping cost that an in-order layout inherently avoids.

So the cache is logically 90% smaller and still physically the same size. Many compression results miss this because they measure on pre-allocated contiguous tensors rather than a paged server.

There's another problem.

Eviction methods pick which tokens to keep by looking at the attention scores themselves (as expected).

But fast attention kernels used in production, like FlashAttention, never save those scores.

They compute attention in small pieces and throw the full score grid away as they go, which is also why they're fast.

So the exact signal eviction methods need isn't available in memory. The workaround is to fall back to eager attention and build the full matrix, which gives up the speed FlashAttention was there to provide.

NVIDIA published a method called TriAttention to solve both these problems.

It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied, where those vectors sit in stable clusters.

For the memory problem, it runs a compaction pass every 128 decoded tokens.

The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order.

On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory.

KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens.

You can find the NVIDIA write-up here: https://t.co/ZwXv7VezVu

I wrote a first-principles breakdown of how the KV cache works. It walks through why the model stores keys and values at all, why the cache grows with every token, and a comparison of LLM generation speed with and without KV caching.

Read it below.

[Andrej Karpathy @karpathy]

This is a new paradigm for interacting with Claude that is significantly more "inline" with all the other human activity org-wide. Once you do all of the under the hood engineering work to make this "just work" (e.g. across tools, integrations, compute environments, memory, security, etc.), Claude basically joins the team in a seamless way - you can talk to it as you would talk to a person and it can help with a very large variety of workloads.

Imo this is the 3rd major redesign of LLM UIUX. The first paradigm was that the LLM is a website you go to, the second was that it is an app you download to your computer. This third one is that it is a self-contained, persistent, asynchronous entity with org-wide tools and context, working alongside teams of humans. It really takes a while to wrap your head around it, but it works and it is awesome.

[Jason Zhou @jasonzhou1993]

Loop Engineering is getting hype now.
But not many talks about how to actually do it

So I open-sourced the template my team uses to build agent loops:
- a shared artifact / knowledge layer
- logging, verification
- and a codebase harness so work compounds across runs

Plus a 20-min deep dive on how to think about it and set it up for real: https://t.co/b3m22eX8oI

Copy the template. Adapt it to your own loops.

[Movez @0xMovez]

Anthropic engineers just showed how to build agentic systems that run for days using "loops."

"At Anthropic, >30% of our code is already written by loops - that's how we ship so fast.

in this 40-minute workshop, they reveal the whole stack:

agent loop + harness + memory + sub-agents.

Worth more than any $500 vibe-coding course.

Watch workshop today, then read article below.

[Vizuara @VizuaraAI]

Eigenvectors aren't just linear algebra-they're the hidden directions that stay stable under transformation.

Powering PCA, compression, spectral clustering, and optimisation, they reveal structure hidden in data. Learn from first principles: https://t.co/Dr2a8jNtnD

[Vizuara @VizuaraAI]

How small can a Vision-Language Model be and still remain useful?

In our latest Research Paper Dissection, we explore NanoVLMs, showing how child-inspired datasets and simple language structures enable surprisingly capable multimodal models without massive compute.

Paper: https://t.co/BwAB0343Fg

Research: https://t.co/6ysZco9BUo

[Piyush Trades @piyush_trades]

Jio made a net profit of ₹30,000 crores, and Airtel made a net profit of ₹38,000 crores last year.

That comes to an average of ₹185 crores extracted per day from Indians in profit (not revenue).

There’s nothing wrong with making profits, but raising plan prices even after making such huge profits is not morally right.

These companies must be forced by the govt to offer plans with 0.5 GB/day, 1 GB/day, and voice-only options.

In the current duopoly, both companies are in a race to see who can extract more from Indians.

The govt needs to wake up from its deep sleep and strengthen BSNL to restore healthy competition.

In today’s age, the internet is a human right, and forcing people to pay more for something they need less of is no less than a crime.