Vipin | Building with AI

Building prod AI apps | LLMs · RAG · Agentic Systems
AWS & Cloud Architect
Sharing real code, architecture & experiments
🧵 Follow for practical AI builds

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while. speculative decoding verifies tokens in one pass, but drafting is still autoregressive. generating 5 candidates takes 5 sequential draft passes. that overhead eats your speedup unless the draft model itself runs in parallel.

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while: transformers didn't beat RNNs on memory. they won on training parallelism. RNNs force token t to wait for t-1, idling the GPU. attention routes all tokens at once with matmuls. it was a hardware throughput win all along.

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while. RNNs are like telephone: token 50 only knows token 1 if 49 steps pass it along cleanly. Attention is a group chat where every token talks directly to every other token. Turns out an O(1) path length beats sequential handoffs.

[Vipin | Building with AI @VipinAILabs]

Hit an OOM wall fine-tuning 70B locally, and cloud GPU costs explode fast. Turns out QLoRA fits it on a single 24GB card: 4-bit NF4 base, bfloat16 adapters, and paged 8-bit Adam so memory spikes swap to CPU instead of crashing. Anyone else running this setup?

[Vipin | Building with AI @VipinAILabs]

220GB to serve one model meant a brutal cost blowup: renting three 80GB GPUs. Turns out INT4 quantization is like reducing a photo palette: it maps 16-bit floats into 4-bit bins, shrinking 220GB to 55GB on one GPU. What's your experience with this?

[Vipin | Building with AI @VipinAILabs]

Changed 'discuss' to 'write' and my agent fired live APIs, risking a leak. Turns out tool calling is just a railroad switch: a single hidden vector flips on one verb. It was never reasoning. Anyone else hit accidental tool triggers like this?

[Vipin | Building with AI @VipinAILabs]

I assumed speculative decoding was free speed. turns out drafting K tokens still takes K sequential forward passes, creating a hard ceiling. generating the whole candidate block in a single pass is what actually unlocks 3x. noting this for future me.

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while: thought tokens changed shape across layers. Tracing the forward pass, a 3x6 tensor stays 3x6 in every block. Attention only mixes rows across tokens. MLP only mixes columns. The shape never changes until the logits projection.

[Vipin | Building with AI @VipinAILabs]

Built a local agent that aced step 1 then derailed on step 10. A 4-bit quant matches benchmark scores, but tiny probability shifts compound each loop until tool calls fail. Turns out measuring KL divergence catches the drift. Anyone else run into this?

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while. When loss hits NaN mid-training, backprop is multiplying partial derivatives that grow exponentially backward through layers. Gradient descent takes massive steps off a cliff and destroys the weights. Anyone else hit this?

[Vipin | Building with AI @VipinAILabs]

I initialized weights to zero once and thought backprop was broken. turns out if weights are identical, every neuron gets the exact same gradient and updates in lockstep. the whole layer collapses into one neuron. random init exists just to break symmetry.

[Vipin | Building with AI @VipinAILabs]

turns out autoregressive models throw away the top hidden state each step and only pass the sampled token back to layer 1. fusing that state into the next input gives non-verbalized compute a full depth budget again. makes so much sense.

[Vipin | Building with AI @VipinAILabs]

turns out finding LLM circuits isn't a measurement problem. it's an optimization problem. Learning differentiable top-k masks over frozen weights gets 2.9x better benchmark scores than standard pruning, and isolated refusal behavior to 1% of weights.

[Vipin | Building with AI @VipinAILabs]

Watching a code agent hallucinate in prod taught me this: transformers discard the top hidden state each step, crushing thought into one token. Like texting one word after solving a proof. Latent feedback fuses that top vector via a GLU. Hit this yet?

[Vipin | Building with AI @VipinAILabs]

pushed chain of thought to a small model in prod and hallucinations spiked. feature patching shows CoT needs scale to form sparse, modular activations (2.8B log-probs jumped 1.2 to 4.3, 70M did nothing). anyone else hit this?

[Vipin | Building with AI @VipinAILabs]

RAG latency tanked in prod once vectors hit 1M. Exact scans timed out API calls. Turns out HNSW fixes this like a skip list: greedy routing over sparse upper graph layers, then beam search at layer 0. No more timeouts. What index do you run in prod?

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while: RAG failures often start with bad chunking. Fixed-size splits sever semantic units. Context fragments, so the LLM gets partial data. It "hallucinates" due to an incomplete view, not bad reasoning. That's the whole trick.

[Vipin | Building with AI @VipinAILabs]

I got this wrong for a while. cosine similarity doesn't measure distance between embeddings. it projects them onto a unit sphere and checks the angle, throwing out magnitude entirely. two vectors can be miles apart in space and still score 1.0.

[Vipin | Building with AI @VipinAILabs]

You ship a prompt filter, but odd spacing doubles token bills and lets injections slip. Turns out BPE never merges across pre-token regex boundaries. Text fragments before attention even runs. It is a tokenizer bug, not the model. Anyone else hit this?

[Vipin | Building with AI @VipinAILabs]

Pushed context past training length and prod output turned to total gibberish. Turns out RoPE does not get distance for free. Unseen rotation angles break Q and K dot products unless you rescale frequencies. Anyone else hit this extrapolation wall?