Biplove Yadav | AI Engineer

AI Engineer | RAG, LLM apps & workflow automation. Building at Continuum Technologies. Sharing LLM notes and practical lessons in public. Let’s connect.

[Biplove Yadav | AI Engineer @BiploveYadav]

I'm Biplove, an AI engineer building at Continuum Technologies. I share LLM learning notes and project work around RAG and workflow automation. Follow along for concrete examples, code, and what I learn as I build. Projects and contact: https://t.co/3lXDm9EdmW

[Biplove Yadav | AI Engineer @BiploveYadav]

A smaller KV cache is useful, but I’d also want to see latency, throughput, and output quality. Those numbers together would tell me more about an attention change than a single speedup claim.

[Biplove Yadav | AI Engineer @BiploveYadav]

For a KV-cache comparison, I’d write down sequence length, layer count, head dimensions, and data type. Without those details, two memory numbers might describe very different workloads. #LLMInference

[Biplove Yadav | AI Engineer @BiploveYadav]

Multi-query attention keeps multiple query heads but shares one key head and one value head. That cuts KV-cache size compared with comparable multi-head attention. How much it helps in practice still depends on the setup.

[Biplove Yadav | AI Engineer @BiploveYadav]

What’s your smallest useful test for future-token leakage? I’d start by changing later tokens and checking earlier outputs in evaluation mode. A mask can have the right shape and still let the wrong information through.

[Biplove Yadav | AI Engineer @BiploveYadav]

A causal-mask check I’d try: change future tokens and see whether earlier outputs stay the same, with dropout off. That tests what information reaches an output, rather than only checking the mask’s shape.

[Biplove Yadav | AI Engineer @BiploveYadav]

A causal attention mask prevents a token from attending to future positions. Without it, a model could peek at what it’s supposed to predict. A falling training loss would then be a misleading signal. #LLM

[Biplove Yadav | AI Engineer @BiploveYadav]

I’m putting assumptions beside equations in my learning notes. It’s easier to see where a derivation applies when you don’t have to search another section for the conditions behind it.

[Biplove Yadav | AI Engineer @BiploveYadav]

The sqrt(d_k) explanation makes more sense when the assumptions are written next to it. Independence, zero means, and unit variances are doing work in that argument. I don’t want to leave them out of my notes.

[Biplove Yadav | AI Engineer @BiploveYadav]

Why divide attention scores by sqrt(d_k)? Under independent, zero-mean, unit-variance assumptions, dot-product variance grows with d_k. Scaling counteracts that growth and helps avoid softmax saturation. #DeepLearning

[Biplove Yadav | AI Engineer @BiploveYadav]

The transformer paper is useful to keep open while studying Q/K/V. I can check the scoring, scaling, masking, and value aggregation against the actual equation. https://t.co/lftGNYiXCG

[Biplove Yadav | AI Engineer @BiploveYadav]

I’m breaking attention into two parts in my notes: work out the weights using Q and K, then use them to combine V. That gives me something concrete to trace through the equation.

[Biplove Yadav | AI Engineer @BiploveYadav]

Queries and keys produce attention scores. Those scores become weights, and the weights combine values. Separating those two steps helps me follow Q/K/V without reading too much into the usual analogies. #SelfAttention

[Biplove Yadav | AI Engineer @BiploveYadav]

The toy arithmetic is .9×.5 + .1×.5 + .1×.1 = .51, and .1×.5 + .9×.5 + .1×.1 = .51. Equal scores make this example easy to inspect. They don’t tell us how a trained model would handle the sentence.

[Biplove Yadav | AI Engineer @BiploveYadav]

The .51 result in my attention notes depends on the vectors chosen for the example. Change those vectors and the scores change. I want to keep that visible when sharing it, so the arithmetic doesn’t look like a model result.

[Biplove Yadav | AI Engineer @BiploveYadav]

A toy example in my notes uses dog=[.9,.1,.1], ball=[.1,.9,.1], and it=[.5,.5,.1]. Both dot products with ‘it’ are .51. These are made-up vectors for an example, not measurements from a trained model.

[Biplove Yadav | AI Engineer @BiploveYadav]

There are a lot of arrows in a transformer diagram. I find it easier to start with the tensor sources and shapes, then follow the operations. Starting with the full equation makes it harder to see what is being combined.

[Biplove Yadav | AI Engineer @BiploveYadav]

Before working through attention math, I’m labeling where Q, K, and V come from. It’s a simple step, but it helps distinguish self-attention from cross-attention before the matrix multiplications start.

[Biplove Yadav | AI Engineer @BiploveYadav]

For self-attention, Q, K, and V come from the same sequence representation. For cross-attention, the queries come from one representation and the keys/values from another. Following those inputs makes the diagram much easier to read.

[Biplove Yadav | AI Engineer @BiploveYadav]

‘One summary vector’ is a helpful way to explain a basic encoder-decoder bottleneck. I just need to name the setup when I use that analogy. Otherwise, a simple explanation can accidentally turn into a claim about every recurrent model.

[Biplove Yadav | AI Engineer @BiploveYadav]

What I find useful about encoder-decoder attention is that each output step can draw on different source states. The source doesn’t have to be represented by one fixed summary alone. Paper: https://t.co/bbasf0qO7P