hey @kunalb11 it would be soo cool if i can query for a particular link or password someone sent me on whatsapp few weeks ago!
like a personal whatsapp RAG agent that knows all my chats.
will save soo much time.
@BaluGorade In our family, roles are simple:
- Father can't take care of himself.
- Mother takes care of father.
- Wife takes care of mother and father.
- I take care of wife, mother and father.
That's our family portfolio.
7/7
My biggest takeaway:
LLM inference optimization isn't just about faster GPUs or smaller models.
How you structure prompts and reuse computation matters too.
Reading about KV caching is one thing.
Seeing a ~67x prefill speedup on your own machine is another.
1/7 🧵
I thought KV caching was only useful during LLM decoding.
Then I ran Qwen3 4B locally on my Mac M4 and noticed something strange:
The same 810-token prompt went from 2,042 ms to just 30.6 ms in prefill.
That's ~67x faster.
Here's what I discovered
6/7
Why does this matter?
Many LLM applications repeatedly send:
• System instructions
• Tool definitions
• Conversation history
• Shared context
Keeping common prefixes stable can dramatically reduce prefill latency when the inference engine supports prefix caching.