Bhuvanesh S

AI infrastructure, agents and inference systems | Founder, 5ivedots Digital | Teaching teams to build with AI.

[Bhuvanesh S @bhuvanesh123]

@joybhattacharj Most important international matches and finals get defaulted to the Ahmedabad venue just because it’s the biggest stadium. Having one more option may not be a bad idea.

If the problem is less matches being played there then maybe this’ll bring more matches to the venue.

[Bhuvanesh S @bhuvanesh123]

The more I learn about LLM inference, the less it looks like an ML problem.

A single model request sounds simple: send a prompt in, generate tokens out.

But once many people use the model at the same time, you suddenly have to think about scheduling, memory allocation, caching, batching and networking.

One user sends a 200-token prompt. Another sends 20,000 tokens. A third is already halfway through a long generation. All three want low latency.

The GPU has to decide what work gets done together, what waits, how much KV-cache memory each request gets, and how to avoid wasting compute while requests start and finish at different times.

This is why systems like vLLM and SGLang are interesting.

They are not making the underlying model smarter. They are figuring out how to use expensive hardware much more efficiently.

PagedAttention improves memory management. Continuous batching keeps adding and removing requests while the GPU is working. Prefix caching avoids repeating work. Speculative decoding tries to generate multiple tokens for the cost of fewer expensive model steps.

At some point, serving an LLM starts looking much closer to building a database or distributed system than calling an ML model.

That is probably why inference engineering is becoming such an interesting field.

[Bhuvanesh S @bhuvanesh123]

RAG WAS NEVER JUST “CHAT WITH YOUR PDF”

The deeper idea behind RAG is that an LLM does not need every piece of knowledge coded into its weights. Some knowledge can live outside the model and be retrieved only when it is needed.

Instead of sending a question directly to the model, the system first searches an external source, retrieves the most relevant evidence, places that evidence into the context, and then asks the model to answer.

That external knowledge can live almost anywhere: documents, databases, Git repositories, APIs, Slack, the web, or a vector store. The model does not need your entire company knowledge base inside the prompt. It only needs the small subset relevant to the current decision.

That is why RAG is more important than the common “chat with your PDF” framing suggests. It introduced a clean separation between intelligence and memory.

The model does the reasoning. The system decides what knowledge to put in front of it.

Context engineering is just an extension of this idea to decide what/when/how an information should be loaded into memory for an agent.

[Bhuvanesh S @bhuvanesh123]

Not all context is created equally. Some are “Lost in the Middle”.

Just because information is inside the context window does not mean the model will use it well.

So a 1M-token context window is not 1M tokens of effective working memory.

Context capacity ≠ context usability.

Good AI systems retrieve, rank, compress, and surface the important parts (and structure it) instead of dumping everything into the prompt.

accuracy
^
| \ /
| \ /
| \_________/
|
+--------------------> position
start middle end

[Bhuvanesh S @bhuvanesh123]

@iannuttall Wow. Just the thing I was looking for. Ran into the same problem.

[Bhuvanesh S @bhuvanesh123]

Haven't touched tmux since I started using herdr. Just the right agent native setup for a dev env.

[Bhuvanesh S @bhuvanesh123]

@championswimmer GPT-5.5 on the $20 plan felt like unlimited. Now at least the limits feel tangible 😅

[Bhuvanesh S @bhuvanesh123]

Reading about context compaction and it feels a lot like manual memory management.

Handle it yourself and you spend real time on it. Let the tool do it automatically and it will occasionally drop the one thing that mattered.

Most people using agents couldn't tell you what their compaction strategy is, which is roughly where we were with memory before garbage collection.

https://t.co/cVJcydNR3s

[Bhuvanesh S @bhuvanesh123]

https://t.co/ktbn52Vi4T

[Bhuvanesh S @bhuvanesh123]

DeepSeek is moving to peak and off-peak API pricing on August 16.

off-peak is 50% of peak pricing, so it looks like a discount. But it is still higher than the flat rate they charge today. your bill goes up either way.

[Bhuvanesh S @bhuvanesh123]

From what I understand it works based on nudging the next token probabilities using a secret key + hashing. So Google may not be able to detect it unless they have the secret key + hash to verify the token probabilities.

So claude can verify claude generated content. But not others.

[Bhuvanesh S @bhuvanesh123]

Claude watermarking, explained simply.

How it works during token sampling, how detection works, and why paraphrasing can weaken it. https://t.co/0ORe1CuZF2

[Tibo @thsottiaux]

Rest assured that GPT 5.6 Sol will stay in the ChatGPT subscription you pay for. Including Go, Plus and Pro subscriptions. At least until we ship an even better model.

[Bhuvanesh S @bhuvanesh123]

With fable gone and 50% less usage limit from July 19 why should anyone continue the Claude max subscription?

This is not going to end well for anthropic. Somewhere down the line they’ve fumbled the ball.

[Bhuvanesh S @bhuvanesh123]

ChatGPT plus/pro is the best value for money subscription one could have. It has generous limits and separate usage for both ChatGPT and Codex. This is nuts when you compare it how Claude models are priced.

With the release of GPT 5.6 Sol, and given its general availability (subscription) unlike Fable, OpenAI has a real edge now.

[Bhuvanesh S @bhuvanesh123]

@clairevo @ClaudeDevs @sentry I’m curious how this differs from, or improves upon, a CLI tool that can already be invoked to perform bug fixes. What advantages does this approach offer?

[Julia Turc @juliarturc]

Imagine if Apple said: Your iPhone takes crappy pictures because you're using it wrong. Most people just press the camera button.

You need to do 3 jumping jacks, hold your iPhone at 12.5 degrees and whisper a mayan incantation to Siri so that she can call the tool that takes a picture.

Also, please stay tuned for updates to these shenanigans every 3 months or so.

[Bhuvanesh S @bhuvanesh123]

@wesbos It's just annoying when the screen scrolls down to the cursor after copying . Otherwise it is cool. Just need a little tweaking.

[Bhuvanesh S @bhuvanesh123]

@jon_stokes This has been happening since 4.7 noticeably. Regressed further with 4.8.

[Bhuvanesh S @bhuvanesh123]

@pmarca Enna man solra. Purira maari pesu bro.

Purila la? Same 😂