Ishwar | Infrastructure Systems

Tracing production infrastructure code. Contributing to Meshery & llm-d. Exploring how modern systems actually work.

[Ishwar | Infrastructure Systems @Ishwarinfra]

Most RAG systems don't have an LLM problem.

They have a retrieval problem.

I kept swapping embedding models with little improvement.

The biggest gains came from redesigning the retrieval architecture.

Same LLM.

Results:
• 466 ms → 12 ms
• 320 MB → 48 MB
• Top-5 accuracy: 68% → 91%
The biggest gains didn't come from changing the model.
They came from treating retrieval as a search problem first.

Full write-up:
https://t.co/WbV7sMtFCW

[Ishwar | Infrastructure Systems @Ishwarinfra]

I scraped 271 live GPU listings across 19 cloud providers on Shadeform's marketplace.

One result surprised me:

Despite 19 providers, several GPU SKUs are effectively single-provider markets.

Marketplace diversity ≠ supply diversity.

I'm writing up the data and methodology.
@shadeformai @edsgoode @Cholical

[Ishwar | Infrastructure Systems @Ishwarinfra]

@JensenHuang @nvidia It's less about the letter itself and more about the narrative it's trying to establish.

If openness becomes the accepted policy direction, it shapes regulation, enterprise adoption, and ultimately who gets to build AI.

[Ishwar | Infrastructure Systems @Ishwarinfra]

@humzaakhalid It also signals what NVIDIA thinks the next battle is. Less about benchmarks, more about the rules that shape AI adoption.

[Ishwar | Infrastructure Systems @Ishwarinfra]

The debate isn't "open vs. closed." It's whether AI becomes infrastructure everyone can build on, or infrastructure everyone rents.

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Ishwar | Infrastructure Systems @Ishwarinfra]

You can block a GitHub repository. You can't easily block an idea once the code has been mirrored and distributed. That's one of the defining properties of open source.

[jack @jack]

the government of india does not like technologies like bitchat and wants it taken down

[Ishwar | Infrastructure Systems @Ishwarinfra]

Today I spent more time validating a Hugo shortcode change in a downstream site than reading the diff.

The diff explained what changed. The downstream project told me whether it actually worked.

I'll probably trust that workflow over another pass through the code when reviewing shared tooling.

[Ishwar | Infrastructure Systems @Ishwarinfra]

"Prompt engineering is dead" has been all over my timeline.

While building context-audit, I analyzed 27 coding-agent sessions expecting prompt optimization and caching to be the biggest wins.
They weren’t.

Static prompt caching barely moved the needle. The dominant cost came from long-running context and accumulated conversation history.

That's why the recent discussions around "loop engineering" caught my attention.

As agents get better, the hard problems are shifting toward:
• context & memory
• orchestration
• verification
• execution loops

The prompt is becoming one component. The system around the model is becoming the product.

Builders working on agents: where are you seeing the biggest bottlenecks today?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@oleksantoniv @mvanhorn You mentioned re-discovering architecture less after moving to pull-based retrieval. Was that something you observed qualitatively, or were you measuring anything during those sessions?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@DanielGlejzner You mentioned matched A/B pilots for evaluating context strategies.

Beyond task success, what behavioral signals would convince you an intervention is actually improving the agent?
For example, less repeated exploration, fewer recovery steps, or something else?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@_overment Have you tried the same feature both ways—.md planning vs executable artifacts—and compared how the agent behaved? I'm curious whether the win is better outputs, or simply less repository exploration and iteration.

[Ishwar | Infrastructure Systems @Ishwarinfra]

@0xMovez One thing I'm still unsure about: as these graph workflows get larger, where does most of the context overhead actually come from? Shared state, tool outputs, or subagents repeatedly gathering the same information?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@ree1dist @0xMovez If the execution model isn't the differentiator, what do you think ends up mattering more in practice—context management, retrieval, or evaluation?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@MatureIntellect @0xMovez Curious what you mean by continuous fields here. Are you talking about replacing explicit orchestration graphs, or changing how context/state is represented?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@swollenpollen @0xMovez Have you found the bottleneck is the number of subagents, or that they repeatedly rediscover the same context independently?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@DanielGlejzner Your telemetry mentions measuring quality signals. Have you found a practical way to attribute improvements to the routed context itself, rather than the model or prompt changes?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@erictronai @levie Have you found that's mainly because retrieval misses relevant information, or because some of the context only exists in the ongoing reasoning and decisions?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@mylifcc @levie Curious where you've seen the biggest source of conflicting context in practice. Multiple retrieved documents, accumulated conversation history, or tool outputs?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@dsing42 @levie When you say "context starvation," what tends to be missing in practice? Repository knowledge, business context, or previous decisions made during the session?

[Ishwar | Infrastructure Systems @Ishwarinfra]

@MayankV_Tweets @kunchenguid Curious whether you think TTL alone is enough. Have you run into cases where important decisions expire even though they're still relevant?