Thejas Manjunath

Building production AI systems.

LLMs • RAG • Agents • AI Infra

Learning in public. Interested in what happens after the demo works.

[Thejas Manjunath @imThejas18]

🚨 Hiring at @Cyient | Bengaluru

Multiple openings across:

• AI/ML & Full-Stack
• Embedded Software & AUTOSAR
• Automotive & Hardware
• Robotics / ROS
• Testing & Validation
• Medical Imaging

Immediate joiners preferred.

DM me your role, experience & notice period for a referral.

Reposts appreciated!

#Hiring #BengaluruJobs

[Thejas Manjunath @imThejas18]

@RumiTweetz @grok, is this real?

[Thejas Manjunath @imThejas18]

I used to think making an AI agent more capable mostly meant giving it more tools.

I’m starting to think the harder problem is deciding when it shouldn’t use them.

Imagine an incident-response agent finds a likely cause of an outage.

It can restart the service.

But should it?

Maybe.

Or maybe it should keep investigating, ask for approval, escalate to an engineer, or do nothing because the evidence isn’t strong enough yet.

The tool call is the easy part.

The decision around it is harder.

Once agents can take real actions, “can it do this?” isn’t enough.

We also need to think about:

permission
confidence
reversibility
cost of being wrong
whether a human should approve it

And importantly, I don’t think the model itself should be the final authority on all of those.

Some boundaries need to be enforced by the system around it.

I’m starting to see agent autonomy less as “how much can we let the AI do?”

and more as:

“For this particular action, how much autonomy is actually appropriate?”

Sometimes the best next action is no action at all.

[Thejas Manjunath @imThejas18]

I built a RAG system because I realised I understood most of the pieces individually, but not how they actually worked together.

Embeddings? Fine.

BM25? Fine.

Reranking? Fine.

Citations? Fine.

Then I tried answering:

What actually happens between a user asking a question and getting an evidence-backed answer?

Mine now looks roughly like:

Question → plan → retrieve → rerank → build evidence → generate → cite

And the biggest thing building it taught me is that “the LLM gave a bad answer” is almost useless as a diagnosis.

Maybe the right evidence was never retrieved.

Maybe it was retrieved but ranked badly.

Maybe the context we constructed was poor.

Or maybe everything upstream worked and the model still generated a bad answer.

Same bad output. Completely different failures.

That’s what made RAG click for me:

it’s not one AI call. It’s a chain of decisions, and every stage can quietly make the next one worse.

[Thejas Manjunath @imThejas18]

What if the problem with RAG isn’t retrieval itself, but how we’ve been doing retrieval?

The RAG pipeline I first learned looked something like:

documents → chunks → embeddings → vector DB → similarity search → LLM

PageIndex takes a very different route.

Instead of chopping a document into chunks and searching for semantically similar pieces, it builds a hierarchical tree from the document’s structure.

Then at query time, the LLM reasons through that tree to figure out where the relevant information is.

So rather than asking:

“Which chunks look most similar to this query?”

it’s closer to:

“Where in this document would I expect the answer to be?”

That difference is what caught my attention.

Because chunking has always felt like a necessary compromise in RAG. We break a document apart so it becomes searchable, but in doing that we can also lose some of the structure and context that made the document understandable in the first place.

PageIndex is basically asking: what if we preserve that structure and use it for retrieval instead?

I wouldn’t jump to “vector RAG is dead” from one benchmark.

But I do think this is a much more interesting question:

Do we really need to turn every document into vectors before an AI system can retrieve from it well?

Apparently, that assumption is worth challenging.

[Thejas Manjunath @imThejas18]

@daleverett @daltonmeon @damienhe @polygres @ycombinator @fdotinc Curious where the crossover point is here. If the same corpus is being queried heavily, does pre-embedding eventually become cheaper/faster than doing more work per search? Or does the query-time approach still hold up at high QPS?

[Thejas Manjunath @imThejas18]

The part I find interesting isn’t “embeddings are dead.”

It’s the economics of the architecture.

Pre-embedding:
pay upfront → maintain an index → make queries cheap

Query-time search:
avoid some upfront work → keep raw data easier to update → pay more when someone actually searches

Very different tradeoff depending on how often the corpus changes vs how often it gets queried.

[Thejas Manjunath @imThejas18]

I’ve spent a lot of time learning RAG as:

documents → chunks → embeddings → vector index → retrieval

So this caught my attention.

WonderSearch is basically asking: what if we don’t pay the cost of embedding the entire corpus before anyone searches it?

Instead, do more of the work when the query actually arrives.

That doesn’t make vector search obsolete. It changes where the compute happens.

And that creates a tradeoff I want to understand better: at what query volume does doing more work at search time become worse than precomputing the index?

[dale @daleverett]

Today we’re launching WonderSearch (YC F26)

Search millions of unstructured documents without pre-embedding the corpus. With comparable/better retrieval recall than embedding-based search.

WonderSearch searches at query time:

> No corpus-wide embedding step.
> No vector index to maintain.
> More compute only when needed.
> Pay per search intead of embeddings you may never use.

Go from raw data to useful answers in seconds.

Link + FAQ below (+ free launch credits!) ↓

[Thejas Manjunath @imThejas18]

The more I learn about AI engineering, the less it feels like “just AI.”

Getting an LLM to answer a prompt is relatively easy.

Getting an AI application to survive real users is a very different problem.

Because the model is only one component ↓

[Thejas Manjunath @imThejas18]

The human handoff is actually the part I find most interesting here.

A useful enterprise agent isn’t just one that knows how to act, it needs to know when it has enough authority/confidence to act, when to request approval, and when to hand the task back to a human.

Curious how you determine that boundary as the agent takes on more workflows.

[Thejas Manjunath @imThejas18]

This is helpful. Thanks for checking.

So the ~40% cost reduction is observable in that run, but we can’t yet attribute it cleanly to context/token reduction vs fewer generations/tool steps.

That breakdown would be really interesting because it would show where the context-selection layer is actually creating the efficiency.

[Thejas Manjunath @imThejas18]

Which is why “just add a reranker” isn’t a universal RAG fix.

Reranking also adds latency, compute and cost.

The better debugging question is:

At what stage did the relevant information disappear?

Sometimes the LLM is only where the failure becomes visible, not where it began.

[Thejas Manjunath @imThejas18]

Here’s the failure mode that made this click for me:

The correct document exists.

Retriever misses it.

Reranker receives 20 wrong candidates.

It can rank those 20 perfectly.

The correct answer is still missing.

Reranking can’t manufacture evidence it never received.

[Thejas Manjunath @imThejas18]

This creates two different optimization problems.

Retrieval: Did we FIND the relevant evidence?

Reranking: Given what we found, did we ORDER the best evidence highly enough?

Roughly:

Retrieval → recall + speed
Reranking → precision + ordering

[Thejas Manjunath @imThejas18]

Retrieval has to search a huge collection quickly.

Its first job is to make sure useful evidence actually makes it into those candidates.

Then reranking can afford to look at that much smaller set more carefully:

20 candidates → rerank → best 5 → LLM

[Thejas Manjunath @imThejas18]

Imagine your knowledge base contains 100,000 chunks.

A user asks a question.

You don’t want an expensive model carefully reading all 100,000.

So retrieval creates a much smaller candidate set:

100,000 → retrieve → 20 candidates

[Thejas Manjunath @imThejas18]

A reranker can improve ordering.

It cannot rescue a document your retriever never found.

That one distinction explains a lot about why RAG systems separate retrieval from reranking.

Here’s the simple version ↓

[Thejas Manjunath @imThejas18]

This feels less like a prompting problem and more like a workflow-design problem.

Instead of trying to make one agent simultaneously build, self-critique and stay minimal, separate the roles:

builder → fresh-context reviewer → simplify

Curious whether you’ve found the second agent actually removes unnecessary abstractions reliably, rather than just replacing them with different ones?

[Thejas Manjunath @imThejas18]

And that’s the bigger idea I find interesting.

We spend a lot of time asking:

“Which AI model should I use?”

Agentic systems may increasingly ask a different question on every task:

“Which model should handle this work, right now, given what the system already knows?”

That’s a much more dynamic architecture.

[Thejas Manjunath @imThejas18]

The cheapest AI model isn’t always the cheapest AI call.

That sounds backwards.

But imagine your agent has already spent an hour working with one model. That model now has a huge amount of context ready to reuse.

Switch to a “cheaper” model and you may have to process much of that context again.

That’s what makes the cache-aware part of this Jev + OpenRouter release interesting.

Here’s what’s actually happening ↓

[OpenRouter @OpenRouter]

Introducing typesafe/jev-router: a cache-aware model router powered by Jev and @typesafeai

The Jev Router picks the best model and reasoning effort for each request, balancing quality, speed, and cost.

Here's how it works 👇🏻

[Thejas Manjunath @imThejas18]

But “pick the cheapest model capable of answering” isn’t enough.

A production router may need to think about:

task difficulty + model price + reasoning effort + latency + confidence + existing cache

Suddenly model selection becomes an engineering problem of its own.