Pradeep Thiyyagura

Researcher by the week, Adventurer by the weekend. Chasing dreams and celebrating life.

[jason @jxnlco]

Yesterday I did an expert call about building AI agents for messy government PDFs and RFP-style workflows.

A few ideas that might be useful if you’re in the same bucket:

1. Your “agent problem” is probably a search/recall problem

If the model “misses” facts, 9/10 times it’s not the reasoning model — it’s recall:

No full-text search for exact strings (e.g. “13.1.2” legal clauses).

No decent metadata filters (city, date, doc type, attachment relationships).

Search fan-out is too timid (15 queries instead of 60).

The generation model almost never has a chance to be smart if the search tool is starving it.

2. Stop vibing, get a number

Right now most teams are flying blind:

“Search feels okay.”
“Sometimes it misses stuff.”
“It seems worse on X.”
That’s useless.
You can get a rough recall number in 1–2 days:

Randomly sample pages from your corpus.

For each page, use GPT-5 (or similar) to generate 3–5 realistic user questions that should retrieve that page.

Run those questions through your search tool.

Check: “Is the original page in the top N results?”

Now you have a number like “recall@20 = 0.72”.

Not perfect. Good enough to track:

Add full-text search → did recall move?

Fix layout parsing → did recall move?

Add better metadata filters → did recall move?

Then group by question type and document type. Something will be embarrassingly bad. That’s where you focus.

3. Use existing harnesses as your baseline, not your end state

Everyone wants to build their own “deep research” agent orchestration from scratch.

Most of you shouldn’t.

Use:

OpenAI Responses + Deep Research as a baseline harness.

Or Claude Code with a simple CLI search tool.

Wire your search as:

search --query "..." --filters ...

Give that to a serious agent (Deep Research / Claude Code) and just watch what it does:

How many times does it call search?

Does it fan out per question/section?

If their harness + your search tool crushes your in-house agent, the problem is not “LLMs are bad” — it’s your orchestration.

4. Remember what semantic search actually is

At scale:

ls, read_file, grep would work in the limit.

Semantic search is basically a cost + latency optimization on top of that.

If you think of it that way, it becomes obvious when you’re abusing embeddings for something that should really be solved with:

Better string filters

Proper normalized metadata

Table-aware tools

Time-window filters

Ticker/state/entity resolution

5. Hiring “LLM engineers”

The only real filter that matters: have they actually built and shipped things?

Failure modes:

“AI FOMO” hire who wants to get paid to learn on your dime.

Research-brained person who realizes 3 months in they actually just want to join a lab.

You want the middle: someone who already built agents, evals, or workflows on their own time and can walk you through the tradeoffs.

If you’re building retrieval/agent systems for messy real-world docs (gov, medical, finance, etc.), start with:

Make search recall measurable.

Use Deep Research / Claude Code as baselines.

Iterate on tools and data, not vibes.

Everything else gets a lot easier once you have those numbers.

[Greg Kamradt @GregKamradt]

A few, depending on what you're looking for

* State level actions (very ambitious): Situational awareness by @leopoldasch + complimentary pod with @dwarkesh_sp
https://t.co/JNWfJA101A
https://t.co/D3eP58LwCg

* Science predictions w/ emphasis on biology, Machines Of Loving Grace by @DarioAmodei. The framework of "intelligence constraints" was illuminating
https://t.co/QQi6JhCNpH

* Datacenter build out plans and tokenomics, most content by @dylan522p. My favorite is his stanford talk
https://t.co/YCsuAOG9pI

* Safety view by someone with a rational mind, @eshear redpoint segment
https://t.co/Ek16DMtpTB

* Practical takes on short/long term economy by @tylercowen with Dwarkesh again
https://t.co/R0u1zcoT8r

[Pradeep Thiyyagura @tpradeep]

@williamfalcon Can we attend virtually?

[LangChain @LangChain]

🕸️Building a GraphRAG Agent With Neo4j and Milvus

This agent combines the power of graph databases and vector search to provide accurate and relevant answers to user queries. In this example, we will use LangGraph, Llama 3.1 8B with Ollama and GPT-4o.

https://t.co/CxA4kFNji7

[Sundar Pichai @sundarpichai]

Introducing Willow, our new state-of-the-art quantum computing chip with a breakthrough that can reduce errors exponentially as we scale up using more qubits, cracking a 30-year challenge in the field. In benchmark tests, Willow solved a standard computation in <5 mins that would take a leading supercomputer over 10^25 years, far beyond the age of the universe(!).

[Pradeep Thiyyagura @tpradeep]

@jeremyphoward @tsarnick

[Sarah Chieng @MilksandMatcha]

I created a notes version of @karpathy's "Intro to LLM" video for folks who don't have 1 hour 😀 (~5 min read)

I focused on making the document as holistic and easy to follow as possible, for all understanding levels.

For example,
> significant restructuring of the video content (he presents the steps of creating an LLM out of order)
> enriching with additional very-important topics
> making it less OpenAI/ ChatGPT focused (he makes it sound like the assistant model is the only output format of fine-tuning)

Would love and appreciate any feedback on how to make these notes as useful and thorough as possible, and any recommendations for future videos/folks to watch. Hoping to continue making these documents and please share!

full notes: https://t.co/RdVgqkWc1w

Also, thanks to @scikud and @HarperSCarroll for their feedback ❤️

[Pradeep Thiyyagura @tpradeep]

Cleveland is awesome! #Autogenerated

[Lior Alexander @LiorOnAI]

Finally, someone did it. The ultimate "gpt-prompt-engineer", an AI agent that creates optimal GPT prompts.

Simply describe a task, and a chain of AI systems will:
- Generate many possible prompts
- Test and rank each of them
- Return the best prompt

After the generation, the system tests each prompt against all the test cases, comparing their performance and ranking them using an ELO rating system.

[Pradeep Thiyyagura @tpradeep]

@popcsev @Azure @MicrosoftEdge GPT-4 models are available upon request. Existing customers of Azure OpenAI can apply by filling out the form below, while existing customers of GPT-4 will have access to the new version (0613) https://t.co/sXgUHzCYDd.

[Pradeep Thiyyagura @tpradeep]

The latest version (0613) of GPT-3.5 turbo and GPT-4 are now available in Azure Open AI. Consider upgrading to these new models because they are cheaper and armed with powerful function calling capability. @Azure @MicrosoftEdge

[Pradeep Thiyyagura @tpradeep]

If you’re not familiar with function calling yet, check out this blog https://t.co/tLs9h6muEK

[The Full Stack @full_stack_dl]

🥞🦜 Full Stack LLM Bootcamp 🦜🥞

tl;dr We're releasing our lectures on building LLM-powered apps, for FREE.

🚀 Launch an LLM App in One Hour
✨ Prompt Engineering
🗿 LLM Foundations
🔨 Augmented LLMs
🤷 UX for LUIs
🏎️ LLMOps
🔮 What's Next?
👷 Project Walkthrough

Learn more:

[Pradeep Thiyyagura @tpradeep]

@MicrosoftEdge I just love it!

[Pradeep Thiyyagura @tpradeep]

@MicrosoftEdge is bringing you more exciting new features. It’s your copilot as you explore the web in a whole new way. Generating high quality images or creative content for your posts is just a prompt away.

https://t.co/iG19gwKeBG

[Pradeep Thiyyagura @tpradeep]

@GregKamradt Looks like you are already going this route, saw your second comment just now.

[Pradeep Thiyyagura @tpradeep]

@GregKamradt If you want to preserve the original structure or flow of events, the initial set of summaries should be numbered bullet points along with the chapter number. This can be used to re-rank the top n search results if summarizing at a character or event level.

[Pradeep Thiyyagura @tpradeep]

@mattlungrenMD @StanfordAIMI @StanfordHAI @erichorvitz @drnigam @curtlanglotz @Idasim https://t.co/8PETRPQ76O
Link to the talk by @drnigam referenced in the post above. I completely agree that it is the best LLM 101 talk, not just in healthcare, but in general. Currently, there are too many models and too few actual real-world applications.

[Pradeep Thiyyagura @tpradeep]

I highly recommend taking a break from whatever you're doing to and check out this awesome TED talk by @gdb , the co-founder and CTO of @OpenAI. https://t.co/nJnSTtPyPz

[Pradeep Thiyyagura @tpradeep]

@swyx @babyAGI_ @AutoGpt I think I learn and remember the information better when I ask follow-up questions to an LLM on my own or click on some of the ‘learn more’ options. For me, the integration of Bing chat in Microsoft Edge browser is more helpful than babyAGI/AutoGPT for learning/research purposes.