Nilesh Barla

AI engineer studying agent reliability. I run experiments, write eval harnesses, and share the patterns.

[Nilesh Barla @thenielspace]

This week I was busy developing a custom GPT for research purposes. I aimed to create a foundational model that I experiment with on the go without requiring me to rent out GPUs. I wanted to see how my M1 Macbook Air would perform.

Here are some insights:

[Nilesh Barla @thenielspace]

I keep going back to this one thing Boris Cherny said on Lenny's podcast.

Boris built Claude Code at Anthropic. He hasn't written a line of code by hand since November 2025.

Same underlying model anyone can hit through the raw API. What he has that the raw API does not is the harness. A filesystem with persistent context. A deterministic shell. A structured tool surface. A write-test-fix loop. A way to ask for help when it gets stuck.

Same model. Wildly different agent.

That one anecdote is the whole thesis of the piece I just shipped in one sentence. The model is constant. The harness is the variable. The agent that emerges from that combination behaves like a different system entirely.

Inside the full piece:

Five layers of the harness: Instructions, tools, retrieval, orchestration, and evaluators.
> Two proof points: Voyager (2023) and Airbnb Data Flywheel (2025).
> The prompt trap: Why static prompts stop improving after v1.
> Ownership: Who on your team should own the harness.

Honestly, if you're still tuning prompts in isolation in 2026, you're optimizing the wrong layer. The interesting engineering is one abstraction up.

Full piece: https://t.co/XkhTW4gPZc

[Dwarkesh Patel @dwarkesh_sp]

Narration: the data efficiency black hole.

00:00:00 – What is really driving AI progress?
00:03:11 – Comparing human vs AI sample efficiency
00:08:46 – Does sample efficiency matter?

Also on pod and YouTube feed.

[Nilesh Barla @thenielspace]

Today I was exploring harness engineering with local MLX models.

Five things the harness handles that the model can't:
> When to stop generating.
> When to call a tool.
> What to do when the tool fails.
> How to keep context from blowing up.
> How to retry without losing state.

Get those right and a 4B local model holds. Skip them and the model drifts.

[Nilesh Barla @thenielspace]

Spent today fixing my agent. Barely wrote any prompts. Most of the work was harness engineering— guardrails, memory, what tools the model can call.

Note to myself: The harness decides what the agent is capable of. The prompt just decides what it tries.

[Nilesh Barla @thenielspace]

So when you're working on a project with agents, the system evolves and you keep adding instructions. What happened with me was at one point I had ~700 lines, and Claude got worse.

It would keep missing things. I'd keep reminding it. The context kept getting lost. Because every task pulled the same monolith in. Maybe 70% of what loaded was dead weight.

The model wasn't stupid. It was overstuffed.

1. One way I fixed it was using an instruction architecture. I cut the file into three layers:

2. A global file (~155 lines) that holds what genuinely applies everywhere — voice, banned words, competitor handling, where outputs save.

3. Task files under .claude/instructions/tasks/, one per content type, with only their format-specific specs.

An agent that acts as the router: /<task> loads global plus the right markdown and stops there.

The lesson? More instructions doesn't make the model better. It needs less context, but sharper.

[Nilesh Barla @thenielspace]

Introducing the self-improvement layer from Adaline 2.0. Static agents lose ground every week. I have watched this at several companies. There is no version where the static agent wins. This is what we are here to solve.

[Arsh Shah Dilbagi @arshdilbagi]

Introducing Adaline 2.0 - The Agent Self-Improvement Layer

Adaline turns Traces into Behaviors,
Behaviors surface Issues,
Issues become auto-generated Evals + Data,
Adaline then generates new agent candidates and tests them.

You review the winners and ship!

[Dwarkesh Patel @dwarkesh_sp]

Recently met @srush_nlp and he started giving me an impromptu lecture on how targeted on-policy self-distillation works.

I asked him if I could record it on my iPhone.

The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory.

So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made.

Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required.

The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

[Nilesh Barla @thenielspace]

These days spinning up an agent feels so good.

[Nilesh Barla @thenielspace]

Writing prompt for agentic workflow is like coding a transformer layer by layer.

[Nilesh Barla @thenielspace]

What production reliability actually requires beyond MCP https://t.co/NzL6VNmaZD

[Nilesh Barla @thenielspace]

I keep seeing the same mistake with MCP, i.e., confusing standardized tool access with reliable tool behavior. MCP handles the connection. Retries, silent failures, and observability are still your engineering problem.

[Nilesh Barla @thenielspace]

What production-ready agents actually require — the five failure modes and the measurement infrastructure that catches them before users do: https://t.co/h3Y7A2splD

[Nilesh Barla @thenielspace]

I've talked to a lot of engineers debugging production agents. The ones shipping stable systems share one thing: they decided measurement was part of the design, not something to add after users started complaining.

[Nilesh Barla @thenielspace]

Full breakdown on why AI agents call the wrong tool — the four failure modes and how to fix each at the description level: https://t.co/EfBeiCgbp2

[Nilesh Barla @thenielspace]

Looked at the AgentProp-Bench data. A wrong tool call cascades to a wrong final answer 62% of the time. Engineers reach for the model as the fix. The actual fix is the tool description — and those are completely different problems.

[Nilesh Barla @thenielspace]

https://t.co/wdg9aHVGCt

[Nilesh Barla @thenielspace]

Last week I added a fifth tool to a minimal coding agent, and selection fell apart immediately.

Here is why.
The model was not broken. It was resolving ambiguity. And when two descriptions are both a match, the model has no clean signal to act on, so it picks arbitrarily. You see this in any production agent with six or more tools.

The root cause of most wrong tool calls is not model capability, but description quality. And that means it has a description-level fix.

Full mechanism in the first reply.

[Nilesh Barla @thenielspace]

I wrote about why embeddings matter for AI agents here: https://t.co/2CqXbtN2Jq
Curious how others are evaluating retrieval quality inside agent workflows.

[Nilesh Barla @thenielspace]

Question for people building AI agents: How much time are you spending on the retrieval layer?
I think many agent failures get misdiagnosed as model failures.

But often the model is working with weak context.

Examples:
• Wrong document retrieved
• Stale memory surfaced
• Similar but irrelevant chunk selected
• Important constraint missed
• Tool input generated from partial context

At that point, improving the model may not fix the product.

The agent needs better context selection.

[Nilesh Barla @thenielspace]

Read the entire blog here: https://t.co/0zNfdTUQf3

[Nilesh Barla @thenielspace]

Tool-using agents need recovery paths.

The hard question is not: “Can the model call the tool?”
It is: “What happens if the action partially succeeds?”

Reliable agents need:
&gt; Idempotency
&gt; Typed errors
&gt; State tracking
&gt; Approvals
&gt; Retries
&gt; Trace replay