Shiv Mohith

AI is extremely capable but not reliable. I want it to be reliable so I tweet about whenever I can.

Building confidence layer for conversational agents

[Viv @Vtrivedy10]

experts in their fields know
1. exactly how to ask for what they want
2. how to check if work is good

think this is my favorite explanation for why AI seems to accelerate experts in their own fields (coding, design, market research, etc) more than novices

the best ppl in their fields have spent thousands of hours understanding exactly what they need and important and underrated part…

they literally know the words to describe their thing properly. and LLMs are trained on those words so they can reproduce that

ex: “make this look nicer”
vs
“Increase section padding to 48–64px, limit the UI to a neutral gray palette plus a single blue accent color, reduce border usage, and use subtle shadows instead of heavy outlines.”

this is true for all domains imo, being good at specifying intent in a detailed way is a superpower with agents

and the more we can corral agents on the outputs we need, the better outputs we get

[Shiv Mohith @shiv_mohith]

@TommiPedruzzi @NavalismHQ Claude

[Shiv Mohith @shiv_mohith]

@hwchase17 I would like an example that does document analysis. Given a set of documents and rules, it analyzes those documents and provides a report following the rules. Like contract analysis in legal domain.

[Shiv Mohith @shiv_mohith]

@NirantK Do we really need a harness for this use case? Wouldn't a workflow solve it? Memory extraction could be a step in the workflow. Just thinking is harness an overkill or maybe your use cases needed it 🤔

[Shiv Mohith @shiv_mohith]

@agupta @ycombinator Building the confidence layer for conversational agents.

Simulate, Evaluate & Monitor agents.

https://t.co/VbO6DFsaYs

[Shiv Mohith @shiv_mohith]

@0xN1nja @grafana Thanks! Will take a look.

[Shiv Mohith @shiv_mohith]

@javarevisited https://t.co/aGq5uW6k7O

[Shiv Mohith @shiv_mohith]

Do you time your coding breaks around your Claude Code session?

If you have an hour left and have used only 50% of your limit, it feels wasteful to step away when you can still get a lot done.

[Shiv Mohith @shiv_mohith]

Before LLMs, junior devs mostly learned from their immediate team. Your growth depended on who you worked with.

Now, skills + GenAI break that boundary. You can learn workflows and thinking directly from the best, like Andrej Karpathy, without being in the same room.

[Shiv Mohith @shiv_mohith]

@IfeanyiArinze10 @ionleu It's a testing & monitoring platform for chat/voice AI agents (customer support, scheduling, etc.). Before launch, simulate real users across scenarios, run them repeatedly since LLMs are unpredictable. After launch, get insights on failures and iterate.

[Shiv Mohith @shiv_mohith]

@TriCast_ https://t.co/FME9DSmcvt - confidence loop for the agent loop.

Please do connect!

[Shiv Mohith @shiv_mohith]

@kakatohesss 101%

[Shiv Mohith @shiv_mohith]

@1Umairshaikh Your first 10 users want something different from what you built.

[Shiv Mohith @shiv_mohith]

Fix -> Ship -> Learn loop is the only reliable way to improve agent loop

[Shiv Mohith @shiv_mohith]

@FlippedRay https://t.co/FME9DSmcvt - confidence loop for the agent loop

[LangChain @LangChain]

LangSmith 🤝 San Francisco

You don't know what your agents will do until you actually run them. What works in demos can break in the real world.

Without tracing and evals, you're just guessing at why. Track what your agent actually does. Optimize and fix your agents. Then measure whether your fixes work.

That loop is how agents get better, and LangSmith is built to power that workflow.

[Shiv Mohith @shiv_mohith]

@DhravyaShah Just a curious thought. When memory is enabled, the agent becomes much more stateful. How do you or your customers do evaluation? Since you have almost 5k members, somebody must have faced this issue 🤔

[Shiv Mohith @shiv_mohith]

@hwchase17 @LangChain So true. That's why I'm building the loop for conversational agents specifically - https://t.co/FME9DSmcvt