Deepak Sharma

AI Engineering for Enterprise at Microsoft

@[email protected]

[Deepak Sharma @deepaks4077]

@boardyai Count me in for Boardy Pro

[Deepak Sharma @deepaks4077]

Generational Montreal summer this year <3

[Deepak Sharma @deepaks4077]

@natolambert @JoshPurtell @allen_ai Thank you so much for all the work you’ve done!

[Deepak Sharma @deepaks4077]

@Vtrivedy10 - I think it would be super helpful to see how folks at langchain add/remove evals over time or optimize eval sets for different constraints. For example, evals for PR builds can’t take too long so you need to build the smallest subset that has coverage of core user scenarios, provides enough signal, isn’t flaky, quick and cheap enough to run on each PR.

[Deepak Sharma @deepaks4077]

@charles_irl oh my god

[Deepak Sharma @deepaks4077]

@SOURADIPCHAKR18 @lateinteraction @amritsinghbedi3 @NoahZiems Is this Active Learning + RL?

[Yann LeCun @ylecun]

@eladgil BS.
Attention was born in Montréal
PyTorch in NYC.
AlphaGo in London
AlphaFold in London
ESMFold in NYC
Llama 1 in Paris.
Llama 2 in Paris+NYC+SV
DeepSeek in Hangzhou

Plus:
DINO in Paris
JEPA in Montréal+Paris+NYC

SV is 3 mos ahead on topics SV is singularly obsessed with.

[Deepak Sharma @deepaks4077]

I’ve had loadsss of fun building an agent for multi-record entity extraction from enterprise documents. It uses gpt41mini, ~20turns with parallel tool calling of 5-7 tools.

Biggest surprise has been how much agentic and reasoning behaviour you can get out of ‘legacy’ chat models <3

[Justin Schroeder @jpschroeder]

what. what. what.
gpt-image-2 almost passes the pelican test...in a screenshot of a code editor.

[Deepak Sharma @deepaks4077]

@IntuitMachine Is this built with Codex?

[Deepak Sharma @deepaks4077]

I’ve found this to be a super useful mental model for agent harnesses.

[Viv @Vtrivedy10]

Harness, Memory, Context Fragments, & the Bitter Lesson

this is a work in progress mental dump on interesting intersections between how we use and design a harness, implications for memory being accumulated over long timescales, and the search bitter lesson we can’t escape

this is v30+, HTML diagrams help me iteratively refine + chat to roughly “see” and alter the mental model

Harnesses & Context Fragments:
a very important job of the harness is to efficiently & correctly route data within its boundaries into the context window boundary for computation to happen

the context window is a precious artifact. Harnesses make decisions on how to populate, manage, edit, and organize it so agents can do work. Each loaded object can be thought of as a Context Fragment and represents an explicit decision by the user and harness designer of what needs a model needs to do work at any given time.

many ideas on externalizing objects + loading into the context window are pioneered and very well described by @a1zhang with RLMs

Experiential Memory:
we’re in the very early days of deploying agents and agents produce massive amounts of data in every interaction they have. this is akin to humans doing things and remembering things they did.

however agent memory has a massive advantage as it can be accumulated across all agents which are easily forked and duplicated (unlike humans). @dwarkesh_sp does a good talking about this massive benefit of artificial systems

memory can be treated as an externalized object. the harness is tasked with doing good contextualized retrieval which means pulling in the right data from accumulated memories across all agent interactions

Search & The Bitter Lesson:
As we deploy agents in our world over year timescales, there is going to be a hyper-exponential in the amount of data produced by those agents. We should want to:
1. Own that data for ourselves. Open ecosystems are important here
2. Use that data

This means that we’ll have to search over, distill, and organize massive amounts of data. Our brain is exceptional at doing this. Both contextually using prior experience and mostly committing the right stuff to memory with enough intentional practice.

Our current infrastructure systems and algorithms will be put to the test and often break as we get used to this new data regime

some open questions:
- how do we efficiently distill experiences (Traces) into higher level memory primitives that capture the important parts? How do we do this over ultra long time horizons?

- How much of the future is Search just-in-time vs Search that gets integrated into model weights?

- How do we make models much better at self-managing their context window? How do we reduce error rates in recursively allowing agents to operate over external objects?

i’ll be expanding on, altering, and adjusting these mental models but these feel like an important subset to me on the future of designing agents practically

[Deepak Sharma @deepaks4077]

@ivanburazin I think @_ardensh has the right idea for this!

[Arden @_ardensh]

AI agents + Stripe = real risk

In this demo, an agent attempts unsafe actions

Arden intercepts and blocks them at runtime

First end-to-end guardrails demo

[Deepak Sharma @deepaks4077]

@Kimi_Moonshot @cursor_ai Lmaooo

[Sherwood @shcallaway]

GEPA for pre-training

[Omar Khattab @lateinteraction]

I still find it borderline stupid that coding agents seem inclined to use APIs or libraries in complex scripts before tinkering at small scale, as in bottom-up notebooks, to make sure they're modeling these APIs correctly.

Who is responsible for this and what are they thinking.

[Deepak Sharma @deepaks4077]

Orchestrating multiple coding agents helps you speedrun through software best practices. It teaches you best practices the right way.

[Deepak Sharma @deepaks4077]

Hire candidates based off of their Claude /insights ;)

[Deepak Sharma @deepaks4077]

Every knowledge worker will become a process engineer by 2027? 2030?