Jacob Newman

Principal AI PM @ Amplitude

[Jacob Newman @Jacob__Newman]

(1/6) i'm the pm on Wave at @Amplitude, our agent for self-improving products. for the past few months we've had it running on our own product. it finds problems, ships fixes, and measures whether they worked. here are some highlights:

[Jacob Newman @Jacob__Newman]

@frankdotlee @AnthropicAI small feature request: make all claude models perfectly aligned to the well-being and long-term success of human civilization

congrats on the move 🫶

[Jacob Newman @Jacob__Newman]

@spenserskates we’re all having the same convos huh? 😂

[Jacob Newman @Jacob__Newman]

and as all great product origin stories go, this was yet another case where we built it first to serve our own needs:

my team was building our first data analytics agent (Global Agent) last year and realized that despite being an analytics company for 15 years, i had no clue how to assess quality.

traditional analytics told me about messages sent + retention, but it couldn’t tell me if the user got frustrated mid-chat or our agent hallucinated data. and i couldn’t connect those quality signals to what really mattered, which is whether an account was going to churn or renew. this product was the solve.

if anyone was curious how @Amplitude_HQ built first-class AI agents *and* drove their adoption across an AI-skeptical enterprise customer-base (>50%) this is it.

[Spenser Skates @spenserskates]

Today we're launching Agent Analytics

Every team shipping an AI agent has the same blind spot. Offline evals pass, you ship, and then you have no idea what's happening in production. AI fails silently. Users ask a question and get different answers. They all look 'engaged' in a classic dashboard. You don't know who got a great response and who got a terrible one.

Agent Analytics solves it:
- Every session scored out of the box on task completion, response quality, friction, safety, and negative feedback
- Topic clustering across thousands of conversations, so you know if a failure hits 1 user or 10,000
- Eval agents that watch for regressions, and if you want will file a Linear ticket or the pull request themselves
- Agent quality sits next to product data, so 'payment scheduling fails 31%' becomes 'which renewals did that cost us?'

The Economist got their agent to a 96.9% task success rate and cut weekly failures 84%.

Included on every plan. Free tier included. https://t.co/K5vqrlMLvd

[Jacob Newman @Jacob__Newman]

@spenserskates it’s SO important to get your entire team comfortable with writing and managing evals if you’re investing in agentic experiences in your product - love this!

[Jacob Newman @Jacob__Newman]

not just product analytics data, but customer feedback, session replay, and contextual data in your warehouse too :)

at @Amplitude_HQ we’ve built Wave, an agent that finds opportunities based on your data, pushes the code, and measures the impact in a full loop. check out my latest post for some real examples!

[Jacob Newman @Jacob__Newman]

one of the great learnings for us in terms of agent lifecycle is the idea of “triangulating evidence” during problem identification - it can often be tempting to take a single observation in data and spin off a recommendation, but humans don’t work that way. we often want to see the quant behavioral data, cross-reference with SR, validate across your customer feedback sources, etc. that’s what makes for a super compelling problem, and if that foundation is solid the agent can do a pretty good job at solutioning

[Jacob Newman @Jacob__Newman]

(1/6) i'm the pm on Wave at @Amplitude, our agent for self-improving products. for the past few months we've had it running on our own product. it finds problems, ships fixes, and measures whether they worked. here are some highlights:

[Jacob Newman @Jacob__Newman]

@quinn_darling @kristaletz @Amplitude there are different modes of autonomy and we want customers to select what is right for them. “by type” means that you might want to auto-approve bug fixes but require larger features to get human sign-off

[Jacob Newman @Jacob__Newman]

@andypshi @Amplitude of course! in general, Wave can query agent traces as well as the traditional event-based data. so in our case, it ingests its own traces and usage data to come up with opportunities to improve itself

[Jacob Newman @Jacob__Newman]

@barelyknown @Amplitude HAH thanks for flagging, sadly too late to edit 😅

[Jacob Newman @Jacob__Newman]

(6/6) the most important takeaway here is that these improvements weren't the result of a human-curated roadmap or hours of steering an agent. Wave simply identified real opportunities based on behavioral data while we went about our other work. i truly believe Wave will be the first agent to meaningfully crack the "self-improving products" problem

→ https://t.co/SaWFyFuh8r

[Jacob Newman @Jacob__Newman]

(5/6) we were just shooting ourselves in the foot on this one... Wave found that readers of our blog posts had no meaningful way to engage with us after they finished reading. it simply added a cta matched to each post's topic. early signal is about 45 extra sign-ups a week from the blog. these are high-intent leads!

[Sam Broner @SamBroner]

The fact that SaaS co's are giving us agents instead of MCPs shows just how self-centered / narcissistic they are.

I never wanted to "live in your product." Your product was the best way to get something done. If the best way to get it done is now headless, please let me do that.

[Jacob Newman @Jacob__Newman]

i think the passion, personality and beauty of great writing has always come from the individual behind the pen/computer

i’m excited for a future where AI allows more passionate people to articulate and share their perspective on the world - not in stale Claudlish but a faithful reproduction of what’s in their head

[Jacob Newman @Jacob__Newman]

i believe we will look back on the panic around AI-generated writing at work with some humility in 5 years time

here’s my defense of AI slop: https://t.co/MvbDhtBTN3

[Chandra @NCResq]

As someone tired of reading AI slop documents all the time love this policy from Clay

https://t.co/d2kmc66kch

[Jacob Newman @Jacob__Newman]

good take. i’ve noticed a surprising trend in enterprise AI products *towards* complexity.

this is especially true in marketing - we used to love simplicity in storytelling and now we’ve reverted to overly technical, architecture-as-copy language. can we stop talking about “harnesses” yet?

probably an overcorrection to a period in early 2025 where everyone overpromised simple stories of AI value and under-delivered

[Fletcher Richman @fletchrichman]

It's actually crazy watching everyone pivot in the complete wrong direction from what companies want.

Having an army of agents/bots is counterproductive. It's a vanity metric.

What teams want is a shared workspace where they can:
- work with any model
- build a "company brain" of skills, integrations, and context/memory mapped to their permissions
- build and host custom apps
- interact from slack, email, or wherever they work

[Fletcher Richman @fletchrichman]

It's actually crazy watching everyone pivot in the complete wrong direction from what companies want.

Having an army of agents/bots is counterproductive. It's a vanity metric.

What teams want is a shared workspace where they can:
- work with any model
- build a "company brain" of skills, integrations, and context/memory mapped to their permissions
- build and host custom apps
- interact from slack, email, or wherever they work