Jacob Newman

Principal AI PM @ Amplitude

[Jacob Newman @Jacob__Newman]

(1/6) i'm the pm on Wave at @Amplitude, our agent for self-improving products. for the past few months we've had it running on our own product. it finds problems, ships fixes, and measures whether they worked. here are some highlights:

[Jacob Newman @Jacob__Newman]

@spenserskates we’re all having the same convos huh? 😂

[Jacob Newman @Jacob__Newman]

and as all great product origin stories go, this was yet another case where we built it first to serve our own needs:

my team was building our first data analytics agent (Global Agent) last year and realized that despite being an analytics company for 15 years, i had no clue how to assess quality.

traditional analytics told me about messages sent + retention, but it couldn’t tell me if the user got frustrated mid-chat or our agent hallucinated data. and i couldn’t connect those quality signals to what really mattered, which is whether an account was going to churn or renew. this product was the solve.

if anyone was curious how @Amplitude_HQ built first-class AI agents *and* drove their adoption across an AI-skeptical enterprise customer-base (>50%) this is it.

[Spenser Skates @spenserskates]

Today we're launching Agent Analytics

Every team shipping an AI agent has the same blind spot. Offline evals pass, you ship, and then you have no idea what's happening in production. AI fails silently. Users ask a question and get different answers. They all look 'engaged' in a classic dashboard. You don't know who got a great response and who got a terrible one.

Agent Analytics solves it:
- Every session scored out of the box on task completion, response quality, friction, safety, and negative feedback
- Topic clustering across thousands of conversations, so you know if a failure hits 1 user or 10,000
- Eval agents that watch for regressions, and if you want will file a Linear ticket or the pull request themselves
- Agent quality sits next to product data, so 'payment scheduling fails 31%' becomes 'which renewals did that cost us?'

The Economist got their agent to a 96.9% task success rate and cut weekly failures 84%.

Included on every plan. Free tier included. https://t.co/K5vqrlMLvd

[Jacob Newman @Jacob__Newman]

@spenserskates it’s SO important to get your entire team comfortable with writing and managing evals if you’re investing in agentic experiences in your product - love this!

[Jacob Newman @Jacob__Newman]

not just product analytics data, but customer feedback, session replay, and contextual data in your warehouse too :)

at @Amplitude_HQ we’ve built Wave, an agent that finds opportunities based on your data, pushes the code, and measures the impact in a full loop. check out my latest post for some real examples!

[Jacob Newman @Jacob__Newman]

one of the great learnings for us in terms of agent lifecycle is the idea of “triangulating evidence” during problem identification - it can often be tempting to take a single observation in data and spin off a recommendation, but humans don’t work that way. we often want to see the quant behavioral data, cross-reference with SR, validate across your customer feedback sources, etc. that’s what makes for a super compelling problem, and if that foundation is solid the agent can do a pretty good job at solutioning

[Jacob Newman @Jacob__Newman]

(1/6) i'm the pm on Wave at @Amplitude, our agent for self-improving products. for the past few months we've had it running on our own product. it finds problems, ships fixes, and measures whether they worked. here are some highlights:

[Jacob Newman @Jacob__Newman]

@quinn_darling @kristaletz @Amplitude there are different modes of autonomy and we want customers to select what is right for them. “by type” means that you might want to auto-approve bug fixes but require larger features to get human sign-off

[Jacob Newman @Jacob__Newman]

@andypshi @Amplitude of course! in general, Wave can query agent traces as well as the traditional event-based data. so in our case, it ingests its own traces and usage data to come up with opportunities to improve itself

[Jacob Newman @Jacob__Newman]

@barelyknown @Amplitude HAH thanks for flagging, sadly too late to edit 😅

[Jacob Newman @Jacob__Newman]

(6/6) the most important takeaway here is that these improvements weren't the result of a human-curated roadmap or hours of steering an agent. Wave simply identified real opportunities based on behavioral data while we went about our other work. i truly believe Wave will be the first agent to meaningfully crack the "self-improving products" problem

→ https://t.co/SaWFyFuh8r

[Jacob Newman @Jacob__Newman]

(5/6) we were just shooting ourselves in the foot on this one... Wave found that readers of our blog posts had no meaningful way to engage with us after they finished reading. it simply added a cta matched to each post's topic. early signal is about 45 extra sign-ups a week from the blog. these are high-intent leads!

[Sam Broner @SamBroner]

The fact that SaaS co's are giving us agents instead of MCPs shows just how self-centered / narcissistic they are.

I never wanted to "live in your product." Your product was the best way to get something done. If the best way to get it done is now headless, please let me do that.

[Jacob Newman @Jacob__Newman]

i think the passion, personality and beauty of great writing has always come from the individual behind the pen/computer

i’m excited for a future where AI allows more passionate people to articulate and share their perspective on the world - not in stale Claudlish but a faithful reproduction of what’s in their head

[Jacob Newman @Jacob__Newman]

i believe we will look back on the panic around AI-generated writing at work with some humility in 5 years time

here’s my defense of AI slop: https://t.co/MvbDhtBTN3

[Chandra @NCResq]

As someone tired of reading AI slop documents all the time love this policy from Clay

https://t.co/d2kmc66kch

[Jacob Newman @Jacob__Newman]

good take. i’ve noticed a surprising trend in enterprise AI products *towards* complexity.

this is especially true in marketing - we used to love simplicity in storytelling and now we’ve reverted to overly technical, architecture-as-copy language. can we stop talking about “harnesses” yet?

probably an overcorrection to a period in early 2025 where everyone overpromised simple stories of AI value and under-delivered

[Fletcher Richman @fletchrichman]

It's actually crazy watching everyone pivot in the complete wrong direction from what companies want.

Having an army of agents/bots is counterproductive. It's a vanity metric.

What teams want is a shared workspace where they can:
- work with any model
- build a "company brain" of skills, integrations, and context/memory mapped to their permissions
- build and host custom apps
- interact from slack, email, or wherever they work

[Fletcher Richman @fletchrichman]

It's actually crazy watching everyone pivot in the complete wrong direction from what companies want.

Having an army of agents/bots is counterproductive. It's a vanity metric.

What teams want is a shared workspace where they can:
- work with any model
- build a "company brain" of skills, integrations, and context/memory mapped to their permissions
- build and host custom apps
- interact from slack, email, or wherever they work

[Jacob Newman @Jacob__Newman]

we have the most honest policy, which is free and unlimited ai for our customers :)

but even if your business charges for ai, you’re likely using a credit model that abstracts tokens. if you optimize cost, it’s to improve your own margins. no one is passing those savings on, hence your customers don’t care

imo best long-term-greedy strategy is to obsess over agent quality and only address cost as required by COGS pressure

[Jacob Newman @Jacob__Newman]

unpopular truth: your customers don't care how much your agent costs to run

the real downside of a successful, well-subscribed agent is that cost balloons fast. we know this well at @Amplitude_HQ , because our Global Agent is increasingly the primary way our customers use the platform and the inference is putting pressure on margins

the knee-jerk reaction for any team is jumping to open-weight models and blindly cost-cutting

but running an agent is pareto optimization across 3 variables: cost, latency, quality. cost and latency are easy to measure, so teams over-index on them. quality has been largely vibes-based, and yet it's the most important to your users

we built agent analytics to make quality quantitative: live evals, accessible to everyone on the team, measuring every dimension of quality. it let us ask the right question: which model cuts cost without sacrificing quality or latency?

spoiler: a blind model switch doesn't work. kimi k2.7 was cheap and fast, but it scored worse than our incumbent (sonnet 4.6) out of the box on the dimension i cared most about as the PM: answering analytics questions accurately

agent analytics pointed us to the top failure modes introduced by kimi - so we remediated and tweaked the harness

the outcome? a faster, more accurate Global Agent at a third of the cost

and we’re just getting started with fine-tuning 😉