I'll be frank. A lot of marketing tools are rushing out AI insight features because they demo well. But there's a dirty secret: AI won't tell you what it missed.
We found out the hard way.
We were a few months into building Kleerr's recommendation engine.
The goal was simple: use AI agents to find and qualify opportunities and fixes from the data.
The agents had everything they needed:
✅ Clean, quality data.
✅ Safeguards.
✅ Verification layers to prevent hallucinations and fake data.
✅ Thousands of datapoints for analysis.
Some results were great. They found patterns we never told them to look for.
But run the analysis twice and you get two different investigations.
One run catches the drop in qualified lead rate. The next focuses on a broken ad audience. Both mattered. But each run only caught one.
When you're using AI yourself, YOU are the quality control.
You notice when something looks off. You push back. You ask follow-ups.
Take the human out of the loop, run it at scale, and things slip:
❌ Accuracy drops
❌ Findings change from run to run
❌ Made-up numbers still show up
❌ Important checks get skipped
I can hear you now:
📢 "Write better prompts."
📢 "Give it a semantic layer."
📢 "Add evals and an LLM-as-judge."
📢 "Use multi-agent verification."
We tested a lot of this. None of it got results we were happy to put in front of customers, so it never shipped.
We're strict on this because marketers are going to make real decisions based on what we tell them.
Where to move budget. What to fix first. Whether a campaign is working.
If we miss something, they miss it too. Software has to be repeatable. Trustable.
So we changed the architecture.
We built a deterministic layer underneath: 300+ algorithms across 5,000+ datapoints, run every week. Same calculations, same checks, every time. It makes sure every check that matters runs and finds consistent opportunities.
Only then do we add an AI layer on top, doing what it's great at:
→ Finding unexpected relationships
→ Digging into anomalies
→ Connecting signals across channels
→ Explaining what might be happening
Deterministic analysis gives you coverage. AI gives you discovery.
You need both.
So, if you're evaluating AI analytics tools, here are 5 questions I'd ask:
1. What does it check every time, no matter what?
2. How does it verify its numbers and conclusions are correct?
3. Can it show me what it checked, not just what it flagged?
4. If I run it again on the same data, will it catch the same important issues?
5. Can I trace every insight back to the underlying data?
"What can the AI find?" is the exciting question.
"What can you guarantee it checks?" is the important one.
Have you caught AI missing something important in your data?
@gavin0922 Exactly. Controlled tests are the referee, but you still need a clean scoreboard. Deduped orders, refunds, new vs returning, one source of truth. Then test lift against that, not against each platform’s version of a conversion.
@marcobatt Do you optimize the ad conversion event around this too, or mainly use it for reporting?
Also curious if you’ve tested optimizing for new buyers > certain cart values?
Always wanting to see how the algos are responding to those signals.
Google laid out more of its measurement direction today.
The bigger direction is clear: attribution, incrementality and MMM are meant to work together, not compete for one “correct” answer.
The harder questions are still:
- Which one actually drives the budget decision?
- How confident do you need to be before changing spend?
- And after you make the change, how do you know it worked?
Making good decisions from all of it still looks like the harder problem.
Today, we’re announcing additional updates across our measurement suite designed to unify first-party data, eliminate integration friction, and give you causal proof of what is actually driving incremental revenue.
This builds on our recent Google Tag updates (details at the end).
Here is what’s rolling out, what problems they solve, and how to put them to work: 🧵
@adsliaison Really interesting to see MMM, geo experiments and attribution getting pulled closer together.
When those methods disagree, does Google have a point of view on which one marketers should use to actually make the budget decision?
@emaildeepdive Good points. I keep finding AI is great and all, but it tends toward mediocrity or the middle of the bell curve. Especially with content.
This is exactly what we've built into Kleerr.
We look across the customer journey before each purchase, determine which platforms contributed, and can send the appropriate conversion signal back.
No custom attribution pipeline. No complicated setup.
Meta's case study:
https://t.co/bvWkH4805g
Meta just published a case study validating a simple idea:
Don't train the algorithm on every sale it can claim credit for. Train it on the sales it actually contributed to.
The Oodie tested it:
+12% new-customer ROAS
-15.8% CAC
Signal quality > signal quantity.
Everyone talks about budget allocation.
Almost nobody talks about audience allocation.
It’s not just: where should the next dollar go?
It’s also: who should the next dollar reach?
Marketing optimization needs both.
Most martech is built around the first.
Kleerr is building around both.