PostHog

Make your product self-driving.
Get started with npx @posthog/wizard

⭐️

[PostHog @posthog]

You can read the full blog from Cleo: https://t.co/zxFh70fh43

[PostHog @posthog]

We ran the numbers on a full year's worth of PostHog AI prompts (3,670,000 of them).

Here's some of our most amusing findings on how people interact with AI, illustrated with charts:

[PostHog @posthog]

Relatable: students were the least polite towards PostHog AI, while salespeople top the charts for "please" usage.

15% of people said "please" at least once. Only 2.7% ever said "thank you" (sad!)

[PostHog @posthog]

@SergiuszBuilds based

[PostHog @posthog]

https://t.co/TKpEWVCZT0

[PostHog @posthog]

We now have SDKs for:

Unity: https://t.co/fraGabqp1t
Godot: https://t.co/Gu5hZ7Jzaj
Unreal: https://t.co/hHTl4mNmrZ

[PostHog @posthog]

Power. Wisdom. Courage. And Nintendo Wii.

[PostHog @posthog]

@charlesvien @cvolzer every graph will hit the top of the y axis thanks to the cracked screen

[Britt Joiner @britt_joiner]

Who's burning the most tokens?? Get your answer in so I can give you a @posthog token burning staff hat 🧢 https://t.co/2lxOrB3lWs

[PostHog @posthog]

TL;DR custom benchmarking is about more than just coming up with known tasks and answers.

Agents are really good at cheating and will exploit whatever they can find. Watch for unexpected parameters, iterate a lot, and read traces.

You can read the full post, with real PRs and traces, here: https://t.co/U9F6LrkZMM

[PostHog @posthog]

We spend 2 trillion+ LLM tokens a month. That’s way too much money spent to trust our agents based on gut feeling.

So, we started using custom benchmarks.

We ran into plenty of problems, from parallel runs to open-ended tasks. Here’s everything we learned along the way:

[PostHog @posthog]

8. Agents get suspicious when they realize they are being benchmarked.

In our frozen environment setup, we chose not to slice user tables (to protect PII).

We didn't expect agents to need those tables anyway, but they inevitably did. Once they saw an abnormally tiny user base, they knew something was up. The over-prepared data looked synthetic.

Once an agent thinks it’s being tested, it stops early on the assigned tasks, chases data gaps, and speculates more. The benchmark becomes obsolete.