1/ Test #1: I ran 4 search APIs through 24 queries my AI agents actually send.
Google came last.
Yes, the Google. I thought my code was broken, so I reran it. Still last 🧵
I run a lot of AI tests while building: models, search APIs, agents, evals.
So I decided to log them here in public.
Hoping to build in public with the community here. Stay tuned.
@brocmayo Per test: relevance (nDCG@5, P@5 with an LLM judge), latency, cost per call, all broken down by query type.
First test was search APIs, not models. Biggest surprise: Google came last, and Brave put the official site at #1 on every navigational query https://t.co/HBncnSJQgz
1/ Test #1: I ran 4 search APIs through 24 queries my AI agents actually send.
Google came last.
Yes, the Google. I thought my code was broken, so I reran it. Still last 🧵
1/ Test #1: I ran 4 search APIs through 24 queries my AI agents actually send.
Google came last.
Yes, the Google. I thought my code was broken, so I reran it. Still last 🧵
@talesbytim totally. just posted Test #1: I ran 4 search APIs on agent queries, and google came last. i thought my code was broken so reran. still last.
6/ My setup:
Langfuse dataset: 24 queries tagged by type, with expected domains on the navigational ones
1 experiment per provider, 1 trace per query
Search call as a child span with latency, cost and result count
Ranking scores (nDCG, P@5) on the trace
5/ Plot twist: Google is shutting down its search API on Jan 1, 2027 anyway.
So the company that won the human web won't sell agents its raw results.
Test #2 is coming. Tell me what to break next.
Just published a quick read on how predictive analytics can totally level up your product roadmapping game! 👇🏼
#DataScience#ProductRoadmap
https://t.co/QlVqGoIc8R
We continue to make a difference one tab at a time, but as ad revenues fluctuate with market trends, we faced lower-than-anticipated revenues in Q4 2023. Therefore, we've decided to wait until Q1 2024 to merge earnings and maximize our impact. Stay tuned!
Join us in making a difference!
HyeTab's Q2 2023 earnings report is out, showcasing our investments in Armenia and Artsakh.
Your continued support has a direct impact on the development of our homeland. Open a tab, open opportunities.
#HyeTab#Armenia#Artsakh#InvestInArmenia
Our existence is a form of resistance against hate and intolerance. Let us honor the victims of the Armenian Genocide by standing strong, fighting for justice, promoting peace, and building a better future for all Armenians.
#ArmenianGenocide#1915NeverForget