Vikram Chatterji

Co-founder @rungalileo GenAI Development Platform
// Prev Product @GoogleAI

[Vikram Chatterji @vikramchatterji]

Excited to finally talk about what we have been working on at @rungalileo for the past year! Building the data intelligence toolchain for ML developers working with unstructured data.

Launch announcement: https://t.co/zJFsIKUuFf
More here: https://t.co/2gOgVqM8T2

[Galileo, now part of Cisco @rungalileo]

Scaling agentic systems means preparing for hundreds or even thousands of agents operating in production. But achieving this level of scale brings critical new questions:

– How will you provision and orchestrate these agents?
– What about authentication and authorization?
– How will you evaluate, measure, and ensure reliability?

@crewAIInc CEO and Co-founder @joaomdmoura joined us on the Chain of Thought podcast to discuss the emerging agentic stack, and why the agentic future will require an entire ecosystem, from your databases to your user interfaces.

We’re proud to partner with industry leaders like CrewAI, learn more in this week’s episode with João, @vikramchatterji, and @ConorBronsdon 👇

[AMD @AMD]

ICYMI: Recorded live at #AdvancingAI, check out this special episode of Chain of Thought podcast hosted by @ConorBronsdon with @AnushElangovan and @realSharonZhou from @AIatAMD

[Galileo, now part of Cisco @rungalileo]

Debugging agents shouldn’t feel like detective work.

Today, we’re excited to release two new AI agent interfaces that make agent observability & evaluations even more effective.

🔎 Timeline View – See execution flow and bottlenecks at a glance. No more guessing where your agent gets stuck.

💬 Conversation View – Experience exactly what your users see. Debug from the user's perspective, not just the system's.

Combined with last week's Graph View, you now have three complementary ways to debug your agents:

→ Graph: Visualize decision paths and tool usage
→ Timeline: Spot performance bottlenecks instantly
→ Conversation: See the user experience end-to-end

AI evaluations + observability are crucial to building reliable AI. These interfaces make it simpler to identify blockers and improve your agents faster.

See all three views in action, and try it free with the link below 👇

[Galileo, now part of Cisco @rungalileo]

🔥 𝗝𝗨𝗦𝗧 𝗥𝗘𝗟𝗘𝗔𝗦𝗘𝗗: 𝗢𝘂𝗿 𝗟𝗮𝘁𝗲𝘀𝘁 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁 𝗟𝗲𝗮𝗱𝗲𝗿𝗯𝗼𝗮𝗿𝗱 𝗦𝗵𝗼𝘄𝘀 𝗦𝘂𝗿𝗽𝗿𝗶𝘀𝗶𝗻𝗴 𝗥𝗲𝘀𝘂𝗹𝘁𝘀

We've just updated our AI Agent Leaderboard at Galileo, and the performance rankings challenge conventional wisdom about which models deliver the best value for AI agents.

The headline finding: Gemini-2.0-flash-lite dominates with a 0.933 performance score, outperforming GPT-4.5 at a fraction of the cost.

Three critical insights from our comprehensive evaluation:

• 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲-𝘁𝗼-𝗖𝗼𝘀𝘁 𝗥𝗮𝘁𝗶𝗼: The top 3 models and GPT-4.5 span a staggering 1000x price difference while showing only a 2% performance gap. This raises important questions about cost efficiency in production AI agents.

• 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲 𝗣𝗿𝗼𝗴𝗿𝗲𝘀𝘀: Mistral-small-2501 leads the open source category at 0.83, performing on par with GPT-4o-mini. This signals the growing maturity of open source models for tool-calling capabilities.

• 𝗠𝗼𝗱𝗲𝗹 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗛𝗶𝗲𝗿𝗮𝗿𝗰𝗵𝘆: Claude-3.7-sonnet (0.953) > Gemini-2.0-flash (0.938) > GPT-4.5 preview (0.900) demonstrates a clear performance ranking across the major AI providers.

Our evaluation covered 20 models across 14 diverse datasets, assessing real-world AI agent capabilities and tool selection quality.

𝚆̲𝚑̲𝚊̲𝚝̲'̲𝚜̲ ̲𝙽̲𝚎̲𝚡̲𝚝̲?̲
We're raising the bar. Our upcoming evaluations will incorporate more challenging metrics focused on real-world scenarios with additional complex and specific datasets.

As AI agents grow more sophisticated, the foundation models powering them must improve in decision quality, goal alignment, and task completion—all while maintaining reasonable costs for builders.

What other metrics or test cases would you like to see in our next evaluation?

Check out the full updated leaderboard and methodology below 👇

[Galileo, now part of Cisco @rungalileo]

𝗕𝗿𝗲𝗮𝗸𝗶𝗻𝗴: Claude 3.7 Sonnet claims the top spot on our AI Agent Leaderboard!

Our comprehensive evaluation shows @AnthropicAI's newest model achieving a 0.953 TSQ score, narrowly edging out Gemini 2.0-flash (0.938) and GPT-4o (0.900).

Looking at the performance-to-cost ratio reveals an interesting story: While Claude 3.7 delivers exceptional performance, Gemini 2.0-flash still offers remarkable value at just $0.15/$0.60 per million tokens—20x cheaper than some competitors with comparable capabilities.

As @OfficialLoganK + @ConorBronsdon discussed in our recent podcast, tool calling capabilities continue to evolve rapidly. The competitive landscape shows how quickly models are advancing in their ability to accurately select tools, orchestrate multi-step processes, and handle edge cases.

Check out our updated leaderboard to see how your preferred model stacks up 👇

[Vikram Chatterji @vikramchatterji]

With the cost of intelligence and compute going down fast, the 'quality' of your AI product's is the moat. A well thought through layer of offline and online evals make the difference between otherwise commoditized product experiences across AI verticals.

[Garry Tan @garrytan]

Evals are emerging as the real moat for AI startups

Hard won insights about customers and their business logic discovered by founders acting almost as ethnographers spelunking in the underserved slices of the GDP pie chart

[Alex Rudall @alexrudall]

Chain of Thought from @rungalileo is the best podcast I’ve found so far that’s focused on productionising genAI. Worth a listen

[Bob van Luijt @bobvanluijt]

🤩 Excited to join the @rungalileo virtual panel tomorrow!
💡 Join us as we dive into Agentic Architectures, Generative Feedback Loops, and more
🗣️ with: @_brian_raymond , @joaomdmoura, @vikramchatterji, and yours truly
📅 Details: October 29th, Virtual, Free
🎟️ Register here: https://t.co/TlGfGUyxoU

[Unstructured @UnstructuredIO]

Excited for @rungalileo GenAI Productionize - happening tomorrow! Sign up below to watch @_Brian_Raymond and other leaders chat on the latest in AI agents and GenAI.

📅 Details: October 29th, Virtual, Free
🎟️ Register here: https://t.co/vxNmGqe7YM

[Forbes @Forbes]

This AI Startup Raises $45 Million To Make Sure AI Models Don’t Hallucinate Or Leak Data https://t.co/mw1PhWrmtO https://t.co/mw1PhWrmtO

[Stack Overflow @StackOverflow]

🎙️We chat with @vikramchatterji, founder and CEO of @rungalileo, about the challenges of evaluating GenAI models, the importance of data quality in AI systems, and the trade-offs between using pre-trained models and fine-tuning models with custom data.

https://t.co/mXkZQYbmFQ

[Vikram Chatterji @vikramchatterji]

@surojit Congrats @surojit !!

[Vikram Chatterji @vikramchatterji]

@ashah0052 @TheTuringPost @ashah0052 here's more on the methodology behind the index: https://t.co/b6NSZTUInU

Tl;dr The ranking is based on a combo of the ChainPoll method (desc in the link above), and human reviewers as an additional check, across multiple popular datasets.

[agentnative @agentnativedev]

LLM Hallucination Index is practitioner focused, intuitive and straight to the point.

“Open AI's GPT-4-0613 performed the best and was least likely to hallucinate for Question & Answer with RAG. Huggingface's Zephyr-7b was the best-performing open-source model, outperforming Meta's 10x larger Llama-2-70b, proving larger models are not always better.”

It will take sometime to build trust in enterprise deployments, but we are getting there!

Great work from @rungalileo!

https://t.co/GjFxAFt7kW

[Galileo, now part of Cisco @rungalileo]

1/ This week, we launched the Hallucination Index, which ranks popular LLMs on their propensity to hallucinate for common GenAI tasks.

We evaluated 11 LLMs across 3 GenAI tasks using 2 powerful metrics.

Here’s what we found👇

#AI #LLM #Hallucinations #HallucinationIndex

[Vikram Chatterji @vikramchatterji]

Excited to unveil the Hallucination Index!

Dive into the rankings and the methodology: https://t.co/wJ4pq9IBjs
The goal is purely to help builders work with the right LLM for their nuanced tasks.

[Galileo, now part of Cisco @rungalileo]

🚀 Unveiling the Hallucination Index! 🚀 Evaluating LLM output quality across real-world tasks, it addresses the challenge of hallucinations with a structured framework.

Learn more: https://t.co/vQbPGGSOqe 🌐✨

#AI #Hallucinations #GenerativeAI

[Battery Ventures @BatteryVentures]

What we're reading today: @fortunemagazine's profile of Battery portfolio company @rungalileo and its co-founders @YashSheth46, @atinsanyal and @vikramchatterji!

Check out the full story from @polina_marinova here: https://t.co/8u17yKODiW

[Vikram Chatterji @vikramchatterji]

@karpathy Super critical to build custom LMs with the *right*, *high-quality* data that is *contextual* to the use case -- https://t.co/pp03fI14xb is a LLM prompt and diagnostics tool that aims to turbocharge exactly that!

[Vikram Chatterji @vikramchatterji]

Thousands of people at the exploratorium for a ML meetup? This is what a new wave in technology looks like. Such great ideas and energy at the @huggingface open source AI event today!
#WoodstockAI

[Pete Warden @petewarden]

I'm excited that @rungalileo are finally public! They've built a wonderful tool for automatically cleaning up NLP training data, it's an easy way to boost your model accuracy.