Ankit Mathur

Engineering Lead for Agentic Infrastructure @databricks | Scout @greylockvc (angel investing in data/ML/cloud) | prev: @stanford @ucbrise

[Rishabh Singh @rishabhs]

Enterprise context is key to making knowledge agents perform well in the real world.

Existing data and knowledge agent benchmarks primarily focus on tabular data, whereas actual enterprise tasks often require leveraging business context present in multiple sources such as dashboards, notebooks, historical queries, documents, and metric views.

We partnered with @turingcom to create a new benchmark that reflects this real-world enterprise complexity. We observed that Genie (powered by Genie Ontology) significantly outperforms frontier agents:
• 1.6x accuracy
• 4x faster
• 50% fewer tokens

More details: https://t.co/uOCoIFF1Hl

[{Jules Damji}=💻📚📝 @2twitme]

Count the commands between an empty directory and a running agent. That number is your developer experience.

A decade ago, I argued that any SDK or platform that wants adoption should respect the rule of three.

[Ali Ghodsi @alighodsi]

A year ago, I ran @databricks totally differently from today. Getting stuff done at Databricks involved calling ppl, scheduling meetings, and lots of back and forth.

Today, I just wake up and use Genie, which works for me in the background. I often just use the phone app to queue up work in the background.

Three examples from just this week:
→ Org charts. How do we flatten the org? It literally found me the hotspots where many projects had gotten stuck. Turns out the data was already in the ontology graph.

→ New internal project everyone raves about (Keanu). Genie monitored the slack channel, got all the insights, correlated them with the source code, and gave me a dashboard of what I needed.

→ New culture survey. Used ontology to figure out from the org chart, work output, and survey scores, which leaders are AI pilled. Why certain scores (e.g. work-life balance) are so low.

These are things that used to take me hours and many meetings, often waiting for the data science team. This is possible because Genie has access to an Agentic Ontology, so it proactively does all the work in the background for me.

Learn more about our next-generation release of Genie:
https://t.co/H5nRGviy15

[Debnil Sur @debnilsur]

forgot ex-@BellarmineBells (and our secret co-founder 👀 )

[Ali Ghodsi @alighodsi]

Day 0 launch of Claude Fable 5.2 on @databricks! You can just get the tokens from Unity Gateway on Databricks.

[Ankit Mathur @ankit_math]

I’m speaking at the @aiDotEngineer about agent infra for the next generation of agents!

Join me Wednesday, 11:45AM on the main stage

Session details here: https://t.co/tJVlGN9A7j

[Databricks @databricks]

Claude Fable 5.2 lands on Databricks on Day 0.

Built for coding, finance and science, it sets a new high on our OfficeQA Pro v2 benchmark.

Run long-running agents on governed data, with Unity Gateway controlling calls across 70+ models.

[Erik Bernhardsson @bernhardsson]

Feels like a lot of Jevon’s paradox arguments are a bit sloppy.

Prices go up: “great for revenue”
Prices go down: “Jevon’s paradox, great for revenue”

Like, you should be honest about the sign of the partial derivative of revenue wrt prices

[Jamin Ball @jaminball]

https://t.co/rojJ2ix09C

[Sabhya Chhabria @sabhyac267]

New @databricks office dropped in sf, come say hi 👋

[Ali Ghodsi @alighodsi]

Zero Day Launch of Kimi 3.1! This is an amazing open source model. It seems particularly good at coding! Try it on Databricks. Even better, connect your models to Unity Gateway and use Smart Routing to control your cost vs quality.

[Yuchen Jin @Yuchenj_UW]

We're launching Kimi K3.1 Pro on Databricks on Day 0!

It's the best oss model to date. Our team has already started using it for internal coding.

Excited to be a launch partner with Moonshot AI!

[AI.Engineer @ CODE @aiDotEngineer]

Cheap tokens can still mean an expensive coding task.

Hear @ankit_math of @databricks at AIE CODE.

Thank you to Databricks, a Platinum Sponsor.

November 10 to 12, San Francisco.
Apply: https://t.co/N46r5dfdGz

[Yuchen Jin @Yuchenj_UW]

AI bills are exploding everywhere.

Since joining Databricks, I've seen brilliant engineers across the industry (Distinguished Engineers, Berkeley/MIT/Stanford PhDs) spend months figuring out where to buy AI tokens.

Let me save you some time.

Choose Databricks AI Gateway:
- Fast open-source and proprietary models, available on Day 0.
- Smart Routing cut our internal coding costs by 35%.
- Easily track spending and set budgets across teams.

Spend less time managing AI tokens. Spend more time building.

[Kuman @KUMAN_R]

やっぱり Databricks の DB ブランチ、いいな、、

> ブランチの作成は数秒で完了し、数分もかかりません。データベースのサイズにも依存しません。

エージェントの実行ごとにブランチを1本。何十も並べても、費用は DB の複製ぶんではなく、ほぼ変更したぶんだけとのこと。
https://t.co/uZn5A1i0Ix

[Ankit Mathur @ankit_math]

Great analysis by @calcsam - @alighodsi and the founders have never stopped being in founder mode, and they’ve always backed innovation by putting together a world class engineering team

[Sam Bhagwat @calcsam]

The reason Databricks is the largest open source company in history is both structural and execution focused. Ben gives a great overview but as an open source founder I’ll try my hand as well.

Databricks is at its core a compute company. Their project Spark beat out all the other distributed data processing architectures like Hadoop and Cassandra to emerge victorious in the big data wars of the 2010s.

Partially this was because the other architectures were flawed and partially because they invested more in fixing them through hiring incredible engineers. This company did come out of a Berkeley PhD lab.

Still they struggled to make money. So in the mid 2010s they began making their next investment in a proprietary execution engine for Spark they called Photon.

As if building the leading distributing data processing library wasn’t hard enough they also shipped a native C++ vectorized query engine and continued both the public Spark roadmap and an internal proprietary roadmap as well.

It’s worth pointing out that this company is a decent example of the a16z thesis that you can kingmake with capital. Ben dropped a $10m check on a company that only asked for $250k, which is how the company had the war chest to make these investments.

If that’s all they had done, my guess is they’d be a $20B or $30B company today. I’ve made the point before that open source infrastructure expands forever.

That said there was a big second act. Circa 2018 they pioneered lakehouse architectures. The way to think about it that the traditional way you provision databases is you reserve some bucket on specialized compute and storage for them on your infrastructure. This made sense in a pre-cloud world where you had separate special-purpose servers, but less so in a world where you have scalable services like S3, EC2 etc.

So instead of reserving expensive DB compute and storage you reserve cheap S3 storage and compute that can spin up and down based on variable usage.

And now there’s a third act, namely building a cloud with AI primitives. The new gateway is a big piece of that but last month they also released a backend as a service with the primitives for building agents.

So again, why is it such a big company? Again, I think this is both structural and execution oriented.

The structural reasons: they started in an extremely lucrative market with a barrier to entry (it is very difficult to create popular open source projects in valuable markets!), that expands forever.

Second, they are a general purpose orchestration framework.

What I mean here is that every open source / devtools market is split between general purpose orchestration and specialized single purpose utilities. Think auth (Clerk) or email (Resend) for web apps (Next.js), or integrations (Composio) and sandboxes for agents (Mastra). While it is simpler from a product execution standpoint to build a single purpose utility, it is usually more valuable to build the general purpose orchestration framework and Databricks bears that out when you compare them to single purpose utilities doing things like ingestion and transformation (dbt, Airbyte).

Lastly as an orchestration framework / compute company, unlike database companies they are not limited to a particular workload (Supabase etc for transactional data, Clickhouse etc for columnar data), and they are in the core infra layer as opposed to being stuck on the client (Vercel/Next.js). It’s the $10B vs $200b difference

The execution reasons: they are technical geniuses who really want to win. I’ve spent the last decade building open source companies (Gatsby and now Mastra) and the typical open source founders are brilliant tinkerers and, in the best case, it takes a while to develop a ruthlessly efficient competitive commercial DNA. With Databricks they had both sets of DNA within the founding team — incredibly rare.

Congrats to @alighodsi and team on everything they’ve built so far!

[Diptanu Choudhury @diptanu]

One of the tweaks we recently made in our internal software factory, Foundry, is giving the agents the ability to spin off new sessions by cloning the sandbox it’s running in. This lets agents run many investigations and implementations in parallel.

We’ve also exposed tools that allow sessions to discover other sessions and send messages back and forth, creating a community of agents learning from each other and sharing outcomes from performance benchmarks, etc.

Turns out agents LOVE cloning themselves and messaging each other to coordinate work! They seem to be good at referencing work done by other agents in the community and also send each other relevant tasks related to the area of code they were working on.

[Ali Ghodsi @alighodsi]

@hwchase17 bro, databricks very much is databricks for agents

[Yuchen Jin @Yuchenj_UW]

@RamaswmySridhar We’d love to have Snowflake as a customer of our AI Gateway! my DM is open.

https://t.co/mMBVJGayJD

[Sarah Wang @sarahdingwang]

“Since you're not doing this AI stuff”

🔥🔥🔥