Divya Ranganathan

[Divya Ranganathan @divs1101]

We’re back and this time with @LangChain!

After a packed Apache DataFusion meetup in Feb and May, we’re making this a quarterly series in San Francisco.

August 13 @LangChain office, SOMA
Featuring Talks from @columnar_tech, @hotdata_dev, @LangChain, @datadoghq, @LakeSail covering observability, APIs, distributed execution + more.
RSVP: https://t.co/gWYt7fiZE0

See you there!
@eddietejeda @andrewlamb1111 @ankush_gola11 @amadaecheverria

[Ankush Gola @ankush_gola11]

We're really excited to be teaming up with our friends at Hotdata and hosting an @ApacheDataFusio meetup at our new SF office on Aug 13th!

The LangChain team will be giving a talk on how we built SmithDB.

We'll also have some great speakers: Divya Ranganathan, Alexander Bianchi, Shehab Amin, and Emil Sadek

If you're interested in modern data systems, you won't want to miss this. Register here: https://t.co/e2hFstXm7Y

[Divya Ranganathan @divs1101]

We’re back! 🚀

After a packed Apache DataFusion meetup in Feb, we’re making this a quarterly series in San Francisco.

📍 May 11 @ Chroma office, SF
🎤 Talks from @RisingWaveLabs, @wherobots, @lancedb, @paradedb covering streaming, geospatial, distributed execution + more.
👉 RSVP: https://t.co/euOy3WqEO0

See you there! 🙌
@eddietejeda @andrewlamb1111

[Divya Ranganathan @divs1101]

When most databases are created by AI and agents collaborate autonomously, query engines aren’t just static interfaces to data anymore - they become the interface humans use to understand and steer agent behavior.

[Divya Ranganathan @divs1101]

Parquet is a boxed meal - to get the fries, you open everything. FlatBuffers (F3) is a vending machine - press B4, only fries come out 🍟

The width of data has exploded - ML tables now have thousands of columns.

But Parquet still forces you to deserialize all metadata just to read a few columns. Even if you query 10 columns, you pay for 50k.

That’s why I find F3 (Future-proof File Format) from CMU interesting. It uses FlatBuffers so agents can randomly access only the metadata they need - no full deserialization.

Result: ~100ms metadata cost in Parquet vs ~10ms in F3 on wide tables.
https://t.co/OKdueiDhO4

[Divya Ranganathan @divs1101]

A win for the DuckDB community! What makes Lance special is that it skips Parquet style row groups for page-level parallelism making it 100x faster than Parquet for point lookups and querying small slices.

[Xuanwo @xuanwo]

Added a full reference for lance in duckdb!

[Divya Ranganathan @divs1101]

@andy_pavlo @andrewlamb1111 @andy_pavlo This is cool. Would love to connect to learn more.

[Divya Ranganathan @divs1101]

Parquet decoupled query engines from physical layout, but its success introduced a new constraint: many reader and writer implementations across engines and languages.

AnyBlox pushes the abstraction further by questioning why engines need to understand encodings at all.

By bundling WASM decoders with the data, formats can evolve independently of query engines. This feels like a logical next step in file format design.
https://t.co/lJnEPGgPGz

[Divya Ranganathan @divs1101]

Read a great paper from DeepSeek on Manifold-Constrained Hyper-Connections. Most LLMs pass information through narrow “express lanes” between layers. This work turns that into a multi-lane highway - with guardrails.

Think music production:
• Layers = tracks
• Residuals = reference track
• Letting every track hit every channel can distort the mix

mHC acts like a mastering engineer applying rules to keep the mix balanced: layers influence each other, but no part dominates. https://t.co/vjXPg91jdt
Result: more capable LLMs, at lower cost.

[Divya Ranganathan @divs1101]

After you push query efficiency far enough, you hit the next bottleneck: S3 itself.

GPUs are the most expensive resource in a modern platform, and waiting on data is costly.

Vortex treats data movement as the core problem - rather than optimizing layout for access patterns, it aims to make decoding from S3 to GPU so fast it disappears behind I/O.

https://t.co/QrFbwhRwTr

[Divya Ranganathan @divs1101]

I see FastLanes as a bridge between OLAP and AI workloads. AI metadata - latency, token count, cost, traces is now critical to route, filter, rank, and gate. It belongs next to the GPU: selectively decoded, partially computed & queried at machine speed. https://t.co/wc0l3UPPd9

[Divya Ranganathan @divs1101]

Database scale used to mean monster VMs and consolidation. Agents create a new dimension of database scalability: hyper-tenancy. Not big data, but monster scale number of databases.

[Glauber Costa @glcst]

I believe agents redefine what do we mean by "scalability" in databases. In the same way boot time of an OS didn't matter until we started creating VMs by the thousands, agents will require databases by the billions.

How fast they can show up? How many of them you can manage?

[Divya Ranganathan @divs1101]

This breakdown of ChatGPT’s memory is a nice reminder that you can often get far with simple data structures, plain text and a clear understanding of what users actually need. https://t.co/rRIoTEzd79

[Divya Ranganathan @divs1101]

Highly recommend this piece by @andrewlamb1111 on Apache DataFusion. Powerful data platforms usually get locked inside proprietary products. DataFusion flips that - giving engineers the building blocks to design their own. https://t.co/G9X8y3HsFb

[Divya Ranganathan @divs1101]

@DarrelFrater Can you try now?

[Divya Ranganathan @divs1101]

@guilleflorvs FOUNDER

[Divya Ranganathan @divs1101]

@samuelcolvin @pydantic Welcome to the bay area! It was great meeting you at the Zero Prime Halloween event.

[Divya Ranganathan @divs1101]

Agents blur the line between compute and data. Caching isn’t an optimization layer anymore - it’s the execution layer.

[Eddie A Tejeda @eddietejeda]

The yearly @smalldatasf gathering is a breath of fresh air.

One interesting take away:

Small data is now pretty big. A single node machine can now outperform the benchmarks that led to creation of Dremel/Big Query.

Slide credit @jrdntgn