Vinoth Chandar

Founder @Onehousehq, Creator of @apachehudi, Built the World's first #DataLakehouse, Distributed/Data Systems, Linkedin, Uber, Confluent alum. (views are mine)

[Vinoth Chandar @byte_array]

Full breakdown: https://t.co/qEkmsvSu0h

[Vinoth Chandar @byte_array]

GROUP BY ROLLUP over N keys computes N+1 levels of subtotals. Spark computes that by copying every input row N+1 times before it aggregates anything. That copying is the Expand operator, and it's the real cost of your rollup query.

[Vinoth Chandar @byte_array]

Q67 is the worst offender: 8 grouping keys, 9 levels. Expand fans every input row into 9 copies; at 10TB the partial aggregation hashes and probes roughly 9x as many rows, each level re-shuffled across the cluster.

#ApacheSpark #DataEngineering

[Vinoth Chandar @byte_array]

Full benchmark methodology and figures: https://t.co/cVFIorQsPX

[Vinoth Chandar @byte_array]

Latency claims are cheap, so here are the receipts. A real customer's 12 hour Athena trace replayed through Lakegres, across 570 production tables: p0.001 goes from 1000ms to 119.86ms.

[Vinoth Chandar @byte_array]

That's a consistent ~6x, on exactly the narrow point-lookups and joins that agent query loops hammer.

#Lakehouse #DataEngineering

[Vinoth Chandar @byte_array]

Full breakdown: https://t.co/dlcOD3J7hD

[Vinoth Chandar @byte_array]

Every cost-conscious data engineer has tried to reduce the number of S3 GET calls their pipeline makes. That instinct is good, but needs nuance.

[Vinoth Chandar @byte_array]

The fix isn't fewer S3 calls. It's more of them, in parallel. Maximize concurrency, keep compute busy, and stop paying for the expensive side of that 27.5x comparison: compute that's waiting instead of working.

#DataEngineering #AWS

[Vinoth Chandar @byte_array]

⚡ Streaming foundation upgrades: Record Level Index on Flink, dynamic bucket scaling, FLIP-27 Source V2. Low-latency incremental processing matters more in an agent-driven world.

#ApacheHudi #AgenticAI #VectorSearch

[Vinoth Chandar @byte_array]

🚀 Hosting the next Apache Hudi Meetup in Bangalore this Wednesday. I'll be talking agentic lakehouse, vector search, and what's landing in the repo.

Wed July 29, 4-7pm IST
COWRKS Ecoworld, Bellandur
Free — RSVP: https://t.co/f7mFTq9PMI

[Vinoth Chandar @byte_array]

🤖 Agentic lakehouse work is already landing: built-in AI agent gateway, self-hostable inference (Ollama/vLLM), MCP tooling. Tables that agents can query, tune, and operate — not humans writing SQL.

[Vinoth Chandar @byte_array]

📦 New in Hudi 1.2: native VECTOR, BLOB, and VARIANT types. Vector search in Spark SQL. Lance format support. Multimodal data is now first-class in the lakehouse, not an afterthought.