Vinoth Chandar

Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.

[Vinoth Chandar @byte_array]

If you want spare EC2 capacity, look at your Apache Spark clusters and then go give https://t.co/sglXKfuuWn a shot.

[MTS @MTSlive]

SITUATION DETECTED: Amazon CEO Andy Jassy says that AWS capacity for 2026 is completely sold out, 2027 is largely sold out, and "a good chunk" of 2028 is already sold as well.

[Vinoth Chandar @byte_array]

sign up link: https://t.co/aaZekYguUF

[Vinoth Chandar @byte_array]

Spark on Kubernetes at scale is hard. Onehouse runs a lot of it, and tomorrow we're sharing what we've learned.

In most places, the knowledge required to get all of this right lives across the data team and the Kubernetes/infra team.
We've collected a list of things that looked simple at first, but hard in production:

- OOM kills
- Shuffle storage
- Spot interruptions
- Autoscaling
- Long-running drivers
- Cluster upgrades
- Keeping streaming jobs alive while the infra underneath changes

Most of it lives in the gap between the data team and the infra team.

Webinar tomorrow. Bring your worst production problem.
Link in comments.

[Vinoth Chandar @byte_array]

@ClaudeDevs This has got to be the most hype that gzip has had in a decade or more.

[Vinoth Chandar @byte_array]

Alright . That is in fact pretty freakin' cool.

[Benny Johnson @bennyjohnson]

🚨 The new Tesla Roadster can FLY.

This is INSANE.

Only Elon Musk and only in America 🚀

[Vinoth Chandar @byte_array]

@rabata_io I feel this is AI generated. 🙂
What tool are you using?

[Vinoth Chandar @byte_array]

Last week, I posted on why warehouses still default to closed formats. What I learned: they won't flip the default. Open formats are opt-in.

So the fix is with the users. Give every open format (Delta, Hudi, Lance, Vortex, ClickHouse, DuckLake) the same consideration on performance and flexibility that you give the closed ones.

Your data will thank you.

Back to building on open formats.

[Vinoth Chandar @byte_array]

@rmahindra Does the technical problem simply boil down to a very large table on object storage with fast writes and fairly fast analytical and point lookups?

[Vinoth Chandar @byte_array]

I like that the ChatGPT Mac app has a built-in markdown editor. It definitely reduces friction in review cycles.

[Vinoth Chandar @byte_array]

@jaykreps My wrists are happy that I have to type less. This feeling might be same elsewhere too

[Vinoth Chandar @byte_array]

Not at all. Customers are stuck. We speak to dozens every week. They have to pay hefty egress costs to do the migration.

The question I raised is more about first principle and forward-looking. Why not make open formats the default for any new table created from this point forward? That costs the vendor nothing.

Btw the LinkedIn counterpart for this x post has a lot more thoughts if you are curious about this topic .

[Vinoth Chandar @byte_array]

Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a post. Then I caught myself.

Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats?

Two years of Iceberg announcements. Still no public retirement date for their closed formats.

Open-source projects get scrutinized for interoperability on every release. The largest warehouses should meet the same bar.

We’ve spent a decade building open formats and data lakes. I think it’s time we asked.

[Vinoth Chandar @byte_array]

My thoughts and receipts are in the blog https://t.co/nTvqO28KT6

[Vinoth Chandar @byte_array]

@rabata_io lol it's actually pretty basic when you see your servers only have 50% CPU utilization.

[Vinoth Chandar @byte_array]

To squeeze the performance out of our runtime components, we implemented a good amount of concepts from - SEDA: an architecture for well-conditioned, scalable internet services https://t.co/RhXodpXFX7

The seminal work at MIT around writing high-performance server software and the gift that keeps on giving.

[Onehouse @Onehousehq]

A cloud VM’s network I/O, CPU, and storage IOPS taking turns instead of running concurrently is bad news: the CPU sits idle while a file downloads, then works through it once the download finishes.

[Vinoth Chandar @byte_array]

@shcallaway @sazabi I don't have extreme opinions on unit tests, but I hate the false sense of security they give developers, who only later realize the end-to-end functionality doesn't work at all.

[Vinoth Chandar @byte_array]

Muse, Dots, Grok bot.

Eagerly awaiting the consumer AI/agent app explosion, to watch agents melt our databases at scale.

[Vinoth Chandar @byte_array]

Impressive numbers.

[Daniel Lemire @lemire]

You can consume compressed JSON documents in a multithreaded manner if you use zstd for compression. This allows crazy speeds. See my article.

[Vinoth Chandar @byte_array]

@samuel_spitz What if you need a super innovative idea to collect the $0.01 ?

[Vinoth Chandar @byte_array]

@a16z Consumer lives and livelihood improving will the ultimate test for this AI wave? Once we start to run out of ideas to use AI to make enterprises efficient.