Apache Hudi

Official twitter handle of Apache Hudi, an open data lakehouse platform.

[Apache Hudi @apachehudi]

Hudi 1.0 is the most powerful release to date for data lakehouses. Read the blog for details:

Secondary Indexing, Expression Indexes, Partial Updates, Non-blocking Concurrency Control, New LSM timeline, +more: https://t.co/QHi7rnNsn9

#datalakehouse #opentableformat

[Apache Hudi @apachehudi]

AWS Architecture Blog (co-authored with Johnathan Hwang, Zendesk): https://t.co/srpt6ooTO3

[Apache Hudi @apachehudi]

Zendesk ran CDC from 1,800+ Aurora MySQL databases across 8 AWS regions into one Apache Hudi data lake.

10+ billion ticket events. Petabytes of data ↓

[Apache Hudi @apachehudi]

Cross-region, multi-database CDC at meaningful scale.

Links ↓

#ApacheHudi #CDC

[Apache Hudi @apachehudi]

Schema evolution docs: https://t.co/HF4eqhFVvf

Mastering Schema Evolution blog: https://t.co/5wBkk5I18F

[Apache Hudi @apachehudi]

Unsafe (workarounds required): non-nullable column (make it nullable), datatype demotion (rewrite), field reorder (breaks partial-coverage upsert reads).

[Apache Hudi @apachehudi]

For column renames + deletes + complex evolution: opt into schema-on-read with https://t.co/psHuYs25bV.enable=true. Spark 3.2.1+, can't be disabled once on.

Proven at scale at Uber, Walmart, LinkedIn.

Links ↓

#ApacheHudi #SchemaEvolution

[Apache Hudi @apachehudi]

Safe (writes succeed, reads return the full dataset): add nullable column at root or nested level, add complex type with default, promote int→long / float→double / string↔bytes, add Hudi meta columns.

[Apache Hudi @apachehudi]

Upstream added a column. Did your pipeline survive?

Hudi supports schema evolution on write out of the box — the changes that happen in production all the time.

[Apache Hudi @apachehudi]

Upstox engineering blog https://t.co/vgdbmYZ8sc

Hudi concurrency control: https://t.co/CeS2q5UKUy

[Apache Hudi @apachehudi]

Upstox — one of India's largest online trading platforms — ingested 300 GB of new data per day for 12 million customers on Apache Hudi.

Capital markets workloads stress every part of a lakehouse ↓

[Apache Hudi @apachehudi]

Streaming-first lakehouse, built for upserts on S3 — what trading-day workloads require. 300k data requests served daily.

Links ↓

#ApacheHudi #FinTech

[Apache Hudi @apachehudi]

Streaming reads docs: https://t.co/vdaa3J4YsC

Incremental queries docs: https://t.co/KuRxN3U7XI

[Apache Hudi @apachehudi]

Hudi tables are streaming sources. Read changes, not whole tables.

Spark Structured Streaming on Hudi is spark.readStream.format("hudi").load(basePath). That's it.

[Apache Hudi @apachehudi]

Pair with KEEP_LATEST_COMMITS retention long enough that your longest checkpoint lag never falls off the table.

Flink streaming reads work the same way — incremental read mode, checkpoint-driven, commit-aware.

Links ↓

#ApacheHudi #StreamingData