Scott Haines

OSS Engineer | Speaker • Trainer | #DatabricksMVP | Author @OReillyMed | ❤️ #ApacheSpark. ❤️ #Dogs. #DatabricksMVP. Views are my own

[Delta Lake @DeltaLakeOSS]

Data + AI Summit session: Tyler Croy (delta-rs maintainer) will introduce Virtual Delta Tables and the associated open source code designed for multimodal inference.👇

🗓️ June 15-18
📍 San Francisco

🔗 Session details: https://t.co/WY89dsCvEK

#DeltaLake #MultimodalAI #DataAndAISummit #Lakehouse

[Alexy 🤍💙🤍 @ChiefScientist]

Exciting!

[Apache Spark @ApacheSpark]

#DataAISummit Session Spotlight ➡️ Learn how to build agentic workflows with OSS Spark Declarative Pipelines, with patterns for deterministic, testable, production-ready data workflows.

🗓️ June 15–18
📍 San Francisco

🔗 Session details: https://t.co/fWzAL6uUcQ

#ApacheSpark #DataAISummit

[Scott Haines @newfront]

@motherduck This is a really interesting. I like how responsive the 3d globe is with the quakes over time.

[Scott Haines @newfront]

Advancing #ApacheIceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance

Big drop from the UC team comes in the form of multiple Apache Iceberg capabilities, all landing at once, and the throughline is real: the catalog is the center of gravity for the open lakehouse.

What’s GA:

1.⁠Managed Iceberg (with Predictive Optimization + Liquid Clustering handling the maintenance)

2.⁠Iceberg v3 across managed/foreign/UniForm tables (deletion vectors, row tracking, VARIANT).

3.⁠Foreign Iceberg + credential vending, and external sharing to any Iceberg REST client over Delta Sharing.

What’s New:
1. Foreign Iceberg sharing is in Public Preview.

2. Iceberg-compatible materialized views are in Gated Preview — CREATE MATERIALIZED VIEW ... USING ICEBERG, which is the ergonomics you’d actually want.

3. Cross-engine ABAC is in Beta, enforced server-side through the Iceberg REST scan planning API, so any 1.11+ client gets row filters and column masks essentially for free.

4. Federation now reaches Google Cloud Lakehouse and Palantir too.

https://t.co/bHPSp1JKwm

[Scott Haines @newfront]

@DeltaLakeOSS @Data_AI_Summit Looking forward to this one. ☝️

[Delta Lake @DeltaLakeOSS]

Headed to Data + AI Summit (June 15–18)? 🗓️ Don't miss Your Guide to Open Table Formats — Delta, Iceberg, Best Practices, and What's Next!

Benjamin Mathew & Scott Sandre will cover Delta 5.0, Iceberg v4, Unified Delta Kernel (GA), and best practices across formats. Add it to your agenda. 👇
https://t.co/toMqbklq4Q

#DeltaLake #ApacheIceberg #DataAISummit

[Delta Lake @DeltaLakeOSS]

Delta Lake and Apache Iceberg™ have converged on similar ideas—columnar metadata, manifest trees, deletion vectors—but two separate metadata structures still duplicate work.

At DAIS (June 15–18, San Francisco): the next evolution of Delta Lake metadata. 👇
🔹 Delta commits → Iceberg v4’s adaptive metadata tree
🔹 Tree-structured manifests + Iceberg interoperability
🔹 Transactional guarantees Delta users depend on — preserved

🔗 https://t.co/rUwZZtYwl6

#DeltaLake #ApacheIceberg #DataandAISummit

[Delta Lake @DeltaLakeOSS]

Headed to Data + AI Summit? Don't miss New Foundations of Delta Lake with Kernel + Spark's DSv2.

Delta on Spark DSv1 set the standard for the past 8 years. Rahul Potharaju & Tathagata Das will highlight the move to DSv2 and new foundations for the next decade.

🗓️June 15-18
📍San Francisco

Add it to your agenda 👇
https://t.co/qg9iXIAdPO

#DeltaLake #DataAISummit

[Scott Haines @newfront]

If you’re in the Bay Area tomorrow and looking for an excuse to hang out with the @ApacheIceberg community. Come join us at the Bay Area Apache Iceberg meetup (https://t.co/920tD5c1tk)

Engineers from @awscloud, @puppyquery , @databricks, @Snowflake, @Cisco, hyperparam, supermetal, @LakeSailHQ, and @ZetaGlobal will keep you up to date on the latest and greatest.

[Apache Spark @ApacheSpark]

Apache Spark 4.1 introduces Spark Declarative Pipelines (SDP), a built-in framework that infers dataset dependencies to automatically handle orchestration, parallelism, and retries.

⚡ Highlights:
• Python/SQL: pyspark.pipelines module & CREATE STREAMING TABLE/MATERIALIZED VIEW DDLs
• New spark-pipelines CLI with init, run, and dry-run subcommands
• Pipeline projects configured via a spark-pipeline.yml spec file

🔗 Programming guide: https://t.co/4niBy4suQ0

#ApacheSpark #OpenSource #SDP #Pipelines

[Scott Haines @newfront]

@DeltaLakeOSS Thanks everyone for coming out for this today!

[Delta Lake @DeltaLakeOSS]

Delta Lake Community Meetup [May 2026] https://t.co/W6Ws69ZHqF

[Alexy 🤍💙🤍 @ChiefScientist]

What are some notable Rust AI startups in the Bay Area and worldwide? We’re starting a Rust AI meetup at https://t.co/3H4lGMh1HE — keep an eye on it! @pydantic @zeddotdev @cocoindex_io @lancedb @helixdb @SurrealDB @qdrant_engine cc @newfront

[Matei Zaharia @matei_zaharia]

This is a fantastic team. Check it out if you want to help build agents and models that reliably answer the most challenging analytical questions in biotech, finance, etc for our customers.

[DuckDB @duckdb]

The Delta and Unity Catalog extensions in the latest DuckDB release come with a fresh set of features and have shed their experimental labels. In today's blog post, Ben Fleis walks you through the key improvements:

✍️ You can now write Delta tables with DuckDB. Multiple inserts within a transaction produce a single atomic version in the Delta table.

🤝 The Unity Catalog unlocks multi-writer access. DuckDB and other clients such as Spark can now perform writes alongside each other with the catalog handling concurrency control.

⏪ You can use the coolest feature of data lake formats: time travel. This lets you query any Delta table at a specific historical version. Thanks to incremental snapshot loading, this is fast even across large Delta logs.

Read the full blog post for more – link in the thread 🧵

[Delta Lake @DeltaLakeOSS]

📣 Join us for the next Delta Lake community meetup on May 19!

Agenda:
🔹 Issues & PR backlog
🔹 Unified Kernel
🔹Community highlights

Register: https://t.co/Kwvv3d9lVE

#DeltaLake #OpenSource #Rust #OpenLakehouse

[Unity Catalog @unitycatalog_io]

[Ep 1] Open Lakehouse + AI: The Catalog Layer, Interoperability & AI-Agent Governance

Learn how composable lakehouses center the catalog for metadata, versioning, and commit coordination. 👇

🔸 Interop: Iceberg REST, Unity Catalog (OSS), Spark, Delta Lake, Delta-RS, Iceberg.
🔸 Governance: credential vending, row/col filters, column masks, audits, and why “no governance was the easiest governance.”
🔸 Agents as data customers (Temporal). Fine-grained access beats blanket credentials on 24/7 workloads.

🎥 Full episode: https://t.co/9UZYnY39fJ

#OpenLakehouse #UnityCatalog @ApacheSpark @ApacheIceberg @DeltaLakeOSS

[Esteban Suárez @EstebanSuarez]

You can just Ship 26 ▲

See you there!

[#DataAISummit @Data_AI_Summit]

#DataAISummit early bird pricing ends tomorrow, April 30!

Don’t miss your chance to join the leaders shaping the future of data + AI. Lock in 50% savings today: https://t.co/9PhDU1IurZ