Scott Haines

OSS Engineer | Speaker • Trainer | #DatabricksMVP | Author @OReillyMed | ❤️ #ApacheSpark. ❤️ #Dogs. #DatabricksMVP. Views are my own

[Unity Catalog @unitycatalog_io]

📣 Our next Unity Catalog Community Sync is coming up on Tuesday, November 17!

This is a dedicated space for maintainers and developers to connect on roadmap updates, dive into ongoing development, and work through technical challenges together. Bring your questions and join the discussion.

🗓️ Tues, Nov 17
🕓 8:00 AM PT

Save your spot ⬇️
https://t.co/FfJ6U0CK0f

[Jacek Laskowski @jaceklaskowski]

#ApacheSpark 4.3.0 hit the shelves! 🔥🔥🔥

➡️ https://t.co/iJvH9f688K

[Anthony Giuliano @ant_giuliano]

As a longtime @neondatabase user, there is one feature that I have always wanted.

Well...

[Buf @bufbuild]

We experimented with TypeSafe AI's Jev the only way we know how: using Protobuf. That turned into protoc-gen-jev, an experimental plugin that generates typed Jev clients for Go, TypeScript, and Python.

https://t.co/kq7QQb4kgv

[Delta Lake @DeltaLakeOSS]

Join us tomorrow, Oct 13 at 9 AM PT for the Delta Lake Community Meetup! 🎉

On the agenda:
🔹 Parquet FILE data type protocol support
🔹 Concurrent identity column generation RFC
🔹 delta-rs integration with Unity Catalog Delta APIs

🔗 RSVP: https://t.co/pet7ozOn6Z

#DeltaLake #OpenSource

[Scott Haines @newfront]

@ChiefScientist There is a drum and quick cymbal crash there. But you are correct in the analogy too

[Buf @bufbuild]

connect-go v2 is here.

We took four years of production experience and redesigned the API: simpler method signatures (no more generic Request/Response wrappers), pluggable transports, a unified interceptor model, and a migration tool to automate the upgrade.

Learn more: https://t.co/dyT5VveT1C

[Scott Haines @newfront]

@lhoestq @coyohf @databricks @KhalilKafrouni @DeltaLakeOSS @unitycatalog_io @omnigent_ai I really liked your presentation. Everything made complete practical sense, and you introduced me to Xet deduplication which was a technique I’d never encountered before. Cheers

[Scott Haines @newfront]

@YoussefMrini Thinking of you. Not sure what this is like but if anyone can handle this it is you.

[Apache Spark @ApacheSpark]

Join us next Wednesday for Schema-on-Read Without Regrets!

Geir Alstad (Gabler AS) and @newfront (@databricks) will walk through Spark Variant shredding in a real pipeline.

🗓️ Wed, Sept 30 · 8:00 AM PT

🎟️ Register: https://t.co/9S3AvF68nd

#ApacheSpark

[Buf @bufbuild]

How well do you actually know Protocol Buffers? We put together a quick interactive quiz covering schema design, field rules, and wire format mechanics.

https://t.co/TghGexYZTZ

The questions are now randomly selected from a question bank so retrying will give you a few new questions.

#protobuf #grpc #quiz

[Omnigent @omnigent_ai]

Omnigent v0.14.0 is now available! 🎉

This release makes it easier to work across GitHub repositories, configure sessions, and keep track of ongoing work.

🔀 Multi-repo GitHub PRs and sandboxes
🎛️ Redesigned composer
🪟 Claude SDK /compact
🗄️ Gensee, Kubernetes, CockroachDB
🔁 Approvals that survive restarts

Learn more 👉 https://t.co/amzQ9EpRMk

#Omnigent #AIAgents #OpenSource

[Apache Spark @ApacheSpark]

At Airflow Summit, Meni Shmueli from DataFlint covers how #ApacheSpark and @ApacheAirflow run together in production, including splitting CPU work (like reading Parquet) off GPU machines so AI jobs aren’t paying GPU rates for CPU-only steps.

Apache Spark 4.2 is out: auto CDC and Real-Time Mode for streaming, DataSketches as first-class operators, and Spark Connect so client/server queries come back in milliseconds instead of spinning up a full app.

#SparkConnect #DataSketches #ApacheAirflow

[Scott Haines @newfront]

@bufbuild I am a really big fan of this. I’ve been using it for a new project I’ve been working on for @unitycatalog_io’s new UI.

[Scott Haines @newfront]

What’s New in @ApacheSpark 4.2: Spark Connect, DataSketches, and Airflow https://t.co/09mvDXybBV via @YouTube

[Scott Haines @newfront]

@YoussefMrini Congrats Youssef!

[Scott Haines @newfront]

RT @ApacheSpark: 📣 Schema-on-Read Without Regrets: Lessons from Apache Spark™ Variant shredding in Production

Join Geir E. Alstad (Gabler…

[Scott Haines @newfront]

@ApacheSpark @databricks This session should be eye opening for people looking for best practices for using the Variant type. I can be honest, my first impression was "this is going to lead to unmanageable tables over time"

[Databricks @databricks]

The Replit-Databricks integration is now generally available, with new native Lakebase support!

Now, you can use @Replit to build full-stack Databricks Apps on live, governed enterprise data. Replit Agent can automatically provision a Lakebase database, while AI-proposed schema changes require human approval.

Build, deploy, and iterate in one flow.

See it in action: https://t.co/6G46vaAfFU

[MLflow @MLflow]

What’s New in MLflow: September 2026 Roundup https://t.co/V9EJELV9XV