When someone tells you "we use Unity Catalog," what do you think they mean?
Turns out there are two!
• Unity Catalog OSS, an LF AI & Data sandbox-stage project
• Unity Catalog, Databricks' proprietary, closed-source governance service
I decided to dig into the data.
With Sail you can run the Spark workloads you already have (much faster and cheaper), plus build the AI data pipelines you need next... all in one engine. Pretty cool!
What can you build when Python, SQL, and DataFrames run in the same engine?
With Sail, your data pipeline becomes the AI pipeline. The same workflow that reads, cleans, and transforms your data can also generate embeddings, run model inference, and prepare training datasets for physical AI.
Python runs natively, directly inside Sail's Rust engine, so your Python libraries, models, and SQL transformations work together through familiar DataFrame and SQL APIs.
You can build multimodal pipelines that search images using text, find scenes in video, extract text from scanned documents, and transcribe audio. You can turn humanoid robot recordings into training datasets, then query and validate the results in SQL.
The same data, new consumers. Your pipelines can support analytics and feature engineering, provide context for agents, and prepare data for physical AI, training, and inference.
This is what we built Sail for. One engine for data and AI.
See the workloads in action:
https://t.co/dBdSy3CR0C
Get the full examples:
https://t.co/JpSQsG2xXu
Why is Apache Arrow becoming the universal columnar foundation for modern data and AI infrastructure?
Apache Arrow is great because of its language-independent columnar memory format, zero-copy data interchange, efficient in-memory processing, and broad multi-language ecosystem.
That’s why projects like Apache Spark, Apache Parquet, ClickHouse, Dremio, Polars, Pandas, InfluxData, OpenObserve, Greptime, LakeSail, Spice AI, IOMETE, Daft, Apache Doris, and so many other great systems use it.
For data and AI systems, Arrow is not just another format. It is becoming a foundational layer for moving and processing data efficiently across the entire ecosystem.
RisingWave, a unified data platform for real-time and Iceberg-based lakehouse workloads, also uses Arrow extensively, particularly for interoperability with Apache Iceberg, Apache DataFusion, Parquet, UDFs, and some connectors.
Shoutout to our partner @columnar_tech for advancing Apache Arrow and ADBC. They're really doing great work and redefining database access and connectivity. 🙌
@superlogical's Rex terminal multiplexer is incredible, and feels insanely fast.
Running agents in the terminal on my laptop and mac mini (and phone) has literally never felt better.
Also one of the most inspiring UI/UX apps I've seen to date.
Great work @mitchellh@almonk!
Highly recommend @LakeSail
Spark rebuilt in Rust + Python native for multimodal and AI in one engine. Spark-compatible so no code rewrite.
The platform is fully agent native. Mint a PAT and have your agents build and monitor everything you want!
Faster than Spark and every Spark accelerator too.
LakeSail announced a strategic partnership with Oxide Computer Company to help enterprises modernize data and AI infrastructure while maintaining control over security, performance, and cost.
The partnership follows the successful validation of Sail on Oxide’s platform. Sail, LakeSail’s open-source, Rust-native, Spark-compatible compute engine, ran on Oxide’s integrated, rack-scale cloud platform, establishing a validated foundation for secure, customer-controlled AI environments.
Oxide delivers a cloud-like experience within a customer’s data center through an integrated platform and unified control plane. Its architecture includes a hardware root of trust and mechanisms for verifying the integrity of the software stack.
Sail preserves Spark APIs, SQL, and DataFrames while replacing the traditional JVM execution layer with a Rust-native architecture. It supports Python, Apache Iceberg, and Delta Lake, allowing enterprises to modernize without rewriting existing applications.
By eliminating the JVM execution layer, Sail reduces costs and performance bottlenecks associated with cluster startup, garbage collection, memory tuning, idle capacity, and overprovisioning. Depending on workload and configuration, LakeSail has demonstrated performance improvements of up to 10x and infrastructure-cost reductions of up to 98% compared with equivalent Spark and JVM-based deployments.
Enterprises can use Sail as open-source software or purchase the commercial LakeSail Platform for management, governance, security, and support capabilities.
“We believe data processing and AI belong in the same engine rather than in separate systems customers must stitch together,” said Shehab Amin, Co-Founder and CEO of LakeSail. “Combining Sail’s efficiency with Oxide’s customer-owned cloud infrastructure gives enterprises more freedom to build without inheriting more complexity or traditional JVM economics.”
“As data management and processing become more central to AI, we're excited to see LakeSail running on Oxide," said Steve Tuck, Co-Founder and CEO of Oxide Computer Company. "Running LakeSail's modern execution engine on Oxide's on-prem Cloud Computer allows customers to gain insights from their data faster while maintaining the control over where that data lives."
Read the full press release: https://t.co/r7mqqHdvyU
The other day my brokerage account rep recommended me to activate "voice authentication" claiming it's a "very secure new technology", obviously just a new feature she's prompted to tell customers... Needless to say, I couldn't be more skeptical having my "voice" trained on like a few minutes of conversation grant access to my financial accounts.
https://t.co/BQ4treR0Qr
@steipete Excited for this. Big fan of OpenClaw. Been trying the web app for the first time lately and it’s quite buggy compared to my other daily drivers.
@TrungTPhan I very much enjoy reading AI content. The thrill of learning is truly insane. As long as it’s clear it’s AI.
Listening to AI content is a whole nother ball game. Immediate click out on YouTube.
A data platform that's simple, transparently priced, and ridiculously fast.
The LakeSail Platform runs your Spark SQL and Python jobs on a Rust-native runtime. It's faster than Spark, drastically cuts hardware costs, and scales from a single node to a multi-node cluster. It reads and writes your Delta Lake and Iceberg tables where they already live, so your lakehouse stays open and your data stays in place.
For those who want to run Sail in production most efficiently, with all the operations handled for you, this platform was built for you.