LakeSail

Spark rewritten in Rust. A new standard for unified compute, reimagined for modern data and AI infrastructure.

[LakeSail @LakeSail]

Spark rebuilt in Rust — no JVM, 8x faster, 94% less cost.

[CocoIndex @cocoindex_io]

Full house at the first ever inaugural Rust AI Begins meetup in SF AWS Loft 🔥. With an amazing community pushing the boundaries of what’s possible with Rust and AI.

We are excited to share our work about high permanence Rust Engine for fresh context @LinghuaJ along with infra friends at @databricks , @temporalio , @valkey_io , @LakeSail , @ApacheIggy / laserdata and many amazing Rust projects that defines today's landscape for Rust AI Infrastructure. We’re also thrilled to see such passionate and in-depth discussions from the community.

Thanks a lot for hosting the amazing mini conf @ChiefScientist @newfront and we are excited to build along with the amazing Rust community 🩷 !

[LakeSail @LakeSail]

https://t.co/fBwkmVo5EH

[LakeSail @LakeSail]

Sail in Spark docs would have been cool 😞

[LakeSail @LakeSail]

Read more in our deep dive: https://t.co/wv5m7KnEZ9

[LakeSail @LakeSail]

The lakehouse format is an engine contract.

[LakeSail @LakeSail]

Months of Delta and Iceberg integration work have taught us a few things. Most importantly, that a lakehouse format is actually an engine contract.

For a simple scan, the library view holds: load the table, list the files, read the Parquet. But a row-level MERGE is not a step at the end of a query. Reading the source, joining it against the target, deciding which files to overwrite, and committing atomically have to be planned and executed as one thing, inside the engine.

In Sail, Delta and Iceberg run inside the engine, not as an external library. This allows us to give writes everything queries already get: optimization, distribution, and observability.

Read more in our deep dive: https://t.co/9GCtnZ4IBJ

[LakeSail @LakeSail]

Deep dive: https://t.co/sKxlHkMBOG

[LakeSail @LakeSail]

Sessions bring secure execution for humans and agents alike.

[LakeSail @LakeSail]

Production data sits in a private cluster in your own cloud, no public address. Reaching it from outside has usually meant putting yourself on the network first: a VPN, a bastion, something.

So we shipped Sessions: with a scoped, time-boxed token, you can reach the cluster from anywhere through one hardened gate, while the cluster stays private and your data never leaves your account.

The same token works from your laptop or a coding agent. So you can give an agent real compute on real production data, scoped to one workload and time-boxed, without ever granting it standing access.

Check it out here: https://t.co/eo8Fsfy65r

[LakeSail @LakeSail]

In 2026, you no longer have to be stuck tuning the JVM. Run your Spark workloads on a Rust-native runtime for 94% less hardware cost and zero garbage collection.

pip install pysail

[LakeSail @LakeSail]

Keep your Spark code, just plug in a Rust-native runtime underneath and say goodbye to the JVM forever. That's Sail in a nutshell.

[LakeSail @LakeSail]

We built Sail so you don't have to keep paying the JVM tax.

[LakeSail @LakeSail]

We dive deeper into how Sail compares to Spark on this front in our blog: https://t.co/iFMK1WXCvs

[LakeSail @LakeSail]

Slow Python is simply unacceptable for AI workloads. Agents on a compute engine need native speeds, and Sail was built from the ground up in Rust with exactly that in mind.

[Alexy 🤍💙🤍 @ChiefScientist]

If you are at the @Snowflake Summit today or SF at large, come join us tonight for a panel on the future of Data=>AI at Yes SF!

https://t.co/ViioE1VlY2

Your truly will join the illustrious engineering and devrel leaders Mark Hoerth of Datastrato / Apache Gravitino, Andrew Madson of the newly combined juggernaut of Fivetran / dbt Labs, and Josh Wo of LiveRamp. I will represent LakeSail with our founder and CEO Shehab Amin keeping me honest from the bench!

Come learn all about our the real Rust AI future and the Semantic Layer going much beyond OSI metrics.

[LakeSail @LakeSail]

Check out Sail on GitHub: https://t.co/PCmvi4iN0e

[LakeSail @LakeSail]

Spark accelerators like Photon speed up portions of execution, but are still tied to the JVM.

Sail, written in Rust, removes the JVM entirely. The result is drastically better performance with no JVM tuning required.

The same familiar Spark API on a Rust-native runtime. That’s Sail.

[LakeSail @LakeSail]

Read our full breakdown how Sail compares to Spark accelerators here → https://t.co/zUI1vmswWT

[LakeSail @LakeSail]

Why fully rebuild Spark in Rust instead of just accelerating it?

Spark accelerators speed up parts of execution, but they still inherit Spark's JVM control plane, memory model, shuffle path, and Python serialization costs. You're still tuning a JVM.

With Sail, we took a clean-slate approach: a fully Rust-native runtime, with no JVM, no heap tuning, and no GC pauses. Same Spark interface with an entirely new runtime underneath.

[Alexy 🤍💙🤍 @ChiefScientist]

We wrote a book, with Codex, delving deep into @lakesail architecture. Sail is a modern data ecosystem with deep roots in @ApacheArrow, @ApacheDataFusio, and rebuilding the whole @ApacheSpark ecosystem in @rustlang. One of the key advantages of an AI-native stack is extensibility. We are convening the community to build Sail extensions. A proposal is on the table in https://t.co/3bqv7MWi5B. In order to extend the engine as profoundly efficient as Sail, you need to operate at several levels — physical, logical plans, loading and linking, and performing at the top both in a single node and in cluster mode. The book is written as an exploration of the codebase, of the overall Sail architecture, its use of Arrow and DataFusion, its implementation of SparkConnect protocol, and everything else pertinent to the extensions.

https://t.co/j9CQEsfOie