vortex

An extensible, state of the art columnar file format. Formerly at @spiraldb, now a Linux Foundation project (@LFAIDataFdn). Apache-2.0

[Quentin Lhoest 🤗 @lhoestq]

@willmanning @vortexdotdev Vortex support in the HF Dataset Viewer coming soon ! 😁

[Will Manning @willmanning]

fast, distributed determinism is hard. really cool work from our friends at @datologyai!

(also... integrates with @vortexdotdev 👀)

[Jamie Turner @jamwt]

We've always called it software engineering, but we've actually been builder...gineers.

In civil engineering, aerospace, or architecture, the people who do the engineering and the people who do the building are obviously different.

But in software, we collapsed them into the same role. With agents, we're finally separating the jobs.

We're going to specify and encode more upfront, add more guardrails, and design architectures that bias agents toward success.

In many ways, we're finally starting to look like actual engineers.

[Tigris Data @TigrisData]

9ms point lookups without running your own database in every region

That's what @goldskyio built with Tigris, @vortexdotdev, and SlateDB

Get our side of the story 👇

[Luke Kim @lukekim]

Cayenne stores data in @vortexdotdev, which computes directly on compressed data. Pretty cool!

Many operations skip decompression. And when a decode is needed, bytes land straight in Arrow arrays with no intermediate copy.

https://t.co/CCjv6KGTxf

[Will Manning @willmanning]

More like Hugging Fast 🚀🚀

@vortexdotdev

[Andrew Lamb @andrewlamb1111]

The DAVE stack: the age of domain-optimized analytical databases on Datafusion, Arrow, and Vortex, Embedded
https://t.co/ygqaOsJEHK

[Marko Bakovic @mbakovic__]

If you happen to be looking for more reasons why you should choose @vortexdotdev as your storage layer:

[vortex @vortexdotdev]

❤️ DataFusion

[RisingWave @RisingWaveLabs]

Apache DataFusion is becoming a foundational building block for modern data systems.

From RisingWave, InfluxDB, LanceDB, and GreptimeDB to OpenObserve, ParadeDB, https://t.co/OXE9rMjohZ, Vortex, Apache Comet, Ballista, and many others, a growing ecosystem is building on the same shared query foundation.

Why?

Building a high-performance query engine from scratch requires a huge amount of engineering.

A modern engine needs:
SQL and DataFrame APIs
Logical and physical query planning
Query optimization
Vectorized execution
Multithreaded processing
Streaming execution
File-format support
Object-storage integration
Memory management
Extensible functions and operators

Apache DataFusion provides these core capabilities in a modular and extensible engine written in Rust and built around the Apache Arrow in-memory format.

The real value is not just performance.

It is extensibility.

This allows teams to reuse the common foundations of query processing without giving up control over the parts that make their systems unique.

Instead of rebuilding the same SQL parser, optimizer, execution engine, and file-format support again and again, projects can focus their engineering effort on their specific workloads and product differentiation.

RisingWave follows the same approach.

RisingWave uses Apache DataFusion as its default batch query engine while continuing to use its streaming-native engine for incremental and stateful stream processing.

Apache DataFusion handles the query engine, allowing projects to focus on the unique requirements of their specific use cases.

[Spice AI @spice_ai]

AI agents and real-time apps increasingly need analytical answers from data that was written seconds ago. Pointing those queries straight at an operational database slows production and widens security risk. The usual alternative, an ETL pipeline into a warehouse, is costly to operate and can lead to data that's stale by hours or days.

Spice 2.0 offers a new approach, and is now available.

It stands up a sandboxed analytic replica beside your operational store and keeps it current with high-throughput change data capture directly from those sources. You point Spice at a database and start running analytical queries in minutes, while production never carries the analytical load.

High-throughput CDC replication is complemented by the rest of the 2.0 platform: multi-node distributed compute on @ApacheDataFusio Ballista for petabyte-scale queries, Spice Cayenne columnar acceleration on the open @vortexdotdev format, a cluster-sidecar architecture that pairs isolated local sidecars with a shared cluster, and enterprise security enforced in the engine with OIDC, Cedar policies, PII masking, and mTLS.

"Spice is the data plane behind Summation. Every connector we ship, from Snowflake to a generic REST-as-a-table, collapses into one SQL surface, which is what makes our AI agents portable across a customer's stack. 2.0's distributed query takes the scaling concern off the table."

-Ramachandra Ramarathinam, CTO at @summation

It's open source, portable, scalable, and fast.

Read the launch blog for more details: https://t.co/xMB090iIg0

And, check out the thread ⬇️ for breakdowns of the major features in 2.0.

[vortex @vortexdotdev]

but trust me we are very powerful tho

[Will Manning @willmanning]

so cool to see another blazing fast database built on vortex!

[Ankush Gola @ankush_gola11]

We leveraged two amazing open source projects when building SmithDB.

One is @ApacheDataFusio: an extensible Rust based query engine. We built custom execution plans specifically tuned for our workloads and storage backend, and DataFusion made it straightforward to plumb everything together.

The other is @vortexdotdev: an extensible file format that allows you to build custom layouts with specific encoding and chunking strategies for different columns.

I would highly recommend checking out both of these projects if you're interested in modern data systems.

[Spice AI @spice_ai]

The Research Behind Modern Data Compression & @vortexdotdev

When we chose Vortex as the storage layer for Spice Cayenne (the data accelerator engine in Spice), we were betting on decades of database research finally reaching production-ready maturity.

Here's the research behind Vortex:

📄 BtrBlocks (SIGMOD 2023) - The core algorithm from the Technical University of Munich. Cascading multiple lightweight encodings outperforms monolithic compression. Optimize for decompression speed, not just compression ratio.

📄 FastLanes (VLDB 2023) - Hardware-friendly integer compression. Structures bit-packing to maximize SIMD utilization across AVX-512, AVX2, and ARM NEON. Near-memory-bandwidth decompression.

📄 FSST (VLDB 2020) - Fast Static Symbol Table for strings. Near-LZ4 ratios at 5-10× faster decompression. Critical for string-heavy columns.

📄 ALP (CWI Amsterdam) - Adaptive Lossless floating-Point compression. Exploits real-world float patterns (prices with 2 decimals, sensor readings with limited precision).

📄 MonetDB/X100 + Morsel-Driven Parallelism - Foundations for vectorized, NUMA-aware query execution that Vortex builds on.

The result? Compression that is tailored to your data:

• Integers via FastLanes bit-packing

• Floats via ALP adaptive encoding

• Strings via FSST symbol tables

• Timestamps via delta encoding

• Sorted columns via run-length encoding

Why does this matter for production systems?

1️⃣ Query performance scales with decompression speed. Focus on decode performance translates directly to faster queries.

2️⃣ Automatic encoding selection means zero configuration. The algorithm samples your data and picks optimal strategies per column.

3️⃣ SIMD acceleration is baked in. FastLanes was designed for vectorized, hardware accelerated execution from day one.

4️⃣ Zero-copy Arrow access. Data decompresses directly to Arrow arrays with no intermediate copies.

Vortex is now a Linux Foundation AI & Data project, and researchers are building on it (Anyblox, F3). You get SOTA research in production systems.

The future of data storage is exciting.

To learn more about our Vortex implementation, check out the blog: https://t.co/f1AWwnsREK

#datafusion #ai #data #vortex #spiceai #arrow #parquet

[Will Manning @willmanning]

Connor Tsui & I just merged a first cut of TurboQuant into @vortexdotdev , already validated on production embeddings 🚀🚀🚀

[vortex @vortexdotdev]

Fastest OSS file format, in both performance and velocity

[Will Manning @willmanning]

Connor Tsui & I just merged a first cut of TurboQuant into @vortexdotdev , already validated on production embeddings 🚀🚀🚀

[vortex @vortexdotdev]

you took up with Weasley, but he can't afford sliceable cascaded encodings.

now your random access is dogged, and your cortisol is properly spiked, potter

[vortex @vortexdotdev]

hey man, thrilled that you're interested in contributing. we'll be waiting for you in slack

https://t.co/dj3ThP4MTY

[MeekMill @MeekMill]

I need a GitHub too! Is it like that or nah?

[Luke Kim @lukekim]

CASE-WHEN support coming to @vortexdotdev

Guess I'm a Vortex contributor now!

[vortex @vortexdotdev]

🦆❤️🚀

[DuckDB @duckdb]

DuckDB now supports reading from and writing to the Vortex file format! The DuckDB Labs and Spiral teams have worked together to make Vortex available as a core extension in DuckDB.

Vortex is an open source, columnar file format whose design is heavily influenced by recent research in lightweight compression encodings, computing and IO techniques.

We gave it a test drive, and it performed very well. Read the full article to learn more https://t.co/qBZUrQ81Ni