polars data

Dataframes powered by a multithreaded, vectorized query engine, written in Rust.

[Mim @mim_djo]

First successful run of Polars with dbt 🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉🎉

#OneLake #ApacheIceberg #Polars #dbt #SQL

[polars data @DataPolars]

We ran Polars 2.0 SQL, DuckDB and DataFusion on an @nvidia Vera CPU and compared it to an Intel Xeon 6, m8id.metal-96xl.

On the NVIDIA Vera, Polars ran 2 to 2.5x faster compared to Xeon, and 2 to 3x faster than the runner-up.

https://t.co/hKDzWMzO55

[polars data @DataPolars]

@naveenpalli @sh_reya That memory isn't correct. It's mmap. The OS doesn't purge mmap pages unless under memory pressure. On a smaller machine, you'll see smaller memory usage, as there is less mmap cache.

[Mim @mim_djo]

#polars 2 is very good
https://t.co/aPSifNq00f

[polars data @DataPolars]

@IntenseRealist That's ours of course! https://t.co/mO7KMU6e4t

[polars data @DataPolars]

We just released Polars 2.0.

It removed many of our legacy decisions makes the streaming engine our default and promotes SQL to a first class citizen within Polars.

It comes with initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://t.co/ATc6UzJqby

[polars data @DataPolars]

Polars 2.0.0rc2 has been released.
We expect this to be the last release candidate and that Polars 2.0 will go live next week.

Install now by running `pip install --pre polars`

Link to the full release guide: https://t.co/K1rHqtWEBV

[polars data @DataPolars]

Profile any Polars query to view its progress and optimize performance with one line change:

pl.Config.enable_monitoring()

Read the full blog: https://t.co/hhheJ8EUhN

[polars data @DataPolars]

Come join us at the second day of Pydata Amsterdam!

Learn about our new query profiler and let an agent optimize your queries!

[polars data @DataPolars]

We are happy to announce the first pre-release of Polars 2.0.

Polars 2.0 will bring the streaming engine as a default and improves overall strictness of Polars.

Give it a spin and share your thoughts. We expect the final Polars 2.0 release in 3-4 weeks.

https://t.co/0KyuugiDqq

[polars data @DataPolars]

The Polars GPU engine now runs on a new streaming backend, built with NVIDIA on their RapidsMPF library.

• Queries are no longer bound by GPU memory. Data flows through the query graph in morsels that spill to host memory when device memory gets tight.

• The same query scales from one GPU to many. Construct a RayEngine, pass it to .collect(), and there is no cluster to configure and no change to the query itself.

Full post: https://t.co/AvXRN4MFIT

[polars data @DataPolars]

We've released Python Polars 1.44. Some of the highlights:

• Schema evolution when writing to Iceberg
sink_iceberg() gained schema_mode="merge", so appending a frame that has a new column extends the table schema instead of raising. Rows written earlier read back as null for that column.

• Adaptive rate limiting for cloud I/O
Polars now learns how many requests per second S3, GCS, or Azure will actually serve and paces itself to stay under it, so large scans spend less time in retries and are far less likely to fail on throttling.

• Correlated subqueries in SQL
Subqueries that reference the outer query now work in WHERE, in the select list, and in EXISTS and IN.

Blog post: https://t.co/sGbXf5ljfY
Full changelog: https://t.co/4uIgLeiBEm

[polars data @DataPolars]

As datasets grow, your tools can limit what analysis is possible. That causes teams to slows down, or forces them to work with samples rather than the whole dataset.

BMLL Technologies tackled this issue by migrating their market data workloads to Polars, resulting in a 48x performance improvement. At that speed, a single on-demand machine can aggregate over 1.5 TB of market data in under 4 minutes, removing the need for overnight batch processing infra.
Work now fits on a single machine, allowing teams to iterate daily instead of weekly, compounding small wins faster and capturing more basis points.

Read the full case study here: https://t.co/M085hi5Lew

[polars data @DataPolars]

You can turn a column into labeled buckets. With if/elif/else logic takes one chained expression: pl.when().then().otherwise().

Conditions are checked top to bottom, and the first match wins, exactly like Python's if/elif/else. Chain as many .when() branches as you need, and every branch runs vectorized in Rust rather than row by row.

Note: a bare string inside then() is read as a column name, so .then("adult") looks for a column called "adult" and raises a ColumnNotFoundError. If you want to fill a string value instead, use pl.lit().

[Ritchie Vink @RitchieVink]

We will ship the first Polars 2.0 release candidate next week. Polars 2.0 will be great for casual Polars users as it will default to the streaming engine, meaning your code will be faster without doing anything.

So the first release candidate next week. Give it a spin!

[polars data @DataPolars]

Time series datasets are seldom perfect. Polars gives you a whole menu for repairing them, and the right pick depends on what you need.

`fill_null` accepts more than a constant value to fill. `strategy="forward"` carries the last value forward, the natural choice for state that persists until it changes, like a stock ticker. `strategy="backward"` pulls the next known value back, and there is also `"mean"`, `"min"`, `"max"`, `"zero"`, `"one"`, plus a `limit` parameter to cap how many consecutive nulls a fill may bridge.

For values that were moving smoothly while a sensor was offline, for example, `interpolate()` draws a straight line between the surrounding points. Combine it with `.over("sensor_id")` and each sensor is filled using only its own history.

[polars data @DataPolars]

When migrating your pandas pipeline to Polars there are several ways to go about it.
Read which one works best for you and the tips and tricks to make it successful here: https://t.co/xuEpiMimv1

[polars data @DataPolars]

We've released Polars Cloud client 0.10.0. Some of the highlights:

• Stream query results into Python with `sink_batches()`
• A new experimental query planner: Miso
• Distributed `pl.collect_all()`
• Hive-partition aware scans
• On-Prem HDFS support

Blog post: https://t.co/B93JGTWWbG
Full changelog: https://t.co/YobYV2x5DS

[Plotly @plotlygraphs]

What does it take to serve a month of Polymarket orderbook data, with 16 billion events and 500 GB of Parquet, as a live interactive dashboard?

Less than you'd think.

We partnered with the Polars Cloud team to take this dataset to production:
- Polars Cloud handles the heavy compute; the same code that processes an hour of data on your laptop scales to a full month across a cluster, unchanged
- A Dash app serves it as a thin, stateless layer reading query-optimized Parquet
- Plotly Cloud handles the ops: machine sizing, Always-On compute, and metered billing

And with Dash 4.3, the app doubles as an MCP server. Claude, Cursor, or ChatGPT can connect and query the actual dataset, to answer users' questions.

Explore the live dashboard and read the full breakdown 👇
https://t.co/exP8zlqw4i

https://t.co/IrCrFAQ7da

[polars data @DataPolars]

Aggregating 500GB of compressed parquet with 16,000,000,000 rows to display it in an enterprise-ready Plotly dashboard.

We explored a sample of the Polymarket dataset locally on a laptop, and once we got the pipeline like we needed it to be for the visualisations, we scaled it up to its full month size, without changing the query, to Polars Cloud. The results can be viewed on a live dashboard on @plotlygraphs Cloud!

Check the dashboard: https://t.co/Duy7q6Vbxk
And the blog: https://t.co/51o2Vr4cd9