Polars 2.0

Polars has released version 2.0, making its streaming engine the default when users collect a LazyFrame and enabling an initial form of out-of-core processing that can spill supported work to disk. The major-version change matters because streaming can alter observable row order for some operations, so users who depend on ordering may need to set an explicit option.

According to the release post, calling collect on a LazyFrame now uses the streaming engine by default. Polars says this can improve memory use and performance on many queries. However, the engine does not guarantee row order by default for certain operations, including joins, group-by operations and unpivot. Users who need to preserve order can set maintain_order=True for supported operations.

Version 2.0 also turns on an initial spill-to-disk capability. Polars says spilling begins at about 80% of available RAM, a threshold the project says may need tuning, and the default disk budget is 64GB. At release, supported workloads include sorts, window functions and many expressions. The post says joins and group-by operations are not yet covered by out-of-core processing, but are planned for later work.

Other changes include broader SQL support, improvements to the optimizer and engine, and a new Map dtype for Arrow MapType data. The release post lists join reordering, common-subplan elimination and dynamic predicates or bloom filters among its performance work. It also describes stricter handling of data types and explicitness, intended to provide earlier feedback during development.

Streaming Changes Query Behavior

The new default could affect both query performance and the way applications handle results. Streaming is intended to process data without requiring the full working set to remain in memory, while disk spilling gives some supported operations a way to complete when memory is constrained. That may help users working with datasets or transformations that exceed available RAM, although the release does not claim that every operation can now run out of core.

The row-order change is a compatibility issue users should test. Code that relies on a particular order after joins, grouping or unpivoting may produce a different sequence unless it explicitly requests order preservation. The release post presents the major version as the point to make this default change, so teams upgrading existing pipelines will need to check assumptions as well as benchmark performance.

Polars also presents SQL as a first-class interface, broadening the ways people can use its query engine. The project’s benchmark results may be relevant to teams comparing analytical tools, but they are not independent evaluations: the measurements were run and reported by Polars under specified hardware and query conditions.

Benchmarks and Release Scope

To support its performance claims, Polars compared its SQL engine with DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0 on TPC-H- and TPC-DS-derived data. The project reports tests on two AWS machines: a c7a.4xlarge with 16 virtual CPUs and 32GB of RAM, and a c7a.metal with 192 virtual CPUs and 384GB. Each query ran five times in a hot setting, with the best run used for comparisons. The post says Polars and both DuckDB versions completed all queries; DataFusion timed out on TPC-DS query 72, timed out once on query 67, and ran out of memory on TPC-H query 18 on the smaller machine. Those queries were excluded from results for all engines.

Polars says its default configuration was fastest in all but one benchmark, while also reporting that its overhead at 192 threads hurt smaller queries. It says a 32-core configuration was competitive or leading in all benchmarks and that the cause of the scaling behavior had been diagnosed, with a possible fix expected in a later release. These are Polars-reported results, not a general guarantee of performance across workloads. The project published a benchmark repository so others can replicate the tests.

The release also changes how Arrow MapType data is represented. Before version 2.0, the post says Polars read that type as a list of key-value structs; the new Map dtype supports dictionary-like operations such as looking up a key, checking whether a key exists, and retrieving keys or values.

“Calling collect on a LazyFrame will now default to the streaming engine.”

— Polars, in its version 2.0 release post

Limits Still Affect Performance

The release post does not specify a publication date, and it provides no independent validation of the benchmark results. Performance can vary with data, query shape, hardware and configuration, so the reported comparisons should not be read as a ranking for every deployment. The source describes the results and test conditions but does not provide enough information here to establish how representative they are of users’ production workloads.

Some operational details also remain subject to change. Polars says the spill threshold of about 80% of RAM may need tuning; it does not state how that threshold will behave across different environments. The default disk budget is 64GB, and support for spilling joins and group-by operations is future work rather than a capability available in this release. The post also says the 192-thread overhead has been diagnosed, but does not name a confirmed fix or release date.

More Spill Support Planned

Polars says it plans to add out-of-core support for joins and group-by operations, which would extend disk-spilling beyond the operations covered at launch. It also says it hopes to address the performance overhead observed when scaling to 192 threads in a subsequent release, without committing to a date in the supplied post.

For users adopting version 2.0, the immediate next step is to test existing queries for ordering assumptions and workload-specific performance. The project’s public benchmark repository gives other teams a way to reproduce its comparison, while further releases should clarify how the spill threshold, operation coverage and large-machine scaling develop.

Key Questions

What is the main change in Polars 2.0?

LazyFrame.collect now defaults to the streaming engine. Polars says this can improve memory use and performance on many queries, but some operations no longer guarantee row order unless users request it.

Which operations can currently spill to disk?

The release post lists sorts, window functions and many expressions as supported. It says spill support for joins and group-by operations is planned, not available in this initial release.

How can users preserve row order?

For supported operations where order matters, Polars says users can set maintain_order=True. Teams should test their own queries to identify where their code depends on ordering.

Did Polars independently verify its benchmark advantage?

No independent verification is described in the release material. Polars ran and reported the benchmarks, comparing its SQL engine with DuckDB and DataFusion under specified hardware and test conditions.

What is the new Map dtype for?

It represents Arrow MapType data directly as a Polars Map dtype, rather than as a list of key-value structs. The release describes dictionary-like operations including key lookup, key-presence checks, and access to keys and values.

Source: hn