r/dataengineering • • 9d ago

Open Source Python Polars 2.0 release

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

Disclosure: I am involved with Polars

355 Upvotes

19 comments sorted by

81

u/kvlonge 9d ago

Amazing stuff guys! Polars is a stunning project.

Just saw the SQL 'first class citizen' comment as well. That is brilliant news!

56

u/Efficient_Shoe_6646 9d ago

I love that Rust is taking Data Engineering by swarm. DataFusion and Polars are going to be industry standards pretty soon.

9

u/commenterzero 9d ago

It can be very.....polarizing

9

u/Wojtkie 9d ago

Sweeeet!!! I’m stoked. I’ll have to give it a look later today

9

u/alt_acc2020 9d ago

>out of core support

Yay! Polars’ iceberg lib being broken + lack of spilling made it really hard to pick it over Duckdb for some of our app-layer processing. Really glad to see these in finally :)

7

u/zmanvv 9d ago

Fantastic work as always, congratulations to you and the team Ritchie!

5

u/necrohobo 8d ago

I have standardized the use of polars at work, many now see the light.

6

u/CrayonUpMyNose 9d ago

A loot of tools - and LLM coding assistants - still default to pandas. Is publishing how-to guides and code samples, especially for library developers, the way to overcome this inertia in adoption by having LLM training data scrapers pick them up?

20

u/VipeholmsCola 9d ago

If you point any llm to the polars docs they write polars fine 

1

u/CrayonUpMyNose 9d ago

That's understood. The question is how to get to the point where it becomes the default.

3

u/VipeholmsCola 9d ago

Its goingto default to the collective knowledge which pandas currently has, because its older. Just tell the LLM to switch to polars and its no problem

1

u/imanexpertama 9d ago

Since when? Genuinely asking, it wasn’t working too good for me 1/2 - 1 year ago and I wonder if that has changed or I went wrong somewhere else.
I felt it especially annoying with differences between versions, the generated code would default to older versions and Claude kept telling me to update

5

u/kvothethechandrian 9d ago

There is a official polars skill for claude that pulls docs matching your polars version and helps writing better code with polars. Honestly for me it's been a breeze, and the API is much more readable than pandas

2

u/wildjackalope 9d ago

A lots happened in a year, might be worth another shot. Honestly, I’ll take the LLM friction if I don’t have to deal with Pandas API.

4

u/Past_Huckleberry5571 9d ago

pandas in 2026 omegalul

1

u/isaac-harvey 5d ago

Wondering how to choose between DuckDB and Polars for SQL processing? Both seem pretty fantastic.

1

u/Western_Rub_7236 3d ago

And I am already shamelessly using it already to build another product. https://github.com/det-Labs/det[det](https://github.com/det-Labs/det)

1

u/espero 2d ago

Data frames with Polars or Pandas?