Dataframes powered by a multithreaded, vectorized query engine, written in Rust.
Amsterdam
- Tweets 453
- Following 6
- Followers 7,356
- Likes 396
We are happy to announce Python Polars 1.0!
pola.rs/posts/announcing-pol…
6
141
628
58,746
We're very happy to announce that @nvidia and Polars will work together to bring GPU acceleration to Polars DataFrames! 🚀💯
pola.rs/posts/polars-on-gpu/
11
99
552
49,536
Together with NVIDIA RAPIDS we bring GPU acceleration to Polars!
Read more in our release blog:
pola.rs/posts/gpu-engine-rel…
7
58
305
41,041
Polars 0.16.0 is out!
github.com/pola-rs/polars/re…
A few breaking changes that lead to more strictness and more consistency. It feels good to ditch mistakes!
37
276
52,310
Happy New Year!
With a new Python Polars release:
github.com/pola-rs/polars/re…
A new Rust Polars release:
github.com/pola-rs/polars/re…
And a new release for Polars plugins:
github.com/pola-rs/pyo3-pola…
Oh.. And polars now supports plotting😎
6
29
250
28,348
🤗 x Polars!
Polars now supports native reading from @huggingface datasets. Check out our latest blog to learn more about it: pola.rs/posts/polars-hugging…
4
35
188
18,254
We surpassed 25.000 Github Stars yesterday!!
🐻❄️🚀
github.com/pola-rs/polars
2
16
178
6,412
Export your Polars DataFrame directly to pytorch with the new `to_torch` export.
See more in this weeks release guide:
github.com/pola-rs/polars/re…
1
23
166
8,717
You can read the pre-release of the first chapters of the upcoming O'Reilly polars book.
Join our discord to download the PDF: discord.com/invite/4UfP5cf…
The first two chapters of our book Python Polars: The Definitive Guide have just been released!
It ain't much, but it's honest work.
@thijsnieuwdorp and I are working hard on the remaining 16 chapters. We'll keep you posted.
6
32
161
24,272
Polars 0.15 is released.🐻❄️
Parts of the query engine are rewritten and we have seen up to 20% performance increase because of it and huge decreases of RAM usage.
Next releases will increase performance of parquet reading. 🚀
github.com/pola-rs/polars/re…
26
162
Python polars 0.20.4 is released with as highlights:
- Full support for the DataFrame interchange protocol.
- A lot of extra SQL functions
- Better hive partition pruning
- Many new expressions for the Array datatype
See the full changelog here:
github.com/pola-rs/polars/re…
2
25
157
11,035
Polars 0.16.10 is out. Only 5 days in the making, but you wouldn't say that...
- full out-of-core sort
- writing to excel
- writing to databases
- streamable UDFS
- skipping of entire parquet pages on `is_in` and equality queries.
- decimal datatypes
...
github.com/pola-rs/polars/re…
2
19
153
10,802
Python polars 0.18.0 is released. This is likely the latest 0.x release before we land 1.0
Notable changes:
.arr -> Array
.list -> List
.implode <-> explode symmetry
github.com/pola-rs/polars/re…
4
24
150
10,680
Plotting in Polars 1.6 now requires no extra heavy dependencies. Read how we were able to, thanks to Narhwals and Altair: pola.rs/posts/lightweight_pl…
1
21
145
12,883
Rust polars and python polars release all in one day...
Have a nice weekend!
github.com/pola-rs/polars/re…
github.com/pola-rs/polars/re…
2
11
136
8,065
Python Polars 0.19.8 is released!🐻❄️
Another release with great performance improvements on our cloud parquet reading.
And more improvements are in the pipeline.
github.com/pola-rs/polars/re…
7
8
127
17,859
Polars backed time series ML library. According to the authors it is the fastest in the world👀
1
12
125
15,855
The new polars release ships with native cloud readers for `scan_parquet` AND support for reading hive partitioned datasets (with pushed down optimizations).
This is in beta-release, so give it a spin and do provide feedback, so that in next releases all will run smoothly. :)
7
18
126
10,432
Polars will start a new phase!
I am very excited to announce that Polars raised a $4M seed round!
Chiel Peters and I co-founded Polars the company. Read more on what we will build!
pola.rs/posts/company-announ…
1
13
117
8,920
Enhanced zero-copy interop with NumPy, faster boolean filters and fast path for COUNT(*) queries and much more in our new Polars in Aggregate:
pola.rs/posts/polars-in-aggr…
12
117
9,184
With python polars release 0.19.9 you can compile and link your own custom expressions into the default polars engine.
Those user provided expressions will enjoy:
- Rust speed
- Parallelism
- Query optimizations
Read more here:
pola-rs.github.io/polars/use…
1
20
107
36,637
Polars 0.16.14 has landed!
github.com/pola-rs/polars/re…
Most string functions have been optimized with performance increases ranging from 2-15x!
And the import times has been reduced drastically to < 50 seconds.
For comparison:
- numpy: 104ms
- pandas: 520ms
5
19
118
9,178
We are happy to share more about what we are building and our goal to run Polars on any dataset size!
A managed Distributed Polars compute cluster to ensure a single DataFrame API for all your needs.
pola.rs/posts/polars-cloud-w…
13
107
6,780
A new release of Polars in Aggregate!
TLDR:
- Full plan CSE
- Benchmark of the updated CSV writer
- Dead expression elimination
- Ad-hoc SQL on DataFrame and LazyFrame
- Support for export to PyTorch and JAX
pola.rs/posts/polars-in-aggr…
2
12
106
4,851
New python polars release. Interesting changes since last week.
- Zero-copy interchange between numpy/ polars.
Moving data between polars and numpy will now only move pointers around.
4
9
100
13,276
Polars 1.19 comes with support for arbitrary predicates in join_where.
This means that inequality joins are now more flexible than ever!
Here is a small example of something you couldn't do before:
ALT A diagram showing how `join_where` now supports arbitrary predicate expressions. The example uses an expression with the string namespace function `str.starts_with` to match URLs in one dataframe to the domain names in another dataframe.
1
10
97
4,651
Polars supports dynamic aggregations based on time windows via the function `group_by_dynamic`.
To use it, you specify a date(time) column to group by, and then determine the windows over which values are aggregated.
Note: data points can fall within 2+ windows 👇
ALT Diagram showing how the window boundaries of 10-year long windows align neatly with the decades and the decade halfpoints, 1980, 1985, 1990, etc, although the first datapoint is from 1981.
3
5
95
4,627
Last few months we've made a lot of improvements for the Polars in-memory engine.
And it shows in the latest TPC-H benchmark run!
pola.rs/posts/benchmarks/
4
91
6,219
The team at @dbsystel (subsidiary of Deutsche Bahn) optimizing Germany’s train schedules realized 20x speedups after switching to Polars. Lowering cloud costs and supporting the organization's sustainability goals: pola.rs/posts/case-db-systel…
5
7
86
5,429
Try duckdb AND polars.
DuckDB 0.7.0 "Labradorius" released with #JSON support, parallel and partitioned export to CSV and Parquet, UPSERT, @DataPolars integration, and much more in our release announcement blog post: duckdb.org/2023/02/13/announ…
9
85
15,940
Recent research from the @VUamsterdam shows that Polars can be up to 8 times more energy efficient than pandas. Making Polars a sustainable choice for large scale data analytics.
pola.rs/posts/benchmark-ener…
1
8
85
4,772
Latest polars vs pandas 2.0 with pyarrow datatypes.
Queries as a whole tell a more realistic story.
Pandas 2.0 vs Polars 0.16.10
I see people taking the wrong conclusions from this microbenchmark. Here are the results of pandas with arrow backed datatypes vs polars on TPCH queries.
Micro benchmarks don't show the whole query runtime. Code here:
github.com/pola-rs/tpch/pull…
1
9
80
14,901
Check rewrote their airflow DAGs from pandas to Polars and:
- Got rid of their out of memory SIGTERMs
- Made their architecture simpler
- Reduced the ETL time by ~50%
- Scaled down their cluster, saving 25% in cloud bills.
pola.rs/posts/case-check-tec…
3
84
5,396
Python Polars 0.19 is released!
See the migration guide: pola.rs/posts/polars-0-19-up…
And full changelog: github.com/pola-rs/polars/re…
1/6
1
8
75
3,189
Polars latest release comes with FULL plan Common Subplan Eliminiation (CSE). Duplicate parts of the query will automatically be cached/shared.
Here is the difference on TPCH q2:
1
9
73
6,999
The Polars team has been hard at work!
We've just published a post with an overview of some of the additions and changes in Polars from 1.3.0 to 1.15.0.
We cover performance improvements to I/O, the addition of inequality joins, and much more!
pola.rs/posts/polars-in-aggr…
1
8
72
2,425
Always wondered what happens when you run a query in Polars? In our next blog post 'A birds eye of Polars' by Chiel Peters we take a look at the internals of Polars.
pola.rs/posts/polars_birds_e…
4
15
71
7,969
Rust polars 0.34 is released!
Find all the goodies below:
github.com/pola-rs/polars/re…
4
6
70
4,856
Why is there a `struct` data type?
A single expression produces a single column, so expressions like `value_counts` need to output structs to map the values to their counts.
With that said, do you understand why `.struct.unnest` doesn't break the 1 expr = 1 column principle?
1
5
68
3,345
Polars 0.15.17 is released.
github.com/pola-rs/polars/re…
6
63
5,698
Today we are launching the first open Crash Course training sessions with a limited time discount. These instructor-led sessions are open to everyone looking to get up and running with Polars.
Find a date and sign up via our Academy: pola.rs/academy/
1
8
62
4,580
Pandera just shipped Polars support!
pandera--1373.org.readthedoc…
2
6
53
5,804
Polars 0.17.15!
This weeks release adds:
- Arrows FixedTypeList (we call them Arrray)
- writing to delta tables
- UNION, EXPLAIN, CASE, EXCLUDE to its SQL vocab
- SIMD acceleration for all json parsing
See all changes: github.com/pola-rs/polars/re…
11
61
6,485
Can't remember how many days each month has?
(Me neither!)
Memorise this Polars snippet instead.
Using some calendar-aware functions, we can get the answer in a tidy dataframe, as the diagram below shows.
4
5
57
3,810
Polars in Aggregate: A small subselection on what we have been working on.
pola.rs/posts/polars_in_aggr…
2
5
54
7,723
That's
`pl.scan_csv(..).sink_parquet(..)`
While building Data Engineering or Machine Learning pipelines using CSV data, you should first:
🔥Convert the CSV data to parquet and save it using @DataPolars.
Using Parquet + @ApacheArrow combination with almost any data engine is significantly faster than CSV
3
5
56
7,658
The context filter lets you filter out rows from a dataframe based on some conditions.
Within an aggregation, you can also use filter to filter values from aggregated groups.
In this example we ignore unverified times when computing the current record.
ALT Diagram showing how filter works inside aggregations. We use an aggregation to compute the current record for different swimming race types and we use a filter to ignore unverified times, while also using information about unverified times to compute another column.
2
11
56
3,683
In our latest blog we uncover predicate pushdown, a query optimization technique. Read more on what it is and how you can apply at the link below.
pola.rs/posts/predicate-push…
4
52
3,870
Polars 0.15.3 is out!
Coming with support for reading @DeltaLakeOSS delta-tables AND polars is able to push optimizations into the delta reader.
github.com/pola-rs/polars/re…
1
9
53
Watch Marco Gorelli's Pydata London talk about Polars plugins. A thorough guide on how to get started and bring your custom expressions powered by crates.io!
piped.video/j2N_YD5vbOs?si=-N9e…
7
52
3,881
Polars has essentially 18 different data types.
If you are unsure what each type is, the conversion table below might help you.
Each Polars data type is presented next to the **most similar** Python type.
ALT Diagram establishing a connection between Polars data types and similar Python types. The connections are as follows: Boolean is most similar to bool Int8, Int16, Int32, and Int64 is most similar to int with restrictions UInt8, UInt16, UInt32, and UInt64 is most similar to int with restrictions Float32 and Float64 is most similar to float Decimal is most similar to decimal.Decimal String is most similar to str Binary is most similar to bytes Date is most similar to datetime.date Time is most similar to datetime.time Datetime is most similar to datetime.datetime Duration is most similar to datetime.timedelta Array is most similar to numpy.array List is most similar to list Categorical is most similar to enum.StrEnum Enum is most similar to enum.StrEnum Struct is most similar to typing.TypedDict Object is most similar to object Null is most similar to None
3
4
47
2,377
The expression `clip` is pretty straightforward:
You provide a lower and an upper bound, and Polars makes sure all values fall within those bounds.
If a value is too small/too large, it's replaced by the bound.
Bounds can be literals, other columns, or arbitrary expressions.
ALT Diagram showing how the expression clip works. The diagram shows `clip` used with integers and with `datetime` objects. The diagram also shows the usage of literals, other columns, and arbitrary expressions, as the bounds.
5
46
2,569
This week in polars:
github.com/pola-rs/polars/re…
1
3
47
4,525
Extend Polars and run your custom functions as fast as native expressions.
Double River re-implemented two key models as plugins and achieved 10x speedups with half the memory usage.
pola.rs/posts/case-double-ri…
4
45
2,542
The expression over can be used to compute expressions within isolated groups.
This means you can do computations per group without having to group first and then explode after.
In this example, we rank swimmers based on their time, but within their race type.
7
46
2,441
A polars LazyFrame knows its schema before running it.
That means you can assert the output data types and catch errrors before running your query.
1
2
44
#Python polars 0.14.19 is released. This is the first of many releases where we will improve the capabilities to process larger than RAM data.
This release on that topic:
- first parts of streaming engine
- batched csv reader
github.com/pola-rs/polars/re…
1
9
42
Rust polars 0.31 is released!
github.com/pola-rs/polars/re…
6
41
2,849
You want to join two tables on their ID column, but only when the dates in one table fall within the range of the other table.
Polars lets you do that with `join_where`, which supports inequality joins through the use of inequality predicates.
Here's an example 👇
ALT Diagram exemplifying how `join_where` performs inequality joins. We're joining two dataframes that have 3 rows each and each row has a corresponding row on the other dataframe because they have matching IDs. The dataframe on the right has a column `dt` with date values, which fall within the range created by the columns `start` and `end` of the other dataframe, except for the third row, in which `dt` falls outside the range. When using `join_where` with the predicate `pl.col("dt").is_between("start", "end")`, the third row of each dataframe will not be matched and won't show up in the join result, which only joined the first 2 rows of each dataframe.
3
38
1,661
Polars support in PyCharm! 💯
📣 PyCharm 2023.2 EAP 2 is now out!
Try these new features:
🥁 Live templates for Django forms and models
🥁 Support for @DataPolars DataFrames
🥁 Initial @GitLab integration
Read more and download 👉
jb.gg/t1xv3m
1
37
2,936
Scaling up any Polars (or other) query done easy with coiled!
This blog shows how it easy it is to run something on a big VM.
New blog post on how to process large datasets with @DataPolars in the cloud.
Polars is a great tool for querying large datasets. When you need more memory, you can use Coiled serverless functions to run Polars on a big VM in the cloud. medium.com/coiled-hq/process…
ALT Code snippet of using the coiled.function decorator to run a query with Polars on a large VM in the cloud.
3
31
4,361
How to “expand” ranges like "3-5" to three different rows with the integers 3, 4, and 5, and all other variables are the same?
ALT A diagram shows how to use str.split, list.first / list.last, cast, pl.int_ranges, and explode, all together, to turn a dataframe where a column may contain ranges like "3-5" into a similar dataframe where all ranges have been expanded, or exploded, across multiple rows. The full code is: range_start = pl.col("nrs").str.split("-").list.first().cast(pl.Int64) range_end = pl.col("nrs").str.split("-").list.last().cast(pl.Int64) df.with_columns(pl.int_ranges(range_start, range_end + 1)).explode("nrs")
2
2
33
1,959
Join our webinar with @nvidia on January 28 for an in-depth session on how the GPU engine works, from collecting your query to parallel execution on the GPU.
Sign up at info.nvidia.com/nvidia-polar…
1
3
33
1,978
#Python #polars 0.14.27 is released. github.com/pola-rs/polars/re…
This release launches the alpha version of our streaming engine.
Need to process 250GB on you laptop? Try:
my_query().collect(allow_streaming=True)
Next we release we will be able to | the result to different files!
5
33
We are looking for a Rust Engineer based in the Netherlands. Are you interested in the job, or do you know someone who is? Please reach out.
hiring.pola.rs/o/rust-softwa…
2
13
31
7,251
Polars and Narwhals now temporarily are visible as street art in Amsterdam.
Big shoutout to @Anopsy
2
5
28
3,527
Congratulations Matt🎉
Today is the pre-sale of "Effective Polars". A great boost for Polars content!
🥧 Happy Pi Day, Data Enthusiasts! 📊
I hope you are enjoying (or will enjoy) some literal pie, and perhaps you are enjoying some "pie"thon today! 🐍
While I love math, today marks another big day for me.
The presale launch of "Effective Polars".
Folks have been clamoring for this book for a while. I expect to have it released next week.
🚀 Why dive into "Effective Polars"?
• Speed and Efficiency: Learn how to leverage Polars for high-speed data processing, transforming how you handle large datasets.
• Real-World Applications: Packed with practical examples, this book guides you through implementing Polars in your projects, making data work for you.
• Expert Insights: Benefit from my journey with Polars, sharing the nuances, best practices, and tips that will elevate your data analysis skills to new heights.
🎉 Exclusive Pi Day Presale Offer:
I’m offering an exclusive discount for the Pi Day presale to celebrate. There are a few options available. If you buy a bundle, the video course is already available.
🔗 Secure your copy now store.metasnake.com/effectiv…
Use the code PIDAY for a 31% discount today only!
1
27
3,291
Researchers and engineers at G-Research are using Polars in their projects to handle data more efficiently. Reporting 150x speedups over earlier implementations.
Read our latest case study: pola.rs/posts/case-gresearch…
1
6
27
5,558
And next week our new streaming engine will be usable. We're already beating the in-memory engine on TPC-H and we know we can improve much more in the coming months. Stay tuned: github.com/pola-rs/polars/is…
1
2
28
1,012
New article on the Polars blog!
“Breaking the rules with expression expansion” delves into how `.struct.unnest` seems to break one of Polars'most fundamental principles but doesn't:
A single expression must always produce a single column as a result.
👉 pola.rs/posts/breaking-the-r…
3
27
1,675
Python polars 0.14.22 is released.
github.com/pola-rs/polars/re…
3
26
Win a Polars swag box in the @posit_pbc 2024 Table Contest.
The top table entry that involves Polars will win the box, containing a Polars hoodie, stickers and more.
Find out how to participate: posit.co/blog/announcing-the…
1
4
24
4,154
And you are in luck!
@braaannigan has made his course free today!
udemy.com/course/data-analys…
1
3
24
1,078
Polars core algorithms have large materialization speed-ups. This will most likely be significant on tables in the millions of rows.
The re-run of the db-benchmark made me look at our groupby and join materialization again and I have mode some solid performance improvements last 2 weeks. Some queries are ~2x faster!
23
2,434
Today we added an @kapa_ai LLM Plugin in our docs that has RAG access to our reference and user guide.
This drastically improves Polars code generation!
docs.pola.rs/api/python/stab…
2
3
24
2,677
Extend python polars with UDFs compiled in rust!
I just added pyo3 bindings to polars `Series` and `DataFrame`.
Extending python polars with your own rust functions is now super easy!
crates.io/crates/pyo3-polars
20
4,058
#Python polars 0.15.2 release🚀
Most notable this week is a perf improvement in deserializing RLE columns in parquet. Reading string columns can be up to 40% faster!
github.com/pola-rs/polars/re…
2
22
Jetbrians DataSpell has polars support!
DataSpell 2023.2 EAP 1 is out! Highlights include:
- Polars column-name completion
- Interactive tables for Polars DataFrames
- Easier column data type identification
Learn more and download DataSpell here:jb.gg/b026ov
3
1
22
2,266
Great example of plotting with polars.
I find fascinating:
- How beautiful @DataPolars pipelines look like (and how fast they are)
- How nicely @plotlygraphs works with #Polars
- That all real estate data in #Dubai is open (also other data published by @DigitalDubai)
5
14
2,691
Discover how Joulen uses Polars to build a highly scalable, cost-effective infrastructure for analyzing data from renewable energy assets in this case study: pola.rs/posts/case-joulen/
1
14
1,680
📢 Hot off the press, new blog article:
“Understanding Polars data types”
Many dtypes you already know or understand intuitively.
This article will push your intuition further, so you can confidently pick the appropriate data types when you are working with data in Polars.
1
14
1,268
Interested in machine learning pipelines with polars?
@marktenenholtz course covers that!
The “Advanced Topics” week of my course lets students build a whole feature engineering pipeline with it.
Everyone who tried it *loved* the library and a few said they were immediately bringing it to their work projects.
Check out the course here: corise.com/go/forcasting-wit…
1
10
1,820
@rasbt the string.replace edge will be resolved on next release:
github.com/pola-rs/polars/pu…
1
12
1,276
Come chat with use in the discord channel:
discord.gg/4UfP5cfBE7
You don't have to bring cake. 👋
1
3
12