Photo by the editor
# Entry
Pandas is undoubtedly a powerful and comprehensive library manage and analyze data flowssomething fundamental in the bigger picture of data science. However, when dataset sizes become very immense, this may not be the most effective option as it mainly runs on a single thread and relies heavily on the Python interpreter, which can lead to significant processing time.
This article focuses on a newer library that speeds up Pandas-like operations: polar. In particular, I will share with you 10 insightful one-liners from Polars to improve and accelerate every day data manipulation and processing tasks.
Before you start, don’t forget import polars as pl First!
# 1. Loading CSV files
The Polars method of reading a set of data from a CSV file looks very similar to its Pandas counterpart, except that it is multi-threaded (and internally written in Rust), which allows it to load data in a much more effective way. This example shows how to load a CSV file into a Polars data frame.
df = pl.read_csv("dataset.csv")Even for a medium-sized dataset (not just a very immense one), the difference in time it takes to read a file using Polars can be about 5 times greater than using Pandas.
# 2. Lethargic loading for more scalable workflows
Creating a so-called “lazy data frame” rather than quickly reading it in one go is an approach that allows you to chain successive operations throughout your data workflow, executing them only when collect() eventually the method is called – a very useful strategy for immense scale data pipelines! Here’s how to apply inactive loading of a data frame using scan_csv() method:
df_lazy = pl.scan_csv("dataset.csv")# 3. Select and rename the appropriate columns
To make later processing easier and clearer, it’s a good idea to make sure you’re only dealing with dataset columns that are relevant to your data science or analytics project. Here’s how to do it effectively with Polars data frames. Let’s assume you’re using a customer dataset like this. You can then employ the following one-liner to select the appropriate columns as follows:
df = df.select([pl.col("Customer Id"), pl.col("First Name")])# 4. Filtering a subset of rows
Of course, we can also filter specific rows, e.g. customers, using the Polars method. This one-line filter is used to filter customers who live in a specific city.
df_filtered = df.filter(pl.col("City") == "Hatfieldshire")You can employ a method like display() Or head() to see the result of this “query”, i.e. rows that meet certain criteria.
# 5. Grouping by category and computational aggregations
With operations like grouping and aggregation, the performance value of Polars really starts to show in larger datasets. Take this one line for example: The key here is linking group_by in the categorical z column agg() to perform an aggregation of all rows in each group, e.g. the average of a numeric column or simply the number of rows in each group, as shown below:
df_city = df.group_by("City").agg([pl.len().alias("num_customers")])Be careful! In Pandas, groupby() there is no underscore symbol, but in Polars there is.
# 6. Creating derived columns (straightforward feature engineering)
Thanks to Polars’ vector computing capabilities, creating novel columns based on arithmetic operations on existing ones is much faster. This one-line snippet shows this (now considering the popular file Housing in California dataset for examples below!):
df = df.with_columns((pl.col("total_rooms") / pl.col("households")).alias("rooms_per_household"))# 7. Using conditional logic
Continuous attributes such as income levels or similar attributes can be categorized and transformed into tagged segments, all in a vectorized and overhead-free manner. This example does this to create income_category column based on average income per district in California:
df = df.with_columns(pl.when(pl.col("median_income") > 5).then(pl.lit("High")).otherwise(pl.lit("Low")).alias("income_category"))
# 8. Doing a inactive stream
This one-liner, although slightly larger, combines several of the ideas presented in the previous examples to create a inactive pipeline executed with collect method. Remember: for this inactive approach to work, you need to employ single-line number 2 to read the dataset file “lazily”.
result = (pl.scan_csv("https://raw.githubusercontent.com/ageron/handson-ml/master/datasets/housing/housing.csv")
.filter(pl.col("median_house_value") > 200000)
.with_columns((pl.col("total_rooms") / pl.col("households")).alias("rooms_per_household"))
.group_by("ocean_proximity").agg(pl.mean("rooms_per_household").alias("avg_rooms_per_household"))
.sort("avg_rooms_per_household", descending=True)
.collect())# 9. Combining data sets
Suppose we have an additional dataset called region_stats.csv with statistical information collected for California counties. We could then employ such a single-line expression to apply join operations to a specific categorical column as follows:
df_joined = df.join(pl.read_csv("region_stats.csv"), on="ocean_proximity", how="left")
The result would be to efficiently connect housing data with neighborhood-level metadata through multi-threaded Polars connections that maintain performance even on larger datasets.
# 10. Performing rolling calculations
For highly volatile data, rolling aggregates are useful for smoothing, for example, average house values at different latitudes and longitudes. This one-line illustration illustrates how to apply such a brisk, vectorized operation: ideal for temporal or geographic sequences.
df = df.sort("longitude").with_columns(pl.col("median_house_value").rolling_mean(window_size=7).alias("rolling_value_avg"))
# Summary
In this article, we have listed 10 useful, straightforward tips for efficiently using Polar as a brisk alternative to Pandas for handling substantial data. These one-line strategies include brisk, optimized strategies for handling immense volumes of data in less time. Take advantage of them the next time you work with Polars on your projects and you will undoubtedly notice a number of improvements.
Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the employ of artificial intelligence in the real world.