Processing gigantic data with Dask and Scikit-learn

Photo by the editor

# Entry

dask is a set of packages that take advantage of parallel computing capabilities – extremely useful when handling immense data sets or building proficient data-intensive applications, such as advanced analytics and machine learning systems. Its most essential advantages include polished integration of Dask with existing Python frameworks, including support for processing immense data sets scikit-learn modules through parallel workflows. In this article, you’ll learn how to operate Dask for scalable data processing, even with restricted hardware constraints.

# Step-by-step walkthrough

While not particularly massive, it is California Housing Dataset is quite immense, making it an excellent choice for a gentle, illustrative coding example that demonstrates how to operate Dask and scikit-learn together for large-scale data processing.

Dask assures dataframe a module that mimics many aspects of Pandas DataFrame objects for handling immense data sets that may not all fit in memory. We will operate this Dask DataFrame structure to load our data from the CSV file in the GitHub repository as follows:

import dask.dataframe as dd

url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/refs/heads/main/housing.csv"
df = dd.read_csv(url)

df.head()

An essential note here. If you want to see the “shape” of a data set – the number of rows and columns – the method is a bit more arduous than the usual operate df.shape. Instead you should do something like this:

num_rows = df.shape[0].compute()
num_cols = df.shape[1]
print(f"Number of rows: {num_rows}")
print(f"Number of columns: {num_cols}")

Exit:

Number of rows: 20640
Number of columns: 10

Note that we used Dask compute() lazily calculate the number of rows but not the number of columns. Data set metadata allows us to immediately obtain the number of columns (features), while determining the number of rows in a data set that may (hypothetically) be larger than memory – and therefore partitioned – requires distributed computation: something that compute() is handled transparently for us.

Data preprocessing is most often the previous step to building a machine learning model or estimator. Before we get to this part, and since the main purpose of this hands-on article is to show how Dask can be used for data processing, let’s spotless and prepare it.

One of the common steps in data preparation is dealing with missing values. With Dask, the process is as polished as if we were using Pandas. For example, the following code removes rows for instances that contain missing values ​​in any of their attributes:

df = df.dropna()

num_rows = df.shape[0].compute()
num_cols = df.shape[1]
print(f"Number of rows: {num_rows}")
print(f"Number of columns: {num_cols}")

Now the dataset has been reduced by over 200 instances, with a total of 20,433 rows.

We can then scale some of the numerical features in the dataset by enabling scikit-learn StandardScaler or other appropriate scaling method: :

from sklearn.preprocessing import StandardScaler

numeric_df = df.select_dtypes(include=["number"])
X_pd = numeric_df.drop("median_house_value", axis=1).compute()

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_pd)

Importantly, note that for sequences of data-intensive operations we perform in Dask, such as deleting rows containing missing values ​​and then deleting the target column "median_house_value"we have to add compute() at the end of a sequence of linked operations. That’s why data set transformations in Dask are performed lazily. Once compute() is called, the result of the chain transformation of the dataset materializes as Panda DataFrame (Dask depends on Pandas, so you don’t need to explicitly import the Pandas library into your code unless you’re calling a Pandas-exclusive function directly.)

What if we want to? train a machine learning model? Then we should extract the target variable "median_house_value" and apply the same principle to convert it to a Pandas object:

y = df["median_house_value"]
y_pd = y.compute()

From now on, the process of dividing the data set into training and test sets trains the regression model RandomForestRegressorand evaluate its error on test data, fully resembles the established approach using Pandas and learning scikit in a structured way. Since tree-based models are insensitive to feature scaling, either unscaled features can be used (X_pd) or scaled (X_scaled). Below we proceed with the scaled features calculated above:

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

# Utilize the scaled feature matrix produced earlier
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_pd, test_size=0.2, random_state=42)

model = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
print(f"RMSE: {rmse:.2f}")

Exit:

# Summary

Dask and scikit-learn can be used together to leverage scalable, parallel data processing workflows, for example to efficiently pre-process immense data sets for building machine learning models. This article shows you how to load, spotless, prepare, and transform data with Dask, and then apply scikit-learn’s standard machine learning modeling tools – all while optimizing memory usage and accelerating your pipeline for massive datasets.

Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the operate of artificial intelligence in the real world.