Photo by the editor
# Entry
Feature engineering it is an indispensable process in data analytics and machine learning workflows, as well as in any AI system as a whole. This entails constructing meaningful explanatory variables from raw – and often quite unstructured – data. The processes behind feature engineering can be extremely basic or overly convoluted, depending on the volume, structure, and heterogeneity of the datasets, as well as the goals of the machine learning modeling. Although the most popular Python libraries for data manipulation and modeling, such as Pandas AND scikit-learnthey enable some basic and moderately scalable feature engineering, there are specialized libraries that do their best to deal with huge datasets and automate convoluted transformations, but they are largely unknown to many.
This article lists 7 hidden Python libraries that push the boundaries of large-scale feature engineering processes.
# 1. Speeding up with NVTabular
First, we have NVIDIA-Merlin NVTable: a library designed to apply pre-processing and feature engineering to datasets that are – yes, you guessed it! — tabular. Its distinctive feature is its GPU-accelerated approach, designed to easily manipulate the very large-scale datasets needed to train large-scale deep learning models. The library is specifically designed to facilitate scaling pipelines for contemporary deep neural network (DNN) based recommendation engines.
# 2. Automation with FeatureTools
This code snippet shows an example of using DFS with featuretools the library looks like on the customer dataset:
customers_df = pd.DataFrame({'customer_id': [101, 102]})
es = es.add_dataframe(
dataframe_name="customers",
dataframe=customers_df,
index="customer_id"
)
es = es.add_relationship(
parent_dataframe_name="customers",
parent_column_name="customer_id",
child_dataframe_name="transactions",
child_column_name="customer_id"
)# 3. In parallel with Dask
dask is gaining popularity as a library that enables faster and simpler parallel computations in Python. The main recipe for Dask is to scale time-honored Pandas and scikit-learn based feature transformations through cluster-based computation, thus facilitating faster and low-cost feature engineering pipelines on huge datasets that would otherwise exhaust memory.
This article provides a practical Dask guide to data preprocessing.
# 4. Optimize with Polars
Competing with Dask in terms of growing popularity and with Pandas aspiring to the Python podium, we have polar: a Rust-based data frame library that leverages the slothful expression API and slothful computation to provide competent, scalable feature engineering and transformations on very huge datasets. Considered by many to be the competent equivalent of Pandas, Polars is very effortless to learn and get familiar with if you know Pandas well.
Want to learn more about fleeces? This article presents several practical, single-sided solutions from Polars for common data science tasks, including feature engineering.
# 5. Storage with the Holiday
Holiday is an open source library intended as a feature store to support provide structured data sources for production-level or production-ready large-scale AI applications, especially those based on Vast Language Models (LLM), for both model training and inference tasks. One of its attractive properties is that it provides consistency between both stages: training and inference in production. Its exploit as a feature store has been closely linked to feature engineering processes, namely by using it in conjunction with other open source frameworks, for example denormalized.
# 6. Extraction with tsfresh
features_rolled = extract_features(
rolled_df,
column_id='id',
column_sort="time",
default_fc_parameters=settings,
n_jobs=0
)# 7. Improvement using the river
Let’s finish dipping our toes into the river (pun intended) with: River library, designed to streamline online machine learning workflows. As part of the feature suite, it enables online or streaming feature transformation and feature learning techniques. This can support you effectively deal with issues such as unconstrained data and concept drift in production. River is built to reliably deal with problems rarely encountered in batch machine learning systems, such as the appearance and disappearance of data features over time.
# Summary
This article lists 7 noteworthy Python libraries that can support you raise the scalability of your feature engineering processes. Some of them focus directly on providing distinctive feature engineering approaches, while others can be used to further support feature engineering tasks in specific scenarios, in combination with other frameworks.
Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the exploit of artificial intelligence in the real world.