# Entry
One sec data pre-processing is critical in data science and machine learning workflows, these processes are often not performed correctly, mainly because they are perceived as too sophisticated, time-consuming or require extensive custom code. As a result, practitioners may delay critical tasks such as data cleansing, rely on breakable ad hoc solutions that are unsustainable in the long run, or overwork solutions to problems that may be straightforward at their core.
This article shares 8 Python tricks that let you turn raw, messy data into spotless, neatly processed data with minimal effort.
Before we look at specific tricks and their accompanying code examples, the following preamble code sets up the necessary libraries and defines a toy dataset to illustrate each trick:
import pandas as pd
import numpy as np
# A diminutive, intentionally messy dataset
df = pd.DataFrame({
" User Name ": [" Alice ", "bob", "Bob", "alice", None],
"Age": ["25", "30", "?", "120", "28"],
"Income$": ["50000", "60000", None, "1000000", "55000"],
"Join Date": ["2023-01-01", "01/15/2023", "not a date", None, "2023-02-01"],
"City": ["New York", "new york ", "NYC", "New York", "nyc"],
})# 1. Instant normalization of column names
This is a very useful one-line trick: in one line of code, it normalizes the names of all the columns in the dataset. The details depend on how thoroughly you want to normalize attribute names, but the following example shows how to replace whitespace with underscores and all lowercase letters, thus ensuring a consistent, standardized naming convention. This is critical to prevent annoying errors in subsequent tasks or to fix possible typos. There is no need to iterate column by column!
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")# 2. Removing whitespace from strings on a immense scale
Sometimes you may just want to ensure that certain garbage that is hidden to the human eye, such as white space at the beginning or end of (categorical) string values, is systematically removed from the entire data set. This strategy neatly does this for all columns containing strings, leaving other columns, such as numeric ones, unchanged.
df = df.apply(lambda s: s.str.strip() if s.dtype == "object" else s)# 3. Secure conversion of numeric columns
If we’re not 100% sure that all the values in a numeric column have an identical format, it’s generally a good idea to explicitly convert those values to a numeric format, turning sometimes messy strings that look like numbers into actual numbers. In one line we can do what would otherwise require trying, except with blocks and a more manual cleaning procedure.
df["age"] = pd.to_numeric(df["age"], errors="coerce")
df["income$"] = pd.to_numeric(df["income$"], errors="coerce")It should be noted that other classic approaches such as df['columna'].astype(float) may sometimes crash if invalid raw values are found that cannot be easily converted to numeric values.
# 4. Analyzing dates with errors="coerce"
Similar validation-oriented procedure, separate data type. This trick converts valid date and time values, invalidating those that are not. Using errors="coerce" is the key to saying Pandas that if invalid, non-convertible values are found, they should be converted NaT (More than once), instead of generating an error and crashing the program during execution.
df["join_date"] = pd.to_datetime(df["join_date"], errors="coerce")# 5. Fix missing values with astute defaults
For those who are not familiar with strategies for handling missing values other than deleting entire rows containing them, this strategy assigns these values - fills in the gaps – using statistically based defaults such as median Or mode. A powerful single-line strategy that can be customized with a variety of default aggregates. The [0] the index accompanying this mode is used to obtain only one value in the case of associations between two or more “most common values”.
df["age"] = df["age"].fillna(df["age"].median())
df["city"] = df["city"].fillna(df["city"].mode()[0])# 6. Standardization of categories using a map
In categorical columns with different values, such as cities, it is also necessary to standardize the names and eliminate possible inconsistencies in order to obtain cleaner group names and create further group aggregations, such as groupby() reliable and effective. This dictionary-aided example applies a one-to-one mapping to string values related to Modern York, ensuring that all of them are uniformly labeled “NYC”.
city_map = {"new york": "NYC", "nyc": "NYC"}
df["city"] = df["city"].str.lower().map(city_map).fillna(df["city"])# 7. Astute and malleable removal of duplicates
The key to this highly customizable deduplication strategy is usage subset=["user_name"]. In this example it is used to tell Panda to consider a row as duplicate just by looking at it "user_name" column and check if the value in the column is identical to the value in another row. A great way to ensure that each unique user is only represented once in the dataset, preventing double counting and doing it all in one statement.
df = df.drop_duplicates(subset=["user_name"])# 8. Trimming quantiles to remove outliers
The final trick is to automatically reduce extreme values or outliers rather than removing them completely. This is particularly useful when it is assumed that outliers are due to, for example, manually entered errors in the data. Truncation sets extreme values that fall below (and above) two percentiles (1 and 99 in the example), at such percentile values, while keeping the original values lying between the two specified percentiles unchanged. Simply put, it’s like keeping values too immense or too miniature within certain limits.
q_low, q_high = df["income$"].quantile([0.01, 0.99])
df["income$"] = df["income$"].clip(q_low, q_high)# Summary
This article shares eight useful tricks, tips, and strategies that will streamline your Python data preprocessing pipelines, making them more competent, effective, and resilient: all at the same time.
Ivan Palomares Carrascosa is a thought leader, writer, speaker and advisor in the fields of Artificial Intelligence, Machine Learning, Deep Learning and LLM. Trains and advises others on the apply of artificial intelligence in the real world.