VerdictPal
Join
GuidesPlaybook

Data cleaning pipeline

A documented, reproducible data cleaning pipeline from raw to analysis-ready.

Intermediate
Start now
Playbook

Data cleaning pipeline

A documented, reproducible data cleaning pipeline from raw to analysis-ready.

Steps5Time~65 minArtifactCleaning notebookLevelintermediate
Before you start
  • A messy dataset ready for cleaning
  • Basic Python or R familiarity
Tools in this guide
Claude
0%0/5 steps

Data researcher · 60–90 minutes

Most data analysis time is cleaning — but nobody documents it.

This runbook makes every cleaning step explicit, documented, and reversible.

Cleaning notebook

Cleaning notebook

A Jupyter notebook with profiled data, documented transformations, and a before/after comparison.

Profile the raw data

10 min

Run shape, dtypes, missing value counts, and basic statistics. Document everything before touching the data.

Data profile summary with column types and missing counts.
Pitfall
Skipping the profile — you'll miss silent type mismatches and encoding issues.
Verification
The profile shows every column's type, non-null count, and sample values.

Handle missing values

20 minClaude

For each column with missing values: document the pattern, choose a strategy (drop, impute, flag), and justify the choice.

Missing value strategy table per column.
Pitfall
Dropping all rows with any missing value — you may lose 50% of your data silently.
Verification
Each imputation choice has a justification tied to the missing mechanism (MCAR, MAR, MNAR).

Standardize formats

15 min

Normalize date formats, string casing, numeric units, and categorical labels across all columns.

Consistent format documentation.
Pitfall
Assuming dates are in the same format across columns — they never are.
Verification
All date columns parse to the same datetime format. All strings are consistently cased.

Validate the cleaned data

10 minClaude

Re-run the profile on cleaned data and compare with the original. Verify no unexpected rows were lost.

Before/after comparison with row counts and value distributions.
Pitfall
Not comparing before/after — cleaning bugs often delete valid rows silently.
Verification
Row count change is documented and justified. Value distributions are plausible.

Document the pipeline

10 min

Write a summary cell listing every transformation applied, in order, with the rationale for each.

Cleaning pipeline documentation.
Pitfall
Leaving the notebook undocumented — six months from now you won't remember the order.
Verification
A colleague could run the notebook top-to-bottom and reproduce the cleaned dataset.
Finish line

You should now have: Cleaning notebook.

  • The data profile is documented before and after cleaning.
  • Every missing value has a documented strategy and justification.
  • The cleaned data validates against the expected schema.

PrivacyDo not upload raw datasets with PII to cloud tools. Anonymize or sample before using AI assistants for cleaning code review.