Data Cleaning
Also called Data Cleansing, Data Scrubbing
The process of finding and fixing or removing wrong, duplicate or incomplete records in a dataset before using it.
What is data cleaning?
Data cleaning is the work of identifying and correcting or removing corrupt, inaccurate or irrelevant records in a dataset. It is a standard first step before training a model or running an analysis.
How does it work?
Typical tasks include removing duplicate records, deciding how to handle missing values, fixing typing errors and wrong entries, and making formats consistent, such as using one date format or one unit of measurement. Statistical checks can flag values that look out of range.
What is a simple example?
A customer list contains "Lagos", "lagos" and "Lagos " as three different cities, plus two copies of the same person. Cleaning merges the spellings and removes the duplicate before the data is used to train a model.
Why does it matter?
A model learns whatever its training data contains, including the mistakes. Cleaner data usually means more reliable results, and careful sourcing also helps guard against data poisoning. Adding tags to examples is a separate step, called data labeling or data annotation.