What is Data Cleaning?

S7-SA3-0394

Grade Level:

Class 12

AI/ML, Physics, Biotechnology, FinTech, EVs, Space Technology, Climate Science, Blockchain, Medicine, Engineering, Law, Economics

Definition

What is it?

Data cleaning is the process of finding and fixing errors, inconsistencies, and missing values in a dataset. It makes sure the data is accurate, complete, and ready for analysis, just like cleaning your room before guests arrive.

Simple Example

Quick Example

Imagine you have a list of your friends' favourite ice cream flavours. If one friend's entry says 'Choclate' instead of 'Chocolate', or another's entry is simply blank, that's 'dirty data'. Data cleaning would involve correcting 'Choclate' to 'Chocolate' and asking the friend for their missing flavour.

Worked Example

Step-by-Step

Let's clean a small dataset of student marks (out of 100) from a science test:

Original Data: [85, 92, 'ninety', 78, -5, 105, 60, '', 90]

---Step 1: Identify incorrect data types. 'ninety' is text, not a number.
Correction: Replace 'ninety' with its numerical value, 90.
Data now: [85, 92, 90, 78, -5, 105, 60, '', 90]

---Step 2: Identify out-of-range values. Marks cannot be negative or above 100.
Correction: Replace -5 (negative mark) with 0 (assuming minimum mark). Replace 105 (above 100) with 100 (assuming maximum mark).
Data now: [85, 92, 90, 78, 0, 100, 60, '', 90]

---Step 3: Handle missing values. The empty string '' is a missing mark.
Correction: Decide how to handle it. For this example, let's replace it with the average of the other marks (mean imputation). Average of [85, 92, 90, 78, 0, 100, 60, 90] is (85+92+90+78+0+100+60+90)/8 = 595/8 = 74.375. Let's round to 74.
Data now: [85, 92, 90, 78, 0, 100, 60, 74, 90]

---Step 4: Review and verify. All values are now numbers, within the valid range (0-100), and there are no missing entries.

Cleaned Data: [85, 92, 90, 78, 0, 100, 60, 74, 90]

Why It Matters

Clean data is crucial for making good decisions, whether in AI/ML models predicting weather or in FinTech apps managing your money. Data scientists, engineers, and even doctors rely on clean data to ensure their work is accurate and reliable, helping us build better technologies and solve real-world problems.

Common Mistakes

MISTAKE: Ignoring missing values or replacing them with a random number. | CORRECTION: Understand why data is missing (e.g., not recorded, error) and choose an appropriate method like removing the entry, replacing with average/median, or predicting the value.

MISTAKE: Not checking for inconsistent data formats (e.g., 'Delhi', 'delhi', 'New Delhi' for the same city). | CORRECTION: Standardize data formats using techniques like converting all text to lowercase, trimming extra spaces, and mapping similar entries to a single standard value.

MISTAKE: Correcting all 'outlier' data points without understanding their context. | CORRECTION: First, investigate if an outlier is a genuine extreme value (e.g., a record-breaking cricket score) or a data entry error. Only correct it if it's an error; otherwise, keep it.

Practice Questions

Try It Yourself

QUESTION: You have a list of phone numbers: ['9876543210', '876543210', '9988776655', 'abcde']. Which entry is likely 'dirty' and why? | ANSWER: '876543210' (it's only 9 digits, Indian numbers are usually 10) and 'abcde' (it's not a number).

QUESTION: A dataset of student ages has entries: [15, 16, 14, 150, 13, 16]. What is the most obvious error and how would you likely fix it? | ANSWER: The entry '150' is an obvious error (a student cannot be 150 years old). You would likely replace it with a reasonable value like the average age, or mark it as missing if the correct age is unknown.

QUESTION: You are cleaning a dataset of product prices. One entry is '₹500', another is '500 INR', and a third is 'Five Hundred Rupees'. How would you make these consistent? | ANSWER: Convert all entries to a standard numerical format, for example, just '500'. This involves removing currency symbols (₹, INR), converting text to numbers ('Five Hundred Rupees' to 500), and ensuring all are numbers.

MCQ

Quick Quiz

Which of the following is NOT a common task in data cleaning?

Handling missing values

Correcting inconsistent data formats

Finding and fixing spelling mistakes

Creating visual graphs from the data

The Correct Answer Is:

Data cleaning focuses on improving the quality of raw data by fixing errors and inconsistencies. Creating visual graphs (data visualization) is typically done AFTER data cleaning, once the data is reliable.

Real World Connection

In the Real World

Imagine an app like Zomato or Swiggy. When you search for restaurants, they rely on clean data. If a restaurant's location is entered incorrectly, or its menu prices have typos, it would lead to a bad user experience. Data scientists at these companies constantly clean data to ensure you see accurate restaurant details and prices, making your food ordering smooth.

Key Vocabulary

Key Terms

MISSING DATA: Information that is not recorded or is absent from a dataset. | INCONSISTENT DATA: Data that is not uniform in format or value (e.g., 'Mumbai' vs 'Bombay'). | OUTLIER: A data point that is significantly different from other observations. | DATA VALIDATION: Checking if data meets certain rules or standards.

What's Next

What to Learn Next

Now that you understand data cleaning, you're ready to explore 'Data Preprocessing'. This next step involves transforming your clean data into a format suitable for machine learning models, which is essential for building smart AI applications.