Real data is never perfect. Sensors malfunction, survey respondents skip questions, spreadsheets get corrupted. Before you train any AI model, you must deal with missing values — or your model will either crash or produce wrong predictions.
This guide covers the complete missing data workflow required in Class 11, Unit 5: Data Literacy — Data Pre-processing and Class 12, Unit 1: Python Programming – II of the CBSE AI syllabus (Subject Code 843, 2025-26).
What You’ll Learn
- What missing data is and why it breaks AI models
- How to detect missing values with Pandas
- Three strategies: fill, drop, or replace — when to use each
- Complete programs ready for your practical file and Lab Test
Why Missing Data Matters in AI
Every machine learning algorithm expects complete, clean data. When values are missing, algorithms either crash with an error or silently produce wrong results — both outcomes are worse than taking 10 minutes to clean the data first.
In India’s agricultural data collected by state governments, district-level crop yield records routinely have missing entries for remote areas. A predictive model trained on that data without handling missing values would produce unreliable yield forecasts — the same problem at a school scale is a failing practical program.
Missing values in Python are represented as NaN — Not a Number. Pandas uses NaN for any missing numeric or text value.
Part 1 — Detecting Missing Values
Before fixing missing data, you must find it.
python
# Program to detect missing values in a dataset
import pandas as pd
import numpy as np
# Create a dataset with deliberate missing values
data = {
"Student" : ["Arjun", "Priya", "Kiran", "Meena", "Rohan", "Sneha"],
"Maths" : [85, np.nan, 78, 95, np.nan, 88],
"Science" : [90, 82, np.nan, 88, 74, np.nan],
"AI" : [92, 95, 88, np.nan, 79, 91],
"City" : ["Delhi", "Mumbai", np.nan, "Chennai", "Delhi", "Mumbai"]
}
df = pd.DataFrame(data)
print("Dataset:")
print(df)
print("\n--- Missing Value Analysis ---")
# Count missing values per column
print("\nMissing values per column:")
print(df.isnull().sum())
# Total missing values across entire DataFrame
print("\nTotal missing values:", df.isnull().sum().sum())
# Percentage missing per column
print("\nPercentage missing per column:")
print((df.isnull().sum() / len(df) * 100).round(1))
# Which rows have at least one missing value
print("\nRows with missing values:")
print(df[df.isnull().any(axis=1)])Expected Output:
Dataset:
Student Maths Science AI City
0 Arjun 85.0 90.0 92.0 Delhi
1 Priya NaN 82.0 95.0 Mumbai
2 Kiran 78.0 NaN 88.0 None
3 Meena 95.0 88.0 NaN Chennai
4 Rohan NaN 74.0 79.0 Delhi
5 Sneha 88.0 NaN 91.0 Mumbai
--- Missing Value Analysis ---
Missing values per column:
Student 0
Maths 2
Science 2
AI 1
City 1
dtype: int64
Total missing values: 6
Percentage missing per column:
Student 0.0
Maths 33.3
Science 33.3
AI 16.7
City 16.7
dtype: float64
Rows with missing values:
Student Maths Science AI City
1 Priya NaN 82.0 95.0 Mumbai
2 Kiran 78.0 NaN 88.0 None
3 Meena 95.0 88.0 NaN Chennai
4 Rohan NaN 74.0 79.0 Delhi
5 Sneha 88.0 NaN 91.0 MumbaiKey methods for detection:
| Method | What It Returns |
|---|---|
df.isnull() | DataFrame of True/False — True where value is missing |
df.isnull().sum() | Count of missing values per column |
df.isnull().sum().sum() | Total missing values in entire DataFrame |
df.info() | Shows non-null counts per column |
df.isnull().any(axis=1) | True for rows that have at least one missing value |
Part 2 — Strategy 1: Fill Missing Values (fillna)
fillna() replaces missing values with something — without losing any rows.
Fill with Column Mean (Most Common for Numeric Data)
python
# Program to fill missing numeric values with column mean
import pandas as pd
import numpy as np
data = {
"Student": ["Arjun","Priya","Kiran","Meena","Rohan"],
"Maths" : [85, np.nan, 78, 95, np.nan],
"Science": [90, 82, np.nan, 88, 74]
}
df = pd.DataFrame(data)
print("Before filling:")
print(df)
print("\nMissing:", df.isnull().sum().sum())
# Fill with column mean
df["Maths"].fillna(df["Maths"].mean(), inplace=True)
df["Science"].fillna(df["Science"].mean(), inplace=True)
print("\nAfter filling with mean:")
print(df.round(2))
print("Missing:", df.isnull().sum().sum())Expected Output:
Before filling:
Student Maths Science
0 Arjun 85.0 90.0
1 Priya NaN 82.0
2 Kiran 78.0 NaN
3 Meena 95.0 88.0
4 Rohan NaN 74.0
Missing: 3
After filling with mean:
Student Maths Science
0 Arjun 85.00 90.0
1 Priya 86.00 82.0
2 Kiran 78.00 83.5
3 Meena 95.00 88.0
4 Rohan 86.00 74.0
Missing: 0Why mean? The mean preserves the overall average of the column — adding a mean value doesn’t shift the centre of your data.
Fill with Median (Better When Data Has Outliers)
python
# Fill with median — better when data has extreme values
import pandas as pd
import numpy as np
salaries = pd.Series([25000, 28000, np.nan, 30000, 500000, np.nan, 27000])
print("Mean :", salaries.mean().round(0)) # pulled up by 500000
print("Median:", salaries.median()) # not affected by 500000
# Fill with median to avoid inflating missing values
salaries_filled = salaries.fillna(salaries.median())
print("\nAfter filling with median:")
print(salaries_filled)Expected Output:
Mean : 122000.0
Median: 28000.0
After filling with median:
0 25000.0
1 28000.0
2 28000.0
3 30000.0
4 500000.0
5 28000.0
6 27000.0Mean vs Median for missing data — when to choose:
| Situation | Use |
|---|---|
| Data is roughly symmetric (most marks, heights) | Mean |
| Data has outliers (salaries, house prices) | Median |
| Categorical column (city names, grades) | Mode or placeholder |
Fill Categorical/Text Columns
python
# Fill text column with mode (most common value) or placeholder
import pandas as pd
import numpy as np
data = {"City": ["Delhi", "Mumbai", np.nan, "Delhi", np.nan, "Chennai"]}
df = pd.DataFrame(data)
# Option A: fill with most common value (mode)
mode_city = df["City"].mode()[0]
df["City"].fillna(mode_city, inplace=True)
print("Filled with mode:", df)Expected Output:
Filled with mode:
City
0 Delhi
1 Mumbai
2 Delhi ← filled with "Delhi" (mode)
3 Delhi
4 Delhi ← filled with "Delhi" (mode)
5 ChennaiPart 3 — Strategy 2: Drop Missing Values (dropna)
dropna() removes rows (or columns) that contain missing values.
python
# Program to demonstrate dropna() with different options
import pandas as pd
import numpy as np
data = {
"Name" : ["Arjun","Priya","Kiran","Meena","Rohan"],
"Maths" : [85, np.nan, 78, 95, np.nan],
"Science" : [90, 82, np.nan, 88, 74],
"AI" : [92, 95, 88, np.nan, 79]
}
df = pd.DataFrame(data)
print("Original shape:", df.shape)
# Drop rows where ANY column has a missing value
df_any = df.dropna()
print("After dropna() — any missing:", df_any.shape)
# Drop rows where ALL columns are missing (rare, more conservative)
df_all = df.dropna(how="all")
print("After dropna(how='all') :", df_all.shape)
# Drop rows only where a specific column is missing
df_maths = df.dropna(subset=["Maths"])
print("After dropna(subset=['Maths']):", df_maths.shape)Expected Output:
Original shape: (5, 4)
After dropna() — any missing: (1, 4)
After dropna(how='all') : (5, 4)
After dropna(subset=['Maths']): (3, 4)Part 4 — Choosing the Right Strategy
This is the decision that most students skip — and it is the most important one.
| Situation | Best Strategy | Reason |
|---|---|---|
| Missing values are few (< 5% of rows) | dropna() | Small loss, clean data |
| Missing values are many (> 20%) | fillna() with mean/median | Can’t afford to lose rows |
| Column is numeric, symmetric distribution | fillna(mean) | Preserves average |
| Column is numeric, has outliers | fillna(median) | Median is robust to extremes |
| Column is text or category | fillna(mode) or "Unknown" | Mean/median don’t apply to text |
| Row is missing most of its values | dropna() for that row | Row adds no useful information |
| Column is missing more than 50% | Drop the column | Column is unreliable |
python
# Program: complete missing data decision workflow
import pandas as pd
import numpy as np
df = pd.read_csv("data.csv") # Replace with your file
# Step 1: Diagnose
missing_pct = (df.isnull().sum() / len(df)) * 100
print("Missing percentage per column:")
print(missing_pct.round(1))
# Step 2: Drop columns with > 50% missing
cols_to_drop = missing_pct[missing_pct > 50].index
df.drop(columns=cols_to_drop, inplace=True)
print(f"\nDropped columns: {list(cols_to_drop)}")
# Step 3: Fill remaining numeric columns with median
numeric_cols = df.select_dtypes(include='number').columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].median())
# Step 4: Fill remaining text columns with placeholder
text_cols = df.select_dtypes(include='object').columns
df[text_cols] = df[text_cols].fillna("Unknown")
# Step 5: Verify
print("\nMissing values after cleaning:")
print(df.isnull().sum())
# Step 6: Save
df.to_csv("data_cleaned.csv", index=False)
print("\nCleaned file saved.")Part 5 — Forward Fill and Backward Fill (Time-Series Data)
These methods are used when data has a natural order — like daily temperatures or monthly rainfall. Instead of filling with a fixed value, you carry forward the last known value or fill backward from the next known value.
python
# Program to demonstrate forward fill and backward fill
import pandas as pd
import numpy as np
rainfall = pd.Series([120, np.nan, np.nan, 95, np.nan, 110, 130])
print("Original :", list(rainfall))
# Forward fill — use last known value
ffill = rainfall.ffill()
print("Forward :", list(ffill))
# Backward fill — use next known value
bfill = rainfall.bfill()
print("Backward :", list(bfill))Expected Output:
Original : [120, nan, nan, 95, nan, 110, 130]
Forward : [120, 120, 120, 95, 95, 110, 130]
Backward : [120, 95, 95, 95, 110, 110, 130]The CBSE Class 11 Unit 5 content on data preprocessing and Class 12 Unit 1 (handling missing values in DataFrames) both expect you to know fillna() and dropna() at minimum. Forward/backward fill is a useful additional technique to demonstrate in your Viva.
Quick Revision Box
| Method | What It Does |
|---|---|
df.isnull() | Returns True where values are missing |
df.isnull().sum() | Count of missing values per column |
df.isnull().sum().sum() | Total missing values in DataFrame |
df.fillna(value) | Replaces NaN with given value |
df["col"].fillna(df["col"].mean()) | Fills column NaN with column mean |
df["col"].fillna(df["col"].median()) | Fills with column median |
df["col"].fillna(df["col"].mode()[0]) | Fills with most common value |
df.dropna() | Drops all rows with any NaN |
df.dropna(how="all") | Drops rows where ALL values are NaN |
df.dropna(subset=["col"]) | Drops rows where specific column is NaN |
df.ffill() | Forward fill — use previous value |
df.bfill() | Backward fill — use next value |
inplace=True | Modifies the original DataFrame directly |
Practice Questions
Q1 (2 marks): Write Python code to read a CSV file, check for missing values in each column, and fill all numeric missing values with the column mean.
Model Answer:
python
import pandas as pd
df = pd.read_csv("data.csv")
print("Missing values:", df.isnull().sum())
numeric_cols = df.select_dtypes(include='number').columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].mean())
print("After filling:", df.isnull().sum())Q2 (MCQ): Which method removes rows from a DataFrame where a specific column has missing values?
a) df.fillna(subset=["col"]) b) df.dropna(subset=["col"]) c) df.isnull(subset=["col"]) d) df.remove(na=["col"])
Answer: b) df.dropna(subset=["col"]) — the subset parameter tells Pandas to only drop rows where the specified column(s) have missing values, leaving other rows intact.
Frequently Asked Questions
Q1: Should I always fill missing values rather than dropping them? Not always. The decision depends on how much data you can afford to lose and why values are missing. If only 2–3 rows out of 500 have missing values, dropping them is clean and harmless. If 30% of your Maths scores are missing, dropping those rows destroys too much data — fill instead. As a general rule: if missing rows are less than 5% of your dataset, dropping is fine.
Q2: Does filling missing values with the mean make my model less accurate? Filling with mean introduces a small inaccuracy — you are guessing values that were not observed. However, the alternative (dropping rows or leaving NaN) is usually worse: models that cannot handle NaN will crash, and dropping rows loses real data. Mean/median imputation is an acceptable trade-off for CBSE practicals and most real-world scenarios.
Q3: fillna() doesn’t seem to work — the NaN values are still there after I run it. This is a very common issue. By default, fillna() returns a new DataFrame without modifying the original. Either use inplace=True: df.fillna(value, inplace=True) — or save the result back: df = df.fillna(value). Both work; pick one and use it consistently.
