Skip to content
LearnPY
pandas

Clean messy data deliberately

Inspect missing values, remove duplicates, normalize text, and convert columns to the types your analysis actually needs.

12 min readbeginnerpandas

By the end

  • Measure missing values and duplicate rows before changing anything.
  • Choose between filling and dropping missing data.
  • Normalize text and convert numeric columns safely.
  • Remove duplicate keys while keeping the row you intend to keep.
  • Verify the cleaned table instead of assuming the operations worked.

Inspect before you clean

Cleaning is a series of decisions, not a single method call. Start by measuring what is wrong: which columns are missing values, which keys repeat, and which columns arrived as text when they should be numbers.

Python

Three questions before changing the table

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Fill, drop, or keep

dropna removes rows with missing values. fillna substitutes a value or a per-column mapping. Neither is universally correct: dropping a mostly empty column can be right, while filling a missing count with zero may be a meaningful business rule.

Python

Different gaps, different decisions

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

  • Use dropna(subset=[...]) when the row cannot be analyzed without that column.
  • Use fillna when a known default is legitimate, such as a missing quantity meaning zero units.
  • Inspect how many rows each choice affects before accepting the result.

Fix types and normalize text

Numbers stored as text cannot be summed, and names with inconsistent whitespace create duplicate-looking groups. Normalize the representation before analyzing the values.

Python

Strip, parse, and fill

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

errors="coerce" turns an unparseable value into NaN instead of raising. That is useful for an audit, but it also means you must count the new missing values and decide what they represent.

Remove duplicate keys carefully

duplicated marks rows whose key has already appeared. drop_duplicates removes them. The keep argument decides whether the first row, the last row, or no rows survive when a key repeats.

Python

One row per order key

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Before removing a duplicate, ask why it exists. If two rows disagree on an important value, silently keeping the first one may discard the very problem you need to investigate.

A cleaning recipe

A reliable pass is: inspect, normalize text, parse types, handle duplicates, fill or drop with a reason, then assert the expected row count and missing-value count. The final assertions are part of the analysis.

Python

Clean and verify in one pass

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

The next lesson treats dates as a special kind of cleaning problem: they must be parsed, ordered, and grouped on a calendar rather than as ordinary strings.

Practice

Make it stick

Run the tests as often as you like. Hints and the solution are available when you need them.

Exercise

Clean the orders table: keep the first row for each order_id, strip whitespace from customer, convert amount to numbers (invalid values become missing), and fill missing quantities with 0 as integers. Assign the result to clean.

Your solution

Ready
Loading editor…

Results

Run the tests to check your solution. The first run downloads Python; later runs reuse the session.

Quiz

Check your understanding

01What does raw.duplicated(subset="order_id").sum() measure?
02What does pd.to_numeric(..., errors="coerce") do with an invalid string?
03When is filling a missing quantity with 0 defensible?

Lesson progress