Inspect before you clean
Cleaning is a series of decisions, not a single method call. Start by measuring what is wrong: which columns are missing values, which keys repeat, and which columns arrived as text when they should be numbers.
Python
Three questions before changing the table
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Fill, drop, or keep
dropna removes rows with missing values. fillna substitutes a value or a per-column mapping. Neither is universally correct: dropping a mostly empty column can be right, while filling a missing count with zero may be a meaningful business rule.
Python
Different gaps, different decisions
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
- Use dropna(subset=[...]) when the row cannot be analyzed without that column.
- Use fillna when a known default is legitimate, such as a missing quantity meaning zero units.
- Inspect how many rows each choice affects before accepting the result.
Fix types and normalize text
Numbers stored as text cannot be summed, and names with inconsistent whitespace create duplicate-looking groups. Normalize the representation before analyzing the values.
Python
Strip, parse, and fill
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
errors="coerce" turns an unparseable value into NaN instead of raising. That is useful for an audit, but it also means you must count the new missing values and decide what they represent.
Remove duplicate keys carefully
duplicated marks rows whose key has already appeared. drop_duplicates removes them. The keep argument decides whether the first row, the last row, or no rows survive when a key repeats.
Python
One row per order key
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Before removing a duplicate, ask why it exists. If two rows disagree on an important value, silently keeping the first one may discard the very problem you need to investigate.
A cleaning recipe
A reliable pass is: inspect, normalize text, parse types, handle duplicates, fill or drop with a reason, then assert the expected row count and missing-value count. The final assertions are part of the analysis.
Python
Clean and verify in one pass
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
The next lesson treats dates as a special kind of cleaning problem: they must be parsed, ordered, and grouped on a calendar rather than as ordinary strings.
Practice
Make it stick
Run the tests as often as you like. Hints and the solution are available when you need them.
Exercise
Clean the orders table: keep the first row for each order_id, strip whitespace from customer, convert amount to numbers (invalid values become missing), and fill missing quantities with 0 as integers. Assign the result to clean.
Your solution
ReadyResults
Run the tests to check your solution. The first run downloads Python; later runs reuse the session.
Quiz
Check your understanding
Lesson progress