Skip to content
LearnPY
scikit-learn

Choose the right classification metric

Read a confusion matrix, balance precision and recall, and use ranking metrics when the threshold is not fixed.

14 min readbeginnerscikit-learn, numpy, matplotlib

By the end

  • Read true positives, false positives, and false negatives.
  • Compute precision, recall, and F1.
  • Use ROC-AUC to evaluate ranking.
  • Relate threshold changes to error types.
  • Choose a metric from the decision, not from habit.

Start with the four outcomes

For binary classification, every prediction falls into one of four cells: true negative, false positive, false negative, or true positive. The confusion matrix keeps those counts separate, which accuracy collapses into one number.

Python

The four cells behind one accuracy number

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Bug hunt

This recall calculation uses the wrong denominator. Fix it so recall matches scikit-learn's recall_score.

Your solution

Ready
Loading editor…

Results

Run the tests to check your solution. The first run downloads Python; later runs reuse the session.

Precision and recall answer different questions

Precision asks: of the rows predicted positive, how many were positive? Recall asks: of the actual positives, how many did we find? Improving one often costs the other when the threshold moves.

Python

Two views of the positive-class errors

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

  • High precision means few false alarms.
  • High recall means few missed positives.
  • A spam filter and a cancer screen usually need different balances.

Summarize with F1, carefully

F1 is the harmonic mean of precision and recall. It drops when either side is poor and is useful when both errors matter similarly. It is not a universal score: it ignores true negatives and assumes precision and recall deserve equal weight.

Python

Per-class precision, recall, and F1

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

classification_report is a useful first look, but still report the confusion matrix when the cost of each error matters. A compact score cannot carry the full decision context.

Evaluate ranking with ROC-AUC

ROC-AUC measures how well the model ranks a random positive above a random negative across all thresholds. It is threshold-independent, so it is useful before committing to a cutoff. It does not tell you the precision at the operating point you will deploy.

Python

Ranking quality across all thresholds

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Connect the threshold to the errors

As the threshold falls, more rows become positive: recall tends to rise and precision tends to fall. Plot the tradeoff on validation data, choose an operating point that matches the cost of errors, then report the chosen threshold.

Python

The threshold is an explicit operating decision

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Choose the threshold on validation folds or a dedicated validation split. If you tune it on the test set, the final reported metric is no longer an honest estimate.

Practice

Make it stick

Run the tests as often as you like. Hints and the solution are available when you need them.

Exercise

Fit the scaled logistic regression pipeline on the training rows. Store the confusion matrix in matrix, unpack tn, fp, fn, tp, and compute precision, recall, F1, and ROC-AUC from the positive-class probabilities.

Your solution

Ready
Loading editor…

Results

Run the tests to check your solution. The first run downloads Python; later runs reuse the session.

Quiz

Check your understanding

01What question does precision answer?
02What question does recall answer?
03Why is ROC-AUC called threshold-independent?

Lesson progress