Skip to content
LearnPY
scikit-learn

Evaluate with validation

Go beyond one train/test split with baselines, cross-validation, and honest comparisons that separate signal from noise.

13 min readbeginnerscikit-learn, numpy

By the end

  • Compare a model with a simple baseline.
  • Use cross-validation to estimate variability.
  • Read a train-versus-validation gap.
  • Use negative-error scoring correctly.
  • Keep test data out of model selection.

Earn the right to be complicated

A baseline is the simplest reasonable prediction. For regression, predicting the training mean is a strong starting point. If a complex model cannot beat that baseline on held-out data, the complexity has not earned its place.

Python

A mean predictor is the floor to beat

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Average over several folds

A single split can be lucky or unlucky. Cross-validation divides the training rows into folds, fits on the other folds, and scores the held-out fold. The mean estimates performance and the spread shows how stable that estimate is.

Python

Five estimates from the same training set

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

cross_val_score refits the entire pipeline in every fold, including the scaler. That is why preprocessing belongs inside the pipeline: it prevents each validation fold from seeing its own holdout statistics.

Mind the sign of error scores

scikit-learn follows the convention that higher is better. Error metrics such as MAE are therefore exposed as negative scores. Select with scoring="neg_mean_absolute_error" and negate the result for interpretation.

Python

Negate negative-error scores before reporting

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Compare training and validation

A large gap between training and validation performance suggests overfitting. cross_validate can return both scores, which is more informative than a single validation number.

Python

Separate memorization from generalization

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

A small gap is healthy. A huge gap with strong training scores means the model is memorizing. A low score in both places usually means the features or model are too weak.

Protect the final test set

Use training data for exploration and cross-validation. Compare models, tune hyperparameters, and choose a preprocessing recipe there. Touch the test set once, after the decisions are made, to estimate final performance.

Python

Choose with validation, report once on test

Ready
Loading editor…

Output

Press Run to execute this cell. The first run downloads Python; later runs reuse the session.

Practice

Make it stick

Run the tests as often as you like. Hints and the solution are available when you need them.

Exercise

Fit DummyRegressor(strategy="mean") on the training rows and store its test MAE in baseline_mae. Build a StandardScaler plus Ridge(alpha=1.0) pipeline, then use 5-fold cross-validation on the training rows with scoring="neg_mean_absolute_error"; store the fold scores in cv_scores and their negated mean in cv_mae.

Your solution

Ready
Loading editor…

Results

Run the tests to check your solution. The first run downloads Python; later runs reuse the session.

Quiz

Check your understanding

01Why compare a model with a mean-prediction baseline?
02Why is neg_mean_absolute_error negative?
03When should the final test set be used?

Lesson progress