Missing values need a rule
Most scikit-learn estimators do not accept NaN. SimpleImputer replaces missing entries with a learned statistic such as the mean or median. Fit the imputer on training data, then use the same learned values on test data.
Python
Learn imputation values on the training split
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Scale on the training distribution
StandardScaler subtracts the training mean and divides by the training standard deviation. The transformed training columns have mean 0 and standard deviation 1; the test columns use those same training values, so they will not necessarily be centered at 0 themselves.
Python
The test set is transformed, not refitted
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Scaling matters for distance-based models such as K-nearest neighbors, for regularized linear models, and for PCA. Tree-based models usually do not need it.
Turn categories into columns
OneHotEncoder creates one binary column per known category. handle_unknown="ignore" keeps prediction working when a new category appears, and sparse_output=False returns a dense array that is easy to inspect.
Python
One binary column per category
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Apply different rules to different columns
Real tables mix numeric and categorical columns. ColumnTransformer applies one preprocessing pipeline to the numeric columns and another to the categorical columns, then concatenates the results.
Python
A mixed-data preprocessing recipe
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
- Numeric columns are imputed with the training median, then standardized.
- The city column becomes one binary column per known city.
- The fitted preprocessor is now ready to sit in front of a model.
Make preprocessing part of the model
The safest place for preprocessing is inside the same Pipeline as the estimator. Then cross-validation, fitting, and prediction all reuse the same learned rules without manual bookkeeping.
Python
Cross-validation would refit this whole recipe
Output
Press Run to execute this cell. The first run downloads Python; later runs reuse the session.
Practice
Make it stick
Run the tests as often as you like. Hints and the solution are available when you need them.
Exercise
Build a ColumnTransformer named preprocessor. Apply median imputation then StandardScaler to age and spend, and OneHotEncoder(handle_unknown="ignore", sparse_output=False) to city. Fit it on X_train and transform both X_train and X_test into X_train_ready and X_test_ready.
Your solution
ReadyResults
Run the tests to check your solution. The first run downloads Python; later runs reuse the session.
Quiz
Check your understanding
Lesson progress