In scikit-learn, a straight line and a forest of trees are used the same way: to learn, then . One pattern opens up the whole library. The harder part is scoring a model honestly. The easy way to get that wrong raises no error and prints a good number.
An learns from data with . A model then has , which labels new rows; a preprocessor such as has , which rewrites rows with what it learned, and does both at once. holds rows back as a , and a chains steps into one . The stepper builds one baseline twice, wrong and then right.
from sklearn.datasets import load_breast_cancerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import accuracy_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScalerX, y = load_breast_cancer(return_X_y=True)# Wrong: the scaler sees every row, test rows included.scaler = StandardScaler()X_scaled = scaler.fit_transform(X)X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y, random_state=0)LogisticRegression().fit(X_tr, y_tr).score(X_te, y_te)# Right: split first, and let a Pipeline fit the scaler on the training rows.X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)model = Pipeline([('scale', StandardScaler()), ('clf', LogisticRegression())])model.fit(X_tr, y_tr)model.predict(X_te[:12])accuracy_score(y_te, model.predict(X_te))X.shape
(569, 30)
scaler
StandardScaler()
X is 569 tumours by 30 measurements. The scaler is an estimator that has learned nothing yet.
This is where most people slip. Scaling every row before the split runs without a warning, but the scaler’s mean_ now includes the test rows: that is . Here it costs little: the leaky score is 0.965 and the honest one 0.958, one test row of 143. A step that reads the labels leaks far more: on pure noise, where honest is about 0.5, keeping the 20 of 2,000 columns that best match the labels before the split scores 0.74.
import numpy as np
from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
# 200 rows of pure noise, labels drawn at random
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 2000))
y = rng.integers(0, 2, size=200)
# Wrong: keep the 20 columns that best match y, using every row
X_best = SelectKBest(k=20).fit_transform(X, y)
X_tr, X_te, y_tr, y_te = train_test_split(X_best, y, random_state=0)
LogisticRegression().fit(X_tr, y_tr).score(X_te, y_te)0.74# Right: the selection is a Pipeline step, fit on the training rows only
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
model = Pipeline([('select', SelectKBest(k=20)), ('clf', LogisticRegression())])
model.fit(X_tr, y_tr).score(X_te, y_te)0.5Inside a , learns every step from the training rows only, and sends new rows through the same steps. Put every step that learns from data in the , and the stays unseen.
Swapping the model changes one line. and take the same and . numbers, and measures how far they land.
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
# Every classifier has the same fit, predict and score
for clf in [LogisticRegression(), RandomForestClassifier(random_state=0)]:
model = Pipeline([('scale', StandardScaler()), ('clf', clf)])
print(type(clf).__name__, model.fit(X_tr, y_tr).score(X_te, y_te))LogisticRegression 0.958041958041958
RandomForestClassifier 0.972027972027972# A regressor predicts numbers: disease progression a year on
X, y = load_diabetes(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
reg = LinearRegression().fit(X_tr, y_tr)
reg.predict(X_te[:3]).round(1), y_te[:3](array([241.8, 250.1, 165. ]), array([321., 215., 127.]))mean_squared_error(y_te, reg.predict(X_te))3180.1596481558445 hides which mistakes a model makes. A counts them: rows are the true class, columns the one. reads down a column (of the tumours malignant, how many were) and along a row (of the malignant tumours, how many were caught). Both score class 1 by default, which here is benign, so pass pos_label=0.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
pred = RandomForestClassifier(random_state=0).fit(X_tr, y_tr).predict(X_te)
# Rows are the true class, columns the prediction: 0 malignant, 1 benign
confusion_matrix(y_te, pred)array([[52, 1],
[ 3, 87]])accuracy_score(y_te, pred)0.972027972027972# By default the positive class is 1, here benign
precision_score(y_te, pred), recall_score(y_te, pred)(0.9886363636363636, 0.9666666666666667)# The class you care about is malignant, 0
precision_score(y_te, pred, pos_label=0), recall_score(y_te, pred, pos_label=0)(0.9454545454545454, 0.9811320754716981)Real tables mix text and numbers. turns a text column into one 0/1 column per category, and sends each of columns to its own step. A category never saw raises an error by default; handle_unknown='ignore' leaves its columns at 0 instead.
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
jobs = pd.DataFrame({'gpu': ['a100', 'h100', 'a100', 'l4'], 'hours': [2.0, 8.0, 4.0, 1.0]})
enc = OneHotEncoder(sparse_output=False)
enc.fit_transform(jobs[['gpu']])array([[1., 0., 0.],
[0., 1., 0.],
[1., 0., 0.],
[0., 0., 1.]])enc.categories_[array(['a100', 'h100', 'l4'], dtype=object)]# A category fit never saw
enc.transform(pd.DataFrame({'gpu': ['b200']}))ValueError: Found unknown categories ['b200'] in column 0 during transform# Each list of columns gets its own transformer; the results sit side by side
prep = ColumnTransformer([
('onehot', OneHotEncoder(handle_unknown='ignore', sparse_output=False), ['gpu']),
('scale', StandardScaler(), ['hours']),
])
prep.fit_transform(jobs).round(2)array([[ 1. , 0. , 0. , -0.65],
[ 0. , 1. , 0. , 1.59],
[ 1. , 0. , 0. , 0.09],
[ 0. , 0. , 1. , -1.03]])prep.transform(pd.DataFrame({'gpu': ['b200'], 'hours': [4.0]})).round(2)array([[0. , 0. , 0. , 0.09]])One split is one roll of the dice. and scores five times, holding out a different fifth of the training rows each time. does that for every setting in a grid, then refits the best on all the training rows. The waits until the end and is scored once.
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)
model = Pipeline([('scale', StandardScaler()), ('clf', LogisticRegression())])
# Five folds of the training rows: fit on four, score the fifth, five times
scores = cross_val_score(model, X_tr, y_tr)
scores.round(3), scores.mean().round(3)(array([0.988, 0.988, 0.953, 1. , 0.976]), np.float64(0.981))# Every C in the grid, scored the same way; a step's setting is step__name
search = GridSearchCV(model, {'clf__C': [0.01, 0.1, 1, 10]})
search.fit(X_tr, y_tr)GridSearchCV(estimator=Pipeline(steps=[('scale', StandardScaler()),
('clf', LogisticRegression())]),
param_grid={'clf__C': [0.01, 0.1, 1, 10]})search.best_params_, round(search.best_score_, 3)({'clf__C': 1}, np.float64(0.981))# refit=True: the best setting refit on all training rows, then the test set, once
search.score(X_te, y_te)0.958041958041958The habit this chapter adds is to split first, then everything inside a , so every score you print was earned on rows the model never saw. How Does a Model Actually Learn? shows what that held-out score guards against, . How Do You Know a Model Got Better? carries the same rule, a that never leaks, into the gate a model must pass to ship.
Homework
The packet has three parts: fix a function that its before the split; build a with a that must be right on more than 78% of a fixed , and still answer for a GPU it never saw; and read and off a .
Chapter 7 workbook
Unzip, follow the README once to set up, then run pytest until it's green.