Chapter 6: Seeing the Data

A training run prints one loss per epoch, and a column of numbers hides what a picture shows at a glance: whether the model is still learning, has stalled, or has started memorizing. Matplotlib draws that picture. Its pyplot module, imported as plt, is the way in.

plt.subplots makes a Figure, the whole image, and an Axes, one plot area with its own scales. You then draw by calling methods on the Axes: ax.plot for lines, scatter for points, hist and bar for counts, imshow for a grid of values. set_xlabel, set_title and legend tell the reader what they are looking at.

import matplotlib.pyplot as pltimport pandas as pdlog = pd.read_csv('data/training-log.csv', index_col='epoch')fig, ax = plt.subplots()ax.plot(log.index, log['train_loss'], label='train')ax.plot(log.index, log['val_loss'], label='validation')ax.set_xlabel('epoch')ax.set_ylabel('loss')ax.set_title('Train vs validation loss')ax.legend()best = log['val_loss'].idxmin()

The stepper shows what each line changed; here is the picture those lines draw, with a dashed line at the lowest validation loss.

import matplotlib.pyplot as plt
import pandas as pd
log = pd.read_csv('data/training-log.csv', index_col='epoch')
fig, ax = plt.subplots(figsize=(6, 3.6), layout='constrained')
# Each drawing call also returns what it drew (a list of lines, a Text);
# the page leaves those out and shows the figure.
ax.plot(log.index, log['train_loss'], label='train')
ax.plot(log.index, log['val_loss'], label='validation')
best = log['val_loss'].idxmin()
ax.axvline(best, color='gray', linestyle='--', label=f'lowest validation (epoch {best})')
ax.set_xlabel('epoch')
ax.set_ylabel('loss')
ax.set_title('Train vs validation loss')
ax.legend()
fig

Read it from the left. Both losses fall while the model learns patterns that hold beyond the training data. From epoch 7 the training loss keeps falling, but the validation loss climbs: the model is now fitting the training examples themselves. That is overfitting, and the weights worth keeping come from epoch 7.

Matplotlib offers two ways in. In pyplot style, plt.plot and plt.title act on the current Axes: the one made last, unless plt.sca(ax) picked another. In the object-oriented style, you call the method on the Axes you want, so every line says which plot it changes.

This is where people slip. With two Axes, plt.title lands on the second one, with no error. And an Axes spells the same jobs differently: ax.title is a Text object, not a function, so calling it raises, and the method is set_title.

import matplotlib.pyplot as plt
fig, axs = plt.subplots(1, 2)
axs.shape

(2,)

# plt.title titles the current Axes, the last one made.
plt.title('train')
[ax.get_title() for ax in axs]

['', 'train']

# On an Axes, the same jobs are set_ methods.
axs[0].title('train')

TypeError: 'Text' object is not callable

axs[0].set_title('train')
[ax.get_title() for ax in axs]

['train', 'train']

plt.close(fig)

Loss often falls fast, then slowly for a long time, and on a linear scale the slow part looks flat. A log scale gives each tenfold drop the same room, so you can see the model is still improving. Asking for two Axes in one row, plt.subplots(1, 2), makes a subplot grid that shows both side by side.

import matplotlib.pyplot as plt
import numpy as np
# A made-up loss over 500 steps: a power law with a little noise.
rng = np.random.default_rng(0)
step = np.arange(1, 501)
loss = 2.5 * step ** -0.7 * np.exp(rng.normal(0, 0.05, 500))
loss[[0, 99, 499]].round(3)

Output:

array([2.516, 0.093, 0.033])

fig, (left, right) = plt.subplots(1, 2, figsize=(8, 3), layout='constrained')
left.plot(step, loss)
left.set_title('linear')
right.plot(step, loss)
right.set_yscale('log')
right.set_title('log')
left.set_xlabel('step')
right.set_xlabel('step')
left.set_ylabel('loss')
fig

The other plots are one call each. A scatter of predictions against targets shows how close they land, a histogram how the errors spread, a bar chart the examples per class, and imshow a confusion matrix. A 2×2 subplot grid returns its Axes as an array of shape (2, 2), so axs[1, 0] is row 1, column 0.

import matplotlib.pyplot as plt
import numpy as np
rng = np.random.default_rng(1)
target = rng.uniform(0, 10, 200)
prediction = target + rng.normal(0, 1, 200)
confusion = np.array([[50, 3, 2], [4, 40, 6], [1, 8, 30]])
fig, axs = plt.subplots(2, 2, figsize=(7, 5.4), layout='constrained')
axs.shape

Output:

(2, 2)

axs[0, 0].scatter(target, prediction, s=6)
axs[0, 0].set_title('scatter: prediction vs target')
axs[0, 1].hist(prediction - target, bins=20)
axs[0, 1].set_title('hist: errors')
axs[1, 0].bar(['cat', 'dog', 'bird'], [120, 80, 30])
axs[1, 0].set_title('bar: examples per class')
axs[1, 1].imshow(confusion)
axs[1, 1].set_title('imshow: confusion matrix')
fig

pandas can do the drawing for you. DataFrame.plot draws one line per column against the index, adds a legend, labels x with the index name and returns the Axes. savefig writes the Figure to a file in the format its extension names, so a training script can leave loss.png behind for you to open.

from pathlib import Path
import pandas as pd
log = pd.read_csv('data/training-log.csv', index_col='epoch')
ax = log.plot(title='Train vs validation loss', ylabel='loss')
ax.get_xlabel(), [text.get_text() for text in ax.get_legend().get_texts()]

Output:

('epoch', ['train_loss', 'val_loss'])

ax.figure

ax.figure.savefig('loss.png', dpi=150)
Path('loss.png').read_bytes()[:4]

Output:

b'\x89PNG'

The habit this chapter adds is to look at the curves before you trust a final number. One loss at the end can’t tell learning from memorizing; two curves side by side can. Give every plot axis labels and a legend, so a saved plot still makes sense when you open it next week.

Homework

The packet has three parts: plot a training log with axis labels, a title and a legend, and savefig it as a PNG; label three runs’ loss curves as overfitting, underfitting or a learning rate too high; and fix a function whose pyplot calls change the wrong Axes.