Chapter 8: Tensors and Autograd

Everything a PyTorch model learns is kept in a tensor. A tensor is NumPy’s array with two additions: it can move to a GPU, and it can remember how it was computed. From that record, autograd works out the gradient of a loss for every weight, which is what training steps on. This chapter is about where a gradient comes from, and the way it goes wrong without an error.

torch.tensor makes a tensor from a list, torch.zeros from a shape, and torch.randn draws random numbers. Each tensor has a shape, a dtype and a device. Python floats become float32 here; NumPy would make float64.

import torch
t = torch.tensor([[1, 2, 3], [4, 5, 6]])
t.shape, t.dtype, t.device

(torch.Size([2, 3]), torch.int64, device(type='cpu'))

torch.zeros(2, 3)

tensor([[0., 0., 0.],
        [0., 0., 0.]])

# Seeded, so a rerun draws the same numbers.
_ = torch.manual_seed(0)
torch.randn(2, 3)

tensor([[ 1.5410, -0.2934, -2.1788],
        [ 0.5684, -1.0845, -1.3986]])

# Python floats become float32; NumPy would make float64.
torch.tensor([0.5, 1.5]).dtype

torch.float32

t.float().mean()

tensor(3.5000)

Mark a tensor with requires_grad, and autograd records every operation computed from it. Call backward on a single-number result and it walks that record in reverse, which is backpropagation. It leaves each gradient in the .grad of the tensor you marked. The stepper computes y = w·x + b and sends the gradients back.

import torchx = torch.tensor([1., 2., 3.])w = torch.tensor([0.5, -1., 2.], requires_grad=True)b = torch.tensor(1., requires_grad=True)y = w @ x + bw.grady.backward()(w @ x + b).backward()w.detach().numpy()

A gradient has the shape of its tensor: w.grad is (3,), like w. Here it equals x, because y grows by x[i] for each unit added to w[i]. Working one out by hand and comparing it with .grad is the quickest check that the graph is the one you meant.

This is where most people slip. backward adds to .grad rather than replacing it, and nothing raises: compute the loss again, call backward again, and both gradients are added together. In a training loop, each step then moves the weights by every earlier gradient too. Clear it before each backward with w.grad = None; Chapter 9’s optimizer.zero_grad() does that for every weight.

import torch
w = torch.tensor(2.0, requires_grad=True)
# The same loss, three times over. d/dw (3w - 1)² = 6(3w - 1) = 30.
for step in range(3):
    loss = (3 * w - 1) ** 2
    loss.backward()
    print(w.grad)

tensor(30.)
tensor(60.)
tensor(90.)

# Clear the gradient before each backward.
for step in range(3):
    w.grad = None
    loss = (3 * w - 1) ** 2
    loss.backward()
    print(w.grad)

tensor(30.)
tensor(30.)
tensor(30.)

Stepping a weight isn’t part of the model, so autograd mustn’t record it. Inside torch.no_grad nothing is recorded; outside it, an in-place update of a weight that has requires_grad raises. detach gives the same values cut off from the graph.

import torch
w = torch.tensor(2.0, requires_grad=True)
loss = (3 * w - 1) ** 2
loss.backward()
w.grad

tensor(30.)

w -= 0.01 * w.grad

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

# Stepping a weight isn't part of the model, so autograd mustn't record it.
with torch.no_grad():
    w -= 0.01 * w.grad
w

tensor(1.7000, requires_grad=True)

(w * 3).requires_grad, w.detach().requires_grad

(True, False)

torch.from_numpy shares a NumPy array’s memory, like a view in Chapter 3, while torch.tensor makes a copy. .numpy() goes the other way, sharing memory too. It refuses a tensor that has requires_grad, so detach first.

import numpy as np
import torch
a = np.zeros(3)
shared = torch.from_numpy(a)
copied = torch.tensor(a)
a[0] = 7
# from_numpy shares a's memory; torch.tensor made a copy.
shared, copied

(tensor([7., 0., 0.], dtype=torch.float64), tensor([0., 0., 0.], dtype=torch.float64))

shared.numpy()

array([7., 0., 0.])

w = torch.ones(3, requires_grad=True)
w.numpy()

RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

w.detach().numpy()

array([1., 1., 1.], dtype=float32)

A tensor’s device is where its values live, and .to() returns a copy on another one. On an NVIDIA GPU the device is 'cuda', and on a Mac’s GPU it’s 'mps' (macOS 14 or later). A result lands on the device its inputs are on, so a model and its data must be moved to the same device. Every example on this page ran on PyTorch’s CPU-only build, so no GPU code runs here.

import torch
t = torch.ones(2, 3)
t.device

device(type='cpu')

# False here: this page runs PyTorch's CPU-only build.
torch.cuda.is_available()

False

device = 'cuda' if torch.cuda.is_available() else 'cpu'
# Already on that device, so .to() hands back t itself.
t.to(device) is t

True

t.to('cuda')

AssertionError: Torch not compiled with CUDA enabled

t.to(torch.float64).dtype

torch.float64

The habit this chapter adds: before you call backward, know which tensors have requires_grad and that their .grad is clear. Then check one gradient by hand on a tiny case, like the stepper’s. Chapter 9 wraps this one step in a training loop.

Homework

The packet has three parts: write down the shape of twelve tensor expressions before running them; compute two gradients by hand, then check them with backward; and fix a gradient descent loop whose gradients pile up across backward calls. Every exercise runs on the CPU.