Chapter 4: Tables

Before a model sees a single array, somebody reads a table: a CSV of training runs, an export of labels, a log. Chapter 1 read one by hand and got strings back. pandas reads it in one call into a DataFrame: named columns, each a Series with its own dtype, all sharing one index that labels the rows.

pd.read_csv reads a CSV, and pd.read_parquet reads Parquet, a binary format that stores each column’s dtype. Then look before you change anything. head shows the first rows, info each column’s dtype and how many values it really has, and describe a summary of the numbers.

import pandas as pd
df = pd.read_csv("data/runs.csv", index_col="run")
df.head(3)

    model    lr  epochs  val_loss device
run                                     
r1    cnn  0.10    10.0      0.42    gpu
r2    cnn  0.01    10.0      0.35    gpu
r3    mlp  0.10     NaN      0.61    cpu

df.info()

<class 'pandas.DataFrame'>
Index: 8 entries, r1 to r8
Data columns (total 5 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   model     8 non-null      str    
 1   lr        8 non-null      float64
 2   epochs    6 non-null      float64
 3   val_loss  7 non-null      float64
 4   device    7 non-null      str    
dtypes: float64(3), str(2)
memory usage: 446.0 bytes

df.describe()

             lr     epochs  val_loss
count  8.000000   6.000000  7.000000
mean   0.053750  14.166667  0.455714
std    0.041726   6.645801  0.155334
min    0.010000   5.000000  0.290000
25%    0.010000  10.000000  0.340000
50%    0.050000  15.000000  0.420000
75%    0.100000  20.000000  0.545000
max    0.100000  20.000000  0.710000

Here is that file of eight runs, cleaned a line at a time. Column selection takes a name in brackets; loc picks rows and columns by label and iloc by position; a boolean mask keeps the rows where a test is True. Then dropna, fillna and astype repair the gaps and the types, and assign adds a column. value_counts counts, and to_numpy turns the result into an array for a model.

import pandas as pddf = pd.read_csv('data/runs.csv', index_col='run')df.dtypesdf.isna()df['val_loss']df.loc['r3':'r5', ['model', 'epochs']]df.iloc[[0, -1], :2]df[df['val_loss'] < 0.4]df = df.dropna(subset=['val_loss'])df = df.fillna({'epochs': 10, 'device': 'cpu'})df = df.astype({'epochs': 'int64'})df = df.assign(steps=df['epochs'] * 250)df['model'].value_counts()X = df[['lr', 'epochs']].to_numpy()

Two of those lines are easy to mix up. loc reads labels, and a label slice includes its end, so 'r3':'r5' is three rows. iloc reads positions and leaves the end out, as NumPy does. After dropna the index has a gap where r4 was, so position 3 now holds r5.

This is where people slip. A missing value reads as NaN, and comparing NaN with <, >= or even == gives False. So df[df['val_loss'] < 0.4] and its opposite, >= 0.4, both leave r4 out, and nothing warns you. Count the gaps with isna before you filter, and check that the rows you kept and the rows you dropped add up.

import pandas as pd
df = pd.read_csv("data/runs.csv", index_col="run")
len(df)

8

len(df[df["val_loss"] < 0.4]), len(df[df["val_loss"] >= 0.4])

(3, 4)

df["val_loss"].isna().sum()

np.int64(1)

nan = float("nan")
nan < 0.4, nan >= 0.4, nan == nan

(False, False, False)

pandas 3 made copy-on-write the rule: anything you take from a DataFrame behaves as a separate copy. That is the opposite of NumPy, where a slice is a view of the original. Changing a Series you selected never changes the table, and neither does a chained assignment like df['epochs']['r3'] = 10, which pandas warns about. To change a table, change the table itself, in one loc step.

import warnings
import pandas as pd
df = pd.read_csv("data/runs.csv", index_col="run")
losses = df["val_loss"]
losses.iloc[0] = 0.0
losses.iloc[0], df.loc["r1", "val_loss"]

(np.float64(0.0), np.float64(0.42))

with warnings.catch_warnings(record=True) as caught:
    df["epochs"]["r3"] = 10
[w.category.__name__ for w in caught]

['ChainedAssignmentError']

df.loc["r3", "epochs"]

np.float64(nan)

df.loc["r3", "epochs"] = 10
df.loc["r3", "epochs"]

np.float64(10.0)

A CSV stores only text, so every read guesses the dtypes again, and a float32 column comes back float64. pd.read_parquet gives back the dtypes that to_parquet wrote.

import pandas as pd
df = pd.read_csv("data/runs.csv", index_col="run")
small = df.astype({"lr": "float32"})
small.to_csv("runs.csv")
small.to_parquet("runs.parquet")
pd.read_csv("runs.csv", index_col="run")["lr"].dtype

dtype('float64')

pd.read_parquet("runs.parquet")["lr"].dtype

dtype('float32')

The habit this chapter adds is to look after every step. Check the dtypes with info, count the gaps with isna, and compare the number of rows before and after each filter. A table that lost a row without a word trains a model that is wrong without anyone noticing.

Homework

The packet has three parts: five one-line exercises with loc, iloc, isna, value_counts and to_numpy; cleaning a messy CSV into the table the test expects; and two bugs to fix, a filter that loses the runs with no accuracy and a chained assignment that changes nothing.