Creating a Digit Classifier (Almost) from Scratch
Published on Feb 21, 2023 12:36 by KE Programmer
Table of Contents
1. Introduction
We're creating a model that can classify any images as a 3 or a 7. We'll use a sample of MNIST that contains just these.
2. Data Prep
First, let's import the necessary libraries:
from fastai.vision.all import * import uuid import os import pathlib
Download and explore the data:
path = untar_data(URLs.MNIST_SAMPLE)
path.ls()
| Path | (home/krm.fastai/data/mnistsample/labels.csv) | Path | (home/krm.fastai/data/mnistsample/train) | Path | (home/krm.fastai/data/mnistsample/valid) |
The sample data is divided into training and validation sets:
(path/"train").ls(), (path/"valid").ls()
| Path | (home/krm.fastai/data/mnistsample/train/7) | Path | (home/krm.fastai/data/mnistsample/train/3) |
| Path | (home/krm.fastai/data/mnistsample/valid/7) | Path | (home/krm.fastai/data/mnistsample/valid/3) |
Get a list of the training set of 3s and 7s:
threes = (path/"train"/"3").ls().sorted() sevens = (path/"train"/"7").ls().sorted()
Have a look at one of the 3s. We use the Image class from PIL:
img3_path = threes[2] img3 = Image.open(img3_path) img3
We can see the image as a collection of digits using numpy arrays or pytorch tensors:
array(img3)[4:10, 4:10]
array([[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 13, 36],
[ 0, 0, 0, 0, 89, 253],
[ 0, 0, 0, 0, 89, 253],
[ 0, 0, 0, 0, 17, 151]], dtype=uint8)
tensor(img3)[4:10, 4:10]
tensor([[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 13, 36],
[ 0, 0, 0, 0, 89, 253],
[ 0, 0, 0, 0, 89, 253],
[ 0, 0, 0, 0, 17, 151]], dtype=torch.uint8)
Next we create tensors for each of the 3s and 7s:
three_tensors = [tensor(Image.open(path)) for path in threes] seven_tensors = [tensor(Image.open(path)) for path in sevens]
len(three_tensors), len(seven_tensors)
| 6131 | 6265 |
Use fastai's show_image to see a seven:
show_image(seven_tensors[6])
We stack each list of tensors into a single one of 3 axes (rank 3), convert to float for some operations. We also scale to values between 0 and 1 (better for the model):
stacked_threes = torch.stack(three_tensors).float() / 255 stacked_sevens = torch.stack(seven_tensors).float() / 255 stacked_threes.shape, stacked_sevens.shape
| torch.Size | ((6131 28 28)) | torch.Size | ((6265 28 28)) |
We create a training input collection by concatenating the two stacked
tensors into one. We use the view method to reshape the tensor into
two dimensions. -1 means making the first dimension as big as
possible to accommodate the new shape:
train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1, 28 * 28)
train_x.shape
torch.Size([12396, 784])
train_x[:2]
tensor([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]])
Each item along axis 0 is now a list of 784 floats representing a single image.
Next we create labels for our training input. Our objective is to classify whether an image is a 3 or not. The labels for 3s will be 1 (True) and for 7s will be 0 (False):
train_y_flat = tensor([1] * len(threes) + [0] * len(sevens)) train_y_flat.shape
torch.Size([12396])
We need to have a 2D tensor with the second dimension being a size
of 1. We use pytorch's unsqueeze for that:
train_y = train_y_flat.unsqueeze(1)
train_y.shape
torch.Size([12396, 1])
In pytorch, a dataset needs to return a tuple of (x, y) when indexed. We therefore zip training input and labels:
dataset = list(zip(train_x, train_y)) x0, y0 = dataset[0] x0.shape, y0.shape
| torch.Size | ((784)) | torch.Size | ((1)) |
We do the same preparation to the validation data as we've done for the training data:
v_threes = (path/"valid"/"3").ls().sorted() v_sevens = (path/"valid"/"7").ls().sorted()
v_three_tensors = [tensor(Image.open(path)) for path in v_threes] v_seven_tensors = [tensor(Image.open(path)) for path in v_sevens]
stacked_v_threes = torch.stack(v_three_tensors).float()/255 stacked_v_sevens = torch.stack(v_seven_tensors).float()/255
valid_x = torch.cat([stacked_v_threes, stacked_v_sevens]).view(-1, 28 * 28)
valid_x.shape
torch.Size([2038, 784])
valid_y = tensor([1] * len(v_threes) + [0] * len(v_sevens)).unsqueeze(1) valid_y.shape
torch.Size([2038, 1])
validation_dataset = list(zip(valid_x, valid_y))
3. Training
The initial model will be a linear function with weights for each
pixel and a bias. We'll need to calculate gradients for each of these,
so we call requires_grad:
def init_params(size, std=1.0): return (torch.randn(size) * std).requires_grad_()
weights = init_params((28*28, 1)) bias = init_params(1) weights.shape, bias.shape
| torch.Size | ((784 1)) | torch.Size | ((1)) |
def linear(x_batch): return x_batch@weights + bias
The @ operator does the dot-product operation
None
Next, we need to define a loss function that is sensitive to small changes in the parameters. For each batch of predictions, we calculate distance from the target values, and get the mean.
We also need to coerce the predictions to values between 0 and 1, so we'll use a sigmoid function for this:
def sigmoid(x): return 1/(1+torch.exp(-x))
How it does this is that for large positive values of x,
torch.exp(-x) approaches zero and the function outputs a value
approaching 1. For large negative values of x, torch.exp(-x)
output a large positive number the the output approaches 0.
sigmoid(tensor(34343424)), sigmoid(tensor(-385984549058))
| tensor | (1) | tensor | (0) |
def mnist_loss(predictions, targets): predictions = sigmoid(predictions) return torch.where(targets==1, 1-predictions, predictions).mean()
The value returned by minst_loss is bounded in [0, 1], so the closer
it is to 1, the worse the prediction is.
We now have enough to calculate gradients. We define a procedure that makes predictions, calculates the loss, then calculates the gradients that would minimise the loss:
def calculate_gradients(x_batch, y_batch, model): predictions = model(x_batch) loss = mnist_loss(predictions, y_batch) loss.backward()
We need to iteratively process mini batches until we exhaust the
entire dataset. The fastai library provides a DataLoader class
that will shuffle the dataset and provide batches of input and their
corresponding targets according to the batch size we provide. We can
iterate through this to process the entire dataset:
train_dl = DataLoader(dataset, batch_size=256) valid_dl = DataLoader(validation_dataset, batch_size=256)
Now we can encode the process of training through an entire epoch. With each mini-batch, we optimize the parameters using the gradients and learning rate. We're adjusting each parameter in opposite direction of the gradient to get closer to a minimized loss:
def train_epoch(model, learning_rate, params): for x, y in train_dl: calculate_gradients(x, y, model) for p in params: p.data -= p.grad * learning_rate p.grad.zero_() # reset gradient
We'll also calculate an accuracy metric for each epoch so that we can observe that the accuracy is improving with each successive epoch.
First we calculate the metric for each batch:
def batch_accuracy(predictions, targets): predictions = sigmoid(predictions) correct = (predictions > 0.5) == targets return correct.float().mean()
Then average it for the entire epoch (rounded off to 4 decimal places):
def validate_epoch(model): accuracies = [batch_accuracy(model(x_batch), y_batch) for x_batch, y_batch in valid_dl] return round(torch.stack(accuracies).mean().item(), 4)
We can now see the model performance for the entire epoch. We define a procedure that takes in the model, parameters, learning rate and number of epochs and prints out the accuracy:
def learn(model, params, learning_rate, epochs): for i in range(epochs): train_epoch(model, learning_rate, params) print(validate_epoch(model))
learn(linear, params=(weights, bias), learning_rate=1.0, epochs=20)
0.6142 0.7987 0.8749 0.9096 0.9252 0.9301 0.9365 0.9413 0.9438 0.9462 0.9496 0.9511 0.954 0.954 0.956 0.9565 0.9574 0.9613 0.9618 0.9628
We see that the accuracy gradually improves to approx 97%.
3.1. Pytorch/FastAI conveniences
Pytorch provides some handy functionality that we can use to simplify
the process above. nn.Linear will combine what init_params and
linear do together:
linear_model = nn.Linear(28 * 28, 1) weights, bias = linear_model.parameters() weights.shape, bias.shape
| torch.Size | ((1 784)) | torch.Size | ((1)) |
We can also create an optimizer class that will optimize parameters using an interface that resembles pytorch's:
class BasicOptimizer: def __init__(self, params, learning_rate): self.params = list(params) self.learning_rate = learning_rate def step(self, *args, **kwargs): for p in self.params: p.data -= p.grad.data * self.learning_rate def zero_grad(self, *args, **kwargs): for p in self.params: p.grad = None
learning_rate = 1.0 optimizer = BasicOptimizer(linear_model.parameters(), learning_rate)
We can re-write train_epoch and learn to use the new optimizer:
def train_epoch(model): for x, y in train_dl: calculate_gradients(x, y, model) optimizer.step() optimizer.zero_grad() def learn(model, epochs): for i in range(epochs): train_epoch(model) print(validate_epoch(model))
We should get similar results to the previous run:
learn(linear_model, epochs=20)
0.4932 0.8354 0.8418 0.9116 0.9331 0.9473 0.9555 0.9619 0.9658 0.9668 0.9687 0.9707 0.9731 0.9746 0.9761 0.977 0.9775 0.9775 0.978 0.9785
FastAI provides the class SGD that does the same thing as BasicOptimizer:
linear_model = nn.Linear(28 * 28, 1) optimizer = SGD(linear_model.parameters(), learning_rate) learn(linear_model, epochs=20)
0.4932 0.8462 0.8262 0.9106 0.9346 0.9463 0.9555 0.9619 0.9658 0.9673 0.9702 0.9717 0.9731 0.9751 0.9756 0.977 0.9775 0.978 0.978 0.9785
Fast AI also provides a Learner.fit method which does the same thing
as our learn. To use it, we combine the training and validation
dataloaders using a DataLoaders object:
dataloaders = DataLoaders(train_dl, valid_dl) learner = Learner( dataloaders, nn.Linear(28 * 28, 1), opt_func=SGD, loss_func=mnist_loss, metrics=batch_accuracy, ) learner.fit(20, lr=learning_rate)
█ epoch train_loss valid_loss batch_accuracy time █ Epoch 1/20 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 1/20 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 1/20 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.4606] Epoch 1/20 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.2281] Epoch 1/20 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.1506] Epoch 1/20 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.1118] Epoch 1/20 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.6246] Epoch 1/20 : Epoch 1/20 : █ Epoch 1/20 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 1/20 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 1/20 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.6365] Epoch 1/20 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.6365] Epoch 1/20 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.6365] Epoch 1/20 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.6365] Epoch 1/20 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.6365] Epoch 1/20 : Epoch 1/20 : 0 0.636500 0.503388 0.495584 00:00 █ Epoch 2/20 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 2/20 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 2/20 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.6165] Epoch 2/20 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.5973] Epoch 2/20 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.5790] Epoch 2/20 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.5613] Epoch 2/20 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.4990] Epoch 2/20 : Epoch 2/20 : █ Epoch 2/20 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 2/20 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 2/20 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.4876] Epoch 2/20 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.4876] Epoch 2/20 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.4876] Epoch 2/20 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.4876] Epoch 2/20 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.4876] Epoch 2/20 : Epoch 2/20 : 1 0.487571 0.209592 0.817468 00:00 --snipped-- █ Epoch 19/20 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 19/20 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 19/20 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.0150] Epoch 19/20 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.0157] Epoch 19/20 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.0161] Epoch 19/20 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.0163] Epoch 19/20 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.0147] Epoch 19/20 : Epoch 19/20 : █ Epoch 19/20 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 19/20 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 19/20 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.0145] Epoch 19/20 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.0145] Epoch 19/20 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.0145] Epoch 19/20 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.0145] Epoch 19/20 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.0145] Epoch 19/20 : Epoch 19/20 : 18 0.014492 0.026389 0.977920 00:00 █ Epoch 20/20 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 20/20 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 20/20 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.0148] Epoch 20/20 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.0155] Epoch 20/20 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.0158] Epoch 20/20 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.0161] Epoch 20/20 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.0146] Epoch 20/20 : Epoch 20/20 : █ Epoch 20/20 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 20/20 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 20/20 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.0143] Epoch 20/20 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.0143] Epoch 20/20 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.0143] Epoch 20/20 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.0143] Epoch 20/20 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.0143] Epoch 20/20 : Epoch 20/20 : 19 0.014336 0.025804 0.978410 00:00
We can now upgrade our model from a linear function to a simple neural network of two linear layers separated by a non-linearity.
The composition of one or more linear functions results in another linear function, but we need the linear functions decoupled from each other to be able to model more complex patterns, hence the use of a non-linearity.
The non-linearity in this case is the rectified linear unit which when given an input tensor X, outputs max(X, 0), meaning any values less than 0 are replaced by 0.
The first layer outputs 20 activations. The second one takes the 20 inputs and produces one activation:
simple_net = nn.Sequential(
nn.Linear(28 * 28, 20),
nn.ReLU(),
nn.Linear(20, 1)
)
Being a deeper network, we can use a lower learning rate and more epochs:
learner = Learner(
dataloaders, simple_net, opt_func=SGD, loss_func=mnist_loss, metrics=batch_accuracy
)
learner.fit(40, 0.1)
█ epoch train_loss valid_loss batch_accuracy time █ Epoch 1/40 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 1/40 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 1/40 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.4877] Epoch 1/40 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.4668] Epoch 1/40 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.4477] Epoch 1/40 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.4263] Epoch 1/40 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.3334] Epoch 1/40 : Epoch 1/40 : █ Epoch 1/40 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 1/40 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 1/40 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.3246] Epoch 1/40 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.3246] Epoch 1/40 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.3246] Epoch 1/40 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.3246] Epoch 1/40 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.3246] Epoch 1/40 : Epoch 1/40 : 0 0.324595 0.405516 0.509814 00:00 █ Epoch 2/40 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 2/40 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 2/40 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.3389] Epoch 2/40 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.3478] Epoch 2/40 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.3479] Epoch 2/40 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.3426] Epoch 2/40 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.1526] Epoch 2/40 : Epoch 2/40 : █ Epoch 2/40 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 2/40 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 2/40 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.1494] Epoch 2/40 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.1494] Epoch 2/40 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.1494] Epoch 2/40 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.1494] Epoch 2/40 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.1494] Epoch 2/40 : Epoch 2/40 : 1 0.149380 0.234102 0.797841 00:00 --snipped-- █ Epoch 39/40 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 39/40 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 39/40 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.0147] Epoch 39/40 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.0152] Epoch 39/40 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.0153] Epoch 39/40 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.0154] Epoch 39/40 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.0148] Epoch 39/40 : Epoch 39/40 : █ Epoch 39/40 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 39/40 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 39/40 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.0146] Epoch 39/40 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.0146] Epoch 39/40 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.0146] Epoch 39/40 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.0146] Epoch 39/40 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.0146] Epoch 39/40 : Epoch 39/40 : 38 0.014574 0.020761 0.982336 00:00 █ Epoch 40/40 : |-------------------------------------------| 0.00% [0/49 00:00<?] Epoch 40/40 : |-------------------------------------------| 2.04% [1/49 00:00<00:00] Epoch 40/40 : |█------------------------------------------| 4.08% [2/49 00:00<00:00... 0.0146] Epoch 40/40 : |██-----------------------------------------| 6.12% [3/49 00:00<00:00... 0.0150] Epoch 40/40 : |███----------------------------------------| 8.16% [4/49 00:00<00:00... 0.0152] Epoch 40/40 : |████---------------------------------------| 10.20% [5/49 00:00<00:00... 0.0153] Epoch 40/40 : |███████████████████████████████████████████| 100.00% [49/49 00:00<00:00... 0.0147] Epoch 40/40 : Epoch 40/40 : █ Epoch 40/40 : |-------------------------------------------| 0.00% [0/8 00:00<?] Epoch 40/40 : |█████--------------------------------------| 12.50% [1/8 00:00<00:00] Epoch 40/40 : |██████████---------------------------------| 25.00% [2/8 00:00<00:00... 0.0145] Epoch 40/40 : |████████████████---------------------------| 37.50% [3/8 00:00<00:00... 0.0145] Epoch 40/40 : |█████████████████████----------------------| 50.00% [4/8 00:00<00:00... 0.0145] Epoch 40/40 : |██████████████████████████-----------------| 62.50% [5/8 00:00<00:00... 0.0145] Epoch 40/40 : |███████████████████████████████████████████| 100.00% [8/8 00:00<00:00... 0.0145] Epoch 40/40 : Epoch 40/40 : 39 0.014454 0.020628 0.982336 00:00
4. Conclusion
At this point we have:
- According to the universal approximation theorem, a function that can approximate any problem to any level of accuracy given the right parameters
- A method of finding the correct parameters via stochastic gradient descent.