How does a neural net really work¶
In this notebook I reimplement core parts of fast.ai's Kaggle notebook "How does a neural net really work" to build intuition step by step. The flow is: start with regression, use gradient descent to optimize coefficients, apply the same ideas to a shallow neural network on Titanic data, and then extend that model into a deeper network.
- Revising Regressions
- Plot a generic quadratic function ($ax^2 + bx + c$)
- Generate some random data points
- Learn the step-by-step process to find the values of
a,b, andcthat make our function represent the random data generated in2 - Use the mean absolute error to manually adjust
a,b, andc.
- Understand and break down the Gradient Descent algorithm
- The Basics of a Neural-Network using the Titanic Survival dataset from Kaggle
- Explore ReLUs and how they differ from a simple linear function
- Build a single-layer neural network using a weighted-sum function $f(x)=m_{Age}x_{Age}+m_{SibSp}x_{SibSp}+m_{Parch}x_{Parch}+m_{Fare}x_{Fare}$
- Do deep learning by stacking layers of weights/neurons into a multi-layer neural network
# Dependencies are managed with uv.
# Run `uv sync` in the project root before executing this notebook.
1. Revising Regressions¶
This section, from the fast.ai course, sets the stage for understanding how neural networks learn "weights". We'll plot some points on a graphic and use visualizations to see how changing the coefficients affects the function to better fit the points.
1.1 Plot a generic quadratic function ($ax^2+bx+c$)¶
from fastai.basics import torch, plt
import numpy as np, pandas as pd
# Wider display settings make notebook tables and tensors easier to inspect
np.set_printoptions(linewidth=140)
torch.set_printoptions(linewidth=140, sci_mode=False, edgeitems=7)
pd.set_option('display.width', 140)
# Set the figure DPI to 90 for better resolution
plt.rc('figure', dpi=90)
# Function to plot a mathematical function over a range
def plot_function(f, title=None, min=-2.1, max=2.1, color='r', ylim=None):
# Create evenly spaced x values as a column vector
x = torch.linspace(min,max, 100)[:,None]
# Set y-axis limits if specified
if ylim: plt.ylim(ylim)
# Plot the function
plt.plot(x, f(x), color)
# Add title if provided
if title is not None: plt.title(title)
# Function with quadratic expression ax^2 + bx + c
def quad(a, b, c, x): return a*x**2 + b*x + c
from functools import partial
# Creates a new function with fixed a,b,c parameters, leaving only x variable
# This allows us to create specific quadratic functions by "fixing" the coefficients
def mk_quad(a,b,c): return partial(quad, a,b,c)
def demo_plot_basic_quadratic():
a = 3
b = 2
c = 1
f = mk_quad(a, b ,c)
plot_function(f, title=f'${a}x^2 + {b}x + {c}$')
demo_plot_basic_quadratic()
1.2. Generate some random data points¶
# Add both multiplicative and additive noise to input x
def add_noise(x, mult, add): return x * (1+torch.randn(x.shape) * mult) + torch.randn(x.shape) * add
def generate_noisy_data(f, x_start=-2, x_end=2, num_datapoints=20, noise_mult=0.15, noise_add=0.15, seed=42):
# Define a static seed, so that the random data is always the same every time we run this
torch.manual_seed(seed)
# Create evenly spaced x values and add a dimension of size 1 at position 1,
# transforming shape from (20,) to (20,1) to make it a column vector
# Example: tensor([1,2,3]) shape=(3,) -> tensor([[1],[2],[3]]) shape=(3,1)
x = torch.linspace(x_start, x_end, steps=num_datapoints).unsqueeze(1)
# Generate noisy y values by applying noise to function output
# mult=0.15 for multiplicative noise, add=1.5 for additive noise
y = add_noise(f(x), noise_mult, noise_add)
return x, y
def demp_plot_random_data():
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y);
demp_plot_random_data()
1.3. Fit the function to the data¶
In this section, we will explore the step-by-step process to find the values of a, b, and c that allow our function to accurately reflect the random data generated in 1.2. The interactive plot below, shows how adjustments to a, b, and c influence the function's shape to better align with our data layout.
from ipywidgets import interact
@interact(a=1.5, b=1.5, c=1.5)
def demo_interactive_plot_quad(a, b, c):
plt.close('all') # Close all existing figures
plt.figure() # Create a new figure
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y)
plot_function(mk_quad(a,b,c), ylim=(0,13))
1.4 Measure the error¶
This is cool and works, but we need to know how close we are to our ideal solution. In regression, we can use fun ways to estimate this, like the "Mean Absolute Error," which averages the distance between predicted and actual values.
The fastai library has a wrapper for some of the most common methods (from scikit-learn).
Here's a quick demo where we can see how it is calculated.
def mean_absolute_error(preds, acts): return (torch.abs(preds-acts)).mean()
def demo_mean_absolute_error():
# Create some example predictions and actual values
preds = torch.tensor([1.0, 2.0, 3.0, 4.0])
actuals = torch.tensor([1.1, 2.1, 2.8, 4.2])
# Calculate and print the mean absolute error
error = mean_absolute_error(preds, actuals)
print(f"Mean Absolute Error: {error:.3f}")
# Let's break down what's happening:
print("\nAbsolute differences between predictions and actuals:")
abs_diffs = torch.abs(preds-actuals)
for i, (p, a, d) in enumerate(zip(preds, actuals, abs_diffs)):
print(f"Prediction: {p:.1f}, Actual: {a:.1f}, Absolute Difference: {d:.3f}")
demo_mean_absolute_error()
@interact(a=1.5, b=1.5, c=1.5)
def plot_quad(a, b, c):
x,y = generate_noisy_data(mk_quad(3,2,1))
f = mk_quad(a, b ,c)
plt.scatter(x,y)
loss = mean_absolute_error(f(x), y)
plot_function(f, ylim=(0,13), title=f"MAE: {loss:.2f}")
2. Understand and break down the Gradient Descent algorithm¶
Now that we can calculate the mean absolute error, the next step is to understand how to adjust our parameters a, b, and c to reduce this error. To do this, we can think about the gradients of the error with respect to each of the a,b,c parameters.
👉 If you were walking on a hill (representing the error surface), the partial derivative with respect to one direction (say, the 'a' direction) tells you the slope of the hill in that specific direction. A steep slope/gradient means a large change in error for a small change in 'a'.
For example, if we consider the partial derivative of the mean absolute error with respect to a (while keeping b and c fixed), a negative value would indicate that increasing a will lead to a decrease in the error (like walking forward-downhill in the 'a' direction). Conversely, a positive value would suggest that decreasing a would reduce the error (ackwardly walking backwards-downhill in the 'a' direction 😄).
Using AI, we can plot this hill. The following plot shows the "error surface" (MAE) for the function $f(x) = m*x + b$. By fixing one variable (e.g., b=2), we can visualize the differentiated function in terms of m and determine whether to increase or decrease m to move "downhill."
import plotly
plotly.offline.init_notebook_mode(connected=True)
import plotly.graph_objects as go
def demo_mae_surface():
# Actual data points
x_actual = torch.tensor([1.0, 2.0])
y_actual = torch.tensor([2.0, 4.0])
# Range of m and b values to explore
m_vals = torch.linspace(-1, 5, 100) # Explore m from -1 to 5
b_vals = torch.linspace(-1, 5, 100) # Explore b from -1 to 5
# Create a meshgrid of m and b values
M, B = torch.meshgrid(m_vals, b_vals, indexing='ij')
# Initialize an array to store MAE values for the surface plot
mae_values_surface = torch.zeros_like(M)
# Calculate MAE for each combination of m and b for the surface plot
for i in range(M.shape[0]):
for j in range(M.shape[1]):
m = M[i, j]
b = B[i, j]
preds = m * x_actual + b
mae = mean_absolute_error(preds, y_actual)
mae_values_surface[i, j] = mae.item() # Store the scalar value
# --- Calculate MAE for fixed b to show gradient of m ---
fixed_b = 2.0 # Fixed b value
mae_values_fixed_b = []
for m in m_vals:
preds = m * x_actual + fixed_b
mae = mean_absolute_error(preds, y_actual)
mae_values_fixed_b.append(mae.item())
# Create the surface plot using Plotly
fig = go.Figure(data=[go.Surface(z=mae_values_surface.numpy(), x=b_vals.numpy(), y=m_vals.numpy(), colorscale='RdBu', name='MAE Surface')])
# Add the line plot for fixed b, offsetting z values slightly
z_offset_line = 0.1 # Offset to lift the line above the surface
fig.add_trace(go.Scatter3d(
x=[fixed_b] * len(m_vals), # Fixed b value for all m values
y=m_vals.numpy(),
z=[z + z_offset_line for z in mae_values_fixed_b], # Add offset to z values
mode='lines',
line=dict(color='green', width=4),
name=f'MAE with fixed b={fixed_b}'
))
# Add annotation - Adjusted for better readability, offsetting z value slightly
annotation_m = 2.0
annotation_mae = mean_absolute_error(annotation_m * x_actual + fixed_b, y_actual).item()
z_offset_point = 0.1 # Offset to lift the point above the surface
fig.add_trace(go.Scatter3d(
x=[fixed_b],
y=[annotation_m],
z=[annotation_mae + z_offset_point], # Add offset to z value
mode='markers+text',
marker=dict(size=8, color='red'),
text=["At m=2, decrease m to go downhill"], # Shortened text
textposition="top center", # Changed text position to top center
textfont=dict(size=10) # Reduced text size
))
fig.update_layout(
title='3D Error Surface: Mean Absolute Error (MAE) for f(x) = mx + b<br>with line showing gradient of m at fixed b=2', # Added <br> for title wrapping
scene=dict(
xaxis_title='b (y-intercept)',
yaxis_title='m (slope)',
zaxis_title='MAE',
camera=dict(eye=dict(x=1.5, y=-1.5, z=0.8)) # Rotate perspective
),
width=700,
height=700
)
fig.show()
demo_mae_surface()
With PyTorch, we can use require_grad=True on tensors, which automatically handles differentiation and the calculation of the gradients for us. To be honest, it looks a bit like dark magic. Let's look at some examples where we can see this functionality from PyTorch being applied to our function $f(x) = m*x + b$, along with two different inputs (x1 and x2)🕵️♂️.
#########################################
# Example 1: Using x1 = [1.0, 2.0, 3.0]
#########################################
x1 = torch.tensor([1.0, 2.0, 3.0])
m1 = torch.tensor(2.0, requires_grad=True)
b1 = torch.tensor(1.0, requires_grad=True)
# Compute f(x1) = m1 * x1 + b1
y1 = m1 * x1 + b1
total1 = y1.sum() # total1 = m1*(1+2+3) + b1*3 = 6*m1 + 3*b1
# Compute gradients for m1 and b1.
total1.backward()
print(f'Example 1 | x: {x1}, m: {m1}, b: {b1}')
print("Example 1 | Gradient with respect to m1 (m1.grad):", m1.grad) # Expected: sum(x1) = 6.0
print("Example 1 | Gradient with respect to b1 (b1.grad):", b1.grad) # Expected: len(x1) = 3
print("--------------")
#########################################
# Example 2: Using x2 = [1.0, 4.0] (different size and values)
#########################################
x2 = torch.tensor([1.0, 4.0])
m2 = torch.tensor(2.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)
# Compute f(x2) = m2 * x2 + b2
y2 = m2 * x2 + b2
total2 = y2.sum() # total2 = m2*(1+4) + b2*2 = 5*m2 + 2*b2
# Compute gradients for m2 and b2.
total2.backward()
print(f'Example 2 | x: {x2}, m: {m2}, b: {b2}')
print("Example 2 | Gradient with respect to m2 (m2.grad):", m2.grad) # Expected: sum(x2) = 5.0
print("Example 2 | Gradient with respect to b2 (b2.grad):", b2.grad) # Expected: len(x2) = 2
print("--------------")
# Print an explanation that details the differentiation steps.
explanation = """
Explanation:
- **Example 1:**
- We start with a list of numbers: x1 = [1.0, 2.0, 3.0].
- We plug each number into our function, which means we multiply it by m (let's say m = 2) and then add b (let's say b = 1):
- For 1.0: f(1.0) = 2 * 1 + 1 = 3
- For 2.0: f(2.0) = 2 * 2 + 1 = 5
- For 3.0: f(3.0) = 2 * 3 + 1 = 7
- So, we end up with y1 = [3.0, 5.0, 7.0].
- Then, we add these numbers together: 3 + 5 + 7 = 15.
- If we look at our equation, we can say: total1 = m1*(1.0+2.0+3.0) + b1*3 = 6 * m1 + 3 * b1.
- Now, when we want to see how changing m1 and b1 affects total1, we find:
- Changing m1 gives us a "gradient" of 6.
- Changing b1 gives us a "gradient" of 3.
- So, m1.grad = 6.0 and b1.grad = 3.
- **Example 2:**
- Now we have a different list: x2 = [1.0, 4.0].
- We use the same m and b:
- For 1.0: f(1.0) = 3 (same as before).
- For 4.0: f(4.0) = 2 * 4 + 1 = 9.
- So, now we have y2 = [3.0, 9.0].
- We add these: 3 + 9 = 12.
- In terms of our equation, total2 = m2*(1.0+4.0) + b2*2 = 5 * m2 + 2 * b2.
- For the gradients:
- Changing m2 gives us a "gradient" of 5.
- Changing b2 gives us a "gradient" of 2.
- So, m2.grad = 5.0 and b2.grad = 2.
"""
print(explanation)
Now we can create an interactive plot where we show the gradient on a, b and c.
If the slope is negative we want to move forward (or downhill).
@interact(a=1.5, b=1.5, c=1.5)
def demo_quadratic_plot_with_gradients(a, b, c):
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y)
a_tensor = torch.tensor([float(a)], requires_grad=True)
b_tensor = torch.tensor([float(b)], requires_grad=True)
c_tensor = torch.tensor([float(c)], requires_grad=True)
f = mk_quad(a_tensor, b_tensor, c_tensor)
loss = torch.abs(f(x) - y).mean()
loss.backward()
a_grad = a_tensor.grad.item()
b_grad = b_tensor.grad.item()
c_grad = c_tensor.grad.item()
plot_function(lambda x: f(x).detach(), ylim=(0,13), title=f"MAE: {loss:.2f}, dLoss/da: {a_grad:.2f}, dLoss/db: {b_grad:.2f}, dLoss/dc: {c_grad:.2f}")
from fastai.metrics import mae
def demo_auto_fit(steps=20):
x, y = generate_noisy_data(mk_quad(3,2,1))
abc = torch.tensor([1.0,1.0,1.0], requires_grad=True)
min_loss = float('inf')
best_abc = abc.clone().detach() # Initialize best_abc with the initial abc
for i in range(steps):
f = mk_quad(*abc)
loss = mae(f(x), y)
loss.backward()
with torch.no_grad():
abc -= abc.grad*0.1
abc.grad.zero_() # Clear gradients after update
print(f'step={i}; loss={loss.item():.2f}; abc={abc}')
if loss < min_loss:
min_loss = loss
best_abc = abc.clone().detach() # Update best_abc when a lower loss is found
return best_abc
best_abc_params = demo_auto_fit()
print(f"Best abc parameters: {best_abc_params}")
3. The Basics of a Neural-Network¶
3.1 Introducing Non-Linearity with ReLU¶
We've seen that simple functions like quadratics can model some data, but real-world data is rarely so straightforward. Imagine trying to predict something complex, like whether a picture is a cat or a dog, based on many pixel values (our 'dimensions'). A simple quadratic or even a single linear function just won't be flexible enough to capture the intricate patterns in such high-dimensional data.
To handle this complexity, we need to build more powerful functions. Simply combining linear functions won't solve the problem because any combination of linear functions is still just a linear function! Linear functions can only model linear relationships in the data. Real-world data, like images of cats and dogs, is highly non-linear.
To introduce non-linearity, we use activation functions. ReLU (Rectified Linear Unit) is a simple yet powerful activation function that introduces non-linearity. By applying ReLU to the output of linear functions, we can create models that can learn complex, non-linear patterns in the data. This non-linearity is what allows neural networks to model intricate relationships that simple linear models cannot. This will lead us to the idea of a ReLU, a simple activation function, and the simplest "neural network" we can build with it.
def rectified_linear(m,b,x):
y = m*x+b
return torch.clip(y, 0.)
plot_function(partial(rectified_linear, 1,1))
Combining two ReLUs allows us to create more complex, piecewise linear functions, as illustrated in the interactive plot below. This combination increases the flexibility of our model, enabling it to capture more intricate relationships in the data.
def double_relu(m1,b1,m2,b2,x):
return rectified_linear(m1,b1,x) + rectified_linear(m2,b2,x)
@interact(m1=-1.5, b1=-1.5, m2=1.5, b2=1.5)
def plot_double_relu(m1, b1, m2, b2):
plot_function(partial(double_relu, m1,b1,m2,b2), ylim=(-1,6))
3.2 Building a Neural Network from-Scratch¶
From this point forward, we will be following this notebook: Linear model and neural net from scratch.
Important ⚠️: For simplicity, I'm skipping all the steps that involve data cleanup and preparation. This means of course that my model will most likely not have a very good performance.
We are using the Titanic competition from Kaggle. I have made a copy in my Hugging Face workspace, which tbh I did to to experiment how Datasets work on Hugging Face.
The goal is to create a model to predict whether a passenger Survived, which is provided in our dataset.
In essence, we will now combine functions like those we've explored above, such as ReLUs, to construct a simple neural network. This network will receive passenger features as input, apply weights (similar to $m$ in our previous examples), and hopefully predict whether the passenger Survived with the lowest possible loss/error.
from datasets import load_dataset
dataset = load_dataset("paulopontesm/titanic", data_files={"train": "train.csv", "test": "test.csv"})
train_dataset_df = dataset["train"].to_pandas()
test_dataset_df = dataset["test"].to_pandas()
train_dataset_df
Since we need numerical data for our model, we'll just use the columns that already contain numbers as predictors. These are the columns that are already numerical.
import numpy as np
train_dataset_df.describe(include=(np.number))
Now that we have numbers for the features, we can create tensors/arrays for our features (aka called independent variables) and target (aka dependent variable).
Even thought I mentioned above that I didn't want to do a lot of data transofrmations, I think we really need to remove the NaNs and to normalize the numbers.
from torch import tensor
from fastai.data.transforms import RandomSplitter
indep_cols = ['Age', 'SibSp', 'Parch', 'Fare']
t_dep = tensor(train_dataset_df.Survived)
trn_split, val_split = RandomSplitter(seed=42)(train_dataset_df)
train_set_targets, validation_set_targets = t_dep[trn_split], t_dep[val_split]
train_df = train_dataset_df.iloc[trn_split].copy()
validation_df = train_dataset_df.iloc[val_split].copy()
# We need to do 2 things before proceeding:
# 1. Replace all the nans by the mode of that column (fit on train split).
for col in indep_cols:
mode_val = train_df[col].mode()[0]
train_df[col] = train_df[col].fillna(mode_val)
validation_df[col] = validation_df[col].fillna(mode_val)
# 2. Scale columns by the max value from the train split.
for col in indep_cols:
max_val = train_df[col].max()
if max_val == 0:
max_val = 1
train_df[col] = train_df[col] / max_val
validation_df[col] = validation_df[col] / max_val
train_set_features = tensor(train_df[indep_cols].values, dtype=torch.float)
validation_set_features = tensor(validation_df[indep_cols].values, dtype=torch.float)
len(train_set_features), len(validation_set_features)
Before training, let's sanity-check the split sizes.
We keep a validation split to estimate generalization during development, while the test set stays untouched for final evaluation.
len(train_set_features),len(validation_set_features)
Now we can generate random weights (ms) for each of our features. We're using a linear model, effectively calculating a weighted sum of the features: $f(x) = m_{Age}*x_{Age} + m_{SibSp}*x_{SibSp} + m_{Parch}*x_{Parch} + m_{Fare}*x_{Fare}$. We will adjust these weights to predict passenger survival based on the features.
def generate_random_coefficients(num_coeffs):
torch.manual_seed(42)
coeffs = torch.rand(num_coeffs)-0.5
return coeffs.requires_grad_()
nn_coeffs=generate_random_coefficients(num_coeffs=train_set_features.shape[1])
nn_coeffs
def calc_preds(coeffs, features): return (features*coeffs).sum(axis=1)
predictions = calc_preds(nn_coeffs, train_set_features)
predictions.topk(3)
def calc_loss(coeffs, features, targets): return torch.abs(calc_preds(coeffs, features)-targets).mean()
loss = calc_loss(coeffs=nn_coeffs, features=train_set_features, targets=train_set_targets)
loss
loss.backward()
nn_coeffs.grad
def one_epoch(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = calc_loss(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
coeffs.sub_(coeffs.grad * lr)
coeffs.grad.zero_()
print(f"{loss:.3f}", end="; ")
def train_model(train_set_features_set, train_set_targets_set, epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients(num_coeffs=train_set_features_set.shape[1])
for i in range(epochs): one_epoch(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs
final_weights = train_model(train_set_features, train_set_targets)
def show_coeffs(coeffs): return dict(zip(indep_cols, coeffs.requires_grad_(False)))
show_coeffs(final_weights)
We have weights, let's do predictions then.
calc_preds(final_weights, validation_set_features)
It's hard not to notice that we should be predicitng a 0 or 1 value, but instead we're getting a lot of negatives.
For simplicity, let's ignore this and say that everything above 0.5 survived.
preds = calc_preds(final_weights, validation_set_features)
print(f"True count was {torch.sum(preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
With this we can use our validation set and calculate the % of predictions that we are getting correctly.
def calc_accuracy(predictions, validation_set_targets):
bool_predictions = predictions > 0.5
bool_validation_set_targets = validation_set_targets.bool()
correct_predictions = bool_validation_set_targets == bool_predictions
accuracy_float = correct_predictions.float()
accuracy_val = accuracy_float.mean()
return accuracy_val
accuracy_result = calc_accuracy(predictions=preds, validation_set_targets=validation_set_targets)
print(accuracy_result)
Looks like we're doing slightly better than throwing a coin. I say this is a success 🤔❓🤔 I don't think so...
From the counts above in the current run, the model predicts "not survived" much more often than "survived". That can still produce moderate accuracy on this split, while missing many passengers who actually survived.
The Linear model and neural net from scratch goes further on this exercise. It uses stronger preprocessing, encodes non-numeric columns, and applies a sigmoid so outputs stay between 0 and 1.
For me, this was enough to get a better overview of what's happening inside a neural network.
3.3 Do Deep Learning¶
In the section above we implemented a simple Neural Network. Now let's explore Deep Learning, which is what truly unlocks the power of Neural Networks.
Deep Learning involves creating Neural Networks with multiple layers. Instead of a single layer, we stack layers of neurons, allowing the network to learn more complex patterns and representations from the data.
def generate_random_coefficients_for_deep_learning(n_coeff, num_neurons_per_hidden_layer=[10, 10]):
torch.manual_seed(42)
# Define the number of neurons for each layer, including input, hidden, and output layers.
# The input layer size is n_coeff, hidden layers sizes are from num_neurons_per_hidden_layer, and output layer size is 1.
num_neurons = [n_coeff] + num_neurons_per_hidden_layer + [1]
layers = []
for i in range(len(num_neurons)-1):
# Determine the size of the input for the current layer from the previous layer's neuron count
layer_input_size = num_neurons[i]
# Determine the size of the output for the current layer from the current layer's neuron count
layer_output_size = num_neurons[i+1]
# Initialize a layer with random weights between -0.5 and 0.5.
# torch.rand generates uniform random numbers between 0 and 1, then we shift and scale to get range [-0.5, 0.5].
# requires_grad_() is set to True to enable gradient tracking for these tensors, which is needed for backpropagation.
layer = (torch.rand(layer_input_size, layer_output_size)-0.5).requires_grad_()
layers.append(layer)
return layers
dnn_layers_coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features.shape[1], num_neurons_per_hidden_layer=[10, 10])
dnn_layers_coeffs
We can test how we do without any training
def calc_preds_for_deep_learning(coeffs, features):
# @ is matrix multiplication in Python
# It was introduced in Python 3.5 as part of [PEP 465](https://peps.python.org/pep-0465/)
layer_features = features
for layer in coeffs[:-1]:
layer_features = layer_features @ layer
layer_features = layer_features @ coeffs[-1]
return layer_features.squeeze()
def calc_loss_for_deep_learning(coeffs, features, targets): return torch.abs(calc_preds_for_deep_learning(coeffs, features)-targets).mean()
dnn_preds = calc_preds_for_deep_learning(coeffs=dnn_layers_coeffs, features=validation_set_features)
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
And we need to do the grandient descent for all the coeffs on each layer.
def one_epoch_for_deep_learning(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = calc_loss_for_deep_learning(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
for layer in coeffs:
layer -= layer.grad * lr
layer.grad.zero_()
def train_model_for_deep_learning(train_set_features_set, train_set_targets_set, num_neurons_per_hidden_layer=[10, 10], epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features_set.shape[1], num_neurons_per_hidden_layer=num_neurons_per_hidden_layer)
for i in range(epochs): one_epoch_for_deep_learning(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs # Returns the trained coefficients, which have the same structure as generate_random_coefficients_for_deep_learning
Let's test it then with different combinations of hidden layers and neurons per layer...
for num_neurons in [[10, 10], [20, 20],[5, 5, 5],[30], [], [2], [50], [2, 2], [50, 50], [5, 10, 5], [2, 2, 2, 2]]:
dnn_final_weights = train_model_for_deep_learning(train_set_features, train_set_targets, num_neurons_per_hidden_layer=num_neurons)
dnn_preds = calc_preds_for_deep_learning(coeffs=dnn_final_weights, features=validation_set_features)
accuracy = calc_accuracy(predictions=dnn_preds, validation_set_targets=validation_set_targets)
print(f"Hidden layers: {num_neurons}")
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
print(f"Accuracy: {accuracy}")
print("-" * 20) # Separator for readability
Not a lot has changed...
Just for fun, we can see how adding a sigmoid and ReLU would affect the results... The code is Ctrl+V Ctrl+C from above, but with a smarter_calc_preds_for_deep_learning.
import torch.nn.functional as F
def smarter_calc_preds_for_deep_learning(coeffs, features):
# @ is matrix multiplication in Python
# It was introduced in Python 3.5 as part of [PEP 465](https://peps.python.org/pep-0465/)
layer_features = features
for layer in coeffs[:-1]:
layer_features = F.relu(layer_features @ layer)
layer_features = layer_features @ coeffs[-1]
return torch.sigmoid(layer_features.squeeze())
def smarter_calc_loss_for_deep_learning(coeffs, features, targets):
predictions = smarter_calc_preds_for_deep_learning(coeffs, features)
return F.binary_cross_entropy(predictions, targets) # Changed loss to Binary Cross Entropy
def smarter_one_epoch_for_deep_learning(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = smarter_calc_loss_for_deep_learning(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
for layer in coeffs:
layer -= layer.grad * lr
layer.grad.zero_()
def smarter_train_model_for_deep_learning(train_set_features_set, train_set_targets_set, num_neurons_per_hidden_layer=[10, 10], epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features_set.shape[1], num_neurons_per_hidden_layer=num_neurons_per_hidden_layer)
for i in range(epochs): smarter_one_epoch_for_deep_learning(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs # Returns the trained coefficients, which have the same structure as generate_random_coefficients_for_deep_learning
for num_neurons in [[10, 10], [20, 20],[5, 5, 5],[30], [], [2], [50], [2, 2], [50, 50], [5, 10, 5], [2, 2, 2, 2]]:
dnn_final_weights = smarter_train_model_for_deep_learning(train_set_features, train_set_targets.float(), num_neurons_per_hidden_layer=num_neurons)
dnn_preds = smarter_calc_preds_for_deep_learning(coeffs=dnn_final_weights, features=validation_set_features)
accuracy = calc_accuracy(predictions=dnn_preds, validation_set_targets=validation_set_targets)
print(f"Hidden layers: {num_neurons}")
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
print(f"Accuracy: {accuracy}")
print("-" * 20) # Separator for readability
Interesting. That definitely improved.
I will stop here for now. However, the next step will likely be to add the boolean variables like is_male, is_female, is_class_1, etc. If I understood that correctly and I'm not making any mistakes, it should bring me to around 80% accuracy, like we see on the fast.ai notebook.
4. Binary Classification Metrics¶
4.1 What BCE Means (Simple to Model)¶
Let's start with BCE itself before talking about NN internals. BCE (Binary Cross-Entropy) is the penalty for predicted probabilities in binary classification.
For one point with target y in {0,1} and predicted probability p for class 1:
$$-\left(y\log(p) + (1-y)\log(1-p)\right)$$
Intuition: confident correct predictions get small loss, confident wrong predictions get large loss.
eps = 1e-7
def bce_for_one_point(y, p, eps=1e-7):
y_t = torch.tensor(float(y))
p_t = torch.tensor(float(p)).clamp(eps, 1 - eps)
return float((-(y_t * torch.log(p_t) + (1 - y_t) * torch.log(1 - p_t))).item())
single_examples = [
("true=1, p=0.95 (good)", 1, 0.95),
("true=1, p=0.55 (uncertain)", 1, 0.55),
("true=1, p=0.10 (confident wrong)", 1, 0.10),
("true=0, p=0.10 (good)", 0, 0.10),
]
for label, y, p in single_examples:
print(f"{label:<34} -> BCE = {bce_for_one_point(y, p):.4f}")
print("\nHand-picked mini-batch (not random):")
# These values are chosen on purpose: mostly good predictions plus one confident wrong case.
mini_targets = torch.tensor([1, 1, 1, 0, 0, 0], dtype=torch.float32)
mini_probs = torch.tensor([0.95, 0.70, 0.40, 0.10, 0.30, 0.85], dtype=torch.float32)
mini_losses = -(
mini_targets * torch.log(mini_probs.clamp(eps, 1 - eps))
+ (1 - mini_targets) * torch.log((1 - mini_probs).clamp(eps, 1 - eps))
)
for i, (y, p, l) in enumerate(zip(mini_targets, mini_probs, mini_losses), start=1):
print(f"row {i}: y={int(y.item())} p={p.item():.2f} -> loss={l.item():.4f}")
print(f"Mean BCE over mini-batch: {mini_losses.mean().item():.4f}")
prob_axis = torch.linspace(0.001, 0.999, 200)
loss_if_true_1 = -torch.log(prob_axis)
loss_if_true_0 = -torch.log(1 - prob_axis)
plt.figure(figsize=(6, 4))
plt.plot(prob_axis, loss_if_true_1, label='If true label is 1')
plt.plot(prob_axis, loss_if_true_0, label='If true label is 0')
plt.title('BCE penalty curve')
plt.xlabel('Predicted probability of class 1')
plt.ylabel('Loss')
plt.legend()
plt.grid(alpha=0.25)
plt.show()
# Hand-picked one-feature example to show where probabilities come from in a model.
toy_x = torch.tensor([-2.5, -1.5, -0.5, 0.5, 1.5, 2.5], dtype=torch.float32)
toy_y = torch.tensor([0, 0, 0, 1, 1, 1], dtype=torch.float32)
toy_w, toy_b = 1.30, -0.20
toy_logits = toy_w * toy_x + toy_b
toy_probs = torch.sigmoid(toy_logits)
toy_bce = -(toy_y * torch.log(toy_probs.clamp(eps, 1 - eps)) + (1 - toy_y) * torch.log((1 - toy_probs).clamp(eps, 1 - eps))).mean()
print(f"Toy sigmoid: p(x) = sigmoid({toy_w:.2f} * x + {toy_b:.2f})")
print(f"Toy mean BCE from model probabilities: {toy_bce.item():.4f}")
x_grid = torch.linspace(float(toy_x.min().item()) - 0.5, float(toy_x.max().item()) + 0.5, 300)
p_grid = torch.sigmoid(toy_w * x_grid + toy_b)
plt.figure(figsize=(7, 4))
plt.scatter(toy_x[toy_y == 0], toy_y[toy_y == 0], color='tomato', s=70, label='Class 0')
plt.scatter(toy_x[toy_y == 1], toy_y[toy_y == 1], color='seagreen', s=70, label='Class 1')
plt.plot(x_grid, p_grid, color='black', linewidth=2, label='Sigmoid probability')
for xi, yi, pi in zip(toy_x, toy_y, toy_probs):
plt.vlines(float(xi.item()), float(yi.item()), float(pi.item()), colors='gray', alpha=0.35, linewidth=1)
plt.title('Toy model: feature x to probability')
plt.xlabel('Feature x (hand-picked values)')
plt.ylabel('Predicted probability of class 1')
plt.ylim(-0.05, 1.05)
plt.grid(alpha=0.25)
plt.legend()
plt.show()
In the NN case, the math is exactly the same:
- Replace
w*x+bwith NN logits. - Apply sigmoid to get probabilities.
- Compute BCE on those probabilities.
Only the source of probabilities changes; BCE itself does not.
4.2 Binary Classification Terms in Plain English¶
Now let's run the same ideas on our NN validation probabilities.
The goal is to make terms like BCE, confusion matrix, ROC, AUC, precision, recall, and F1 feel concrete.
Quick reminder for the next cells: we are using the same BCE formula, now on NN validation probabilities.
example_layers = [50, 50]
example_weights = smarter_train_model_for_deep_learning(
train_set_features,
train_set_targets.float(),
num_neurons_per_hidden_layer=example_layers
)
example_probs = smarter_calc_preds_for_deep_learning(coeffs=example_weights, features=validation_set_features).detach().float()
example_targets = validation_set_targets.float()
print(f"Example model layers: {example_layers}")
print(f"Validation rows: {len(example_targets)}")
def manual_bce(probs, targets, eps=1e-7):
probs = probs.clamp(eps, 1 - eps)
targets = targets.float()
return -(targets * torch.log(probs) + (1 - targets) * torch.log(1 - probs)).mean()
toy_bce_from_helper = manual_bce(toy_probs, toy_y)
nn_bce_manual = manual_bce(example_probs, example_targets)
nn_bce_library = torch.nn.functional.binary_cross_entropy(example_probs, example_targets)
print(f"Toy BCE (manual helper): {toy_bce_from_helper.item():.6f}")
print(f"NN validation BCE (manual): {nn_bce_manual.item():.6f}")
print(f"NN validation BCE (PyTorch): {nn_bce_library.item():.6f}")
prob_axis = torch.linspace(0.001, 0.999, 200)
loss_if_true_1 = -torch.log(prob_axis)
loss_if_true_0 = -torch.log(1 - prob_axis)
plt.figure(figsize=(6, 4))
plt.plot(prob_axis, loss_if_true_1, label='If true label is 1')
plt.plot(prob_axis, loss_if_true_0, label='If true label is 0')
plt.title('BCE penalty curve')
plt.xlabel('Predicted probability of survival')
plt.ylabel('Loss')
plt.legend()
plt.grid(alpha=0.25)
plt.show()
Confusion matrix is a 2x2 table that counts prediction outcomes:
TP(true positive): predicted survived, and actually survived.TN(true negative): predicted not survived, and actually not survived.FP(false positive): predicted survived, but actually not survived.FN(false negative): predicted not survived, but actually survived.
From those counts:
- Precision: out of predicted survivors, how many were truly survivors?
- Recall: out of all true survivors, how many did we catch?
- F1: a balance score between precision and recall.
def preds_from_threshold(probs, threshold):
return (probs >= threshold).int()
def confusion_counts(preds, targets):
targets = targets.int()
tp = int(((preds == 1) & (targets == 1)).sum().item())
tn = int(((preds == 0) & (targets == 0)).sum().item())
fp = int(((preds == 1) & (targets == 0)).sum().item())
fn = int(((preds == 0) & (targets == 1)).sum().item())
return tp, tn, fp, fn
def metrics_from_counts(tp, tn, fp, fn, eps=1e-9):
accuracy = (tp + tn) / (tp + tn + fp + fn + eps)
precision = tp / (tp + fp + eps)
recall = tp / (tp + fn + eps)
f1 = 2 * precision * recall / (precision + recall + eps)
return accuracy, precision, recall, f1
default_threshold = 0.5
default_preds = preds_from_threshold(example_probs, default_threshold)
tp, tn, fp, fn = confusion_counts(default_preds, example_targets)
acc, prec, rec, f1 = metrics_from_counts(tp, tn, fp, fn)
print(f"threshold={default_threshold:.2f}")
print(f"TP={tp} TN={tn} FP={fp} FN={fn}")
print(f"accuracy={acc:.4f} precision={prec:.4f} recall={rec:.4f} f1={f1:.4f}")
print(f"check total rows: {tp + tn + fp + fn} == {len(example_targets)}")
cm = torch.tensor([[tn, fp], [fn, tp]], dtype=torch.float)
plt.figure(figsize=(4.5, 4.2))
plt.imshow(cm, cmap='Blues')
plt.title('Confusion matrix (threshold = 0.50)')
plt.xticks([0, 1], ['Pred: not survived', 'Pred: survived'], rotation=20)
plt.yticks([0, 1], ['Actual: not survived', 'Actual: survived'])
for i in range(2):
for j in range(2):
plt.text(j, i, int(cm[i, j].item()), ha='center', va='center', color='black')
plt.colorbar(fraction=0.046, pad=0.04)
plt.tight_layout()
plt.show()
Threshold is the cutoff used to convert probability into class label. If probability is above threshold, we predict "survived".
Lower threshold usually increases recall (we catch more true survivors), but also increases false positives. Higher threshold usually increases precision, but also increases false negatives.
threshold_grid = torch.linspace(0.05, 0.95, 19)
precision_vals, recall_vals, f1_vals = [], [], []
for thr in threshold_grid:
preds = preds_from_threshold(example_probs, float(thr.item()))
tp, tn, fp, fn = confusion_counts(preds, example_targets)
_, precision, recall, f1 = metrics_from_counts(tp, tn, fp, fn)
precision_vals.append(precision)
recall_vals.append(recall)
f1_vals.append(f1)
best_idx = int(torch.tensor(f1_vals).argmax().item())
best_threshold = float(threshold_grid[best_idx].item())
best_preds = preds_from_threshold(example_probs, best_threshold)
best_tp, best_tn, best_fp, best_fn = confusion_counts(best_preds, example_targets)
best_acc, best_prec, best_rec, best_f1 = metrics_from_counts(best_tp, best_tn, best_fp, best_fn)
plt.figure(figsize=(7, 4))
plt.plot(threshold_grid, precision_vals, marker='o', label='Precision')
plt.plot(threshold_grid, recall_vals, marker='o', label='Recall')
plt.plot(threshold_grid, f1_vals, marker='o', label='F1')
plt.axvline(0.5, linestyle='--', color='gray', label='Default threshold (0.50)')
plt.axvline(best_threshold, linestyle='--', color='green', label=f'Best F1 threshold ({best_threshold:.2f})')
plt.title('Threshold vs precision, recall, and F1')
plt.xlabel('Threshold')
plt.ylabel('Score')
plt.ylim(0, 1.05)
plt.grid(alpha=0.25)
plt.legend()
plt.show()
print(f"Best threshold by validation F1: {best_threshold:.2f}")
print(f"TP={best_tp} TN={best_tn} FP={best_fp} FN={best_fn}")
print(f"accuracy={best_acc:.4f} precision={best_prec:.4f} recall={best_rec:.4f} f1={best_f1:.4f}")
ROC curve (Receiver Operating Characteristic) shows model behavior across all thresholds. Each point is:
TPR(true positive rate): same idea as recall.FPR(false positive rate): how often we wrongly flag non-survivors as survivors.
AUC (Area Under the ROC Curve) is one summary number from 0 to 1. Closer to 1 means better ranking quality; near 0.5 is close to random ranking.
def roc_points_from_thresholds(probs, targets, thresholds):
fpr_vals, tpr_vals = [], []
for thr in thresholds:
preds = preds_from_threshold(probs, float(thr.item()))
tp, tn, fp, fn = confusion_counts(preds, targets)
tpr = tp / (tp + fn + 1e-9)
fpr = fp / (fp + tn + 1e-9)
fpr_vals.append(fpr)
tpr_vals.append(tpr)
fpr_tensor = torch.tensor(fpr_vals, dtype=torch.float)
tpr_tensor = torch.tensor(tpr_vals, dtype=torch.float)
order = torch.argsort(fpr_tensor)
return fpr_tensor[order], tpr_tensor[order]
def manual_auc_trapezoid(fpr, tpr):
auc_val = torch.trapz(tpr, fpr).item()
return float(max(0.0, min(1.0, auc_val)))
roc_thresholds = torch.linspace(0.0, 1.0, 201)
roc_fpr, roc_tpr = roc_points_from_thresholds(example_probs, example_targets, roc_thresholds)
manual_auc = manual_auc_trapezoid(roc_fpr, roc_tpr)
plt.figure(figsize=(5.5, 5))
plt.plot(roc_fpr, roc_tpr, label=f'Model ROC (AUC={manual_auc:.3f})')
plt.plot([0, 1], [0, 1], '--', color='gray', label='Random baseline')
plt.title('ROC curve (manual thresholds)')
plt.xlabel('False Positive Rate (FPR)')
plt.ylabel('True Positive Rate (TPR)')
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.grid(alpha=0.25)
plt.legend()
plt.show()
print(f"Manual ROC-AUC: {manual_auc:.6f}")
Libraries automate the same math shown above, so in practice we often call a helper instead of coding each step by hand.
try:
from sklearn.metrics import roc_auc_score
sklearn_auc = roc_auc_score(example_targets.int().numpy(), example_probs.numpy())
print(f"Manual ROC-AUC : {manual_auc:.6f}")
print(f"sklearn ROC-AUC: {sklearn_auc:.6f}")
except Exception as e:
print(f"sklearn comparison skipped: {e}")
So now we have a clearer checklist:
- BCE tells us probability-quality error.
- Confusion matrix tells us what kinds of mistakes we make.
- Threshold tuning lets us pick the precision/recall tradeoff we want.
- ROC-AUC tells us ranking quality across all thresholds.