How does a neural net really work¶
In this notebook I'm exploring fast.ai's Kaggle notebook on "How does a neural net really work". This relates to Lesson 3 and Lesson 5 of the fast.ai Deep Learning course. While the video provides a solid explanation, the enigmatic imports and variables can be difficult to comprehend. I'm reimplementing some sections to see if if sticks. In a nutshell, this is what is happening in this notebook:
- Revising Regressions
- Plot a generic quadratic function ($ax^2 + bx + c$)
- Generate some random data points
- Learn the step-by-step process to find the values of
a,b, andcthat make our function represent the random data generated in2 - Use the mean absolute error to manually adjust
a,b, andc.
- Understand and break down the Gradient Descent algorithm
- The Basics of a Neural-Network using the Titanic Survival dataset form Kaggle
- Explore ReLUs and how it differs from a simpel linear function
- Build a single-layer neural network using a simple linear function $f(x) = m*x$ (m being a array of weights that we multiply by our features)
- Do Deep Learning by layering coefficients/wheights/neurons to do a multi-layer neural network
# Installing the dependencies within the notebook to make it easier to run on colab
%pip install -Uqq fastai==2.7.18 ipywidgets==8.1.5 plotly==5.24.1 datasets==3.3.2
1. Revising Regressions¶
This section, from the fast.ai course, sets the stage for understanding how neural networks learn "weights". We'll plot some points on a graphic and use visualizations to see how changing the coefficients affects the function to better fit the points.
1.1 Plot a generic quadratic function ($ax^2+bx+c$)¶
from fastai.basics import torch, plt
import numpy as np, pandas as pd
# Make pandas and numpy use the entire screan
np.set_printoptions(linewidth=140)
torch.set_printoptions(linewidth=140, sci_mode=False, edgeitems=7)
pd.set_option('display.width', 140)
# Set the figure DPI to 90 for better resolution
plt.rc('figure', dpi=90)
# Function to plot a mathematical function over a range
def plot_function(f, title=None, min=-2.1, max=2.1, color='r', ylim=None):
# Create evenly spaced x values as a column vector
x = torch.linspace(min,max, 100)[:,None]
# Set y-axis limits if specified
if ylim: plt.ylim(ylim)
# Plot the function
plt.plot(x, f(x), color)
# Add title if provided
if title is not None: plt.title(title)
# Function with quadratic expression ax^2 + bx + c
def quad(a, b, c, x): return a*x**2 + b*x + c
from functools import partial
# Creates a new function with fixed a,b,c parameters, leaving only x variable
# This allows us to create specific quadratic functions by "fixing" the coefficients
def mk_quad(a,b,c): return partial(quad, a,b,c)
def demo_plot_basic_quadratic():
a = 3
b = 2
c = 1
f = mk_quad(a, b ,c)
plot_function(f, title=f'${a}x^2 + {b}x + {c}$')
demo_plot_basic_quadratic()
1.2. Generate some random data points¶
# Add both multiplicative and additive noise to input x
def add_noise(x, mult, add): return x * (1+torch.randn(x.shape) * mult) + torch.randn(x.shape) * add
def generate_noisy_data(f, x_start=-2, x_end=2, num_datapoints=20, noise_mult=0.15, noise_add=0.15, seed=42):
# Define a static seed, so that the random data is always the same every time we run this
torch.manual_seed(seed)
# Create evenly spaced x values and add a dimension of size 1 at position 1,
# transforming shape from (20,) to (20,1) to make it a column vector
# Example: tensor([1,2,3]) shape=(3,) -> tensor([[1],[2],[3]]) shape=(3,1)
x = torch.linspace(x_start, x_end, steps=num_datapoints).unsqueeze(1)
# Generate noisy y values by applying noise to function output
# mult=0.15 for multiplicative noise, add=1.5 for additive noise
y = add_noise(f(x), noise_mult, noise_add)
return x, y
def demp_plot_random_data():
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y);
demp_plot_random_data()
1.3. Fit the function to the data¶
In this section, we will explore the step-by-step process to find the values of a, b, and c that allow our function to accurately reflect the random data generated in 1.2. The interactive plot below, shows how adjustments to a, b, and c influence the function's shape to better align with our data layout.
from ipywidgets import interact
@interact(a=1.5, b=1.5, c=1.5)
def demo_interactive_plot_quad(a, b, c):
plt.close('all') # Close all existing figures
plt.figure() # Create a new figure
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y)
plot_function(mk_quad(a,b,c), ylim=(0,13))
1.4 Measure the error¶
This is cool and works, but we need to know how close we are to our ideal solution. In regression, we can use fun ways to estimate this, like the "Mean Absolute Error," which averages the distance between predicted and actual values.
The fastai library has a wrapper for some of the most common methods (from scikit-learn).
Here's a quick demo where we can see how it is calculated.
def mean_absolute_error(preds, acts): return (torch.abs(preds-acts)).mean()
def demo_mean_absolute_error():
# Create some example predictions and actual values
preds = torch.tensor([1.0, 2.0, 3.0, 4.0])
actuals = torch.tensor([1.1, 2.1, 2.8, 4.2])
# Calculate and print the mean absolute error
error = mean_absolute_error(preds, actuals)
print(f"Mean Absolute Error: {error:.3f}")
# Let's break down what's happening:
print("\nAbsolute differences between predictions and actuals:")
abs_diffs = torch.abs(preds-actuals)
for i, (p, a, d) in enumerate(zip(preds, actuals, abs_diffs)):
print(f"Prediction: {p:.1f}, Actual: {a:.1f}, Absolute Difference: {d:.3f}")
demo_mean_absolute_error()
@interact(a=1.5, b=1.5, c=1.5)
def plot_quad(a, b, c):
x,y = generate_noisy_data(mk_quad(3,2,1))
f = mk_quad(a, b ,c)
plt.scatter(x,y)
loss = mean_absolute_error(f(x), y)
plot_function(f, ylim=(0,13), title=f"MAE: {loss:.2f}")
2. Understand and break down the Gradient Descent algorithm¶
Now that we can calculate the mean absolute error, the next step is to understand how to adjust our parameters a, b, and c to reduce this error. To do this, we can think about the gradients of the error with respect to each of the a,b,c parameters.
👉 If you were walking on a hill (representing the error surface), the partial derivative with respect to one direction (say, the 'a' direction) tells you the slope of the hill in that specific direction. A steep slope/gradient means a large change in error for a small change in 'a'.
For example, if we consider the partial derivative of the mean absolute error with respect to a (while keeping b and c fixed), a negative value would indicate that increasing a will lead to a decrease in the error (like walking forward-downhill in the 'a' direction). Conversely, a positive value would suggest that decreasing a would reduce the error (ackwardly walking backwards-downhill in the 'a' direction 😄).
Using AI, we can plot this hill. The following plot shows the "error surface" (MAE) for the function $f(x) = m*x + b$. By fixing one variable (e.g., b=2), we can visualize the differentiated function in terms of m and determine whether to increase or decrease m to move "downhill."
import plotly
plotly.offline.init_notebook_mode(connected=True)
import plotly.graph_objects as go
def demo_mae_surface():
# Actual data points
x_actual = torch.tensor([1.0, 2.0])
y_actual = torch.tensor([2.0, 4.0])
# Range of m and b values to explore
m_vals = torch.linspace(-1, 5, 100) # Explore m from -1 to 5
b_vals = torch.linspace(-1, 5, 100) # Explore b from -1 to 5
# Create a meshgrid of m and b values
M, B = torch.meshgrid(m_vals, b_vals, indexing='ij')
# Initialize an array to store MAE values for the surface plot
mae_values_surface = torch.zeros_like(M)
# Calculate MAE for each combination of m and b for the surface plot
for i in range(M.shape[0]):
for j in range(M.shape[1]):
m = M[i, j]
b = B[i, j]
preds = m * x_actual + b
mae = mean_absolute_error(preds, y_actual)
mae_values_surface[i, j] = mae.item() # Store the scalar value
# --- Calculate MAE for fixed b to show gradient of m ---
fixed_b = 2.0 # Fixed b value
mae_values_fixed_b = []
for m in m_vals:
preds = m * x_actual + fixed_b
mae = mean_absolute_error(preds, y_actual)
mae_values_fixed_b.append(mae.item())
# Create the surface plot using Plotly
fig = go.Figure(data=[go.Surface(z=mae_values_surface.numpy(), x=b_vals.numpy(), y=m_vals.numpy(), colorscale='RdBu', name='MAE Surface')])
# Add the line plot for fixed b, offsetting z values slightly
z_offset_line = 0.1 # Offset to lift the line above the surface
fig.add_trace(go.Scatter3d(
x=[fixed_b] * len(m_vals), # Fixed b value for all m values
y=m_vals.numpy(),
z=[z + z_offset_line for z in mae_values_fixed_b], # Add offset to z values
mode='lines',
line=dict(color='green', width=4),
name=f'MAE with fixed b={fixed_b}'
))
# Add annotation - Adjusted for better readability, offsetting z value slightly
annotation_m = 2.0
annotation_mae = mean_absolute_error(annotation_m * x_actual + fixed_b, y_actual).item()
z_offset_point = 0.1 # Offset to lift the point above the surface
fig.add_trace(go.Scatter3d(
x=[fixed_b],
y=[annotation_m],
z=[annotation_mae + z_offset_point], # Add offset to z value
mode='markers+text',
marker=dict(size=8, color='red'),
text=["At m=2, decrease m to go dowhill"], # Shortened text
textposition="top center", # Changed text position to top center
textfont=dict(size=10) # Reduced text size
))
fig.update_layout(
title='3D Error Surface: Mean Absolute Error (MAE) for f(x) = mx + b<br>with line showing gradient of m at fixed b=2', # Added <br> for title wrapping
scene=dict(
xaxis_title='b (y-intercept)',
yaxis_title='m (slope)',
zaxis_title='MAE',
camera=dict(eye=dict(x=1.5, y=-1.5, z=0.8)) # Rotate perspective
),
width=700,
height=700
)
fig.show()
demo_mae_surface()
With PyTorch, we can use require_grad=True on tensors, which automatically handles differentiation and the calculation of the gradients for us. To be honest, it looks a bit like dark magic. Let's look at some examples where we can see this functionality from PyTorch being applied to our function $f(x) = m*x + b$, along with two different inputs (x1 and x2)🕵️♂️.
#########################################
# Example 1: Using x1 = [1.0, 2.0, 3.0]
#########################################
x1 = torch.tensor([1.0, 2.0, 3.0])
m1 = torch.tensor(2.0, requires_grad=True)
b1 = torch.tensor(1.0, requires_grad=True)
# Compute f(x1) = m1 * x1 + b1
y1 = m1 * x1 + b1
total1 = y1.sum() # total1 = m1*(1+2+3) + b1*3 = 6*m1 + 3*b1
# Compute gradients for m1 and b1.
total1.backward()
print(f'Example 1 | x: {x1}, m: {m1}, b: {b1}')
print("Example 1 | Gradient with respect to m1 (m1.grad):", m1.grad) # Expected: sum(x1) = 6.0
print("Example 1 | Gradient with respect to b1 (b1.grad):", b1.grad) # Expected: len(x1) = 3
print("--------------")
#########################################
# Example 2: Using x2 = [1.0, 4.0] (different size and values)
#########################################
x2 = torch.tensor([1.0, 4.0])
m2 = torch.tensor(2.0, requires_grad=True)
b2 = torch.tensor(1.0, requires_grad=True)
# Compute f(x2) = m2 * x2 + b2
y2 = m2 * x2 + b2
total2 = y2.sum() # total2 = m2*(1+4) + b2*2 = 5*m2 + 2*b2
# Compute gradients for m2 and b2.
total2.backward()
print(f'Example 2 | x: {x2}, m: {m2}, b: {b2}')
print("Example 2 | Gradient with respect to m2 (m2.grad):", m2.grad) # Expected: sum(x2) = 5.0
print("Example 2 | Gradient with respect to b2 (b2.grad):", b2.grad) # Expected: len(x2) = 2
print("--------------")
# Print an explanation that details the differentiation steps.
explanation = """
Explanation:
- **Example 1:**
- We start with a list of numbers: x1 = [1.0, 2.0, 3.0].
- We plug each number into our function, which means we multiply it by m (let's say m = 2) and then add b (let's say b = 1):
- For 1.0: f(1.0) = 2 * 1 + 1 = 3
- For 2.0: f(2.0) = 2 * 2 + 1 = 5
- For 3.0: f(3.0) = 2 * 3 + 1 = 7
- So, we end up with y1 = [3.0, 5.0, 7.0].
- Then, we add these numbers together: 3 + 5 + 7 = 15.
- If we look at our equation, we can say: total1 = m1*(1.0+2.0+3.0) + b1*3 = 6 * m1 + 3 * b1.
- Now, when we want to see how changing m1 and b1 affects total1, we find:
- Changing m1 gives us a "gradient" of 6.
- Changing b1 gives us a "gradient" of 3.
- So, m1.grad = 6.0 and b1.grad = 3.
- **Example 2:**
- Now we have a different list: x2 = [1.0, 4.0].
- We use the same m and b:
- For 1.0: f(1.0) = 3 (same as before).
- For 4.0: f(4.0) = 2 * 4 + 1 = 9.
- So, now we have y2 = [3.0, 9.0].
- We add these: 3 + 9 = 12.
- In terms of our equation, total2 = m2*(1.0+4.0) + b2*2 = 5 * m2 + 2 * b2.
- For the gradients:
- Changing m2 gives us a "gradient" of 5.
- Changing b2 gives us a "gradient" of 2.
- So, m2.grad = 5.0 and b2.grad = 2.
"""
print(explanation)
Now we can create an interactive plot where we show the gradient on a, b and c.
If the slope is negative we want to move forward (or downhill).
@interact(a=1.5, b=1.5, c=1.5)
def demo_quadratic_plot_with_gradients(a, b, c):
x,y = generate_noisy_data(mk_quad(3,2,1))
plt.scatter(x,y)
a_tensor = torch.tensor([float(a)], requires_grad=True)
b_tensor = torch.tensor([float(b)], requires_grad=True)
c_tensor = torch.tensor([float(c)], requires_grad=True)
f = mk_quad(a_tensor, b_tensor, c_tensor)
loss = torch.abs(f(x) - y).mean()
loss.backward()
a_grad = a_tensor.grad.item()
b_grad = b_tensor.grad.item()
c_grad = c_tensor.grad.item()
plot_function(lambda x: f(x).detach(), ylim=(0,13), title=f"MAE: {loss:.2f}, dLoss/da: {a_grad:.2f}, dLoss/db: {b_grad:.2f}, dLoss/dc: {c_grad:.2f}")
from fastai.metrics import mae
def demo_auto_fit(steps=20):
x, y = generate_noisy_data(mk_quad(3,2,1))
abc = torch.tensor([1.0,1.0,1.0], requires_grad=True)
min_loss = float('inf')
best_abc = abc.clone().detach() # Initialize best_abc with the initial abc
for i in range(steps):
f = mk_quad(*abc)
loss = mae(f(x), y)
loss.backward()
with torch.no_grad():
abc -= abc.grad*0.1
abc.grad.zero_() # Clear gradients after update
print(f'step={i}; loss={loss.item():.2f}; abc={abc}')
if loss < min_loss:
min_loss = loss
best_abc = abc.clone().detach() # Update best_abc when a lower loss is found
return best_abc
best_abc_params = demo_auto_fit()
print(f"Best abc parameters: {best_abc_params}")
3. The Basics of a Neural-Network¶
3.1 Introducing Non-Linearity with ReLU¶
We've seen that simple functions like quadratics can model some data, but real-world data is rarely so straightforward. Imagine trying to predict something complex, like whether a picture is a cat or a dog, based on many pixel values (our 'dimensions'). A simple quadratic or even a single linear function just won't be flexible enough to capture the intricate patterns in such high-dimensional data.
To handle this complexity, we need to build more powerful functions. Simply combining linear functions won't solve the problem because any combination of linear functions is still just a linear function! Linear functions can only model linear relationships in the data. Real-world data, like images of cats and dogs, is highly non-linear.
To introduce non-linearity, we use activation functions. ReLU (Rectified Linear Unit) is a simple yet powerful activation function that introduces non-linearity. By applying ReLU to the output of linear functions, we can create models that can learn complex, non-linear patterns in the data. This non-linearity is what allows neural networks to model intricate relationships that simple linear models cannot. This will lead us to the idea of a ReLU, a simple activation function, and the simplest "neural network" we can build with it.
def rectified_linear(m,b,x):
y = m*x+b
return torch.clip(y, 0.)
plot_function(partial(rectified_linear, 1,1))
Combining two ReLUs allows us to create more complex, piecewise linear functions, as illustrated in the interactive plot below. This combination increases the flexibility of our model, enabling it to capture more intricate relationships in the data.
def double_relu(m1,b1,m2,b2,x):
return rectified_linear(m1,b1,x) + rectified_linear(m2,b2,x)
@interact(m1=-1.5, b1=-1.5, m2=1.5, b2=1.5)
def plot_double_relu(m1, b1, m2, b2):
plot_function(partial(double_relu, m1,b1,m2,b2), ylim=(-1,6))
3.2 Building a Neural Network from-Scratch¶
From this point forward, we will be following this notebook: Linear model and neural net from scratch.
Important ⚠️: For simplicity, I'm skipping all the steps that involve data cleanup and preparation. This means of course that my model will most likely not have a very good performance.
We are using the Titanic competition from Kaggle. I have made a copy in my Hugging Face workspace, which tbh I did to to experiment how Datasets work on Hugging Face.
The goal is to create a model to predict whether a passenger Survived, which is provided in our dataset.
In essence, we will now combine functions like those we've explored above, such as ReLUs, to construct a simple neural network. This network will receive passenger features as input, apply weights (similar to $m$ in our previous examples), and hopefully predict whether the passenger Survived with the lowest possible loss/error.
from datasets import load_dataset
# Load only train and test splits.
dataset = load_dataset("paulopontesm/titanic", data_files={"train": "train.csv", "test": "test.csv"})
# Access the splits
train_dataset_df = dataset["train"].to_pandas()
test_dataset_df = dataset["test"].to_pandas()
train_dataset_df
Since we need numerical data for our model, we'll just use the columns that already contain numbers as predictors. These are the columns that are already numerical.
import numpy as np
train_dataset_df.describe(include=(np.number))
Now that we have numbers for the features, we can create tensors/arrays for our features (aka called independent variables) and target (aka dependent variable).
Even thought I mentioned above that I didn't want to do a lot of data transofrmations, I think we really need to remove the NaNs and to normalize the numbers.
from torch import tensor
# And one for the target. Also known as dependent variables or outputs
t_dep = tensor(train_dataset_df.Survived)
indep_cols = ['Age', 'SibSp', 'Parch', 'Fare']
# We need to do 2 things before proceeding so that we can use our data.
# 1. Replace all the nans by the mode of that column
for col in indep_cols:
mode_val = train_dataset_df[col].mode()[0]
train_dataset_df[col] = train_dataset_df[col].fillna(mode_val)
# 2. to prevent one column from dominating all the others, by making each row range between 0 and 1.
# We can do this by dividing each entry by
# the max value on that column
for col in indep_cols:
max_val = train_dataset_df[col].max()
train_dataset_df[col] = train_dataset_df[col] / max_val
# Create a tensor with our predictors. Also known as independent variables, features, or inputs.
t_indep = tensor(train_dataset_df[indep_cols].values, dtype=torch.float)
t_indep
Looks good (?)...
Because we want to calculate accuracy later, for fun, let's also keep a chunk of the training data for this. This is called a validation set.
Note: This other notebook explains the difference between the validation set and the test set. They seem similar, but looks like it's important not to confuse these two concepts. https://www.kaggle.com/code/jhoward/getting-started-with-nlp-for-absolute-beginners#Test-and-validation-sets
from fastai.data.transforms import RandomSplitter
trn_split,val_split=RandomSplitter(seed=42)(train_dataset_df)
train_set_features,validation_set_features = t_indep[trn_split],t_indep[val_split]
train_set_targets,validation_set_targets = t_dep[trn_split],t_dep[val_split]
len(train_set_features),len(validation_set_features)
Now we can generate random weights (ms) for each of our features. We're using a linear model, effectively calculating a weighted sum of the features: $f(x) = m_{Age}*x_{Age} + m_{SibSp}*x_{SibSp} + m_{Parch}*x_{Parch} + m_{Fare}*x_{Fare}$. We will adjust these weights to predict passenger survival based on the features.
def generate_random_coefficients(num_coeffs):
torch.manual_seed(42)
coeffs = torch.rand(num_coeffs)-0.5 # pick random numbers in the range (-0.5,0.5)
return coeffs.requires_grad_()
nn_coeffs=generate_random_coefficients(num_coeffs=train_set_features.shape[1])
nn_coeffs
def calc_preds(coeffs, features): return (features*coeffs).sum(axis=1)
predictions = calc_preds(nn_coeffs, train_set_features)
predictions.topk(3)
def calc_loss(coeffs, features, targets): return torch.abs(calc_preds(coeffs, features)-targets).mean()
loss = calc_loss(coeffs=nn_coeffs, features=train_set_features, targets=train_set_targets)
loss
loss.backward()
nn_coeffs.grad
def one_epoch(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = calc_loss(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
coeffs.sub_(coeffs.grad * lr)
coeffs.grad.zero_()
print(f"{loss:.3f}", end="; ")
def train_model(train_set_features_set, train_set_targets_set, epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients(num_coeffs=t_indep.shape[1])
for i in range(epochs): one_epoch(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs
final_weights = train_model(train_set_features, train_set_targets)
def show_coeffs(coeffs): return dict(zip(indep_cols, coeffs.requires_grad_(False)))
show_coeffs(nn_coeffs)
We have weights, let's do predictions then.
calc_preds(nn_coeffs, validation_set_features)
It's hard not to notice that we should be predicitng a 0 or 1 value, but instead we're getting a lot of negatives.
For simplicity, let's ignore this and say that everything above 0.5 survived.
preds = calc_preds(nn_coeffs, validation_set_features)
print(f"True count was {torch.sum(preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
With this we can use our validation set and calculate the % of predicitons that we are getting correctly.
def calc_accuracy(predictions, validation_set_features, validation_set_targets):
# Convert predictions to boolean values (True if > 0.5, False otherwise)
bool_predictions = predictions > 0.5
# Convert validation dependent variable to boolean values
bool_validation_set_targets = validation_set_targets.bool()
# Compare boolean predictions with boolean validation dependent variable to find correct predictions
correct_predictions = bool_validation_set_targets == bool_predictions
# Convert correct predictions (boolean) to float (1.0 for True, 0.0 for False)
accuracy_float = correct_predictions.float()
# Calculate the mean of the accuracy_float to get the overall accuracy
accuracy_val = accuracy_float.mean()
return accuracy_val
accuracy_result = calc_accuracy(predictions=preds, validation_set_features=validation_set_features, validation_set_targets=validation_set_targets)
print(accuracy_result)
Looks like we're doing slightly better than throwing a coin. I say this is a success 🤔❓🤔 I don't think so...
As seen in the counts above ("True count was 2 should have been 72", "False count was 176 should have been 106"), the model predicts most instances as False (not survived). It correctly identifies many of the actual 'not survived' cases, contributing to the accuracy, but incorrectly classifies most 'survived' cases as 'not survived'. This bias results in an accuracy that is better than random (50%), but not very high, around 58%.
The Linear model and neural net from scratch goes much further on this exercise. It uses other techniques to clean-up and normalize the data, it also uses the non-numerical by transforming them to numericals, and then uses a sigmoid that differently form our linear function, always give a value between 0 and 1.
For me, this was enought to get an better overview of what's happening inside a neural network.
3.3 Do Deep Learning¶
In the section above we implemented a simple Neural Network. Now let's explore Deep Learning, which is what truly unlocks the power of Neural Networks.
Deep Learning involves creating Neural Networks with multiple layers. Instead of a single layer, we stack layers of neurons, allowing the network to learn more complex patterns and representations from the data.
def generate_random_coefficients_for_deep_learning(n_coeff, num_neurons_per_hidden_layer=[10, 10]):
torch.manual_seed(42)
# Define the number of neurons for each layer, including input, hidden, and output layers.
# The input layer size is n_coeff, hidden layers sizes are from num_neurons_per_hidden_layer, and output layer size is 1.
num_neurons = [n_coeff] + num_neurons_per_hidden_layer + [1]
layers = []
for i in range(len(num_neurons)-1):
# Determine the size of the input for the current layer from the previous layer's neuron count
layer_input_size = num_neurons[i]
# Determine the size of the output for the current layer from the current layer's neuron count
layer_output_size = num_neurons[i+1]
# Initialize a layer with random weights between -0.5 and 0.5.
# torch.rand generates uniform random numbers between 0 and 1, then we shift and scale to get range [-0.5, 0.5].
# requires_grad_() is set to True to enable gradient tracking for these tensors, which is needed for backpropagation.
layer = (torch.rand(layer_input_size, layer_output_size)-0.5).requires_grad_()
layers.append(layer)
return layers
dnn_layers_coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features.shape[1], num_neurons_per_hidden_layer=[10, 10])
dnn_layers_coeffs
We can test how we do without any training
def calc_preds_for_deep_learning(coeffs, features):
# @ is matrix multiplication in Python
# It was introduced in Python 3.5 as part of [PEP 465](https://peps.python.org/pep-0465/)
layer_features = features
for layer in coeffs[:-1]:
layer_features = layer_features @ layer
layer_features = layer_features @ coeffs[-1]
return layer_features.squeeze()
def calc_loss_for_deep_learning(coeffs, features, targets): return torch.abs(calc_preds_for_deep_learning(coeffs, features)-targets).mean()
dnn_preds = calc_preds_for_deep_learning(coeffs=dnn_layers_coeffs, features=validation_set_features)
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
And we need to do the grandient descent for all the coeffs on each layer.
def one_epoch_for_deep_learning(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = calc_loss_for_deep_learning(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
for layer in coeffs:
layer -= layer.grad * lr
layer.grad.zero_()
def train_model_for_deep_learning(train_set_features_set, train_set_targets_set, num_neurons_per_hidden_layer=[10, 10], epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features_set.shape[1], num_neurons_per_hidden_layer=num_neurons_per_hidden_layer)
for i in range(epochs): one_epoch_for_deep_learning(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs # Returns the trained coefficients, which have the same structure as generate_random_coefficients_for_deep_learning
Let's test it then with different combinations of hidden layers and neurons per layer...
for num_neurons in [[10, 10], [20, 20],[5, 5, 5],[30], [], [2], [50], [2, 2], [50, 50], [5, 10, 5], [2, 2, 2, 2]]:
dnn_final_weights = train_model_for_deep_learning(train_set_features, train_set_targets, num_neurons_per_hidden_layer=num_neurons)
dnn_preds = calc_preds_for_deep_learning(coeffs=dnn_final_weights, features=validation_set_features)
accuracy = calc_accuracy(predictions=dnn_preds, validation_set_features=validation_set_features, validation_set_targets=validation_set_targets)
print(f"Hidden layers: {num_neurons}")
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
print(f"Accuracy: {accuracy}")
print("-" * 20) # Separator for readability
Not a lot has changed...
Just for fun, we can see how adding a sigmoid and ReLU would affect the results... The code is Ctrl+V Ctrl+C from above, but with a smarter_calc_preds_for_deep_learning.
import torch.nn.functional as F
def smarter_calc_preds_for_deep_learning(coeffs, features):
# @ is matrix multiplication in Python
# It was introduced in Python 3.5 as part of [PEP 465](https://peps.python.org/pep-0465/)
layer_features = features
for layer in coeffs[:-1]:
layer_features = F.relu(layer_features @ layer)
layer_features = layer_features @ coeffs[-1]
return torch.sigmoid(layer_features.squeeze())
def smarter_calc_loss_for_deep_learning(coeffs, features, targets):
predictions = smarter_calc_preds_for_deep_learning(coeffs, features)
return F.binary_cross_entropy(predictions, targets) # Changed loss to Binary Cross Entropy
def smarter_one_epoch_for_deep_learning(coeffs, lr, train_set_features_set, train_set_targets_set):
loss = smarter_calc_loss_for_deep_learning(coeffs, train_set_features_set, train_set_targets_set)
loss.backward()
with torch.no_grad():
for layer in coeffs:
layer -= layer.grad * lr
layer.grad.zero_()
def smarter_train_model_for_deep_learning(train_set_features_set, train_set_targets_set, num_neurons_per_hidden_layer=[10, 10], epochs=60, lr=4):
torch.manual_seed(442)
coeffs = generate_random_coefficients_for_deep_learning(n_coeff=train_set_features_set.shape[1], num_neurons_per_hidden_layer=num_neurons_per_hidden_layer)
for i in range(epochs): smarter_one_epoch_for_deep_learning(coeffs, lr=lr, train_set_features_set=train_set_features_set, train_set_targets_set=train_set_targets_set)
return coeffs # Returns the trained coefficients, which have the same structure as generate_random_coefficients_for_deep_learning
for num_neurons in [[10, 10], [20, 20],[5, 5, 5],[30], [], [2], [50], [2, 2], [50, 50], [5, 10, 5], [2, 2, 2, 2]]:
dnn_final_weights = smarter_train_model_for_deep_learning(train_set_features, train_set_targets.float(), num_neurons_per_hidden_layer=num_neurons)
dnn_preds = smarter_calc_preds_for_deep_learning(coeffs=dnn_final_weights, features=validation_set_features)
accuracy = calc_accuracy(predictions=dnn_preds, validation_set_features=validation_set_features, validation_set_targets=validation_set_targets)
print(f"Hidden layers: {num_neurons}")
print(f"True count was {torch.sum(dnn_preds>0.5)} should have been {torch.sum(validation_set_targets.bool())}")
print(f"False count was {torch.sum(dnn_preds<=0.5)} should have been {len( validation_set_targets.bool()) - torch.sum(validation_set_targets.bool())}")
print(f"Accuracy: {accuracy}")
print("-" * 20) # Separator for readability
Interesting. That definitely improved.
I will stop here for now. However, the next step will likely be to add the boolean variables like is_male, is_female, is_class_1, etc. If I understood that correctly and I'm not making any mistakes, it should bring me to around 80% accuracy, like we see on the fast.ai notebook.