Do I need to be able to implement backpropagation by hand to work with deep learning?

No — every modern framework (PyTorch, TensorFlow/JAX) computes it automatically via autograd. The practical skill is understanding what it computes and why, well enough to debug training issues, not deriving or implementing the algorithm from scratch.

Is ReLU always the right activation function to reach for?

It is a strong, common default for hidden layers, but not universal — GELU is now more common in transformer architectures, sigmoid remains standard for a binary classification output layer, and softmax is standard for multi-class output. The right choice depends on where in the network the activation sits and what task the network solves.

If more memory is available, should I always use the largest batch size that fits?

Not necessarily — while larger batches give a more accurate gradient estimate and better hardware utilization, some research suggests very large batches can generalize slightly worse than moderate ones, and larger batches typically need a correspondingly larger learning rate to converge at a similar pace. The right batch size is often found empirically for a given problem rather than simply maximized.

Do all deep neural networks need residual connections?

No — they matter most for genuinely deep networks (dozens of layers or more) where vanishing gradients become a real training obstacle. A shallow network (a handful of layers) often trains fine without them; residual connections became standard specifically once architectures started going much deeper.

beginnerintermediateadvanced

AI Full-Stack · Section 9

Deep Learning Fundamentals

Level: intermediate
Read: 65 min
Concepts: 7

The mechanism underneath every neural network, from a single neuron to a training loop that actually converges: how neurons compose into layers and networks around a computational graph; the forward-pass/loss/backpropagation/optimization cycle that repeats every batch; the activation and loss functions that shape what a network can learn and what "good" means for it; the batch/epoch/step arithmetic and gradient accumulation for training at scale; the initialization, normalization and regularization techniques that keep training stable and generalizing; and the vanishing/exploding gradient problem that limits how deep a network can go without them.

What is true here

Without a nonlinearity, any number of stacked layers would still only compute a linear function
Forward pass builds a computational graph; backpropagation walks it backward via the chain rule to get every weight's gradient
Cross-entropy loss specifically penalizes confident wrong predictions far more than uncertain ones
One epoch = dataset_size / batch_size steps; gradient accumulation simulates a larger batch without the memory cost
Vanishing/exploding gradients both come from the chain rule multiplying many per-layer factors together across depth

What you will be able to do

Explain why a network needs a nonlinearity to learn anything beyond a linear function
Walk through the forward → loss → backward → optimizer.step() cycle and what each step computes
Match an activation and loss function to a given task (binary/multi-class classification, regression)
Compute how many steps one epoch takes from dataset size and batch size
Diagnose a training run as likely vanishing or exploding gradients from its symptoms, and name a fix for each

How a neural network actually trains

From a single neuron through the training-step cycle to the stability techniques that make deep networks trainable at all.

Neural network building blocks

standardintermediate

A neuron computes a weighted sum of its inputs, then applies a nonlinear activation function to the result. A layer is a group of neurons applied in parallel. A neural network is layers stacked in sequence. A computational graph is the record of every operation performed, which is what makes automatic differentiation (and therefore training) possible.

Think of it as

Stack these ideas from smallest to largest. A single neuron is a linear operation (a weighted sum plus a bias) followed by a nonlinearity — without that nonlinearity, stacking any number of neurons would still only ever compute a linear function, no more powerful than plain linear regression. A layer runs many neurons on the same input in parallel, each with its own learned weights, so the layer as a whole can learn many different features of the input at once. A network stacks layers sequentially, so later layers combine the features earlier layers extracted — this is what lets a deep network build up from simple patterns (edges, in an image) to complex ones (faces) across its depth. A computational graph is the bookkeeping structure a framework like PyTorch builds automatically as these operations run: every operation is recorded as a node, with edges to whatever it depended on, and that recorded graph is exactly what backpropagation (§9.2) walks backward over to compute every gradient via the chain rule.

python

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 128),   # layer: 784 inputs -> 128 neurons
    nn.ReLU(),              # activation (nonlinearity)
    nn.Linear(128, 10),     # layer: 128 -> 10 neurons
)
# each forward pass builds a computational graph automatically

A real 4-layer fully-connected network, every edge drawn

Remember: A neuron is a weighted sum plus a nonlinearity; without the nonlinearity, stacking layers would still be linear. A layer runs many neurons in parallel; a network stacks layers so later ones combine earlier features. A computational graph records every operation, which backpropagation walks backward over.

Forward pass, loss, and backpropagation

coreintermediate

A forward pass runs input data through a network to produce a prediction. Loss computation compares that prediction to the real answer with a single number. Backpropagation computes how much each individual weight in the network contributed to that loss, using the chain rule. Optimization (§2.5) then nudges every weight to reduce the loss.

Think of it as

This is one repeating cycle, run once per batch, thousands of times over training. The forward pass pushes a batch of inputs through every layer in order, each layer's output feeding the next, ending in a prediction — this is also exactly where the computational graph (§9.1) gets built, recording every operation as it happens. Loss computation compares that prediction to the true label with a loss function (§9.4), producing one number that measures how wrong the network currently is. Backpropagation then runs the chain rule (§2.4) backward through the computational graph the forward pass just built, computing the gradient of the loss with respect to every single weight in the network — not just the last layer's weights, but every layer's, all the way back to the first, each one's gradient computed using the gradients already computed for the layers after it. Optimization then applies the update rule (§2.5) to every weight using its gradient. The entire reason a deep network with millions of parameters can be trained at all is that backpropagation computes all of those gradients in roughly one forward-pass's worth of extra computation, not one gradient computation per parameter.

python

# one training step, the cycle repeated every batch
prediction = model(inputs)            # forward pass
loss = loss_fn(prediction, targets)   # loss computation
loss.backward()                        # backpropagation (fills .grad on every weight)
optimizer.step()                       # optimization: applies the update rule
optimizer.zero_grad()                  # clear gradients before the next forward pass

What we're doing: Trace the training-step cycle across two batches to see how loss decreases as weights update.

training_step_trace.pypython

import torch
import torch.nn as nn

model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.MSELoss()

x = torch.tensor([[2.0]])
y_true = torch.tensor([[10.0]])   # true relationship: y = 4x + 2

for step in range(3):
    prediction = model(x)                    # forward pass
    loss = loss_fn(prediction, y_true)        # loss computation
    optimizer.zero_grad()
    loss.backward()                            # backpropagation
    optimizer.step()                            # optimization
    print(f"step {step}: loss={loss.item():.3f}")

12: The forward pass runs the single input through the model's one linear layer, producing a prediction based on whatever the weights currently are (randomly initialized on the first step).
13: The loss is one number — mean squared error between the prediction and the true target — summarizing how wrong this specific prediction was.
15: This one call computes the gradient of the loss with respect to every parameter in the model, via the chain rule, using the graph the forward pass just built.
16: The optimizer applies the gradient descent update rule to every parameter using the gradient backward() just computed — this is the only line that actually changes the weights.

Why this works: Seeing the four-line cycle — forward, loss, backward, step — run explicitly, rather than hidden inside a `model.fit()` call, is what makes the abstraction "training a neural network" concrete: it is this exact loop, repeated over every batch, for every epoch.

Forgetting to zero gradients before the next backward() call

Wrong

python

for step in range(100):
    prediction = model(x)
    loss = loss_fn(prediction, y_true)
    loss.backward()          # gradients ACCUMULATE, not overwrite
    optimizer.step()
    # optimizer.zero_grad() never called

Better

python

for step in range(100):
    optimizer.zero_grad()     # clear gradients from the previous step
    prediction = model(x)
    loss = loss_fn(prediction, y_true)
    loss.backward()
    optimizer.step()

What you see: Loss decreases erratically or explodes after a few steps, with gradients growing far larger than the loss curve would suggest they should be.

Why: PyTorch accumulates gradients into `.grad` by default on every `backward()` call, rather than replacing them — this is deliberate, to support techniques that need accumulation across several batches, but it means a normal training loop must explicitly zero gradients before each new backward pass, or each step's gradient silently includes every previous step's gradient on top of it.

One real forward pass, one real backward pass

The training-step cycle, one batch at a time

The training-step cycle, one batch at a time
Step	Direction	Produces
Forward pass	Input → output	A prediction, and a computational graph
Loss computation	Prediction vs true label	One number: how wrong the network currently is
Backpropagation	Output → input (backward)	A gradient for every weight in every layer
Optimizer step	Applied to every weight	Updated weights, slightly less wrong next time

Remember: Forward pass: input through every layer → prediction, building a computational graph. Loss: compares prediction to truth with one number. Backpropagation: the chain rule run backward through that graph, computing every weight's gradient in roughly one extra forward pass's worth of work. Optimization then applies the update rule.

Activation functions

standardintermediate

An activation function is the nonlinearity applied after each neuron's weighted sum, and it is what lets a network learn nonlinear patterns at all. ReLU outputs the input directly if positive, zero otherwise. GELU is a smoother version of ReLU. Sigmoid and tanh squash values into a fixed range. Softmax turns a vector of scores into a probability distribution.

Think of it as

Different activations exist because they solve different problems in different parts of a network. ReLU (max(0, x)) is the default for hidden layers in most modern networks — simple, fast to compute, and it avoids a problem sigmoid and tanh have: their gradient shrinks toward zero for very large or very small inputs, which slows or stalls learning in deep networks (an early piece of the vanishing gradient problem, §9.7). ReLU has its own failure mode — a neuron whose input is always negative outputs zero forever and stops learning ('dying ReLU') — which is part of why GELU (a smoother curve, used in most modern transformers including LLMs) is now often preferred over plain ReLU. Sigmoid squashes any input to (0, 1) and is used for binary classification's output layer, where the output needs to be interpreted as a probability. Tanh squashes to (-1, 1) and was common in older recurrent networks. Softmax is different in kind: it turns a whole vector of raw scores into a probability distribution that sums to 1, and is the standard choice for a multi-class classification output layer.

python

import torch.nn as nn

nn.ReLU()      # hidden layers, default
nn.GELU()      # hidden layers, transformers
nn.Sigmoid()   # binary classification output
nn.Softmax(dim=-1)  # multi-class classification output

Five real activation functions, plotted from their actual formulas

Five activations, where each is typically used

Five activations, where each is typically used
Activation	Output range	Typical use
ReLU	[0, ∞)	Hidden layers, the common default
GELU	≈(-0.17, ∞)	Hidden layers in transformers/LLMs
Sigmoid	(0, 1)	Binary classification output layer
Tanh	(-1, 1)	Older recurrent networks
Softmax	(0, 1), sums to 1 across the vector	Multi-class classification output layer

Remember: ReLU is the default hidden-layer activation, fast but can "die"; GELU is the smoother, transformer-standard alternative. Sigmoid outputs one probability (binary); softmax turns a whole vector into a probability distribution (multi-class).

Loss functions, by task

standardintermediate

A loss function turns a prediction and a true target into one number that training tries to minimize. Regression tasks typically use MSE or MAE (§7.1). Classification tasks typically use cross-entropy loss. Generative models often use a loss specific to what they generate, like the token-prediction loss language models train on.

Think of it as

The loss function is the actual objective a network optimizes — everything about what a model learns to prioritize traces back to it, which is why picking the wrong one silently misdirects an entire training run. For regression, MSE and MAE (§7.1) carry over directly, with the same large-error-sensitivity trade-off. For classification, cross-entropy loss compares the model's predicted probability distribution to the true label (as a one-hot distribution), and it specifically penalizes confident wrong predictions much more than uncertain wrong ones — a model that predicts 99% confidence in the wrong class is penalized far more heavily than one that was unsure, which is exactly the incentive that drives models toward calibrated confidence (§7.4) rather than just correct-or-not answers. Generative models train on a loss matched to their generation task — an LLM's pretraining loss is next-token-prediction cross-entropy applied at every position in a sequence, an image generation model's loss measures how well it reconstructs or denoises an image. Across all of these, the shared idea is the same: the loss function IS the definition of 'good' the whole training process is chasing, so it has to actually encode what you want, not just something plausible-sounding.

python

import torch.nn as nn

nn.MSELoss()             # regression
nn.CrossEntropyLoss()    # classification (multi-class)
nn.BCEWithLogitsLoss()   # binary classification

Remember: The loss function is the actual objective training optimizes toward. Regression uses MSE/MAE; classification uses cross-entropy, which specifically penalizes confident wrong predictions. Generative models use a loss matched to what they generate.

Batch size, epochs, steps and gradient accumulation

standardintermediate

A batch is the group of examples processed together in one forward/backward pass. A step is one such pass, updating the weights once. An epoch is one full pass through the entire training dataset. Gradient accumulation lets you simulate a larger batch than fits in memory by summing gradients across several small batches before updating weights.

Think of it as

These terms describe the same training loop from different angles, and the relationship between them is arithmetic: with N training examples and a batch size of B, one epoch takes N/B steps. Batch size trades off memory, speed and gradient quality — a larger batch gives a more accurate estimate of the true gradient (averaged over more examples) and better uses parallel hardware, but needs more memory and, past a point, needs a correspondingly larger learning rate to converge at a similar pace; a smaller batch is noisier per step but that noise sometimes helps escape shallow local minima, and needs much less memory. Gradient accumulation exists specifically for when the batch size you want does not fit in available memory: instead of updating weights after every small batch, gradients are summed across several small batches first, and the optimizer step only happens after enough have accumulated to match the effective batch size you actually wanted — trading extra time for the memory you do not have.

python

steps_per_epoch = len(dataset) // batch_size

# gradient accumulation: simulate batch_size=128 using batches of 32
accumulation_steps = 4
for i, batch in enumerate(dataloader):
    loss = loss_fn(model(batch.x), batch.y) / accumulation_steps
    loss.backward()                      # gradients accumulate
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()                  # update only every 4th batch
        optimizer.zero_grad()

One real loss surface, three real learning rates

Remember: One epoch = dataset_size / batch_size steps. Larger batches: more accurate gradients, more memory, often need a larger learning rate. Gradient accumulation simulates a larger batch by summing gradients across several small ones before updating — trading time for memory.

Initialization, normalization, dropout and regularization

standardintermediate

Initialization sets a network's starting weights before training begins — badly chosen starting values can stall training before it starts. Normalization (like batch normalization) rescales activations inside the network during training, keeping values in a stable range layer to layer. Dropout randomly disables neurons during training. Regularization broadly means any technique that fights overfitting.

Think of it as

These four techniques all exist to make training more stable and to fight overfitting, from different angles. Weight initialization matters more than it looks: weights that start too large cause activations (and gradients) to explode as they pass through many layers; weights that start too small cause them to vanish — both are special cases of the vanishing/exploding gradient problem (§9.7), and modern initialization schemes (like Xavier/He initialization) are specifically designed to keep activation variance roughly stable across layers at the start of training. Normalization does something similar but during training, not just at the start — batch normalization rescales each layer's activations to have a stable mean and variance using statistics from the current batch, which both stabilizes training and, as a side effect, has a mild regularizing effect. Dropout is a more direct regularization technique: randomly zeroing out a fraction of neurons on each training step forces the network to not rely too heavily on any single neuron, which reduces overfitting the same way an ensemble reduces variance — at inference time, dropout is turned off and all neurons are used. Regularization is the umbrella term for all of these plus techniques like L2 weight penalties, which directly penalize large weights in the loss function.

python

import torch.nn as nn

nn.Linear(128, 64)          # PyTorch initializes weights sensibly by default
nn.BatchNorm1d(64)           # normalization layer
nn.Dropout(p=0.3)            # regularization: drop 30% of neurons per step

# L2 regularization is often applied via the optimizer:
optimizer = torch.optim.Adam(model.parameters(), weight_decay=1e-4)

Why initialization scale matters, in real activations

Remember: Good initialization keeps activations stable from step one. Batch normalization keeps them stable throughout training. Dropout forces the network not to over-rely on any one neuron. Regularization is the umbrella term for all techniques, including L2 weight penalties, that fight overfitting.

Vanishing and exploding gradients

standardintermediate

Backpropagation multiplies gradients together layer by layer (the chain rule, §2.4). In a very deep network, if those per-layer factors are consistently smaller than 1, the product shrinks toward zero by the time it reaches early layers — the vanishing gradient problem. If they are consistently larger than 1, the product grows explosively — the exploding gradient problem.

Think of it as

Both problems come from the same mechanism: the chain rule computes a gradient as a product of many per-layer terms, and repeated multiplication is unforgiving — a factor of 0.9 multiplied across 50 layers shrinks to roughly 0.005, effectively zero; a factor of 1.1 across 50 layers grows to roughly 117, effectively exploding. Vanishing gradients mean early layers barely update at all, because the gradient reaching them has shrunk to nearly nothing — the network effectively stops learning in its earlier layers while later layers keep training, which is part of why sigmoid/tanh (whose derivatives are always less than 1, and much less than 1 for large inputs) fell out of favor as hidden-layer activations in deep networks, replaced by ReLU-family activations whose derivative is exactly 1 for positive inputs. Exploding gradients show up as loss suddenly spiking to a huge number or becoming NaN, and are commonly fixed with gradient clipping — capping the gradient's magnitude before the optimizer step, so one large gradient cannot destroy the weights in a single update. Residual connections (skip connections that let a gradient bypass a layer entirely) are the architectural fix that made training very deep networks (dozens to hundreds of layers) practical in the first place.

python

# gradient clipping, applied right before the optimizer step
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

# residual connection, conceptually:
# output = layer(x) + x     <- gradient can flow through the "+ x" path
#                               even if layer's own gradient vanishes

Why gradients vanish: the same real chain rule, two activations

Reading the symptom back to the fix

Reading the symptom back to the fix
Symptom	Likely problem	Common fix
Early layers' weights barely change during training	Vanishing gradients	ReLU-family activations, residual connections, better initialization
Loss spikes suddenly or becomes NaN	Exploding gradients	Gradient clipping, lower learning rate
Deep network (50+ layers) trains worse than a shallower version	Vanishing gradients limiting effective depth	Residual/skip connections

Remember: Vanishing/exploding gradients both come from the chain rule multiplying many per-layer factors together. Vanishing: early layers stop learning (fixed by ReLU-family activations, residual connections, better init). Exploding: loss spikes or NaNs (fixed by gradient clipping).