From a prediction to a parameter update
A neural network can have many parameters, but its computation is a sequence of simple maps. Backpropagation is an efficient way to apply the chain rule to that sequence: first calculate the prediction, then propagate the sensitivity of the loss backwards. It computes a gradient at the current parameters; it does not by itself say that a gradient step will reach a global minimum or that the predictor will generalize.
Definition 22.1.1
(Two-layer network with squared loss)
Take a single observation with target . Let , , and . With an activation applied coordinatewise, define
Here is the hidden preactivation, is the hidden output, is a scalar prediction and is the residual.
We treat the observation and target as fixed when differentiating with respect to the parameters . The factor cancels the from differentiating a square.
Gradients and a proof
Theorem 22.1.2
(Two-layer backpropagation)
Suppose is differentiable at every coordinate of . Put and . Then
The last gradient has the same shape as . These formulas hold at the specified parameter point, including when .
Proof
(Follow one small change backwards)
The differential of the loss is . The prediction changes according to
These expressions include changes in all parameters at once. Substituting the last two into the first gives , and therefore
Since the last term equals , the coefficients of each independent parameter change are exactly the displayed partial derivatives: the coefficient of is , which is the entry of . This is the multivariable chain rule, not an approximation.
Proof X-ray
Where each hypothesis enters the backward pass
Differentiability of is used at exactly one place, the link , and only at the current preactivations; nothing is assumed about elsewhere. The other links are linear or quadratic maps, which are differentiable everywhere. The key move is to factor the common quantity out of every path that passes through unit : it appears once in the bias gradient and again, multiplied by , in each weight of row . Computing once and reusing it is what makes the backward pass cheap. The argument closes by reading off coefficients, which is legitimate because the entries of , , and are independent variables. If one parameter were shared between two places, the coefficients of its occurrences would have to be added before reading off its derivative.
The vector records how sensitive the loss is to each preactivation, not to each weight. A hidden unit influences every weight in its row; the input coordinate converts that sensitivity to the weight derivative . Sharing a parameter in a different architecture would require adding all paths through which it affects the loss.
Training usually minimizes an average over observations rather than a single loss, so we also need the gradient of a mean.
Corollary 22.1.3
(Gradient of an empirical mean)
Let be the losses of observations, each differentiable at the parameter point . Then is differentiable at and
Proof
(Linearity of the derivative)
A finite sum of functions differentiable at is differentiable at , and the derivative of is . Take . Each is computed by the theorem with the th observation in place of , at the same parameters.
Complete numerical passes
Worked example 22.1.4
(Two hidden units)
Let , , , , , , and . Then , , , and . Thus . Since ,
For example, increasing by a small amount while keeping everything else fixed changes the loss by . A step with learning rate changes this entry to ; the minus sign in gradient descent matters.
Worked example 22.1.5
(A ReLU unit that is switched off)
Let , , , , , and . Then . Neither coordinate is , so ReLU is differentiable at both, with . Next , , and . Hence
The whole second row of and the derivatives for and vanish: for this observation the second unit is inactive, and small changes of its incoming weights keep negative and . A gradient step therefore leaves that unit unchanged, although its parameters are not optimal in any sense.
Worked example 22.1.6
(Averaging over two observations)
Take , , and , so that and only is varied. The data are and , and the current weight is .
For the first observation and , so : an observation that is already fitted contributes nothing. For the second, , , and . The corollary gives .
Check directly: , so , which equals at .
Where the rule needs care
Counterexample mode
A kink has no ordinary derivative
For the scalar network , target and , the left derivative at is and the right derivative is . Thus has no ordinary derivative there. Assigning a particular value to the derivative of ReLU at zero is an implementation convention, not a proof that the classical gradient exists. Away from zero, the chain rule applies normally.
A zero gradient is also not evidence of a global optimum. With , and , both partial derivatives vanish at , where the loss is ; at the loss is zero. This two-parameter example exhibits the nonconvexity that can arise even without a nonlinear activation.
Common mistake
(Transposing the weight gradient)
Writing produces a matrix, which cannot be subtracted from unless , and then gives the wrong entries. The coefficient of is : the row index belongs to the hidden unit and the column index to the input coordinate, so the gradient is .
Common mistake
(Evaluating the activation derivative at the output)
The factor in is , the derivative at the preactivation, not . In the first worked example, , while the incorrect would double . For ReLU the two happen to agree when , and also when under the common convention , which hides the error until another activation is used.
Practice with solutions
Quick check
(For the two-hidden-unit example, find the derivative with respect to and explain its sign.)
Use and trace the effect of a small increase.
Solution · Answer
It is . A small positive increase raises and hence ; because the output weight is negative, decreases, and because the residual is already negative, grows. The loss therefore increases to first order.
Quick check
(Suppose , , , , , , and . Compute all four scalar parameter derivatives.)
Compute , , and first.
Solution · Answer
Here , , , and . Consequently , , , and .
Quick check
(In the general network, if , must the loss be independent of the whole th column of for this fixed observation?)
Look at which entries of enter .
Solution · Answer
Yes: contains no contribution from that column. Its derivatives are zero wherever the loss is differentiable. This conclusion for one observation does not imply independence for a dataset containing nonzero th coordinates.
Quick check
(For the two observations in the averaging example, at which does the gradient of the mean loss vanish?)
Use the expression for found in the check.
Solution · Answer
, which vanishes at . Since is a convex quadratic in , this stationary point is its global minimizer; the convexity, not the vanishing gradient alone, is what certifies optimality here.