The chain rule, a line at a timeupdatesback-propagatesmultiplies
The derivative tree, as Serban Liviu and Chris Olah draw it: J at the root, the stem z, a running down through the layers, and at each z two wings, W and b, its leaves; x and y hang off it too, but they are data. Beside each node of the stem and under each wing, its key derivative, ∂J/∂(it), lit by its role: updates once stored in .grad, back-propagates while in flight, faded once dropped, dashed while still to come.
Each edge carries a local derivative, ∂(the node above)/∂(the node below), lit once its node has multiplied it in. A local derivative updates nothing: it multiplies. A key derivative is the product of the edges along the path from J, outlined for the one the step makes; the wings at one layer share every edge above them, so backward() makes those products once, on the stem, and hands them down. The dashed box, the same in all three panes, holds what the step multiplies: from the highest key derivative still held down to the node running, one hop, as a line of the chain rule is, unless retain_grad() keeps what was handed down. A click on an edge runs the pass to its node.
The network as a tape of gates, as cs231n draws a circuit, stood on end: J at the top and x at the foot, one wire up the middle. Each layer’s band holds a × gate, the product with its weight matrix, under a + gate, the bias added; then φ, the activation, and at the top the loss. Left of the wire at each level, its tensor’s values for input 1 of the batch of 8; right of it, ∂J/∂ of that tensor, which the gate above hands down: lit as back-propagates while in flight, faded once dropped, dashed while still to come.
W comes in to its × from the right and b to its + from the left, as leaves: the parameter, then its key derivative, ∂J/∂W or ∂J/∂b, filled as updates once AccumulateGrad has stored it in .grad. One rule runs every gate: the upstream gradient times the gate’s local derivative. + copies δ to both addends, so ∂J/∂b is δ, summed over the batch, framed at b’s step with the entry of ∂J/∂b it makes, [1]; × swaps, so ∂J/∂W is δ times the input it multiplied, framed at W’s steps with the entry of ∂J/∂W they make, [1,1], in its .grad, and the input’s is δ times W; φ scales by its slope. Under a value, a teal line while a node holds it saved, faded once it is freed: a layer’s input for its ×, an activation’s output for φ, Wᵀ under W for ×, z and y for the loss. Each node’s gates are framed and numbered as the graph numbers it.
The graph model(x) records, as torch.autograd builds it (Paszke et al. 2017), numbered in the order the engine runs it. nn.Linear runs addmm(b, a, Wᵀ), so each layer leaves an AddmmBackward0, a TBackward0 for Wᵀ, and an AccumulateGrad for each of W and b, which stores ∂J/∂W or ∂J/∂b in .grad. Each chip is the key derivative an edge carries, ∂J/∂z or ∂J/∂a, lit as back-propagates while in flight, faded once dropped, dashed while still to come. A click on a node runs the pass to it.
What the pass costs. Each node of backward() stands for a Jacobian, the local derivative of its output by its input, but never forms it: it multiplies the gradient it is handed straight through, a vector–Jacobian product, by only the entries that are not 0 by structure, so it costs what its forward op cost. The columns are each node’s matrix products in multiply-adds, as torch.utils.flop_counter counts them, two FLOPs each: δᵀa, which makes ∂J/∂W, in indigo under δW, which makes ∂J/∂a, in red. The elementwise work, the loss’s, φ′’s and Σₙ δ’s, a few dozen numbers a node, is left out. Done solid, the node at work outlined, still to come dashed; a click on a column runs the pass there.
At the right, the totals on a log scale: the forward pass, 624 multiply-adds; backward(), 1,056, 1.69 times it, or twice it with x.requires_grad, when layer 1 also works out δW; and the same 93 partial derivatives one at a time, a forward pass each, as forward mode or finite differences take them, or two each, as gradcheck’s central differences in the readings do. Reverse mode costs a small multiple of the forward pass however many parameters there are, the cheap gradient principle (Baur and Strassen 1983). Under the columns, the node at work as the Jacobian it stands for and what it does instead.
View › Chain Rule: the chain rule a line at a time, a line per key derivative, by the node that makes it, numbered as the graph numbers them. Each is the product a line above made, in red by the name that line gave it, δ or ∂J/∂a, handed down, times the one local derivative this line multiplies in, in teal: red times teal gives the line’s own colour, indigo if it updates, red if it is handed on. A red rail points the way: from a dot at the line that made a product down to an arrow at each line that takes it, lit while the product is held, faded once it is dropped, unless retain_grad() keeps it, dashed before it is made. Follow a name up to the line that made it, and on up to J, for the chain in full.
Right of each chain, the product in matrices, a row per input, with its shape, and what the node saved in the forward pass for its one new factor. Under them, what writing every chain out in full from J would cost against what backward() does. Lit once the pass has run its node, outlined while it runs, faded ahead; a click on a line runs the pass to it.
Across the foot, what is held, step by step, a column a node: each tensor a node saved, from the forward pass until the last node to read it has run, then freed unless retain_graph=True; each .grad from its AccumulateGrad on, kept for optimizer.step() until zero_grad(). The gradients in flight, δ and ∂J/∂a, are the pointers above: held one hop, from the line that makes one to the lines that take it. Faded once gone from memory, as a dropped pointer fades, handed back to the allocator (on a GPU, PyTorch’s caching allocator, which keeps it for the next tensor); dashed, still to come. x and y never fade: the graph lets go of them, but your code still holds them. A click on a column runs the pass there.
The roles, so: updates stops at its own layer, in .grad; back-propagates travels down the layers and is dropped once used, unless retain_grad() keeps it; multiplies stays at its node, used once, what the node saved freed after. Every line above follows one rule, layer l in place of the number: δ_l gives ∂J/∂W_l = δ_lᵀa_l−1, the inputs summed, ∂J/∂b_l = Σₙ δ_l, needing no input, and ∂J/∂a_l−1 = δ_lW_l, handed down, which φ′ turns into δ_l−1.
retain_grad() keeps each δ and ∂J/∂a in the .grad of its z or a. x.requires_grad adds ∂J/∂x, the edge x → z₁ of the tree, and an AccumulateGrad for x. retain_graph=True keeps what the nodes saved, which a second backward would need.
J; what the graph holds, the gradients in flight and the .grad written, in bytes as float32 would hold them; and every ∂J/∂W and ∂J/∂b against central differences in float64, the largest gap over the largest gradient.







About
LinkedIn