L = ½((x − 3)² + 150y²). Momentum’s β moves its edge, 2(1 + β)/150 (Goh, 2017): tuned, β = 0, plain SGD, is under 10⁻³ by step 437, 0.5 by 214, 0.9 by 82, 0.95 by 50, and 0.99 still rings at step 600. RMSProp, AdamW and Muon step about η a step, so one η may reach the floor or settle on it, not both: at a constant η Muon ends ringing 4×10⁻³ above the floor, and with cosine decay 3×10⁻¹² above. AdamW’s weight decay holds it 2×10⁻⁵ above, cosine or not, where its step balances wd·θ rather than where the gradient is 0 (Loshchilov and Hutter, 2019); at wd = 0 it gets under 10⁻¹². RMSProp with Hinton’s α = 0.9 never gets under 10⁻³; with PyTorch’s 0.99, by step 220.

A sharp dip, the deeper, beside a wide one. From the start shown every run settles in the sharp dip, SGD tuned to 0.02 there by step 11, AdamW to 0.1 by 27. From an init the draw decides, not the knobs: the sweep is flat at the wide dip’s 0.106, where 14 of 16 Xavier seeds end at η = 0.01 whichever optimiser runs them. SGD sits still at a dip’s bottom only if η < 2/λ, λ its curvature, 0.045 for the sharp dip and 1 for the wide: at 0.1 noise σ = 1 throws it into the wide one, one account of why large steps and small batches find flat minima (Keskar et al., 2017; Cohen et al., 2021).

The smallest deep network, x to one hidden unit to the output, all linear, fit to y = x: L = ½(ab − 1)², a saddle at 0, lowest all along the dashed ab = 1. From init 0 nothing moves at any η. From σ = 0.01 SGD waits by the saddle about ln(1/|θ₀|)/η steps (Saxe et al., 2014): at 0.01, 3 of 16 seeds are under 10⁻³ by step 600, tuned to 0.5 all by 14. RMSProp divides the tiny gradient by its own size and leaves at once: at torch defaults it is there by step 28 while SGD has not moved. Gradient descent keeps a² − b² as drawn (Du et al., 2018): the rich and lazy regimes (Woodworth et al., 2020).

The loss of the smallest recurrent network, Pascanu et al.’s (2013): one sigmoid unit fed back to itself, x ← wσ(x) + b for 50 steps from 0.5, fit so that σ(x₅₀) = 0.7; their figure 6, drawn wider. Where w gives x two attractors, a little more b tips x₅₀ from one to the other, so the loss steps down a wall where the gradient reaches 10⁴; its lowest points line the lip, dashed. SGD at η = 0.1 climbs to it, steps into the wall at step 32 and is thrown onto the plateau where σ saturates, to stay at (1 − 0.7)² = 0.09; with grad clip on, it slides along the lip to 6×10⁻⁵. Tuned, SGD gets to 3×10⁻² at η = 0.01, and clipped, to 2×10⁻⁶ at η = 1. At torch’s lrs clipping lowers every end but SGD’s, which never climbs as far as the wall.

Clipping at 1 keeps SGD from diverging at any η, and slows it: on Ravine at torch’s lr it ends 5.85 above the floor, not 1.95; on Cliff it keeps SGD on the lip. Under AMP the gradient sees only the weights’ bf16 or fp16 cast, so tuned runs stall above fp32’s, drawn pale (Micikevicius et al., 2018); on Ravine fp16’s GradScaler skips the first 7 steps, ticked at the foot, halving its scale from 2¹⁶. The dotted 2/λ on the sweep is where SGD diverges at L*; a run that diverges ends in an × where it left the map. loss × scales the loss up, as summing it over a batch of that many instead of averaging would, so every gradient is that many times as long and 2/λ that many times lower: on Ravine at torch’s lrs SGD arrives at ×10, and at ×100 it and Momentum diverge at step 2, while RMSProp and AdamW, whose steps are about η whatever the gradient’s size, read as before to the digit at every scale, and Muon to its bf16 rounding.

At first each runs as torch.optim makes it, at PyTorch’s own lr and knobs, the long tick on each slider, and PyTorch defaults is pressed; tuned, at the η the sweep finds best for it, the lowest L − L* at the end, then the earliest under 10⁻³, as a fair comparison tunes each (Choi et al., 2019). The pointer on a card shows its torch.optim call. Momentum, Polyak (1964); RMSProp, Hinton (2012), with no bias correction, as in PyTorch; AdamW, Loshchilov and Hutter (2019), Kingma and Ba’s (2015) Adam with its weight decay, wd, decoupled; Muon, Jordan et al. (2024), orthogonalised in bf16 as torch’s is, θ a diagonal 2×2 on Ravine, two 1×1s on ab = 1 and Cliff, a 1×2 row on Two dips, whose fans set Xavier’s σ and PyTorch’s bound; cosine, Loshchilov and Hutter (2017); warmup, Goyal et al. (2017). Seeds set the init’s draw and the gradient noise. After Alec Radford’s animations and Lili Jiang’s arrows (2020); colours, Okabe and Ito (2008).