A ring of 96 inputs, u cos θ + v sin θ with u and v random, orthogonal and of unit variance a unit, goes through a plain stack of N = 64 units a layer. Each layer’s ring is drawn on its own top two principal components, turned to match the layer before, so the tube shows the ring’s shape and size layer by layer. Poole et al. (2016) drew theirs on three components, at layers 5 and 10.

The ring’s length is the sum of its 96 steps. The distance between two nearby inputs grows by √χ a layer, with χ = σ_w² E φ′(√q z)² at that layer’s q. In chaos the steps grow until neighbours are as far apart as any two inputs, √(2q*(1 − c*)) a unit, and the drawn length levels off while the ring’s own length keeps growing.

The input-output Jacobian ∂h^L/∂h¹ is taken at one input scaled to start at q*. Mean field gives the mean of its σ², χ₁ a layer, but not their spread. Orthogonal weights with tanh near q* = 0 keep them near 1, the dynamical isometry of Pennington, Schoenholz and Ganguli (2017). Gaussian weights spread them over decades. ReLU’s inactive units leave half or more at 0, since the Jacobian’s rank is at most the fewest units active in any layer.