Can statistical physics equations for neuronal avalanches also describe deep neural network learning?

Avalanches as events inside a deep network

The study asks whether the event-based physics used for living brains can be applied to artificial ones. In a biological network, a neuronal avalanche is a burst of activity that spreads and then dies out before the next burst begins. The paper defines the analogous object in a feedforward network by tracking signal strength, the norm of the pre-activations, layer by layer. An avalanche starts when this strength rises above a threshold and ends when it falls back below it.

Two quantities follow from that definition. Avalanche size is the cumulative signal strength above threshold summed over the layers the cascade crosses, and duration is the number of layers it penetrates before dropping below threshold. These are event-level measurements, not the architecture-level proxies such as Lyapunov exponents that earlier work used to argue that deep networks sit near an edge of chaos. The paper's claim is that resolving propagation into discrete avalanches lets crackling-noise theory make testable predictions about learning.

Sizes and durations follow power laws

In Gaussian-initialized networks with zero bias and near-critical weight variance, the distributions of avalanche size and duration follow power laws. The authors fit exponents by maximum likelihood and extrapolate them with finite-size scaling as width grows, obtaining size and duration exponents approaching about 1.34 and 1.53. Those values sit close to the Barkhausen-noise reference values of roughly 1.33 and 1.50.

The stronger test is relational rather than distributional. Crackling-noise theory predicts that mean avalanche size should scale with duration as a power law, with an exponent gamma tied to the size and duration exponents by a specific relation. The fitted gamma is about 1.53 for width 1000, close to the predicted ratio of about 1.58, and the rescaled avalanche profiles collapse onto a single curve at an optimal rescaling value near 1.58. The paper treats this collapse as evidence of a genuine non-equilibrium phase transition rather than a coincidental heavy tail.

Biases act as fields that blur criticality

The clean critical point appears only at zero bias. Each neuron's activity combines a weighted sum from the previous layer with its own bias, and the paper argues that biases play the role of an external field: they tilt the response and remove the sharp transition. With nonzero bias variance, susceptibility no longer diverges; it shows finite peaks around weight variance near 1.

This creates two different notions of optimality that readers should not conflate. The edge of chaos is defined by cross-input correlations, and along that curve the correlation depth diverges. The Widom-like line is defined instead by where weight susceptibility reaches its maximum. The two curves meet only at the exact critical point with zero bias and separate once bias is present, and the susceptibility peaks shrink as bias variance grows until the line vanishes. The paper reports that trainability tracks the Widom-like line, not the whole edge of chaos.

Susceptibility predicts learning better than criticality

The practical comparison comes from training networks at many initialization pairs of weight variance and bias variance. Trainability begins for networks initialized near weight variance of about 1, and the region of high training accuracy narrows as bias variance increases. In deeper networks the high-accuracy region narrows further: with hidden width 300 and depths of 200, 300, 400 and 500 layers trained on MNIST for 10 epochs, the accurate region contracts progressively with depth.

The key claim is comparative. Because networks on the edge of chaos can still train poorly when bias is non-negligible, proximity to the critical point is an unreliable guide. Maximal susceptibility, located on the Widom-like line, is the more reliable predictor of learning in these experiments. The paper describes learning as aligning with a quasi-critical plateau of maximal but finite susceptibility rather than with exact point criticality. This is a statement about initialization-dependent trainability in the tested setups, not a claim that trained networks remain critical after learning.

ResNets fall into a different universality class

The analysis extends beyond Gaussian-initialized fully connected networks to three ResNet variants. Because layer widths and operations vary, the authors use a layer-dependent threshold set a fixed number of standard deviations above the mean signal strength, calibrating it with 10000 runs before analyzing one million perturbations per model. Avalanche size is then computed from the signal strength above that threshold across operations.

ResNet avalanche size and duration distributions again follow power laws, and the exponent relation gives gamma of about 2, consistent with mean-field directed percolation rather than Barkhausen noise. The exponents stay closely clustered across threshold settings. The paper also reports that varying the BatchNorm scale parameter drives a sharp change in activity near a value of 1, with susceptibility peaking there at the onset of the successful training range. The authors read this as evidence that distinct universality classes are identifiable in different architectures, which matters because the class, not just the presence of scaling, may carry functional consequences.

Finite size and architecture limit the exponents

The authors acknowledge that finite system sizes and subsampling impose cutoffs that complicate exponent estimation. They address this with maximum-likelihood fits and finite-size scaling, but note that larger-scale analysis could improve confidence. The measurements also depend on which observable is tracked: the study follows pre-activations in Gaussian networks and module outputs in ResNets, while post-activations, block-level propagation and other candidate signatures are left outside its scope.

The scope is further bounded by architecture. The results cover Gaussian-initialized feedforward networks and ResNet variants, and the paper explicitly invites broader tests of quasi-criticality in other deep learning systems. The diagnostic quantities it proposes, such as exponent-relation mismatch and collapse quality, are therefore tools for the tested settings rather than validated universal monitors.

Tune toward the susceptibility peak, not the edge

The paper's most decision-relevant message is a shift in what to optimize. If a practitioner wants to place a very deep network in a trainable regime, the evidence points to the locus of maximal weight susceptibility, which shifts and weakens as bias variance grows, rather than to the edge of chaos as a whole. The reported narrowing of the trainable region with depth and with bias variance makes this distinction sharper in exactly the regimes where initialization is hardest.

The authors further suggest that standard controls such as dropout, spectral constraints, batch normalization, injected noise and residual depth could be treated as parameters that move a network along this quasi-critical landscape, and that the exponent set could help distinguish tasks or architectures operating in different universality classes. Those are proposed directions rather than demonstrated results. The concrete finding to carry away is narrower: in the networks tested, learning aligned with a quasi-critical plateau of finite maximal susceptibility, and susceptibility outperformed proximity to the critical point as a predictor of trainability.

Sources used in this answer