Power-law escape rate of SGD

Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equation (SDE) with simpler additive noise by performing a random time change. Using this formalism, we show that the log loss barrier \Delta\log L=\log[L(\theta^s)/L(\theta^*)] between a... Show more