MSc Defence: The Angle Degeneracy Phenomenon in Deep Neural Networks: Analysis and Relation to Training Dynamics

CANDIDATE: Cameron Jakub

ABSTRACT: Deep neural networks have proven to be powerful functions with many applications, but the theoretical behaviour of these functions is not fully understood. One such behaviour is the large depth degeneracy phenomenon, where inputs tend to become highly correlated as they travel deeper into a randomly initialized network. This can make the network effectively incapable of distinguishing between inputs, which seemingly has negative impacts on training performance. Through combinatorial expansions, we develop precise formulas to predict the expected value and variance of the angle between inputs at any layer of the initialized network. We provide a detailed analysis of how quickly the angle tends toward zero in a finite width setting, which proves to be qualitatively different than studying the problem in the infinite width limit. We validate our theoretical results through comparison to empirical simulations, and run experiments to explore how network degeneracy can impact training dynamics.