On Signal Propagation in Neural Networks

Neural networks have proven to be powerful methods for various tasks in the field of machine learning. Especially deep neural networks, which consist of multiple layers, have extraordinary predictive capabilities. However, the difficulty of training neural networks typically increases with depth. The main cause for this difficulty is believed to be vanishing gradients and distribution shifts.
Signal propagation theory is the study of how data propagates through the network. By controlling signal propagation through deep neural networks, distribution shifts and vanishing gradients can effectively be alleviated. Therefore, improvements to signal propagation in neural networks have made it possible to train ever deeper models.
Despite extensive studies, there are still numerous boundaries that limit the possibilities of signal propagation. This cumulative thesis presents a set of publications that aim to push the boundaries of signal propagation theory on multiple levels. First, boundaries between different viewpoints on signal propagation are weakened by establishing connections between these viewpoints. Especially the connection between propagation during forward and backward passes is emphasised. Furthermore, the outer boundary of signal propagation theory is expanded by generalising signal propagation theory to layers that have weights with non-zero mean. This enables the creation of a principled weight initialisation for non-convex networks, where weights are constrained to be positive. Finally, boundaries on the application domain are expanded by creating a novel architecture with particular propagation characteristics. The resulting Mass-Conserving LSTM (MC-LSTM) architecture guarantees perfect conservation and propagation of its inputs, unlocking remarkable generalisation performance.