Designing Normalizer-Free EfficientNets

As neural networks grow in complexity, Batch Normalization (BN) has become essential for stabilizing training and improving generalization. However, BN introduces drawbacks such as increased memory usage, computational overhead, and reduced effectiveness for small batch sizes. This study aims to modify EfficientNet to function without normalization while maintaining stable signal propagation.
Signal Propagation Plots (SPPs) were used to visualize activations, gradients, and weights, identifying instability sources and revealing BN’s influence on training dynamics. Testing on feed-forward neural networks showed that BN actively scaled gradients when its learnable parameters were enabled, potentially explaining its regularization effects.
EfficientNet-B0 was redesigned without BN by mainly using Scaled Exponential Linear Units (SELU). While this stabilized signal propagation, a performance trade-off was observed, with variance fluctuations in Squeeze-and-Excitation blocks.
This study confirms stable training without normalization is possible, though alternative regularization techniques are needed to retain performance. Future work could focus on designing novel regularization techniques that mimic how BN is able to influence both the forward and backward pass using its learnable parameters. The full code implementation is available at: https://github.com/PDewyse/MT_Normalizer_free_Neural_Nets