Abstract
The forward dynamics in neural networks for various activation functions has been studied extensively in the context of initialisation and normalisation strategies, by mean field theory, edge of chaos theory, and fixed point analysis. However, the study of the backward dynamics appears to be largely disconnected to the insights obtained from the forward analysis. We argue that many of the ideas from the forward analysis could and should be applied to backward dynamics. We show that the ideas of mean field theory and fixed point analysis apply to the backward pass and allow to characterize activation functions.
| Original language | English |
|---|---|
| Title of host publication | Neural Information Processing Systems (NIPS 2018) |
| Number of pages | 1 |
| Publication status | Published - 2018 |
Fields of science
- 303 Health Sciences
- 304 Medical Biotechnology
- 304003 Genetic engineering
- 305 Other Human Medicine, Health Sciences
- 101004 Biomathematics
- 101018 Statistics
- 102 Computer Sciences
- 102001 Artificial intelligence
- 102004 Bioinformatics
- 102010 Database systems
- 102015 Information systems
- 102019 Machine learning
- 106023 Molecular biology
- 106002 Biochemistry
- 106005 Bioinformatics
- 106007 Biostatistics
- 106041 Structural biology
- 301 Medical-Theoretical Sciences, Pharmacy
- 302 Clinical Medicine
JKU Focus areas
- Computation in Informatics and Mathematics
- Nano-, Bio- and Polymer-Systems: From Structure to Function
- Medical Sciences (in general)
- Health System Research
- Clinical Research on Aging