TY - UNPB
T1 - Rectified Factor Networks
AU - Clevert, Djork-Arné
AU - Unterthiner, Thomas
AU - Mayr, Andreas
AU - Ramsauer, Hubert
AU - Hochreiter, Sepp
PY - 2015/2
Y1 - 2015/2
N2 - We propose rectified factor networks (RFNs) as generative unsupervised models, which learn robust, very sparse, and non-linear codes with many code units. RFN learning is a variational expectation maximization (EM) algorithm with unknown prior which includes (i) rectified posterior means, (ii) normalized signals of hidden units, and (iii) dropout. Like factor analysis, RFNs explain the data variance by their parameters. For pretraining of deep networks on MNIST, rectangle data, convex shapes, NORB, and CIFAR, RFNs were superior to restricted Boltzmann machines (RBMs) and denoising autoencoders. On CIFAR-10 and CIFAR-100, RFN pretraining always improved the results of deep networks for different architectures like AlexNet, deep supervised net (DSN), and a simple "Network In Network" architecture. With RFNs success is guaranteed.
AB - We propose rectified factor networks (RFNs) as generative unsupervised models, which learn robust, very sparse, and non-linear codes with many code units. RFN learning is a variational expectation maximization (EM) algorithm with unknown prior which includes (i) rectified posterior means, (ii) normalized signals of hidden units, and (iii) dropout. Like factor analysis, RFNs explain the data variance by their parameters. For pretraining of deep networks on MNIST, rectangle data, convex shapes, NORB, and CIFAR, RFNs were superior to restricted Boltzmann machines (RBMs) and denoising autoencoders. On CIFAR-10 and CIFAR-100, RFN pretraining always improved the results of deep networks for different architectures like AlexNet, deep supervised net (DSN), and a simple "Network In Network" architecture. With RFNs success is guaranteed.
UR - http://www.bioinf.jku.at/publications
U2 - 10.48550/arXiv.1502.06464
DO - 10.48550/arXiv.1502.06464
M3 - Preprint
T3 - arXiv.org
BT - Rectified Factor Networks
ER -