Modern Hopfield Networks

Description

We propose a new paradigm for deep learning by equipping each layer of a deep-learning architecture with modern Hopfield networks. The new paradigm comprises functionalities like pooling, memory, and attention for each layer. Recently, we saw a renaissance of Hopfield Networks with tremendously increased storage capacity and convergence in one update step while ensuring global convergence a local energy minimum. Surprisingly, the transformer attention mechanism is equal to modern Hopfield Networks. In layers of deep learning architectures, they allow the storage of, and the access to, raw input data, intermediate results, reference data, or learned prototypes. These Hopfield layers enable new ways of deep learning and provide pooling, memory, nearest-neighbor, set association, and attention mechanisms. We apply deep networks with Hopfield layers to various domains, where they improve the state of the art on different tasks and for numerous benchmarks.

Period	14 Apr 2021
Event title	NVIDIA GTC 2021
Event type	Other
Location	AustriaShow on map