A New Generation of Foundation Models Based on xLSTM

Description

Foundation models, such as GPT and vision transformers, are almost all built on the famous Transformer architecture. Since about 2018 these transformers have replaced recurrent neural networks (RNNs) in natural language processing, computer vision, but also in other application areas. However, the computational costs of Transformers scale quadratically with sequence or context length which is one of the main reasons for the huge computational costs of AI systems across the world. xLSTM is a modern RNN, which is parallelizable similar to the Transformer, but its computational costs only scale linearly. Thus, xLSTM-based foundation models have the capacity to overtake Transformers as main components of AI systems. In this talk, we provide an overview of the development of AI systems from RNNs, to Transformers, LLMs and foundation models, and to the xLSTM architecture, its core component and its applications.

Period	19 May 2025
Event title	CAIML Symposium 2025
Event type	Other
Conference number	4
Location	Wien, AustriaShow on map
Degree of Recognition	International