xLSTM: LSTM is so back

Description

Long Short-Term Memory (LSTM) networks have withstood the test of time, forming the foundation of many early deep learning breakthroughs—including the first generation of Large Language Models (LLMs). However, the rise of Transformers has since overshadowed LSTMs, establishing them as the dominant architecture for LLMs.
We revisited the potential of LSTMs and ask: Can LSTMs be scaled up to compete with Transformers?
We introduce xLSTM, a significantly enhanced version of LSTM featuring exponential gating and a novel matrix memory with covariance-based updates. Our kernel implementations of xLSTM adheres to scaling laws and demonstrates faster training than Transformers. More crucially, xLSTM has linear-time inference in the number of produced tokens, in stark contrast to the quadratic complexity of attention mechanisms, making it highly efficient for deployment.
A 7-billion parameter xLSTM model achieves comparable performance to state-of-the-art Transformer models, while offering significantly faster inference. We are currently developing distilled xLSTM variants from large Transformer models with accelerated inference. Additionally, xLSTM time-series foundation models are constructed, which already outperform leading approaches such as Chronos (Amazon), TimesFM (Google), and Moirai (Salesforce).
xLSTM is already seeing real-world adoption: companies like Spleenlab and Festo have successfully integrated it into commercial products.

Period	03 Jul 2025
Event title	International Joint Conference on Neural Networks
Event type	Conference
Location	Rom, ItalyShow on map
Degree of Recognition	International