A new article on Towards AI explains how to scale transformers’ memory up to 262K tokens with minimal changes. This technique involves leveraging language models to memorize information and can be used with available pre-trained models. The article addresses questions about the issue, solution, and results of this approach.

source update: Memorizing Transformer – Towards AI