Attention-based transformers in the AI industry have improved since 2017, with no limitations on input size. These attention models are more powerful than neural networks for generalization and routing through keys and queries. There are numerous variations of attention to decrease computation and optimize complexity. The article explains how this technology functions and how it has evolved.

source update: Long-Range Transformers with Unlimited Length Input – Towards AI