Read the page.
Test what stuck.

A transformer is a deep learning architecture that relies entirely on the self-attention mechanism to compute representations of its input and output, dispensing with recurrence and convolutions entirely.

Self-attention

Self-attention allows each token in a sequence to attend to every other token, producing contextual representations. This is the mechanism that lets transformers model long-range relationships between tokens far more effectively than RNNs.

Positional encoding

Because the architecture contains no recurrence, positional encodings are added to the input embeddings to inject information about the order of tokens in the sequence.

Multi-head attention

Instead of performing a single attention function, transformers use multiple attention heads in parallel, allowing the model to jointly attend to information from different representation subspaces.

Transformers have become the foundation of modern large language models, and their parallelisable structure makes them dramatically faster to train than sequential architectures.