Dictionary of Applied Machine Learning

transformer

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A transformer is an artificial neural network (ANN) that is built by composing layers, each of which transforms the features of a data point that consists of tokens, such as the words of a sentence. The feature vectors of the tokens are stacked into a matrix, and each layer maps this matrix to a transformed matrix of the same shape, so that layers can be composed freely into a deep ANN. The layers alternate between token mixing, which combines information across tokens, and a position-wise multilayer perceptron (MLP), which transforms each token separately. In the original transformer, token mixing is implemented by multi-head attention. Alternative mixing transformations, such as a fixed Fourier transform or a state-space layer, can take the place of attention. Transformers underlie large language models (LLMs).

Definition

P-shapeIn the context of machine learning (ML), the term transformer refers to an artificial neural network (ANN) that is built by composing layers, each of which implements a transformation of the features of a data point that consists of tokens (Vaswani et al., 2017). The feature vectors (or embeddings) $\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)} \in \reals^{\featuredim}$ of the $\contextlen$ tokens are stacked into a matrix $\mX = \big(\featurevec^{(1)}, \ldots, \featurevec^{(\contextlen)}\big)^{\top} \in \reals^{\contextlen \times \featuredim}$. Each layer of a transformer maps such a matrix to a transformed matrix of the same shape. Since input and output have the same shape, the layers can be composed freely into a deep ANN. For example, a transformer that translates a sentence transforms the stacked word embeddings layer by layer into feature vectors from which the translated words are predicted (Vaswani et al., 2017); the same principle underlies large language models (LLMs).

P-blockThe layers of a transformer alternate between two types of transformations of $\mX$ (see Fig. 1). A token-mixing layer combines information across tokens, i.e., across the rows of $\mX$. In the original transformer, token mixing is implemented by multi-head attention; the attention mechanism is what sets transformers apart from previous models for sequential data such as recurrent neural networks (RNNs). A position-wise multilayer perceptron (MLP) then transforms each row of $\mX$ separately, so that tokens interact only through the token-mixing layers. Residual connections and layer normalization are interleaved with these transformations.

Figure 1 of the entry transformer
Figure 1: Signal-flow chart of one transformer block. The input is processed in parallel by a multi-head attention layer (the token-mixing layer) and forwarded by a residual connection to the first addition; layer normalization, a position-wise multilayer perceptron, and a second residual + normalization follow. A transformer ANN stacks many such blocks
Token mixing need not be implemented by attention. Any transformation that combines the rows of $\mX$ can take its place: a multilayer perceptron applied across tokens (Tolstikhin et al., 2021), a fixed Fourier transform (Lee-Thorp et al., 2022), or a state-space layer whose cost grows only linearly in the number $\contextlen$ of tokens (Gu and Dao, 2024). These alternatives replace the pairwise token comparisons of attention, whose cost grows as $\contextlen^{2}$, with cheaper mixing transformations (see attention).

See also: attention, artificial neural network, layer, token, embedding, recurrent neural network, natural language processing, large language model, multilayer perceptron.

References

  1. Vaswani et al. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst.. papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  2. Tolstikhin et al. (2021). MLP-Mixer: An all-MLP Architecture for Vision. Adv. Neural Inf. Process. Syst.. doi.org/10.48550/arXiv.2105.01601
  3. Lee-Thorp et al. (2022). FNet: Mixing Tokens with Fourier Transforms. Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics. doi.org/10.18653/v1/2022.naacl-main.319
  4. Gu and Dao (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. Conf. Language Modeling. openreview.net/forum?id=tEYskw1VY2

Cite this entry

@misc{dictml_transformer,
  author = {Jung, Alexander},
  title = {transformer},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-21},
  url = {https://dictionaryofml.org/terms/transformer.html}
}