Transformer
The neural network architecture behind most modern AI — including GPT-4, AlphaFold, and ESM-2 — built around an attention mechanism that lets the model weigh relationships between all parts of its input simultaneously.
What it means
The Transformer is a neural network architecture introduced by Google researchers in 2017 (“Attention Is All You Need”) that has become the foundation of nearly all modern large AI models. Unlike earlier sequence models that processed input step by step, the Transformer processes all parts of its input simultaneously and uses an attention mechanism to learn which parts of the input are most relevant to each other.
Why transformers matter
Before transformers, different types of data required different specialized architectures (recurrent networks for sequences, convolutional networks for images). Transformers scaled more effectively with more data and compute than their predecessors, and proved adaptable to an enormous range of tasks and data types by treating inputs as sequences of tokens.
The result: virtually every frontier AI model — GPT-4, Claude, Gemini, AlphaFold, ESM-2, Pangu-Weather — is built on transformer architectures or close variants.
The attention mechanism
The core innovation in transformers is self-attention: for each element in the input (a word, an amino acid, a grid cell in a weather model), the model learns how much “attention” to pay to every other element when computing its representation.
In a protein language model (like ESM-2), this means the model can learn that a particular amino acid’s properties depend on distant residues in the sequence — capturing long-range dependencies that earlier architectures struggled with. In a weather model (like Pangu-Weather), it means the model can learn which distant atmospheric cells influence each other.
Transformers in scientific AI
| Model | Domain | What the tokens are |
|---|---|---|
| ESM-2, ESM-3 | Proteins | Amino acids in a sequence |
| ChemBERTa | Chemistry | Atoms/bonds in a SMILES string |
| AlphaFold | Protein structure | Amino acid pairs (via MSA rows) |
| Pangu-Weather | Weather | Grid cells at each pressure level |
| GPT-4, Claude | Language | Subword tokens |
What it doesn’t mean for users
You don’t need to understand transformer architecture to use tools built on it. But knowing that most AI tools are transformer-based helps in a few ways:
- Context window limitations — transformers have a maximum input length; very long sequences (genomes, extremely long documents) hit this limit
- Pattern recognition, not reasoning — transformers learn statistical patterns in training data; they don’t reason from first principles, which explains both their breadth and their failure modes
- Scale matters — transformer performance improves predictably with more data and compute, which is why foundation models keep getting larger