Protein Language Model
An AI model trained on large databases of protein sequences — treating amino acids like words — to learn representations that capture evolutionary and structural information useful for property prediction and design.
What it means
A protein language model (pLM) is a type of neural network trained on large databases of protein sequences using the same self-supervised approach as text language models. The model treats a protein sequence as a sentence of amino acid “letters” and learns to predict masked or missing amino acids from their context.
Through this training on hundreds of millions of sequences from public databases like UniRef, the model learns representations that capture:
- Which amino acids co-occur in certain structural contexts
- Evolutionary constraints on which positions can be mutated without disrupting function
- Structural patterns (secondary structure, domain boundaries) implicit in sequence
- Functional signals encoded in sequence
The resulting representations — called embeddings — can then be used as input features for downstream tasks without the model having explicitly been trained on those tasks.
Key models
| Model | Developer | Scale | Notes |
|---|---|---|---|
| ESM-2 | Meta AI | Up to 15B parameters | Most widely used; strong structural embeddings; open-source |
| ESM-3 | Meta AI | Multimodal | Jointly trained on sequence, structure, and function |
| ProtTrans (ProtBERT, ProtT5) | Technical University Munich | Several variants | Strong benchmark performance; open-source |
| Ankh | — | Efficient; competitive with ESM-2 at lower compute cost |
What they’re used for
Protein language model embeddings are used as features for:
- Property prediction: Predicting thermostability, solubility, binding affinity, or other measured properties from sequence alone — by training a small prediction head on top of frozen or fine-tuned pLM embeddings
- Protein design: Generating novel sequences with target properties, informed by pLM-learned patterns of what sequences are “evolutionary plausible”
- Variant effect prediction: Estimating the likely impact of a mutation without experimental measurement
- Structure prediction: ESMFold generates protein structures directly from ESM-2 embeddings, skipping the multiple sequence alignment step that AlphaFold requires
Limitations
- pLMs learn from evolutionary data — they understand what sequences exist in nature, not necessarily what new sequences could fold or function
- Performance depends on whether your protein family is well-represented in training data; rare or newly evolved families have less coverage
- pLM embeddings capture sequence-level signals; they don’t directly encode 3D structural information unless the model was jointly trained on structure (like ESM-3 or ESMFold’s internal representations)