BERT's power stems from its two carefully designed pre-training tasks, which teach the model to understand and reason about language:
- Masked Language Model (MLM): its goal is to enable deep bidirectional contextual understanding. It randomly masks 15% of the tokens in the input sequence, then asks the model to predict the original words at those masked positions based on context. This process is analogous to a "cloze" fill-in-the-blank task.
- Next Sentence Prediction (NSP): its goal is to teach the model to understand relationships between sentences, which is critical for tasks like question answering and natural language inference.