BERT: Pre-training Model

Progress0%0/11 modules completed

Node overview

BERT adopts the same "pre-training + fine-tuning" paradigm as GPT-1, using large-scale unlabeled data to learn general-purpose language representations, and then adapting to various downstream tasks through fine-tuning.

Learning modules

Encoder-Only

0/3 QPrerequisite: Decoder-OnlyRequired

Tokenization: WordPiece

0/5 QPrerequisite: Tokenization: BPERequired

Learnable Positional Encoding (already introduced in GPT-1)

Prerequisite: Learnable Positional EncodingRequired

FFN (reused from Transformer)

Prerequisite: FFNRequired

Residual Connection (reused from Transformer)

Prerequisite: Residual ConnectionRequired

Layer Normalization (reused from Transformer)

Prerequisite: LayerNormRequired

Activation Function: GELU (introduced in GPT-1)

Prerequisite: Activation Function: GELURequired

Self-Attention Mechanism (reused from Transformer)

Prerequisite: Self-AttentionRequired

Multi-Head Attention (MHA, reused from Transformer)

Prerequisite: Multi-Head Attention (MHA)Required

Scaled Dot-Product Attention (reused from Transformer)

Prerequisite: Scaled Dot-Product AttentionRequired

Softmax (reused from Transformer)

Prerequisite: SoftmaxRequired

Begin the guided reading

Choose a module from the paper guide.

11Modules

8Questions

Notes