BERT's pre-training data is massive in scale, high in quality, and uniformly formatted (document-level text). Through two cleverly designed pre-training tasks — MLM and NSP — the model learned rich linguistic knowledge and reasoning capabilities, laying a solid foundation for its outstanding performance on a wide range of downstream tasks. The data sources are BooksCorpus (a book corpus) and English Wikipedia.