GPT-1: Pre-training Details

-- FLOPs

Paper guide

GUIDED READING

GPT-1: Pre-training Details

Progress0%0/4 modules completed

Node overview

The goal of pre-training is to train a general-purpose language model that can understand and generate natural language by predicting the next token. This is considered a form of unsupervised learning, since no human-annotated labels are required.

Learning modules

1

Adam (Already Present in Transformer)

Prerequisite: AdamRequired

2

Learning Rate Schedule (Already Present in Transformer)

Prerequisite: Learning Rate ScheduleRequired

3

Batching

0/3 QPrerequisite: BatchRequired

4

Dropout (Already Present in Transformer)

Prerequisite: DropoutRequired

Begin the guided reading

Choose a module from the paper guide.

4Modules

3Questions

Notes

01Home 02Papers 03Access