The core innovation of GPT-1 lies in a simple yet effective two-stage framework:
Unsupervised generative pre-training: A general-purpose model is trained on a large-scale, diverse text corpus using a standard language modeling objective (i.e., predicting the next word given the preceding context). This allows the model to learn rich linguistic knowledge, world knowledge, and long-range dependencies.
Supervised discriminative fine-tuning: The pre-trained model is directly applied to downstream tasks (such as text classification, question answering, etc.) using labeled data for fine-tuning. During fine-tuning, only minimal adjustments to the model input are needed, along with the addition of a simple linear output layer.