Whisper's core methodology consists of three components: data preprocessing, model architecture, and multitask format data.
Preprocessing uses a minimalist approach to collect audio-transcript pairs from the internet, constructing a diverse dataset of 680,000 hours. Automated filtering mechanisms — including detection and removal of machine-generated transcripts, audio-language matching verification, and fuzzy deduplication — ensure data quality. Audio is segmented into 30-second chunks, with non-speech segments included for voice activity detection training.
The model architecture adopts a standard encoder-decoder Transformer design. Audio is resampled to 16 kHz and converted to 80-channel log-magnitude Mel spectrograms. The encoder consists of a convolutional stem and Transformer blocks, while the decoder uses learned position encodings and tied representations. Text tokenization employs GPT-2's byte-level BPE.
The multitask format unifies speech recognition, translation, voice activity detection, and language identification through a sequence of special tokens (e.g., <|startoftranscript|>, language tokens, task tokens), enabling a single model to execute the entire speech processing pipeline and leverage historical transcript context to resolve audio ambiguity.