The Experiments section of the Whisper paper systematically evaluates the model's performance in a zero-shot setting. The core finding is that through large-scale (680,000 hours), diverse, weakly supervised multitask pretraining, a single model — without any task- or dataset-specific fine-tuning — can achieve strong generalization and near-human robustness across a wide range of speech tasks.
On English speech recognition, the model achieves performance competitive with fully supervised methods on standard benchmarks such as LibriSpeech (word error rate as low as approximately 3.4%), and demonstrates far superior robustness on out-of-distribution datasets compared to traditional supervised models, reducing the average out-of-distribution error rate by 55.2%.
In multilingual speech recognition, performance is strongly correlated with the amount of training data per language (WER roughly halves for every 16x increase in data), though non-Indo-European languages underperform relative to expectations due to byte-level BPE tokenizer mismatch.
On speech translation (X→en), the model excels on low-resource languages, demonstrating the zero-shot transfer advantage. Language identification accuracy reaches 80.3% on languages covered during training. Robustness tests confirm the model's stronger resistance to additive and environmental noise. Long-form transcription evaluation shows it can effectively process 30-second segments and integrate context.
Analysis and ablation studies further reveal that, except for English recognition where diminishing returns emerge as performance approaches human level, model performance on multilingual recognition, translation, and other tasks continues to improve with model scale. Increasing training data scale benefits all tasks positively. For sufficiently large models, joint multitask and multilingual training incurs no penalty and may even be beneficial.