The defining characteristic of the pretraining data used in the ViT paper is its massive scale. Specifically, the model was pretrained on two key datasets: the proprietary JFT-300M dataset, containing 300 million images labeled across approximately 18,000 classes, and the public ImageNet-21k dataset (roughly 14 million images, 21,000 classes). What these datasets share is immense volume and extraordinarily broad visual category coverage; their format is standard labeled images. The paper's central thesis is precisely that when pretrained with supervision on such massive datasets, the generic visual representations learned by ViT enable it to achieve outstanding performance when transferred to mid-sized and small image recognition benchmarks such as ImageNet, CIFAR-100, and VTAB.
The paper's experimental design uses JFT-300M as the representative of ultra-large-scale data and ImageNet-21k as the representative of relatively smaller-scale public data.
The core argument is that data scale matters decisively. Experimental results show that ViT models pretrained on the massive JFT-300M achieved state-of-the-art performance, significantly outperforming the top convolutional networks of the day.