ViLT's pretraining data integrates four mainstream public datasets — MSCOCO, Visual Genome (VG), Google Conceptual Captions (GCC), and SBU Captions — totaling roughly 4 million image-text pairs. These datasets balance quality and scale: MSCOCO and VG provide precise human annotations (the latter with dense region-level descriptions), while GCC and SBU offer large-scale web-crawled captions that, though noisier, cover a broader range of real-world scenarios.
ViLT's training input x is a multimodal sequence: images are split into 32×32 pixel patches and embedded as vector sequences via linear projection; text is tokenized using the BERT tokenizer and embedded as token sequences; the two are concatenated with special tokens such as [CLS] and [SEP], along with positional and modality type encodings, to form a joint input.
The target y varies by pretraining task — for Image-Text Matching (ITM), y is a binary label (0 or 1) indicating whether the image and text match; for Masked Language Modeling (MLM), y is the original token ID of the masked token, and the model must predict these labels.