37 Self-Supervised & Contrastive Learning

TL;DR

Self-supervised learning manufactures labels from the data itself, no human annotation required: contrastive methods (SimCLR, MoCo) pull together two views of the same image and push apart different images, self-distillation (DINO) matches a student to a teacher instead, and masked reconstruction (MAE) rebuilds hidden patches, vision’s counterpart to BERT’s masked-token objective.

Depends on: Word Embeddings

37.1 Why this matters

Word Embeddings already made this book’s first self-supervised argument: word2vec needs no labels, only raw text, because the label (“predict this masked or nearby word”) is manufactured from the data itself. Vision took much longer to find an equally effective manufactured label; pixels don’t have the same local co-occurrence structure words do, so ImageNet’s human labels remained the standard until the methods in this note closed the gap. That gap-closing matters directly for Multimodal models: CLIP’s image encoder is pretrained with exactly this family of objectives before it ever sees a caption.

37.2 The mechanism

37.2.1 Contrastive learning: pull together, push apart

SimCLR (Chen et al., 2020) takes one image, applies two different random augmentations to get two views, and trains an encoder so the two views land close together in embedding space while every other image in the batch is pushed apart.

The loss is a softmax over similarities, with the matching view as the one positive and every other image in the batch as negatives:

\[ \mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_j)/\tau)} {\sum_{k \neq i} \exp(\text{sim}(z_i, z_k)/\tau)}. \]

No labels are used anywhere; the manufactured label is simply “these two views came from the same source image.” MoCo (He et al., 2020) solves the same objective but keeps a large queue of negative representations across batches, built from a momentum-updated encoder, instead of relying on a big batch size, making the negative pool cheap to scale.

37.2.2 Self-distillation without negatives: DINO

Contrastive learning needs negatives to have something to push away from. DINO (Caron et al., 2021) removes that requirement entirely: a “student” network and a slowly-updated “teacher” network (an exponential moving average of the student’s weights) each see different augmented views of the same image, and the student is trained to match the teacher’s output distribution. Because the teacher is never directly trained, only averaged from the student, there is no adversarial game and no negative sampling, only self-distillation. DINO’s emergent property was that its attention maps, with no segmentation labels ever provided, learned to localize objects.

37.2.3 Masked reconstruction: MAE

MAE (He et al., 2022) applies BERT’s masking idea directly to images: split an image into patches (as in ViT), mask a large fraction of them, and train an encoder-decoder to reconstruct the missing pixels from the visible patches alone. Two design choices stand out for a first-time reader:

A very high masking ratio (75%), far above BERT’s ~15% for text. Images are spatially redundant, so a lower ratio would let the model interpolate trivially from neighboring pixels instead of learning real structure.
An asymmetric encoder-decoder: the (expensive) encoder only ever processes the visible patches, and a small, cheap decoder handles reconstruction. Skipping masked patches in the encoder is what makes MAE affordable to train at scale despite the aggressive masking.

import numpy as np
rng = np.random.default_rng(0)

def mask_patches(patches, mask_ratio=0.75):
    n = patches.shape[0]
    n_mask = int(n * mask_ratio)
    idx = rng.permutation(n)
    masked_idx, visible_idx = idx[:n_mask], idx[n_mask:]
    return visible_idx, masked_idx

patches = np.arange(16)          # a toy 4x4 patch grid, flattened
visible, masked = mask_patches(patches)
print("visible patches:", sorted(visible))
print("masked patches: ", sorted(masked))

visible patches: [np.int64(1), np.int64(8), np.int64(13), np.int64(15)]
masked patches:  [np.int64(0), np.int64(2), np.int64(3), np.int64(4), np.int64(5), np.int64(6), np.int64(7), np.int64(9), np.int64(10), np.int64(11), np.int64(12), np.int64(14)]

37.2.4 What to observe

Contrastive methods (SimCLR, MoCo) need a notion of negative examples; self-distillation (DINO) and masked reconstruction (MAE) do not, two genuinely different ways to manufacture a training signal.
MAE’s masking ratio (75%) is far higher than BERT’s (~15%) precisely because images carry more redundant, spatially local information than text.
All three produce an encoder meant to be reused, fine-tuned or frozen for a downstream task, the same pretrain-then-adapt pattern from BERT & GPT.

37.3 Application & impact

Concept here	What it became	Where you see it today
word2vec’s manufactured label	The general self-supervised pattern	Extended from text to every modality here
SimCLR / MoCo contrastive pretraining	Strong general-purpose image encoders	CLIP’s image tower before caption alignment
DINO self-distillation	Label-free object localization	Vision foundation model pretraining
MAE masked reconstruction	BERT’s objective, applied to pixels	ViT pretraining at scale

Key takeaway

Vision needed several different attempts (contrastive, self-distillation, masked reconstruction) to find an image-native equivalent of word2vec’s manufactured label, unlike text, where masked/next-word prediction worked almost immediately. Each of these encoders is what typically sits underneath a multimodal model’s vision tower before it ever sees language.

← Back to: Multimodal