I-JEPA learns by predicting masked image patch representations: no contrastive pairs, no labels, no explicit notion of similarity between images. CLIP, by contrast, is explicitly trained to align images and text that belong together and push apart pairs that don't. These are fundamentally different objectives.
The question this project asks is simple: does the distinction matter for retrieval? If you take I-JEPA's frozen encoder, extract embeddings, and search for similar images, does it work? And how far behind CLIP does it fall?
Three models, all used as frozen feature extractors: no fine-tuning, no task-specific training. Embeddings are L2 normalized before indexing, and retrieval uses exhaustive cosine similarity search via a FAISS FlatIP index.
| Model | Embedding Dim | Pooling |
|---|---|---|
| I-JEPA ViT-H/14 | 1280 | Mean-pool patch tokens |
| CLIP ViT-B/32 | 512 | Image encoder (official OpenAI library) |
| ResNet-50 | 2048 | Global average pool (IMAGENET1K_V1 weights) |
Evaluation is on Oxford5k (5,062 database images, 55 landmark queries) and Paris6k (6,412 images, 55 queries). Both use a three-tier ground truth (good, ok, junk) and standard mean average precision as the metric.
| Model | Oxford5k mAP | Paris6k mAP |
|---|---|---|
| I-JEPA ViT-H/14 | 0.1822 | 0.3161 |
| ResNet-50 | 0.3706 | 0.5707 |
| CLIP ViT-B/32 | 0.4884 | 0.6555 |
I-JEPA trails ResNet-50 by ~19 points mAP on Oxford5k and ~25 points on Paris6k
CLIP leads by ~30 points mAP on Oxford5k, a model 3x smaller in embedding dimension
I-JEPA significantly underperforms both baselines. A pretrained ResNet-50 with standard ImageNet weights, a much simpler model with no explicit similarity objective, comfortably outperforms it. CLIP's margin is larger still.
The qualitative results clarify where I-JEPA breaks down. It performs best on landmarks with distinctive geometry (the Radcliffe Camera, Hertford College) where the patch-level structure of the building is sufficient to find similar images. It struggles with semantically complex scenes that require understanding of place rather than just shape.
I-JEPA tends to retrieve architecturally similar buildings rather than the specific landmark queried. It has learned something about structure, but not the kind of landmark identity that retrieval benchmarks measure.
Query expansion (α = 3.0) adds the top retrieved embeddings back into the query vector before re-ranking. For CLIP, this provides a +0.011 mAP gain, meaningful and expected, since the embedding space is well-structured enough for nearby points to genuinely refine the query.
For I-JEPA, the gain is +0.002 mAP: essentially nothing. This is the diagnostic finding. If the embedding space were well-structured and the problem were simply a threshold issue, query expansion would help. The near-zero gain points to something more fundamental: the embedding space itself isn't organized in a way that makes retrieval tractable, regardless of how the query is refined.
The gap makes sense once you consider what each objective is actually optimizing for. CLIP's contrastive objective directly trains the model to place matching images close together and non-matching images far apart, which is exactly what retrieval requires. I-JEPA's objective trains for prediction of masked patch representations within a single image, with no signal about inter-image relationships at all.
I-JEPA's embeddings capture local structure well, but the global organization of the embedding space doesn't reflect image similarity in the sense retrieval benchmarks measure. The training objective simply doesn't ask that question.
Full source, evaluation scripts, and replication steps on GitHub →