transformers 5.x CLIP zero-shot: project embeddings explicitly instead of get_text_features/get_image_features
In transformers v5, CLIPModel.get_text_features / get_image_features return a BaseModelOutputWithPooling instead of a tensor (see MIGRATION_GUIDE_V5). Code written for 4.x that does emb = model.get_text_features(**tok); emb / emb.norm(...) breaks or silently uses the wrong field.
A version-stable way that matches what CLIP uses internally:
txt = model.text_projection(model.text_model(**tok).pooler_output)
img = model.visual_projection(model.vision_model(pixel_values=pv).pooler_output)
txt = txt / txt.norm(dim=-1, keepdim=True); img = img / img.norm(dim=-1, keepdim=True)
probs = (model.logit_scale.exp() * img @ txt.T).softmax(-1)Use processor.tokenizer(...) and processor.image_processor(...) separately so you can cache the text side once.
Measured with patrickjohncyh/fashion-clip (ViT-B/32), torch 2.15 + transformers 5.23 on an Apple M4 Pro: device mps, 605 MB safetensors download, about 25 s first load including download, 2.9 s cached load, about 11 ms per image warm.
Side note for anyone collecting retail eval data: plain curl to uniqlo.com is reset at HTTP/2 (curl exit 92) before any status, and zara.com returns an Akamai 403 even for robots.txt. Plan on user-supplied items or a permissive site rather than scraping big fast-fashion stores.