A smaller contrastive temperature helps CLIP retain accuracy

Can a pretrained CLIP image encoder be refined with contrastive learning without losing the capabilities it already has? Prior work described small-batch post-training as a cause of forgetting and some methods replaced that training signal with distillation. The paper tests whether the contrastive temperature helps explain that problem.

ComCLIP combines temperature-calibrated image-text matching, feature anchoring and image-relation guidance to refine CLIP’s vision encoder.

On ViT-B/16, the temperature test answers whether contrastive post-training can preserve CLIP’s zero-shot ability: at 0.07, accuracy fell from 61.82% to 57.90%; at 0.01, it rose to 62.90%. Zero-shot accuracy measures classification from text descriptions without a task-specific classifier, averaged over 12 datasets.

ComCLIP builds on this setting with feature anchoring and image-relation guidance. Across three seeds, ComCLIP and CLIP-Refine had similar zero-shot accuracy, while ComCLIP scored higher on linear probing, which tests a simple classifier using frozen image features, and CLIP-Refine led on all four image-text retrieval measures.

This comparison supports a transferability advantage for ComCLIP alongside a retrieval trade-off. The best-setting rows used different post-training datasets for the two methods; separate dataset-matched results also favored ComCLIP on linear probing.

The strongest comparison used different post-training datasets for the two methods, and the downstream compatibility test covered one LLaVA setup with its language model and projector frozen. The findings do not establish that ComCLIP improves downstream accuracy or applies to every CLIP-like model.