This article was automatically translated from the original Turkish version.
CLIP is a Contrastive Language-Image Pretraining model developed by OpenAI. Its name derives from its ability to compare text and image pairs learning. This enables it to effectively associate textual descriptions with visual content.
CLIP is a multimodal artificial intelligence model that links images and text. It operates without requiring supervised learning based on traditionally labeled data datasets. As a result, it eliminates the need for a labor-intensive labeling phase in data preparation important and time fields stage. By learning from vast amounts of image-text pairs on the internet, it can generalize across a broad spectrum of visual concept categories.
CLIP uses contrastive learning to match text and images. The model processes images and text separately, representing them as vectors in a shared high-dimensional space and then computing their similarity【1】. This allows the model to adapt more broadly to diverse real life applications. Moreover, CLIP’s versatility and zero-shot performance have made it a foundational model for numerous modern artificial intelligence applications, ranging from image generation to search engines.
Visual classification and search: CLIP can recognize and categorize images based on natural language commands
He, Y., Sui, Y., He, X., Liu, Y., Sun, Y., & Hooi, B. (2025). UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs. arXiv preprint arXiv:2502.0080.
Lv, S. L., Chen, Y. Y., Zhou, Z., Li, Y. F., & Guo, L. Z. (2025). Contrast-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment. arXiv preprint arXiv:2501.19060.
OpenAI, GitHub. CLIP Repo. Accessed Adresi.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning.
Tellez, A., Pumperla, M., & Malohlava, M. (n.d.). Mastering Machine learning with SPARK 2.X. Packt Publishing Ltd.
[1]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G. and Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning..
Text and Image Encoding
CLIP uses two separate neural networks to convert both text and images into vector representations (embeddings):
For images: A Vision Transformer (ViT) or ResNet model generates a high-dimensional vector representation (embedding) of the image.
For text: A Transformer-based text encoder converts textual descriptions into similar vector forms.
These encoders are trained on millions of image-text pairs, enabling the model to learn the meaning of both images and text within a shared representation space.
Contrastive Learning and Cosine Distance
During training, CLIP compares all images and texts within a minibatch against each other:
Positive pairs (correct text-image matches) are learned to be closer together in the shared space.
Negative pairs (mismatched text-image pairs) are pushed farther apart in the shared space.
In this process, the Cosine Similarity metric is used to calculate similarities between vectors and guide model training.
【1】
Zero-Shot Learning
Once training is complete, CLIP can match any new image to a textual description without requiring additional training. This is because the model has learned to recognize concepts it has never encountered before, thanks to its generalized representation space. This is one of CLIP’s most powerful features.
Thanks to these principles, CLIP has made significant advances not only in visual classification but also in text-to-image production, content moderation, and robotics perception like applications.
Contribution to Literature
CLIP revolutionized computer vision by eliminating the need for task-specific datasets. Unlike traditional models trained exclusively for specific tasks, CLIP possesses zero-shot learning capability. This means it can recognize and classify images without being explicitly trained on predefined categories
CLIP Function
CLIP Working Principle
Common Use Cases