FG-CLIP: Fine-Grained Visual and Textual Alignment

We have solved the "myopia" problem of CLIP.

360 Artificial Intelligence Research Institute's latest graphic cross-modal model FG-CLIP announces double breakthroughs in "long text in-depth understanding" and "fine-grained visual comparison", which completely solves the "visual myopia" problem of the traditional CLIP model and can accurately identify local details.

How to say it specifically? Let’s start with a vision challenge: Find which sentence on the right correctly describes the content of the image on the left?

The correct answer is: "A light brown wood stool." Note that this wooden stool is located in the center right of the picture, quietly hidden behind the dog.

It can be found that the most matching text description selected by 4 commonly used models - CLIP, EVACLIP, SIGLIP, and FINE-CLIP based on the picture on the left is: A blue dog with a white colored head.

Obviously this description is wrong. This is the "visual myopia" problem of CLIP: because the contrast loss tends to bring the embedding of the global image–text closer, rather than the alignment of local areas, it weakens fine-grained feature learning.

And FG-CLIP accurately hit the answer.

Experimental results show that FG-CLIP significantly outperforms the original CLIP and other state-of-the-art methods in downstream tasks such as fine-grained comprehension, open vocabulary object detection, long and short text image–text retrieval, and general multimodal benchmarks.

On 12 downstream tasks, FG-CLIP achieved significant breakthroughs in key long text understanding + fine-grained comparison compared to existing models.

360 Artificial Intelligence Research Institute also stated that it will comprehensively collect open-source model and its related data.

Questions vision–language model (VLM) faces

In 2021, OpenAI released the CLIP image–text cross-modal model. Through comparative learning, it achieved large-scale image-text alignment for the first time, opening a new era of multimodal pre-trained large models. It enables the model to complete tasks such as zero-shot classification and image retrieval by comparing the embedding spaces of images and texts.

However, CLIP and the models developed later still face the following constraints in practical application:

Text length limit: The text encoder of CLIP only supports 77 tokens, which is difficult to handle detailed descriptions of long texts (such as "a red ceramic tea cup with slight wear on the rim").

Blind spots in global alignment: CLIP aligns the image–text as a whole, ignoring the fine-grained information in local areas (such as the handle shape of the tea cup and the cup body pattern).

Shortcomings of negative samples: In existing datasets, negative samples (unmatched image-text pairs) are mostly rough category errors and lack the ability to distinguish subtle differences.

The importance of understanding detailed descriptions in long texts

Provide rich background information and complex query capabilities: Long text can provide detailed background information, including action status, object attributes and change processes, etc., which is crucial for a comprehensive understanding of events.

Compared with short text analysis, long text allows comprehensive search for information based on multiple conditions (such as objects, character characteristics), and supports more complex query requirements. This allows the model to not only identify what happened, but also understand the full picture of the event and its context.

Supports cross-modal deep semantic matching and fusion capabilities: The cross-modal model needs to establish effective semantic correspondence between different modalities.

Multi-level semantic information in long texts (such as topics, paragraphs, sentences and even vocabulary-level meanings) can help the model perform semantic matching and feature fusion more accurately.

In the image–text retrieval task, long text descriptions can cover comprehensive information from global scenes to local details, allowing the model to compare and match image features at multiple levels, thereby improving the accuracy and relevance of retrieval.

The importance of accurate analysis of local area fine-grained information

Detail Capture: Local image features often contain key information that distinguishes different objects.

For example, when analyzing different characters, differences in attributes such as clothing and movements are crucial to distinguishing individuals. Accurate analysis of these local features can significantly improve the accuracy of recognition systems.

Complex environment adaptability: In complex backgrounds or low-quality images, local features can help the algorithm focus on the most important information and ignore interference factors.

In practical applications, target objects are often partially occluded by other objects. In this case, global features may not be enough to describe the object, and local features are particularly important.

Through precise analysis of local features, the system is able to effectively identify unoccluded parts and use this information to infer the state of the entire object.

Accurate analysis of local image feature attributes: It is of core importance in improving recognition accuracy, enhancing environmental understanding, supporting high-level applications, improving user experience, and ensuring security.

By accurately parsing these detailed information, smarter and more reliable system performance can be achieved, playing a key role in surveillance, autonomous driving, product quality control, or other areas that require detailed image analysis.

The importance of achieving accurate understanding of nuances in images/text

Enhance the robustness and generalization ability of the model: Accurately distinguishing subtle differences in images and text is crucial to enhance the robustness and generalization ability of the model.

Understanding fine-grained enables models to differentiate between objects, scenes, or concepts that are visually or semantically similar but have subtle differences.

This capability is important for real-world applications, where objects may change slightly under different lighting, angles, or backgrounds. Ensure that the model can run reliably in various complex scenarios.

Improving Accuracy in Downstream Tasks: Accurately identifying subtle differences is critical to improving accuracy in downstream tasks such as image description generation, visual question answering (VQA), and medical imaging diagnosis.

For example, in visual question answering (VQA), identifying tiny details in an image and understanding their relevance to the question is key to answering the question correctly. The ability to accurately capture subtle differences in images directly affects system performance and user experience.

Likewise, in natural language processing, identifying subtle differences in text can significantly improve performance on tasks such as sentiment analysis and information retrieval.

model approach

FG-CLIP adopts a two-stage training strategy based on the traditional dual-encoder architecture, which effectively improves the fine-grained understanding ability of vision–language model (VLM).

In the first stage, the initial alignment of image–text representations is achieved through global contrast learning; in the second stage, regional contrast learning and difficult fine-grained negative sample learning are introduced, and regional-text annotation data is used to deepen the model's perception of visual details, thereby achieving accurate capture of local features while maintaining global semantic understanding.

Global comparative learning

Global contrastive learning significantly enhances the model's understanding capabilities by integrating long descriptions generated by large multimodal model (LMM).

This approach not only generates long and informative descriptions, but also provides more complete contextual information and more precise detailed descriptions.

By introducing long descriptions, the model is able to perceive and match semantic details at a global level, thereby greatly improving its context understanding capabilities.

At the same time, FG-CLIP retains the original short description-image alignment mechanism, making long and short descriptions complementary.This dual-track parallel strategy enables the model to obtain complex semantic information from long descriptions and grasp core concepts from short descriptions, thereby comprehensively improving the model's ability to understand and process visual information.

Local contrast learning

Local contrast learning achieves fine-grained visual-linguistic association by accurately aligning local areas of the image with corresponding text descriptions.

Specifically, FG-CLIP first uses RoIAlign to accurately extract regional features from the image, and then applies an average pooling operation to each detection area to obtain a set of representative regional-level visual representations.

These local features are then compared and learned with pre-constructed fine-grained text descriptions, prompting the model to establish an accurate mapping relationship between regional visual content and text semantics, thereby mastering more detailed cross-modal alignment capabilities.

Regional level hard negatives comparative learning

In response to the challenge of fine-grained the scarcity of negative samples, FG-CLIP proposed a difficult fine-grained negative sample learning method.

FG-CLIP defines samples with similar semantics but slight differences from positive samples as hard negatives, and constructs these samples by fine-tuning and rewriting the bounding box description at the attribute level.

In order to make full use of the discriminative information provided by difficult negative samples, a specific negative sample learning strategy is introduced in the loss function.

During the training process, the model needs to simultaneously calculate the similarity between regional features, positive sample descriptions, and their corresponding negative sample descriptions to learn a more refined visual-language alignment relationship.

Data construction

Detailed image description rewriting via LMM

In the initial training stage, FG-CLIP uses the enhanced and optimized LAION-2B dataset, in which the image annotations are regenerated through CogVLM2-19B.

This improvement significantly improves data quality, making descriptions more precise and informative.

The traditional LAION-2B dataset often uses a general description method, which is difficult to support the needs of refined tasks.

Taking bird images as an example, the original annotation may only be "a bird", ignoring species characteristics and environmental details.

By introducing advanced large multimodal model (LMM), the description generated by FG-CLIP not only accurately identifies the target object, but also covers multi-dimensional information such as object characteristics, behavior patterns and scene association.

For example, the simple "a bird" was optimized to "a red-winged blackbird perched on a branch in the park", which greatly increased the information density of the description.

With the help of a 160×910B NPU computing cluster, FG-CLIP completed all data processing work within 30 days.

Experimental results show that this optimization significantly improves the model's performance on multiple tasks, fully proving the key role of high-quality text annotation in improving model accuracy and context understanding capabilities.

Create high-quality visual grounding data

For the second stage of training, FG-CLIP developed a high-quality visual grounding dataset containing precise region-specific descriptions and challenging fine-grained negative samples.

FG-CLIP Make the entire dataset based on the images provided by GRIT.

The process begins with generating detailed image descriptions using CogVLM2-19B, ensuring that the descriptions are comprehensive and detailed, capturing the full context of each image. Subsequently, FG-CLIP uses SpaCy to parse these descriptions and extract referential expressions.

Next, the image and referential expression are input into a pre-trained open vocabulary detection model, here using Yolo-World, to obtain the corresponding bounding box.

Eliminate overlapping bounding box through non-maximum suppression, and only retain bounding box with a prediction confidence score higher than 0.4.

This process produced 12 million images and 40 million bounding box with fine region description.

In order to generate high-quality fine-grained negative samples, FG-CLIP finely adjusts the attributes described by bounding box while keeping the object name unchanged.

Specifically, FG-CLIP constructs 10 corresponding negative samples for each positive sample with the help of Llama-3.1-70Blarge language model (LLM).

To improve the readability of the description, FG-CLIP has removed punctuation marks such as semicolons, commas, and newlines.

After quality assessment of 3,000 negative samples, 98.9% of the samples met the expected standards, and only 1.1% were judged to be noise data. This proportion is in line with the acceptable range of unsupervised methods.

The subtle changes produced by this method are closer to the real scene and can better simulate the situation where objects have differences in specific details while maintaining similar basic categories.

This large-scale dataset consists of 12 million high-quality images, each equipped with precise semantic descriptions.

It contains 40 million bounding box annotations, each bounding box comes with detailed region description, and also integrates 10 million screened difficult fine-grained negative samples.

The data processing stage called an NPU cluster with 160×910B computing power, and it took 7 days to complete efficiently.

This rich and systematic dataset significantly improves the model's ability to identify fine features and lays a solid foundation for FG-CLIP's training, enabling it to perform well in fine-grained understanding of visual and textual features.

Experimental results-quantitative indicators

fine-grainedIdentification

FG-CLIP conducted a systematic evaluation of open source image-text alignment models based on the FG-OVD dataset.

Different from traditional benchmarks such as MSCOCO and Flickr that focus on overall matching, FG-OVD focuses on examining the model's refined ability to identify and locate local areas of the image.

During the evaluation process, each target region is equipped with a precise description and ten carefully designed negative samples, which are generated by strategic modification of the correct description.

The FG-OVD dataset is divided into four subsets with increasing difficulty, and their discrimination is mainly reflected in the degree of similarity between the texts to be matched.

Specifically, the hard, medium, and easy subsets construct negative samples by replacing one, two, and three attribute words respectively, while the trivial subset uses completely irrelevant text descriptions, forming an evaluation system from subtle differences to significant differences.

As can be seen from the table, FG-CLIP can achieve significant improvements in various indicators compared with other methods, which also proves the method's ability to understand fine-grained.

Region identification

FG-CLIP conducts zero-sample testing on the COCO-val2017 dataset to evaluate the model's ability to identify local information. The test plan refers to FineCLIP and CLIPSelf.

This evaluation focuses on the model's performance in classifying objects in bounding box relying solely on text descriptions.

In the specific implementation, FG-CLIP uses the bounding box annotation in the dataset and combines it with ROIAlign technology to extract dense feature representation of local areas.

In the testing phase, all category labels are input as candidate text, each bounding box region is matched and classified, and the performance is evaluated through Top-1 and Top-5 accuracy. FG-CLIP also achieved the best results on this downstream task.

open-vocabulary object detection

In order to further evaluate the fine-grained localization ability of FG-CLIP's method, FG-CLIP was adopted as a Backbone for the downstream open word detection task.

Specifically, FG-CLIP adopts a two-stage detection architecture F-VIT and freezes vision encoder during training.As can be seen from the table, FG-CLIP performs more prominently on the open-vocabulary object detection task, proving that models trained with high-quality data and optimization methods can achieve superior performance on deeper tasks.

image–text retrieval/Classification results

To comprehensively evaluate the task of image strength, FG-CLIP conducted experiments on long and short title image text retrieval tasks and a zero-shot image classification task.

As shown in the table, FG-CLIP achieves significant performance improvements in both long/short title image-text retrieval tasks.

Compared with Long-CLIP and FineCLIP, which aim to improve the recognition ability of fine-grained, FG-CLIP has obvious advantages in accuracy on short text-full image problems such as image classification. The model’s ability to handle different image description lengths highlights its versatility and robustness in multimodal matching.

Experimental results-visual comparison

Comparison of image detail difference effects

FG-CLIP visualizes image features for text input.

In the figure, warm colors (such as yellow) indicate higher correlation, while cool colors (such as blue) indicate lower correlation.

The first is to compare the ViT features of different models for the same input text and images. It can be found that FG-CLIP performs better on this kind of fine-grained understanding problem. As shown in the second line of the figure, when "Black nose" is input, FG-CLIP can achieve accurate recognition of this small target.

Visual diagram under different input text

FG-CLIP also performs correlation analysis on different input texts and the same image.

It can be found that FG-CLIP can give accurate position understanding for different targets in the image, which shows that the model has stable visual grounding and fine-grained understanding capabilities.

Summary

FG-CLIP has made breakthrough progress in the field of fine-grained visual understanding.

This model innovatively integrates cutting-edge technology and achieves multi-level semantic analysis of images based on large-scale selected datasets and difficult negative sample learning strategies.

Its unique advantage lies in its ability to simultaneously grasp the global context and local details, and accurately identify and distinguish subtle feature differences.

A large number of experimental results show that FG-CLIP exhibits excellent performance in various downstream tasks.

360 Artificial Intelligence Research Institute stated:

In order to promote the development of the field, the research team decided to open source the data, code and pre-training models related to FG-CLIP one after another. The relevant content will be released on the homepage of 360 Artificial Intelligence Research Institute and GitHub.

The future research direction of the research team will focus on integrating more advanced multimodal architecture and building a richer and more diverse training dataset to further expand the technical boundaries of fine-grained visual understanding.

Github: https://github.com/360CVGroup