FG-CLIP 2: Fine-Grained Bilingual Vision–Language Understanding
From "seeing" to "seeing clearly", cross-modal the next evolution of understanding
OpenAI's CLIP model is a milestone achievement of the Foundation Model. It successfully proves that through comparative learning of large-scale image–text data, the AI model can master the generalization association ability across vision and language, laying a solid foundation for zero-sample recognition, large multimodal model (LMM), image generation, and downstream models and applications such as search, recommendation, office, and security. It has solved to a large extent the problem of allowing AI to "see" and roughly understand graphic content.
Technological progress is always accompanied by the breakthrough of existing boundaries. The core advantage of the first-generation cross-modal model represented by CLIP lies in the understanding of macro-subjects, such as identifying "a dog in the park", but its limitations become apparent when faced with tasks that require precise detailed cognition. It is difficult for them to stably distinguish complex understanding scenes such as "a golden retriever catching a red Frisbee in mid-air" because the model cannot distinguish the detailed attributes and complex interactions of the object. This "myopia" phenomenon of "understanding but not seeing clearly" is an inherent fine-grained understanding bottleneck of the first generation model.
In order to break through this bottleneck, the visual team of 360 Artificial Intelligence Research Institute led by Dr. Leng Dawei proposed a new generation of image–text cross-modal model FG-CLIP at ICML2025, focusing on fine-grained image–text alignment and understanding capabilities. Now, the fully upgraded FG-CLIP 2 is officially released and open source.
FG-CLIP 2 comprehensively innovates the previous generation model from the training paradigm, objective function to data ecology, solves the myopia and coarse-grained bag-of-words effect of CLIP from the source, and provides the Chinese language ability support that the industry urgently needs. In benchmark evaluations with 29 items in 8 categories, FG-CLIP 2 surpasses SigLIP 2 and SigLIP 2. The latest powerful baseline including MetaCLIP 2, bilingual performance reaches best reported bilingual performance**.
English benchmark comprehensive ranking
Chinese benchmark comprehensive ranking
FG-CLIP 2Performance Radar Chart
FG-CLIP 2 Paper address: https://arxiv.org/pdf/2510.10921
FG-CLIP 2 Model, code and dataset address: https://360cvgroup.github.io/FG-CLIP
FG-CLIP 2 API access address: https://research.360.cn/sass/fg-clip/fg-clipDocument
The "ceiling" of the first generation model: fine-grainedThe root of stubborn diseases
To understand the value of FG-CLIP 2, we must first clearly understand the fundamental obstacles faced by the first-generation model in image–text understanding tasks.
1. Extensive data and single goal
The success of the first-generation models stemmed largely from “number advantage” as they digested billions of Internet image-text pairs. However, the quality of the textual descriptions of these data varies, often with only loose thematic connections to the image content rather than precise details. Based on the training goal of overall feature alignment, the model is repeatedly told that "this picture" and "this paragraph" are related during training. This coarse-grained supervision signal encourages the model to learn a "topic alignment." The model is therefore good at connecting the visual features of “beach” and “ocean” with the textual concepts of “summer” and “vacation”, but this is not sufficient to support deeper understanding.
First-generation models that rely on global features get confused when the task requires distinguishing between "a boy wearing a blue T-shirt on the left" and "a boy wearing a blue T-shirt on the right." Because it has never been systematically trained to pay attention to the local area of "blue T-shirt", nor is it required to accurately map the spatial relationship word "left" to the specific coordinates of the image. Its core contrastive learning goal is to narrow down the global features of matching image-text pairs, while ignoring the precise correspondence of local details.
2. Language barriers and lack of evaluation
Prior to FG-CLIP 2, development of the fine-grained cross-modal model was almost entirely focused on the English-speaking world. Although some work has made progress in improving English fine-grained understanding, the Chinese-speaking world has seriously lagged behind and has made few achievements. Representatives of the first generation Chinese cross-modal models such as Chinese-CLIP and R2D2, their main capabilities are still limited to global tasks such as short text retrieval, and lack effective support for regional level details and long text descriptions.
Another deeper obstacle is that the field of Chinese multimodal lacks even a recognized comprehensive benchmark test that can rigorously assess fine-grained comprehension ability. Most of the existing Chinese datasets are still at the short text retrieval level, which leads to the lack of clear optimization directions for model developers and the inability to accurately measure the true level of the model on the fine-grained task, which restricts the system development in the entire field.
Based on the in-depth insights into the above key issues, the new design of FG-CLIP 2 was born - a second-generation model framework designed to achieve refined and bilingual understanding.
Implementation path of the second generation model: method innovation of FG-CLIP 2
The intergenerational breakthrough of FG-CLIP 2 is reflected in its unique new paradigm of "two-stage training, multi-objective optimization, and high-quality data". This series of designs aims to transform the model from a "macro observer" to a "micro analyst".
1. Two-stage hierarchical learning: from erudition to mastery
The training process of FG-CLIP 2 is divided into two progressive stages, simulating the learning process from broad cognition to in-depth understanding.* Phase 1: Building a Solid Semantic Base In the initial phase, the model's task is to establish strong and extensive cross-modal semantic connections. FG-CLIP 2 introduced the "long and short dual text" strategy from the beginning. For each picture, the model not only learns its original short title, but also simultaneously learns a detailed long description generated by large multimodal model (LMM). Short text provides concise, core labels in the real world, while long descriptions inject rich context, object attributes, and structured language knowledge into the model. By digesting both types of text at the same time, FG-CLIP 2 builds a much deeper foundation of semantic understanding than a single rough label in the early stages of training. At the data level, this stage integrates the enhanced LAION-2B English dataset and a total of 850 million Chinese image–text pairs, ensuring its strong bilingual capabilities.
- The second stage: carefully crafted fine-grained alignment After the model has a solid global understanding ability, the second stage of training will completely shift its focus to "details". This stage introduces the fine-grained dataset with exact bounding box (bounding box) and corresponding region description. In addition, in order to further improve the model's understanding of detailed attributes, FG-CLIP 2 constructed a unique large-scale hard negatives dataset FineHARD. The training objectives have also become rich and three-dimensional, and then jointly optimized through five complementary objective functions to polish the fine-grained capabilities of the model from different dimensions.
2. Five-in-one objective function matrix: the core engine of refined understanding
Different from the first generation model, the optimization goal of FG-CLIP 2 is no longer simple overall feature alignment, but a set of verified collaborative optimization function matrices.
Global Alignment** Learning****(
L_Global)**: Continues and optimizes SigLIP’s sigmoid loss, continues to consolidate macro image–text matching capabilities, and ensures that the model does not lose its grasp of the overall scene while pursuing details.fine-grained visual Learning (
L_FGV): This is the key to achieving "visual focus". The model is trained to align the visual features of a specific region in an image with the textual description of that region. This allows the model to generate dense features that can represent local details, rather than just a general global feature.fine-grained text Learning (
L_FGT): In order to improve the model's language discrimination, this goal introduces the "fine-grained hard negatives" mechanism. For a correct region description (such as "a red cat"), the training data will provide a large number of negative samples with very similar semantics but with key attribute differences (such as "an orange cat", "two red cats"). By training the model to suppress similarities with these "distractors", its ability to distinguish subtle language differences such as color, quantity, and movement is greatly enhanced.cross-modal ranking loss (
L_CMR): This loss function is designed to strengthen the "dominant position" of correct matching. It requires that the similarity score between the correct image–text pairs should not only be higher than the negative sample, but also higher than a dynamically adjusted "safety margin" (margin). This boundary value is consistent on all training nodes through a global synchronization mechanism, ensuring the stability and consistency of training.Contrast loss within the text domain (
L_TIC): This is the key to solving the fine-grained text confusion problem. When dealing with massive region description, many texts themselves are very similar (such as "a green wooden door" and "a door painted green").L``_``TICspecializes in comparative learning within the text encoding space. It treats the descriptions with the most similar semantics in a batch of texts as mutual traps hard negatives, forcing the text encoder to assign discriminative representations to these "synonyms". This greatly improves the model's robustness when dealing with highly similar text descriptions.
Through this carefully designed set of optimization constraints and data ecological matching, FG-CLIP 2 has been systematically shaped into a second-generation cross-modal model that can grasp the overall situation, gain insights into the smallest details, and manage bilingual complexity.
Evidence is King: Defining Performance Benchmarks for Second Generation Models
The advancement of FG-CLIP 2 was finally proven by experimental results across 8 major categories of tasks and 29 public datasets. These performance indicators demonstrate its advantages as a new generation cross-modal model.
1. Set a new benchmark for Chinese evaluation
Before demonstrating the performance, the FG-CLIP 2 team built and launched a new set of Chinese evaluation benchmarks to make up for the shortcomings in Chinese long text evaluation and regional-level classification evaluation, including LIT-CN, DCI-CN and DOCCI-CN for long text retrieval, and BoxClass-CN for regional-level classification.
LIT-CN (Long-text Image-Text Retrieval): Focus on long text retrieval tasks. The average text length exceeds three times the existing short text evaluation dataset, testing the model's global semantic matching ability in complex contexts.
DCI-CN (Detailed Caption-Image): Strengthen detail-level description and local feature correspondence, including rich object attributes, actions and spatial relationships, and test whether the model can recognize the combination of multi-entities and multi-attributes in the image.
DOCCI-CN (Descriptive Object-Centric Caption-Image): Constructed from an "object-centered" perspective, each sample focuses on a single target and its contextual details to evaluate the accuracy of the model's understanding at the object level.
BoxClass-CN (Box-level Classification Benchmark): The first Chinese regional-level classification benchmark, which labels semantic categories and text descriptions for each image area, testing the model's ultimate capabilities in local alignment and fine-grained recognition.
This contribution fills the gap in Chinese multimodal research and provides a solid quantitative evaluation basis for subsequent work.
2. Make a breakthrough in the core "fine-grained understanding" task
In the FG-OVD benchmark that best reflects the capabilities of fine-grained image–text alignment, FG-CLIP 2 performed very well. On the most challenging "hard" subset, FG-CLIP 2 (ViT-B/16) achieved an accuracy of 52.3%, significantly better than the existing baseline, demonstrating its excellent ability to distinguish extremely similar visual-language correspondences.
On the bounding box classification task, whether on the English COCO and LVIS datasets or the newly constructed Chinese BoxClass-CN dataset, FG-CLIP 2 achieved the current best (SOTA) results with a significant lead, fully demonstrating its cross-language and powerful ability to align local content and semantic concepts.
3. Master complex languages and lead in long text retrievalIf short text retrieval tests basic matching ability, then long text retrieval is a comprehensive test of model fine-grained understanding ability. In both Chinese and English long text retrieval tasks, FG-CLIP 2 shows a huge lead. It is particularly worth mentioning that on multiple lists in Chinese and English, the performance of FG-CLIP 2 (ViT-L/16, 1 billion parameters) even surpassed Meta CLIP 2 (ViT-H/14, 1.8 billion parameters), which has a larger number of parameters. This strongly proves the efficiency of the FG-CLIP 2 training paradigm - achieving stronger performance with a smaller model size, which is a manifestation of "generational leadership" rather than "parameter stacking".
4. Empower downstream tasks and demonstrate application potential
The value of FG-CLIP 2 goes beyond refreshing the list. In the open-vocabulary object detection (OVD) scenario, when FG-CLIP 2 is combined with the detector LLMDet, the best performance among open source methods is achieved, proving that its excellent generalization ability can be effectively transferred and improve the accuracy of downstream systems. By using Cat-Seg as the basic architecture, FG-CLIP 2 also achieved better performance in the field of open vocabulary segmentation. More importantly, when FG-CLIP 2 is used as vision encoder and integrated into large multimodal model (LMM), the LMM equipped with FG-CLIP 2 comprehensively surpasses similar models using other open source vision encoder on multiple advanced multimodal inference benchmarks such as GQA and MMMU. This shows that FG-CLIP 2's powerful fine-grained and bilingual abilities can be effectively transferred to higher-level cognitive tasks, making it an ideal cornerstone for building the next generation of LMM.
5. ablation study
In order to verify the effectiveness of the innovative module, the research team conducted rigorous ablation study. The results show that when its original contrast loss in the text domain L``_``TIC is removed, the model's Top-1 accuracy on the COCO classification task plummets by 4.8 percentage points. When L_TIC and cross-modal sorting lossesL``_``CMR are removed at the same time, the accuracy drops significantly. This data clearly proves that it is these modules, carefully designed for the fine-grained mission, that form the core of FG-CLIP 2's success.
Conclusion: Towards a new chapter of precise understanding
CLIP opens the era of cross-modal learning and solves the problem of "whether pictures and texts can be aligned"; while the second generation capability jump led by FG-CLIP 2 is dedicated to solving the core problem of "whether it is possible to understand deeply". FG-CLIP 2’s contributions include:
In terms of paradigm, it replaces a single global alignment with a two-stage learning from "coarse to fine".
Technically, it uses a five-in-one multi-objective function matrix to achieve refined modeling of the relationship between vision, text, and cross-modal.
Ecologically, it not only achieves both Chinese and English bilingualism, but also fills the gap for the development of the Chinese field by building a new evaluation benchmark.
FG-CLIP 2 used solid experimental data to prove that it is a worthy representative of the second generation fine-grained visual language alignment model. With the full open source of FG-CLIP 2's model, code and benchmark, more accurate image search, smarter human-computer interaction, more reliable robot scene understanding, and more powerful multimodal content creation will all have a solid technical base that is trustworthy.