What is most important in the era of large models?

Computing power? As the saying goes, if you have a card, you can sleep in the company, but if you don’t have a card, you can’t talk about relationships.

Talent? Big model talents are in hot demand, and buying horse bones for thousands of dollars is also a legend.

But from the perspective of model developers, the most core factor is the least topical: data, data, or data.

The current increase in computing power density is still growing at an exponential rate year by year. The increase in talent density can also be seen from the rapid explosion in the number of submissions to major AI conferences. However, only data, especially high-quality data, does not follow any scaling law. One of the core points that OpenAI has been criticized for is that the growth rate of real-world data cannot keep up with the growth rate of model parameters, and the scaling law of the model is unsustainable. Data is critical to large models, and high-quality data that incorporates the accumulated knowledge of professional talent is even more important**.

The current large model has evolved from early pure text generation to the current multimodal reasoning. The introduction of image–text multimodal data continues to expand the capabilities of artificial intelligence models. In this context, the team of Dr. Leng Dawei of 360 Artificial Intelligence Research Institute in its latest cross-modal alignment work FG-CLIPBuilding on the open-source FG-CLIP model,, the "Da Vinci Code" behind the success of FG-CLIP is further organized and released as open source: FineHARD high-quality image–text alignment dataset.

"FineHARD dataset focuses on two core features: fine-grained + hard negatives."

Dr. Leng Dawei made a concise summary of the main focus of the work on the FineHARD dataset.

Significantly different from the existing image–text alignment dataset, the FineHARD dataset focuses on the fine-grained alignment of images and text while scaling up. To this end, the FG-CLIP team has made a lot of efforts in data preparation. The FineHARD dataset’s fine-grained is reflected in three aspects:

1. Global fine-grained Alignment: The FineHARD dataset not only contains conventional "short text" descriptions of images (average length is about 20 words), but also in order to make up for the lack of details in short text descriptions, the FG-CLIP team generated "long text" descriptions (average length) for each image in the dataset that contain detailed information such as scene background, object attributes, and spatial relationships based on the multimodalLMM model. 150 words+), significantly improving the global semantic density.

2. Partialfine-grained alignment: The "long text" description mainly lays the data foundation for fine-grained alignment from the text side. In order to further improve the fine-grained capability from the image side, The FG-CLIP team extracted the positions of most target entities in the image based on the open-world object detection model for each image in the FineHARD dataset, and matched the corresponding region description for each target area. The FineHARD dataset contains up to 40 million bounding boxes and their corresponding region-level fine-grained description text.

3. fine-grained hard negatives: Based on the above-mentioned global fine-grained alignment and local fine-grained alignment, in order to further improve the model's ability to understand and differentiate the alignment of image–text details, the FG-CLIP team used the LLM model to construct and clean 10 million groups of fine-grained hard negatives for the FineHARD dataset based on the detail attribute perturbation method. Large-scale hard negatives data is the third important feature that distinguishes the FineHARD dataset from existing data.

In summary, FineHARD is the high-quality image–text alignment dataset behind the FG-CLIP model. It is characterized by scale and refinement, and contains 12 million images and their corresponding long and Short description text, covering 40 million bounding box, each bounding box comes with fine-grainedregion description (Fine-Grained Regional Description). In addition, FineHARD innovatively introduces 10 million groups of fine-grained hard negatives (Hard Fine-grained Negative Samples). These interference samples screened by the algorithm can effectively improve the model's ability to distinguish similar targets.

The construction strategy of the FineHARD dataset directly responds to the core challenges in multimodal learning - cross-modal alignment (Cross-Modal Alignment) and semantic coupling (Semantic Coupling), providing new ideas for solving the "semantic gap" problem. Based on FG-CLIP trained on FineHARD (ICML' 2025) significantly outperforms the original CLIP and other state-of-the-art methods (see figure below) on a variety of downstream tasks, including fine-grained comprehension, open vocabulary object detection, long and short text image–text retrieval, and the general multimodal benchmark, among others.

• Project Github: https://github.com/360CVGroup/FG-CLIP

• Data set address: https://huggingface.co/datasets/qihoo360/FineHARD

FineHARDDataset construction

The FineHARD dataset is based on 12 million high-quality images, and each image is equipped with accurate semantic description text. The dataset contains 40 million bounding box annotations, each bounding box is accompanied by a regional-level fine-grained description (Fine-Grained Regional Description), and 10 million groups of fine-grained hard negatives are integrated through algorithm screening. In the data preprocessing stage, the team adopted a distributed computing architecture and relied on an NPU cluster with 160×910B computing power to complete core operations such as data cleaning, feature extraction and multimodal alignment within 7 days, achieving efficient conversion from original images to structured data.

multimodalDescription generation mechanism

The text description system of FineHARD is deeply optimized based on the GRIT dataset. First, 12 million representative images are retained through strict screening, and then large multimodal model (LMM)CogVLM2-19B with smaller hallucination information is introduced to generate a long description text containing scene background, object attributes and spatial relationships for each image. Compared with the short summary descriptions of the original GRIT dataset (the average length is about 20 words), the text descriptions of this dataset have been extended to more than 150 words on average, significantly improving the semantic density and scene restoration. This description system not only retains the versatility of the original dataset, but also enhances semantic expression capabilities through refined annotation.

Co-construction of bounding box and semantic description

Based on the generated long description text, the natural language processing tool spaCy is used to extract and parse referential expressions (such as "red car", "bottle in the upper left corner"). Although the original GRIT dataset provides the basic bounding box, it has problems such as incomplete category coverage (for example, some images only label people). To this end, the FG-CLIP team designed a dual enhancement strategy:

fine-grained description supplement: fine-grained supplements the basic bounding box to refer to the description.

object detection extension: jointly infer images and referential expressions through the pre-trained Yolo-World model to generate additional bounding box. Non-maximum suppression (NMS) technology is used to filter overlapping areas and only retain high-quality prediction results with a confidence level >0.4.

Finally, a 40 million bounding box system containing regional-level semantic descriptions was constructed to achieve precise alignment of visual elements and semantic information.

fine-grained Negative sample generation and quality verification

In order to improve the model's ability to discriminate similar targets, the FG-CLIP team developed a negative sample generation scheme based on attribute perturbation. The specific process is as follows:

Attribute modification strategy: On the premise of keeping the object name consistent, use the open source large language model (LLM)Llama-3.1-70B to replace the attributes of the positive sample description (such as changing "red car" to "blue car"), and generate 10 negative samples with similar attributes but different semantics;

Text standardization: remove special symbols (semicolons, commas, line breaks, etc.) to ensure uniformity of description format;

Quality assessment system: 3,000 samples were manually reviewed, and the results showed that 98.9% of the samples met the quality standards, and only 1.1% were judged to be noise - this error rate is within the reasonable threshold range of the unsupervised generation method.

Such negative samples with subtle semantic differences are closer to the complex situation in real scenes where objects have similar appearances but different attributes, making the model more robust in the visual grounding task.

FineHARDDataset Analysis

Comparison of common datasets

In order to quantitatively analyze the characteristics of the FineHARD dataset, we compared it with some commonly used datasets in the industry, such as Flickr30k, CC3M and COCO in four dimensions: image, text description, target bounding box and hard negatives, as shown in the figure below.

The FineHARD dataset performs particularly well in terms of size and quality, especially in terms of fine-grained annotations and challenging negative samples. In terms of scale, FineHARD covers 12 million images, 40 million bounding box and corresponding descriptions, far exceeding similar datasets. For example, compared with the widely used COCO dataset, which only provides 1.5 million bounding box, the FineHARD dataset has 40 million bounding box, which greatly enriches the training resources for object localization and recognition. In addition, a notable feature of the FineHARD dataset is that it contains 10 million difficult fine-grained negative samples. These samples are carefully designed to help the model better distinguish subtle differences between semantically similar objects, thereby effectively improving its performance in various downstream tasks. Through such a comprehensive construction, the FineHARD dataset not only has an advantage in quantity, but also provides a solid foundation for the development of advanced visual understanding and object detection technology in terms of quality.

fine-grainedData set comparison

We further compared it with other specialized fine-grained datasets such as LVIS and V3Det. FineHARD extracts and summarizes category labels from the region descriptions generated by CogVLM2-19B and YOLO-World to build object category information covered in the dataset. The following table shows the comparison of different datasets in the number of images, text descriptions and the number of independent category labels summarized from different text descriptions. Note that because of the huge gap, the horizontal and vertical coordinates are all logarithmic coordinates:

In order to further analyze the sample diversity of the FineHARD dataset, we randomly sampled a subset of the same image size (243,000 images) as V3Det for comparison. In this subset, FineHARD contains 21k independent category labels, which is significantly higher than the 13k of V3Det, indicating that the FineHARD dataset has obvious advantages in semantic coverage and diversity. In addition, we use the t-SNEdimensionality reduction method to visually display the category labels of the sampled data:

This figure further verifies that under the same image scale, the FineHARD dataset presents a wider category distribution, indicating its richness and higher diversity in visual semantics. As the dataset expands to 12 million images, the diversity of category labels and description text further increases significantly. This increase in scale not only enhances the model's learning ability for rare categories, but also provides more comprehensive data support for fine-grained visual understanding tasks. This marks an important step in building large-scale, high-quality, and high-diversity visual language datasets.

Conclusion

From the perspective of technological application prospects, the FineHARD dataset will have an impact on multiple frontier fields:

large multimodal model (LMM) Training: Through pre-training of massive image-text-region alignment data, the model's cross-modal understanding and generation capabilities can be significantly improved, especially the understanding and alignment of image details;

embodied AI System development: Combining fine-grained spatial description and action semantics to provide robots with more accurate environmental perception and operation instruction analysis capabilities, and promote the upgrade of industrial automation to the cognitive decision-making level;

3D scene modeling and digital twin: Regional-level fine-grained description can be used as the basis for generating semantic feature point clouds, providing high-precision semantic anchor points for virtual scene reconstruction, and accelerating the development of immersive technologies such as AR/VR;

fine-grained Recognition breakthrough: Through the adversarial training of hard negatives, the model's ability to distinguish similar categories (such as different models of cars, similar varieties of flowers) can be effectively improved, and the application of security, retail and other scenarios can be promoted.