ICCV 2025 | LMM-Det Unlocks Native Object Detection in LMMs

360 AI Research 2025-08-05 720 views
ICCV 2025 | LMM-Det Unlocks Native Object Detection in LMMs

Large-scale multimodal model (LMMs) are undoubtedly the hottest stars in the current AI field. Their powerful understanding and reasoning capabilities in tasks such as image description and visual question answering (VQA) are amazing. However, when faced with a basic but crucial vision task - object detection (Object Detection), the performance of these "generalist" LMMs is often far behind professional "specialist" detectors.

In order to bridge this gap, the traditional method is usually to "plug-in" a powerful and specialized detection module for the LMM. However, this approach not only makes the system complex and bloated, but also deviates from the original intention of building a universal and unified AI model. Recently, a paper "LMM-Det: Make Large Multimodal Models Excel in Object Detection" accepted by the Computer Vision Conference [ICCV 2025] from 360 AI Research Institute proposed a completely different idea. The researchers proposed a concise and efficient framework called LMM-Det, which proved for the first time that LMM can perform the object detection task without any additional detection modules and only by stimulating and optimizing its own potential, and achieved performance comparable to professional detectors.

Research background and significance

Currently, the mainstream method for LMM to have object detection capabilities is usually to integrate a specialized detector (such as DETR series) at its front end or back end. Although this hybrid mode of "LMM+detector" is effective, it has obvious disadvantages:

  • Complex architecture: Additional and heavy detection modules are introduced, increasing the complexity and maintenance cost of the system.
  • Capability fragmentation: The detection capability is not native to LMM, but is provided by external modules, which runs counter to the goal of pursuing model versatility and integration.

The researchers of LMM-Det boldly put forward a core proposition: LMM itself contains powerful detection capabilities, but it has not been "unlocked" correctly. Their goal is to abandon plug-ins and make LMM "self-taught without a teacher" through a series of carefully designed optimization strategiesobject detection.

 

LMM-Det(c) Comparison with other LMMs that require plug-in region generators (a) or professional detectors (b)

 

Core method: LMM-Det

The researchers first analyzed the root cause of the poor performance of standard LMM (taking LLaVA as an example) on the object detection task through in-depth exploratory experiments. They found that the core problem was the sharp drop in recall rate. To put it simply, the model "misses" too many objects that should be detected.

The above picture shows a visual comparison of the detection effects of the professional detector and large multimodal model (LMM) on the COCO verification set. When using the prompt "If there is an area described by this sentence in the image, please provide its bounding box coordinates: ", LLaVA[23] performs poorly in detecting all objects - only a small number of bounding box are generated per query and most of them are inaccurate, resulting in a low recall rate for the object detection task.
In response to this core pain point, LMM-Det proposed a set of combinations aimed at comprehensively improving the recall rate and overall detection performance of the model.

1. Data distribution adjustment

The researchers found that the target distribution of the data that LMM was exposed to during the pre-training and instruction fine-tuning stages was significantly different from that of specialized detection datasets (such as COCO). In order for the model to "adapt" the data patterns of the detection task, they reorganized and enhanced the training data:

  • **Reorganize the command dialogue: **Transform the traditional, descriptive image-text data into a "question and answer" format that is more suitable for detection tasks. For example, disassemble a picture containing multiple objects into multiple command dialogues such as "Is there a cat in the picture?" "Where is the cat?" to force the model to pay attention to and locate each object.
  • Adjust Bbox distribution: Through analysis, it was found that the training data of standard LMM has deviations when dealing with objects of different sizes and numbers. Through a specific sampling strategy, they adjusted the distribution of bounding box (Bounding Box) in the training data to make it closer to the real detection scenario.

Comparison of the distribution of predicted boxes and real boxes

2. inference optimization (inference optimization)

In the inference phase, LMM-Det also introduces an optimization strategy. When faced with a picture containing multiple objects, a standard LMM may describe only part of it due to "lack of concentration". LMM-Det uses a multi-turn query strategy to guide the model to systematically and category-by-category check whether there are objects of a certain category in the image, thereby significantly reducing missed detections.

 

Experimental results and analysis

The effectiveness of LMM-Det has been fully verified on standard datasets such as COCO.

 

The zero-sample detection results on the COCO dataset, LMM-Det far exceed other LMMs without professional detectors Experimental results show that only through the optimization strategy proposed by LMM-Det, the performance of a general-purpose LMM like LLaVA-7B on the zero-sample object detection task far exceeds that of other LMMs that also do not have external detectors, and even approaches some models that require external professional detectors.

 

Further fine-tuning LMM-Det on COCO and comparing it with traditional detection models and multimodal model which relies on external detection experts ablation study also clearly proves the effectiveness of the two core components "data distribution adjustment" and "inference optimization". The combination of the two can bring about the greatest performance improvement.

 

ablation studyThe results prove the effectiveness of the two modules DDA and INO More importantly, while giving LMM powerful detection capabilities, it completely retains its original dialogue and reasoning capabilities, truly realizing one model for multiple uses.

The above figure shows the visualization results of the detection effect of LMM-Det on the COCO verification set, proving that it can achieve object detection without additional professional detectors.

 

Chat examples show that LMM-Det maintains strong conversational capabilities while possessing detection capabilities

Quantitative results for multifunctional LMM-Det

Qualitative (above) and quantitative (above) results together demonstrate that LMM-Det† has multi-task compatibility: it can activate detection capabilities while maintaining the high performance of image description and visual question answering (VQA).

 

Paper contribution and value

The proposal of LMM-Det has brought important inspiration and contribution to the development of multimodalAI:

  1. Paradigm shift: For the first time, it has been systematically proven that LMM does not need to rely on external professional modules, but has powerful native detection capabilities that can be stimulated. This points out a new direction for building a simpler, more versatile, and more integrated AI system.
  2. Profound Insight: Through detailed experimental analysis, it was revealed that low recall rate is the core bottleneck limiting LMM detection capabilities, and targeted and effective solutions were proposed.
  3. Simple and efficient: The optimization strategy of LMM-Det is simple, efficient, and easy to deploy and expand on various existing LMMs.
  4. Comprehensive open source: The research team said that all datasets, models and codes will be open source, which will greatly promote the community's research and exploration of LMM's native capabilities.

In summary, the work of LMM-Det reexamines the potential of LMM. Instead of constantly "patching" and "adding plug-ins" to LMM, it is better to have a deeper understanding and tap into its inherent huge potential that has not yet been fully developed.

Back to Tech Blog