MiniCPM-o.cpp: Bringing Multimodal Models to Edge Devices

Today, with the rapid evolution of large model technology, more and more developers hope to bring multimodal capabilities (text, voice, images, videos) to edge devices. However, reasoning overhead, hardware limitations, and real-time requirements often become the biggest practical obstacles to practical applications. Dr. Leng Dawei from 360 Artificial Intelligence Research Institute believes: **“In the era of large models, inference optimization should be placed in an equally important position as model training.”** With this thought in mind, we officially launch **MiniCPM-o.cpp** - the first open source work of 360 Artificial Intelligence Research Institute in the direction of model inference optimization. ✦ Project address: [GitHub - MiniCPM-o.cpp](https://github.com/360CVGroup/MiniCPM-o.cpp) * * * Why MiniCPM-o.cpp? ------------------- MiniCPM-o.cpp is a lightweight reasoning framework of large multimodal model (LMM), developed based on [ggml](https://github.com/ggml-org/ggml), and completely implemented in pure C/C++. Its goal is straightforward: to allow developers to run large multimodal model (LMM) on edge devices with only 8GB of video memory. Its main features include: * **Lightweight and efficient**: Inference can be completed without relying on heavyweight frameworks. * **Low Hardware Threshold**: 8GB of video memory can run, no expensive GPU required. * **multimodalSupport**: Process audio and video input simultaneously, support streaming inference. * **Device-side optimization**: Special adaptation has been made for NVIDIA Jetson Orin Nano 8GB to meet the requirements of real-time dialogue. * **Easy to integrate**: Provides Python binding, C++ interface, and WebUI demo, covering the complete link from experiment to productization. * * * True end-to-end landing ------- For many developers, the key question is not "can it run", but "can it run in real time". MiniCPM-o.cpp Test results on Jetson Orin Nano Super 8GB show: * In MAXN SUPER mode, the model can achieve real-time video stream inference; * No additional acceleration methods are required, and the first token latency and overall inference speed fully meet the needs of interactive applications. This means that it is not just a research prototype, but a practical tool that has been proven on real hardware. * * * Developer-friendly design -------- Another highlight of MiniCPM-o.cpp is the development experience: * **Simple installation**: Environment configuration can be completed in a few steps; * **Convenient model preparation**: directly use the quantized gguf model to reduce storage and loading overhead; * **Ready to use**: Provides complete Python sample scripts and WebUI demo, developers can quickly experience; * **Flexible integration**: The underlying C++ interface ensures performance and customizability, suitable for embedded and high-performance scenarios. For example, in the Python environment, developers can run through multimodal reasoning with just one command: `python test/test_minicpmo.py --apm-path models/minicpmo-audio-encoder_Q4_K.gguf \` `--vpm-path models/minicpmo-image-encoder_Q4_1.gguf \` `--llm-path models/Model-7.6B-Q4_K_M.gguf \` `--video-path assets/Skiing.mp4` * * * Application scenarios for developers ---------- The capabilities of MiniCPM-o.cpp are particularly suitable for the following types of applications: * **Smart Hardware**: multimodal interaction in smart cameras, robots, and vehicle-mounted systems. * **Edge Computing**: Real-time voice/video understanding in a low-power, low-bandwidth environment. * **Rapid Prototyping**: Use Python binding and WebUI to quickly build experimental environments and application prototypes. * * * Standing on the shoulders of ecology -------- MiniCPM-o.cpp does not start from scratch, but continues the excellent practices of the community: * [llama.cpp](https://github.com/ggml-org/llama.cpp): lightweight LLM inference * [whisper.cpp](https://github.com/ggerganov/whisper.cpp): Device-side speech recognition * [transformers](https://github.com/huggingface/transformers): Mainstream large model training/inference framework Its positioning is to bring multimodal reasoning capabilities to the terminal side based on these tools and fill the application gaps in voice, image, and video interaction. * * * Open source and the future ----- * The code complies with the Apache 2.0 License and can be freely used and re-developed. * Models must comply with official licenses to ensure compliant application. MiniCPM-o.cpp optimizes a multimodal model with 7.6B parameters to only require 8GB of video memory to run on the device side, and achieves real-time inference on actual hardware. This is our preliminary exploration on how to move large models from "cloud laboratory" to "device-side application scenarios". In the future, as the demand for end-side applications grows, we look forward to working with developers to make inference optimization the core force for the implementation of large models. * * *