AAAI | IAA Adds Multimodal Capabilities Without Catastrophic Forgetting
large multimodal model (LMM) What should I do if the embedded language model always appears catastrophic forgetting?
It can be solved with ControlNet like text-to-image (T2I).
This is the core idea of IAA proposed by 360 Artificial Intelligence Research Institute. The research results have been accepted by AAAI and are open source.
The idea behind IAA's work is to add the multimodal understanding ability as a plug-in to the language model of the base, just like ControlNet in the text-to-image (T2I) field, so as to realize the plug-in of multimodal capabilities without affecting the original capabilities of the language model at all, and further form a new language model ecosystem.
Research a new plug-in control mechanism for language models
The current large multimodal model (LMM) (LMM: Large Multimodal Model) mainstream adopts a bridge structure represented by the LLaVA series: vision encoder and LLM are connected through the modal bridge projector to achieve multimodal understanding ability.
The advantages of the bridge structure are simple structure and low training cost (hundreds of thousands of fine-tuned data can achieve basic image understanding capabilities). Currently, mainstream LMM models including QwenVL, DeepSeekVL, internVL, and the institute's self-developed 360VL (https://github.com/360CVGroup/360VL) all use this structure.
However, the bridge structure has always had a shortcoming that is difficult to overcome: The improvement of modelmultimodal capabilities will inevitably bring about the decline of the original text capabilities.
The underlying reason behind this is that in order to improve the performance of LMM on multimodal understanding tasks as much as possible, the LLM language model parameters embedded in mainstream models must be turned on during multimodal training to participate in learning. Although it is easier to improve the indicators on the multimodal task, the original text understanding ability of the language model will have catastrophic forgetting problems due to participation in multimodal training.
This is also the reason why the current mainstream multimodal model exists independently of the language model and is distinguished by -VL.
The above figure clearly compares the performance decline of the embedded language model in text tasks (MMLU and C-eval) due to the occurrence of catastrophic forgetting before and after multimodal training.
In addition, from a practical point of view, the current multimodal model needs to be deployed independently of the language model, which means that the deployment cost will be doubled during application. From an economic point of view, new technology breakthroughs are also urgently needed.
"The inspiration for the IAA work came from the research on the generation direction of multimodal that we are also responsible for." Dr. Leng Dawei said.
"The text-to-image (T2I) field has a prosperous ecology that is completely different from the language model field. In the text-to-image (T2I) field, everyone revolves around one foundation model and completes different tasks by connecting different plug-ins, instead of requiring one model to complete all tasks like the language model field. The IAA work borrows this idea."
In the study of IAA, the author found that simply copying the ControlNet structure in the text-to-image (T2I) field to the language model did not perform well. The reason behind this is that the current mainstream language model is the Transformer stacked structure, which is very different from the UNet structure in the text-to-image (T2I) model. Therefore, a new plug-in control mechanism needs to be studied for the language model.
After experimentally comparing a series of different structures, the v1 version of the IAA plug-in network structure was finally formed as follows:
Compared with the mainstream LLaVA structure, IAA keeps the base language model parameters unchanged in network design, thus avoiding the occurrence of catastrophic forgetting problems; for multimodal knowledge learning, it is specially processed through multiple new insertion layers.
During inference, the IAA network only needs to deploy a set of model weights. The text-only task follows the text-only workflow, while the multimodal task follows the multimodal workflow, thus avoiding the cost problem of deploying both a language model and an additional set of multimodal model.
In addition, the IAA plug-in structure is not only suitable for multimodal tasks, but also tasks that require special enhancement in foundation model capabilities, such as code, math and other tasks, can also be specially enhanced through plug-ins, realizing a new usage and ecology of "foundation model+ professional plug-ins" for handling professional tasks.
The performance of IAA and other related work was compared on the general multimodalbenchmark, and it was verified that IAA can effectively improve its performance on the multimodal task while maintaining the original capabilities of the model.
About 360 Artificial Intelligence Research Institute
All in 360 Group In the context of AI, 360 Artificial Intelligence Research Institute has given full play to its intellectual advantages and undertaken the strategic research and development tasks of understanding and generating large models (commonly known as pictures and text and text-to-image (T2I)). It has continued to exert efforts in both directions and has successively developed large multimodal models, Bridge Diffusion Model (BDM), the layout-controllable HiCo model, and Qihoo-T2X based on a new Diffusion Transformer architecture.
Recently, the institute's work on multimodal understanding direction IAA and multimodal generation direction BDM were respectively accepted by the top conference in the field of AI AAAI. The R&D leader of these two works is Dr. Leng Dawei.
It is reported that this AAAI 2025 conference received nearly 13,000 submissions and accepted 3,032 jobs, with an acceptance rate of only 23.4%.
Arxiv: https://www.arxiv.org/abs/2408.12902 Github: https://github.com/360CVGroup/Inner-Adaptor-Architecture