Preventing Language-Capability Loss in Multimodal Models with IAA
During AI training, have you ever encountered multimodal model ‘forgetting’ your original language ability? The model's answers are not detailed, and the text comprehension is reduced, which is a headache. The 360 team proposed a new IAA structure, which skillfully handles the multimodal task through an internal adapter, significantly improving visual task performance and retaining the original NLP capabilities without modifying the language model parameters.
Introduction: multimodal model Development’s overlooked “forgetting” problem
In the current large multimodal model (LMM) development, although the model represented by the LLaVA architecture performs well in visual tasks, there is a problem that is widely ignored: the embedded language model is prone to "catastrophic forgetting" during the training process. This kind of forgetting will significantly weaken the model's performance on text tasks, resulting in less detailed answers and reduced understanding. In order to verify this problem, the research team found in the LLaVA-1.5 experiment that whether it was based on the Qwen2 or Llama3 language model architecture, the scores in the plain text benchmark test before and after training dropped significantly.
IAA: multimodal framework for forgetting without language ability
The root cause of "catastrophic forgetting" is that in order to improve multimodal model's visual task capabilities, the embedded language model must be adjusted during training, thereby affecting its original language processing capabilities. Although some models try to integrate plain text data to alleviate this problem, the results are not ideal. The 360 team proposed IAA (internal adapter structure) for this purpose. By freezing the language model parameters, it ensures that its original capabilities are not damaged, and at the same time significantly improves the model's multimodal processing performance.
The IAA structure allows the LMM to run two independent workflows simultaneously: one for multimodal interactions and another for plain text conversations. This architectural design not only retains the NLP capabilities of the model, but also effectively improves the performance of multimodal understanding and visual grounding. multimodalThe interactive workflow includes the following three parts:
Image encoder and MLP layer: Use an image encoder (such as CLIP or SigLIP) to extract high-quality image features, and achieve visual and language alignment through the MLP layer.
large language model (LLM): Remain frozen during training to retain original language processing capabilities.
Internal adapter structure: including insertion layer, embedding layer and language model header specially designed for multimodal input.
The plain text conversation workflow only contains the original large language model (LLM) and no other specific modules are added.
In order to conduct better multimodal training, this article also introduces new embedding layers and language model headers, which are also initialized from the corresponding structures of the original language model. In all multimodal training stages, the original language model remains fixed and only the newly added structures are trained. Finally, this article selected the structure shown in Figure 3(c) and named it the internal adapter structure (IAA). Experimental results verify the effectiveness of this strategy.
Training strategy: two-stage pre-training to ensure stability and accuracy
In order to optimize the performance of IAA, the 360 team designed a two-stage pre-training strategy. The first stage focuses on training the image encoder and MLP projection layer to ensure the quality of the projection layer with a high learning rate. In the second stage, the IAA structure is added, the learning rate is reduced, and the model is finely adjusted. After this process, IAA not only performed well in the multimodal task, but also successfully avoided the "catastrophic forgetting" problem.
General multimodal Experimental Evaluation
This article divides the comparison methods into two major categories: one is trained using frozen language models, and the other is trained using unfrozen (i.e., fine-tuned) language models. For in-depth analysis, the 360 team showed in detail the dataset size used by each method, as well as the image encoder, large language model (LLM) and other key components they rely on. Through comparison, we can clearly see that the method proposed by the 360 team shows significant advantages in categories trained using frozen language models. This result shows that the 360 team's method can effectively improve its performance on the multimodal task while maintaining the original capabilities of the model.
Plain text evaluation experiment evaluation
The figure below shows the performance of different models on the plain text dialogue task. It can be clearly seen from this that IAA successfully retains the original conversational capabilities of the language model when processing plain text conversations. In contrast, open source LMMs, such as LLaVA-Llama3 and LLaVA-v1.5, suffer from performance degradation on plain text dialogue tasks due to the influence of multimodal data. When faced with the same questions, LLaVA-Llama3 and LLaVA-v1.5 tend to give shorter answers. This phenomenon is closely related to the fact that the text length is generally shorter in the large amounts of multimodal data they are exposed to during training. Fine-tuning large language model (LLM), especially when processing multimodal data, may impair the model's ability to deeply understand the content and generate detailed responses. Therefore, the design of IAA shows its unique advantages in maintaining the quality of model dialogue.
Sample display
The IAA structure provides an effective solution to the "catastrophic forgetting" problem in multimodal model training. By freezing the language model and introducing the adapter structure, IAA significantly enhances multimodal understanding and visual grounding capabilities while retaining the original natural language processing capabilities of the language model. This innovation not only brings performance improvements to the current multimodal model, but also provides a new direction for future expansion of multimodal model.