This article shares the paper IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities, a new paradigm for freezing language model training LMM. 360 proposes IAA, which enables the language model to losslessly acquire multimodal understanding and grounding capabilities!

• Paper link: https://www.arxiv.org/abs/2408.12902

• Model and code: https://github.com/360CVGroup/Inner-Adaptor-Architecture

Summary

The currently widely popular multimodallarge language model (LLM) (LMM: Large Multimodal Model) based on the LLaVA structure performs well on visual tasks. However, because the embedded language model needs to be fine-tuned during training, its ability in natural language tasks (NLP) is often significantly weakened. Specifically, the model's overall understanding of text will be weakened, and the information given when answering questions is not detailed enough.

In order to prevent this performance degradation, an effective way is to not change the original parameters of the language model when extending the model's multimodal capabilities. However, previous attempts like Flanmingo and BLIP2 have not achieved the desired results.

This article is inspired by the ControlNet idea in the field of image generation. While keeping the language model unchanged, it innovatively adds an internal adapter structure (IAA).

This strategy not only maintains the model's original capabilities in natural language processing (NLP), but also significantly improves the model's general multimodal processing capabilities and visual grounding performance.

Unlike previous methods that relied on frozen language models with large amounts of image–text alignment data, the IAA architecture can still demonstrate excellent performance while using smaller-scale datasets.

In addition, in the face of diverse task requirements, multiple IAA can be easily deployed on a single GPU at the same time, thereby achieving efficient optimization in video memory utilization.

motivation

The mainstream large multimodal model (LMM) uses the structure of vision-encoder + projector + LLM represented by LLaVA. The advantage of this structure is that pre-trained language model embedding can be used to greatly reduce the cost and difficulty of training. Currently, major domestic large multimodal model (LMM) such as Qwen-VL, InternVL, MiniCPM, DeepSeek-VL, etc. all use this architecture.

However, this architecture also has a problem that is very difficult to solve in practice and has been deliberately avoided by everyone. That is, the embedded language model will inevitably have "catastrophic forgetting" problems during the training process of multimodal model, thereby reducing its performance on text tasks.

As a verification, this paper uses the 1.2M open source dataset of LLaVA-1.5 and conducts experiments on its model architecture. In addition to common VQA and instruction fine-tuning data, this dataset also contains a small amount of plain text data.

As shown in Figure 1, whether it is the LLaVA-1.5 architecture built on the two language models Qwen2 or Llama3, before and after training, the scores of the language model in plain text benchmarks such as MMLU and C-Eval have dropped significantly.

Figure 1: Results before and after training the LLaVA-1.5 architecture on the text-only evaluation set MMLU and C-eval based on Qwen2 and Llama3 language models

The fundamental reason why "catastrophic forgetting" of the embedded language model occurs is that in order to emphasize the ability of multimodal model in visual tasks, the embedded language model must participate in the training process.

For multimodalLMM, in order to achieve excellent multimodal understanding ability, a large amount of image-text data is needed for training. The greater the amount of training, the more serious the problem of catastrophic forgetting. Although models like LLaVA try to incorporate some plain text data during their training process, they are still not able to completely avoid this problem.

A straightforward way to prevent large language model (LLM) performance degradation is to freeze the parameters of the language model during LMM training. However, the current model adopting this strategy does not show satisfactory performance in the benchmark test of multimodal capability.

In order to solve these challenges, this paper proposes a training paradigm containing an internal adapter structure (IAA), which aims to significantly improve the model's multimodal performance without affecting the capabilities of the original language model.

method

Figure 2: Overview of IAA, containing two workflows: multimodal interaction and plain text dialogue

In view of the previous discussion, maintaining the original capabilities of LLM becomes crucial. The Inner-Adaptor Architecture (IAA) proposed in this article allows LMM to run two workflows at the same time after deployment: one is multimodal interaction, and the other is plain text dialogue.

multimodalThe interactive workflow includes the following three parts:

1. Image encoder and MLP layer: Use an image encoder (such as CLIP or SigLIP) to extract high-quality image features, and achieve visual and language alignment through the MLP layer.

2. large language model (LLM): Remain frozen during training to retain original language processing capabilities.

3. Internal adapter structure: including insertion layer, embedding layer and language model header specially designed for multimodal input.

The plain text conversation workflow only contains the original large language model (LLM) and no other specific modules are added.

Figure 3: Structural exploration of IAA

In the design of the internal adapter structure (IAA), referring to Figure 3(a), this article is inspired by the ControlNet architecture and performs additional insertion operations of self-attention and other layers, which can be expressed succinctly as follows:

Among them, ϕfl and ϕil represent the frozen language model layer (LM) and the insertion layer respectively. Xin represents multimodal input, Xout represents multimodal output, and G represents a zero-initialized gating layer. The insertion layer includes self-attention layer, layer normalization, feed-forward network, etc., which is consistent with the parameter scale of the transformer layer in large language model (LLM).

For example, if the target is the 22nd language model layer, the initial parameters of the corresponding insertion layer are derived from the 22nd language model layer. Despite this, the design based on ControlNet did not achieve very good performance.

Referring to Figure 3(b), this paper further improves the structure based on ControlNet, eliminating feature propagation between insertion layers, and using the output of the language model layer as the input of the insertion layer.

Each frozen LM layer will accommodate multimodal data through an insertion layer and a gating layer, while the insertion layer is no longer directly affected by subsequent layers. Compared with the design in Figure 3(a), the improved structure shows significant improvements in experimental results.In addition, this article believes that the gating layer may not reach the optimal state through the single-round data training strategy commonly used in LMM. Therefore, this paper proposes a more streamlined solution, as shown in Figure 3(c). The operations of a specific layer in the model can be expressed in the following way:

Similar to scenario (a), if the insertion layer is placed after the 22nd frozen LM layer, the parameters of the 22nd frozen LM layer are used for initialization. In the trained model, the number of inserted layers can be specified arbitrarily.

In addition, in order to conduct better multimodal training, this article also introduces new embedding layers and language model headers, which are also initialized from the corresponding structures of the original language model.

In all multimodal training stages, the original language model remains fixed and only the newly added structures are trained. Finally, this article selected the structure shown in Figure 3(c) and named it the internal adapter structure (IAA).

Experimental results verify the effectiveness of this strategy.

experiment

Training strategy

In the process of training multimodallarge language model (LLM) (LMM), the core task of the pre-training stage is to let the model learn how to correspond visual information to text descriptions.

This stage, often called the image-text alignment stage, builds the bridge between vision encoder and large language model (LLM) (LLM). In our architectural design, the image encoder and LLM are kept constant throughout the training process. This is done to preserve their basic knowledge in visual and language understanding.

The MLP projection layer and IAA internal adapter structure need to be trained to improve the multimodal processing capabilities of the model.

In the experiment, we found that for the IAA internal adapter structure, if a too high learning rate is used, the training loss will be unstable or even overflow.

To solve this problem, we design a two-stage pre-training strategy.

In the first stage, our model structure consists of three parts: image encoder, MLP projection layer and large language model (LLM). At this stage, the parameters of the image encoder and large language model (LLM) are fixed, and we use a high learning rate of 0.001 to focus on training high-quality projection layers.

In the second stage of pre-training, we extended the model and added IAA specifically designed to handle the multimodal task. At this stage, the trainable parameters include the projection layer and the new IAA, and we use a lower 2e-5 learning rate to fine-tune the model.

After completing pre-training, the model will receive further instruction fine-tuning and visual grounding fine-tuning to optimize its performance.

General multimodalbenchmark results

Table 1: Results of multimodal evaluation set

This article divides the comparison methods into two major categories: one is trained using frozen language models, and the other is trained using unfrozen (i.e., fine-tuned) language models.

For in-depth analysis, we show in detail the dataset size used by each method, as well as the image encoder, large language model (LLM) and other key components they rely on. In order to ensure the fairness of the comparison, we selected LMMs with similar parameter sizes, and the performance indicators of these methods are based on officially published data.

By comparison, we can clearly see that the method proposed in this paper shows significant advantages in categories trained using frozen language models.

This result shows that our method can effectively improve its performance on the multimodal task while maintaining the original capabilities of the model.

In addition, as a currently popular practice, LMM without freezing the language model can usually achieve better performance in multimodal performance.

However, this improvement often comes at the expense of some NLP capabilities. When we compared the LLaVA-Llama3 method, we found that IAA achieved better results in multimodal performance indicators using the same vision encoder, language model and training data.

As shown in Table 2, while IAA improves the capabilities of multimodal, its NLP capabilities are not affected. On the contrary, the performance of LLaVA, a method of fine-tuning language models, on natural language processing evaluation sets such as MMLU and C-Eval has been significantly affected, showing a decline in NLP capabilities.

This further proves that IAA effectively enhances the model's multimodal processing capabilities while maintaining the original advantages of the language model.

Table 2: Comparison on plain text evaluation set

Figure 4 shows the performance of different models on the plain text dialogue task.

It can be clearly seen from this that IAA successfully retains the original conversational capabilities of the language model when processing plain text conversations.

In contrast, open source LMMs, such as LLaVA-Llama3 and LLaVA-v1.5, suffer from performance degradation on plain text dialogue tasks due to the influence of multimodal data.

When faced with the same questions, LLaVA-Llama3 and LLaVA-v1.5 tend to give shorter answers. This phenomenon is closely related to the fact that text lengths are generally shorter in the large amounts of multimodal data they are exposed to during training.

Fine-tuning large language model (LLM), especially when processing multimodal data, may impair the model's ability to deeply understand the content and generate detailed responses. Therefore, the design of IAA shows its unique advantages in maintaining the quality of model dialogue.

Figure 4: Structural exploration of IAA

visual groundingbenchmark results

Table 3: Results of visual grounding evaluation set

The results in Table 3 show that IAA is able to achieve competitive visual grounding performance on a limited dataset.

In order to ensure fair comparability, the table excludes models trained using very large datasets, such as CogVLM-grounding, which uses 1.5 billion image-text pairs and 4 billion grounding data, as well as those structures that utilize the pre-trained object detection model, such as LLaVA-Grounding and Groma.

Deployment efficiency

Currently, high-performance multimodal model usually requires unfreezing large language model (LLM) for training. CogVLM highlights the difficulty of developing a model that is good at both comprehension and visual grounding tasks.

To solve this problem, it adopts a dual-model strategy, with one model training the general multimodal ability and the other model training the visual grounding ability. In this case, deploying a language model, a general multimodal model and a visual grounding model simultaneously using FP16 on the GPU requires approximately 50GB of video memory.

The method proposed in this article cleverly combines the general multimodal capabilities and visual grounding capabilities through the internal adapter structure, while protecting the NLP capabilities of the original large language model (LLM).

For example, by using an 8-layer internal adapter configuration, the model can significantly reduce the memory usage. A model deploying these three capabilities at the same time only requires about 30GB of video memory.

ablation study

This article conducts a series of ablation study on the IAA structure, training strategy, number of insertion layers, etc. The specific results can be seen in the table below.

Table 4: Ablation learning for structure exploration of IAA

Table 5: Comparison of different training strategies

Table 6: Ablation of IAA number of inserted layers

Conclusion

This article proposes an internal adapter structure (IAA) designed to improve large language model (LLM) (LLM) capabilities in general multimodal understanding and visual grounding.

Through a series of fine architecture exploration experiments, this paper confirms that LMM using the frozen language model strategy can surpass those fine-tuned similar models in performance.

When deploying applications, IAA simultaneously supports the three workflows of multimodal, target positioning and plain text. This not only maintains the strength of the language model in natural language processing (NLP), but also enhances its flexibility in processing complex tasks. This feature of IAA provides broad possibilities for the model to be expanded to more multimodal in the future.