360 AI Research at AAAI: Perspectives on Multimodal Understanding and Generation

In the wave of technological changes in artificial intelligence, 360 Group fully embraces AI technology from business to research and development. 360 Artificial Intelligence Research Institute leverages its own technological accumulation and intellectual advantages to undertake the strategic research and development tasks of understanding and generating models. Recently, the institute's work on multimodal understanding direction IAA and multimodal generation direction BDM were accepted by the top conference in the AI ​​field AAAI. Let us take a look at these two works of the institute and the in-depth thinking behind them.

§IAA: Innovative architecture that empowers language model multimodal

·Arxiv: https://www.arxiv.org/abs/2408.12902

·Github: https://github.com/360CVGroup/Inner-Adaptor-Architecture

1. Key issues to be solved

1. Problem with catastrophic forgetting embedded language model in large multimodal model (LMM)

Currently, in training, in order to improve multimodal capabilities, it is usually necessary to turn on the embedded language model parameters and participate in learning. Although this can improve the model's indicators on the multimodal task, it also produces "side effects": the original text understanding ability of the language model is catastrophic forgetting due to participation in multimodal training. The mainstream bridge structure represented by the LLaVA series multimodal model, after being trained with multimodal data, the ability of the embedded language model on text tasks will significantly decline.

2. Ecological issues of the multimodal plug-in of the language model

The second key issue that IAA is trying to solve is to add the multimodal understanding ability as a plug-in to the language model of the base like text-to-image (T2I) in the field of text-to-image (T2I). Different from the current way of requiring one model to complete all the work, a new language model ecosystem will be formed through a foundation model+ plug-in model for different tasks.

2. Innovative network structure

1) Keep the base language model parameters unchanged to avoid catastrophic forgetting

IAA keeps the base language model parameters unchanged in the network design. For the learning of multimodal knowledge, it is specially processed through multiple new insertion layers. The new insertion layer interacts with the original text-oriented transformer layer at different depths of the language model, enabling the frozen language model to acquire multimodal capabilities.

2. A set of model weights can handle different tasks and reduce deployment costs.

On the general multimodalbenchmark, comparison of the indicators of IAA with other related work has verified that IAA can effectively improve its performance on the multimodal task while maintaining the original capabilities of the model.

IAA The network only needs to deploy a set of model weights to cope with different tasks. The text-only task and the multimodal task follow different workflows respectively, avoiding doubling the deployment cost. In addition, the IAA plug-in structure is not only applicable to multimodal tasks. Tasks that require special enhancement in foundation model capabilities, such as code, math and other tasks, can also be specially enhanced through plug-ins.

§BDM: Chinese native AI painting compatible with SD ecology

·Arxiv: https://arxiv.org/abs/2309.00952

·Github: https://github.com/360CVGroup/Bridge_Diffusion_Model

1. Key issues to be solved

1) Worldview bias of native Chinese and generative models

English AI image-generation models represented by SD, including but not limited to SD1.4/1.5/2.1/3.5 as well as DALLE, Midjourney, Flux, etc., generally have obvious biases in the English world. The generated characters are more biased towards white people and males, and Chinese concepts are difficult to accurately generate.

There are many ways to develop Chinese AI image-generation models, ranging from easy to difficult, including English model + translation, English model + implicit translation, English model + implicit translation + fine-tuning, and the structure of the English model is trained from scratch using Chinese data, etc. Except for the last method, none of the other methods can completely solve the problem of model bias. This is also the reason behind phenomena such as "braised lion head" and "busy traffic" that have previously caused heated discussions across the Internet.

2. Compatibility with SD ecosystem

In the above solution, completely training from scratch can perfectly solve the worldview bias problem of the AI image-generation model, but it also needs to pay a corresponding price: in addition to the training cost, it also needs to face the problem of being unable to utilize the huge community formed around SD and other open-source model and the large number of excellent plug-ins and other group intellectual assets accumulated in the community. On the basis of overcoming the worldview bias of the AI ​​painting model, further achieving compatibility with the open source community is the second key issue to be solved in the work of BDM.

2. Unique model structure

1) x-language branch network, learning data in different languages

BDM is a branch network idea similar to ControlNet in terms of model structure. It uses different network branches to learn data in different languages. Therefore, in principle, BDM can not only generate native Chinese images, but also generate images in any X language, and ensure that the generated images conform to the cognition of the corresponding language and culture. The English part can directly reuse the existing open-source model, thereby achieving seamless compatibility with the open source community.

2. The generated images are in line with Chinese cultural cognition and compatible with the open source community ecology.

BDM uses 1 billion Chinese image–text data for training and is compatible with the SD1.5 community ecology. The figure below shows that BDM has good compatibility with different SD1.5 style fine-tuning models, especially BDM while maintaining the output capability of Chinese images. For more cases, please see the AAAI paper.

Conclusionmultimodal Understanding and generation is an important direction for the future development of large model technology. While supporting the group's AI strategy, 360 Artificial Intelligence Research Institute also actively participates in promoting industry-university-research exchanges and common progress in the Chinese large model research community. All relevant work shared in this article has been fully open sourced. The institute's Github homepage: https://github.com/360CVGroup. Cooperation and exchanges are welcome.