SEEChat: An Open-Source Chinese Multimodal Dialogue Model
2022 was hailed by the media as "the first year of AIGC". During this year, AI painting and chatGPT successively detonated the global technology community and became two major milestones in the field of artificial intelligence. In particular, the launch of chatGPT has reignited people's new round of expectations for general artificial intelligence (AGI). The unprecedented logic and reasoning capabilities demonstrated by chatGPT have made many experts and researchers in the AI field marvel. At the same time, more companies and institutions are beginning to try to apply chatGPT to their own businesses, hoping to use the power of artificial intelligence to improve work efficiency and solve problems.
ChatGPT is a pure text single-modal language model developed based on GPT3.5. For its next-generation update, we previously speculated that in addition to the continued improvement of text capabilities, the transition from single-modality to multimodal will be a more critical point. The launch of GPT4 on March 15 this year confirmed our speculation: GPT4 will serve as a new generation of GPT. model, adds support for visual modality input, which means it is able to understand images and perform corresponding natural language generation.
GPT4, which adds the multimodal capability, also brings more possibilities at the application level. For example, in the field of e-commerce, merchants can input product images into GPT4 to generate descriptions, thereby providing consumers with more natural product introductions; in the field of entertainment, GPT4 can be used in game design and virtual character creation, bringing players a more personalized gaming experience and interactive happiness. Visual ability has always been regarded as an essential basic ability required by a general artificial intelligence AGI agent, and GPT4 vividly demonstrates the prototype of AGI that integrates visual abilities.
Visual ability integration solutions and comparison of advantages and disadvantages
In fact, GPT4 is not the first work to integrate visual and text modalities. Researchers in the fields of CV, NLP, and robotics have long been exploring various methods to integrate information from multiple different models. VQA, Visual Captioning, Visual Grounding, etc. are already professional research fields subdivided under multimodal.
Specific to the MLLM (Multimodal Large Language Model) that integrates visual capabilities into the language model LLM, the relevant research routes are mainly divided into two: one is the native multimodal route. The model design is specifically adapted to the multimodal data from the beginning. Representative works include MSRA's KOSMOS-1 [1] and Google Robotics' PALM-E [2], both in March this year The other is the single-modal expert model stitching route, which links the pre-trained visual expert model and the pre-trained language model through the bridging layer. Representative works include Deepmind's Flamingo [3], Saleforce's BLIP-2 [4], and recent work such as LLAVA [5] and miniGPT4 [6].
Figure 1 MLLM representative work timeline
For the native multimodal route represented by KOSMOS-1 and PALM-E, the main body of the model structure is a Transformer stack. The figure below shows the model structure and training scheme of KOSMOS-1. Except for the image encoder part, which uses the pre-trained CLIP ViT-L/14, the main MLLM part of the model is a 24-layer Transformer stack, which is trained from scratch using native multimodal data.
multimodal data consists of three parts: a) plain text, mainly Pile and Common Crawl; b) image-text pair data, mainly LAION-2B, LAION-400M, COYO-700M, and Conceptual Captions; c) Interleaved Image-Text Data, including 71M pages of image–text web page data. By concatenating image embedding with text embedding in the following format:
Figure 2 KOSMOS-1 model structure and training plan
Compared with KOSMOS-1, PALM-E has basically the same model structure and modeling method of multimodal data. Figure 3 below shows the model structure and training scheme of PALM-E, where < emb > is the state estimation vector of the robot. The biggest difference compared with KOSMOS-1 is that PALM-E uses the weights of the single-modal language model PALM to initialize the model.
Figure 3 PALM-E model structure and training plan
The advantage of the native multimodal route is that the model structure is natively adapted to multimodal data, and the effect is excellent when the amount of data is sufficient. Experience in related fields also shows that this method has a higher performance limit, but the disadvantages are also obvious. It cannot fully reuse existing results in each single-modal field, and training requires very large computing resources and data resources.
Compared with the original multimodal route, the single-modal expert model stitching route represented by Flamingo, BLIP-2, and LLAVA/miniGPT4 has been designed from the beginning to reuse existing results in various single-modal fields as much as possible, especially the pre-training model of LLM, which has developed rapidly recently. Flamingo is a work published by Deepmind in November 2022. Based on freezing the vision encoder and LM, it achieves alignment and joint learning of visual information and text information by inserting multiple cross-attention layers in the LM.
Figure 4 Flamingo model structure and training plan
Compared with KOSMOS-1, Flamingo's stitching scheme makes full use of existing achievements in the CV and NLP fields. Neither the vision encoder nor the LM require training. It only needs to train the cross-attention layer (GATED XATTN-DENSE in Figure 4) that serves as a bridge between different modal information. Therefore, it has obvious advantages at least in terms of model training costs.The BLIP-2 work published by salesforce in January 23 and the subsequent derived work such as LLAVA and miniGPT4 further simplified this idea to only a single bridge layer between the vision encoder and LM. Figure 5 below is the model structure of BLIP-2, in which the visual side and text side use the pre-trained CLIP ViT-G/14 model and FLAN-T5 respectively. For the model, only the Q-Former that plays a bridging role in the middle participates in the training, and the cost and data volume required for training are further reduced. The training data volume of BLIP-2 is only 129M, and the 16-card A100 training takes 9 days.
Later LLAVA work simplified this idea to the extreme, stitching together CLIP ViT-L/14 and Vicuna language models through only one projection layer. The training data only used 595K image–text pairs and 158K instruction fine-tuning data. miniGPT4 is based on reusing BLIP-2's vision encoder + Q-Former, and stitches the Vicuna language model through a project layer. The training data only uses 5M image–text pair data + 3.5K instruction fine-tuning data.
Figure 5 BLIP-2 model structure
Compared with the native multimodal route, the most obvious advantage of the single-modal expert model stitching route is that it can fully reuse existing results in various single-modal fields, with low cost and quick results. Some researchers speculate that GPT4 may also be based on the visual understanding capabilities achieved by the stitching route. However, the shortcomings of the suture-based route are also obvious, especially for simple shallow fusion solutions such as BLIP-2, LLAVA, and miniGPT4. The MLLM model finally trained can usually only perform single or multiple rounds of dialogue, and does not have the in-context learning capabilities demonstrated by deep fusion solutions such as Flamingo and native multimodal schemes such as KOSMOS-1 and PALM-E.
SEEChat multimodalDialogue Model
SEEChat (https://github.com/360CVGroup/SEEChat) focuses on integrating visual capabilities with an existing large language model to create a multimodal language model. On the implementation route of the multimodal capability, we chose the single-modal Experts Efficient integration route (Single-modal Experts Efficient integration) that can fully reuse existing results in different fields. This is also the multimodal The project's naming source.
The model structure of SEEChat v1.0 is shown in Figure 6 below. Through the projection layer bridging layer, the vision encoder: CLIP-ViT-L/14 and the open source Chinese LM: chatGLM6B are stitched together.
Figure 6 SEEChat Model Structure
The training of SEEChat v1.0 is divided into two stages: the first stage is image–text alignment training, using our previously open source high-quality Chinese image–text dataset Zero [7], with a total of 23 million samples for training; the second stage is human-machine alignment training, using miniGPT4+LLAVA open source instruction fine-tuning data. After English-Chinese translation, the instructions trained in the first stage are fine-tuned.
Figures 7~9 below are simple demonstrations of SEEChat v1.0’s capabilities in picture-text dialogue, code generation and target classification. It can be seen that on the one hand, SEEChat inherits the conversational capabilities of the chatGLM language model (and of course inherits its shortcomings), and on the other hand, it shows impressive image–text alignment and visual understanding capabilities.
Picture 7 SEEChat v1.0 Picture and text dialogue display
Figure 8 SEEChat v1.0 Picture and text dialogue display
Figure 9 SEEChat v1.0 picture and text dialogue display
SEEChat is not the first open source Chinese multimodal dialogue model. In May of the same period, X-LLM [8] from the Institute of Automation of the Chinese Academy of Sciences and VisualGLM [9] from the Tsinghua KEG Group were open sourced one after another. In contrast, SEEChat v1.0 is roughly the same as the previous two works in terms of route selection and model structure, and only differs in model details, training data and methods. We believe that for current shallow fusion solutions, a very critical point is the quality rather than the quantity of data used for training. We compared SEEChat v1.0 with multiple related works on the image captioning task:
1000 Chinese Internet images are randomly selected from the Chinese Zero dataset, and the training set data has been excluded
Use ChineseCLIP [10] to calculate the image-text correlation score (in order to avoid the bias caused by overlapping training data, we did not use the R2D2 Chinese cross-modal model trained by ourselves, but selected the cross-modal model trained by a third party to evaluate the image-text correlation score)
The picture above shows the winning situation of image-text correlation scores among seven disclosure methods (we treat the Internet text native to the data as one of the methods)
It can be seen that the winning rate of SEEChat v1.0 trained using the high-quality image–text dataset Zero even exceeds that of native text by a large proportion.
Figure 10 Comparison of different models on the Image Captioning task
Jobs of the future
The SEEChat project consists of two versions: the internal closed-source version is trained using the company's internal data and focuses on business productivity; the external open-source version is trained using public data and focuses on capability demonstration and community impact. At present, the internal version of SEEChat v1.0 has been implemented in the group's internal business and will be released to the public under the brand of "360 Intelligent Brain - CV large multimodal model (LMM)" on May 31.As mentioned before, the focus of the SEEChat project is to integrate visual capabilities with the existing LLM model to create a multimodal language model that focuses on visual capabilities. MLLM, v1.0 has verified the basic image–text alignment and visual understanding capabilities. Next, we will gradually provide MLLM with With the addition of object detection capabilities, cross-modal capabilities, and the object detection and recognition capabilities of the open vocabulary, the modal fusion solution will also transition from shallow fusion to deep fusion, so stay tuned.
About the author
Leng Dawei: Head of the visual engine department of 360 Artificial Intelligence Research Institute. He currently leads the institute’s visual team to conduct cutting-edge exploration and industrial implementation work in large multimodal model (LMM), AIGC, cross-modal graphic learning, open-world object detection, open vocabulary video analysis, AIoT and other directions.
References
[1] Huang, Shaohan, et al. "Language is not all you need: Aligning perception with language models." arXiv preprint arXiv:2302.14045 (2023).
[2] Driess, Danny, et al. "Palm-e: An embodied multimodal language model." arXiv preprint arXiv:2303.03378 (2023).
[3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in Neural Information Processing Systems 35 (2022): 23716-23736.
[4] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." arXiv preprint arXiv:2301.12597 (2023).
[5] Liu, Haotian, et al. "Visual instruction tuning." arXiv preprint arXiv:2304.08485 (2023).
[6] Zhu, Deyao, et al. "Minigpt-4: Enhancing vision-language understanding with advanced large language models." arXiv preprint arXiv:2304.10592 (2023).
[7] Zero, https://zero.so.com/
[8] Chen, Feilong, et al. "X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages." arXiv preprint arXiv:2305.04160 (2023).
[9] VisualGLM, https://github.com/THUDM/VisualGLM-6B
[10] ChineseCLIP, https://github.com/OFA-Sys/Chinese-CLIP