Introduction

After Llama 3 was released on April 19, the multimodal team at 360 AI Research began evaluating its potential for large multimodal models. The team subsequently released 360VL, the next generation of SEEChat, in 8B and 70B variants based on Llama 3. The project presents the 70B variant as the first open-source large multimodal model built on Llama 3 70B. In addition to the Llama 3 language backbone, 360VL uses a globally aware multi-branch projector to strengthen image understanding. The project reports that 360VL outperforms models including LLaVA 1.6, MM1, and Yi-VL on multiple mainstream benchmarks.

GitHub Address: https://github.com/360CVGroup/360VL

Huggingface address: https://huggingface.co/qihoo360/360VL-70B

Model Architecture

360VL, as a visual multimodallarge language model (LLM), can support multi-level visual tasks. In terms of model structure, 360VL follows the design of vision encoder-bridging layer-large language model (LLM), and fully understands visual tokens through the language model. In addition to using the more powerful foundation model, the researchers focused on improvements to the bridging layer.

In LMM, the bridging layer plays a core role in connecting the previous and the next. Through the alignment training stage, the bridging layer is responsible for transforming the features of vision encoder so that large language model (LLM) can understand the deep meaning of the image. At present, most multimodallarge language model (LLM) mainly use simple linear layers or resampler structures as bridging layers, but they have certain limitations in processing high-resolution images. Limited by the image resolution that the current general vision encoder can handle, sliding window segmentation is usually required when facing high-resolution images. However, it is obvious that when facing multiple image blocks generated by high-resolution images, the linear layer or resampler structure will exponentially increase the number of tokens generated by the image, which brings a certain burden to the training and reasoning of LMM. The researchers investigated the current common bridge layer structures and finally determined to conduct various experiments based on the C-abs structure. Compared with MLP, C-abs adds additional convolution modules to further extract visual features and can also carry more training data with a certain amount of parameters.

For images of different resolutions, we first adjust the size of the image so that the image can obtain four image blocks of 336*336 size through the sliding window, and also adjust the original image to a resolution of 336*336. In the training and inference of the model, the features of 4*L*Dim and 1*L*Dim were obtained from vision encoder for 4 image blocks and 1 original image respectively. Among them, the first four sets of features undergo dimensional conversion to maintain the original spatial state of the large-resolution image, and obtain the features of 1*4L*Dim. Within C-abs, we introduced an interpolation coding module that is adaptive to different sequence lengths to expand the dimensions of position coding features, thereby ensuring the spatial understanding of large-resolution images. Finally, C-abs converts 1*4L*Dim into 1*L*Dim using AdaptivePooling operation. The visual features of the sliding window and the original image are spliced to obtain 2*L*Dim visual features and sent to LLM.

In the training process, a 2-stage training strategy is adopted. Taking into account factors such as training time, the researchers did not use a large amount of training data. The pre-training data was mainly the pre-training data of llava1.5, and a small amount of additional training data was added. On the SFT data, the researchers used the open source data llava_v1_5_mix665k to conduct experiments, and additionally mined multiple types of Chinese and English data to help the model achieve better command-following capabilities for vision-related tasks, especially for Chinese tasks. The Chinese multimodal data improved the Chinese understanding ability of the original llama3. With a limited amount of data, 360VL has achieved good understanding of visual content. It is foreseeable that the introduction of larger amounts of training data will stimulate the potential of the model and achieve better performance.

Experimental results

360VL has been evaluated on various zero-shot lists and has achieved good results in a variety of indicator dimensions. At present, models on the scale of big data have also begun to be trained, and there are plans to open source them in the future.

For different foundation model, researchers conducted comparative experiments under the same data. The visual models here are CLIP-ViT-L, siglip-so400m, and DFN5B-CLIP-ViT. The language models selected Vicuna1.5-7B, Vicuna1.5-13B, Llama3-8B, and Llama3-70B respectively. Judging from the experimental results, siglip-so400m and CLIP-ViT-L perform better. The Llama3 series has been comprehensively improved compared to Vicuna1.5. Llama3-8B also surpassed the effect of Vicuna1.5-13B. Llama3-70B achieved the best performance. The researchers also open sourced the version models equipped with Llama3 and CLIP-ViT-L. In terms of model structure, the researchers also tried to retain the original four sliding windows and expand the image feature dimension to 6*L*Dim, which can achieve a relatively good improvement in the model effect. However, as the image sequence becomes longer, the calculation efficiency will decrease significantly, so this operation is not retained.

Specific effect

Chinese language ability

OCR

Picture understanding

Positioning ability

Outlook

In summary, the researchers explored the multimodal capabilities of llama3 and provided the 360VL model equipped with llama3-70B. As a general-purpose multimodal model, 360VL has demonstrated excellent capabilities in handling standard tasks like visual question answering (VQA) and content creation, and you can download it to experience it. 360VL will also keep each new version open source.