AAAI | Bridge Diffusion Model Connects Native Chinese Generation with the Stable Diffusion Ecosystem
It has the ability to understand native Chinese and is also compatible with the Stable Diffusion ecosystem.
The latest model structure Bridge Diffusion Model is here.
Combined with the Dreambooth model, it generated the Chinese star wearing a Chinese wedding dress looking like this.
It was proposed by 360 Artificial Intelligence Research Institute, recently accepted by AAAI, and has been open source.
A branch network idea similar to ControlNet
The Chinese native problem of the text-to-image (T2I) model has always been a key research issue.
Limited by computing power and data factors, behind a large number of domestic Chinese AI painting products, many are actually based on open source English models and fine-tuned models as their capability base. However, English models include but are not limited to SD1.4/1.5/2.1/3.5 and DALLE, Midjourney, Flux, etc., because the training data of these models are mainly English data, when generating images, the main images include people, objects, buildings, vehicles, clothing, logos, etc. There is a very common and obvious English worldview bias**.
BDM is our relatively early work in the direction of multimodal generation, focusing on two key issues:
- Worldview bias of native Chinese and generative models
- Compatibility with SD ecosystem
Dr. Leng Dawei made the above concise summary of the main focus of the work of BDM.
The issue of "native Chinese" refers to not only that the text-to-image (T2I) model supports Chinese input, but more importantly, it requires that the images of people and objects generated by the model should conform to the cognition of Chinese culture.
The picture below is an example of the worldview bias of the AI image-generation model. From left to right are SDXL, Midjourney, domestic friend B*, and domestic friend V*:
The Chinese AI image-generation model, in terms of implementation route selection, generally has the following methods from easy to difficult:
English model + translation.
Simple and direct, almost no cost except translation. This method can only solve the superficial Chinese input problem, but cannot solve the problem that the English model cannot generate a cognitive image consistent with Chinese culture due to model bias.
English model + implicit translation.
Different from explicitly calling the translation service, this method replaces the text encoder of the English model with the Chinese text encoder, and uses Chinese and English parallel corpora to train the Chinese text encoder so that its output embedding space is aligned with the original English text encoder. It is essentially an implicit translation and a very low-cost solution. It also cannot solve the problem of worldview bias of the model.
English model + implicit translation + fine-tuning.
Based on the above method, the model aligned with the text encoder is further fine-tuned as a whole using Chinese image–text data to improve the model's output capability of Chinese images. It can alleviate the model bias problem caused by the English base model to a certain extent.
Train from scratch on Chinese data.
This is the most thorough Chinese cultural solution: it understands Chinese input and can provide image output results that conform to Chinese cultural cognition, which can perfectly solve the problem of worldview bias of the model.
Of the above four routes, the fourth route looks perfect, but there is still one point worthy of additional research and development efforts: Beyond foundation model, what we need to further consider is the model ecological issues in the era of large models.
A very large open source community ecosystem has been formed around open-source model represented by SD. In this ecosystem, a large number of derivative style models, plug-in models, etc. have accumulated very valuable group intellectual assets.
On the basis of overcoming the worldview bias of the AI painting model, further achieving compatibility with the open source community is the second key issue to be solved in our BDM work.
BDM is a branch network idea similar to ControlNet in terms of model structure. It uses different network branches to learn data in different languages. Therefore, in principle, BDM can not only generate native Chinese images, but also generate images in any X language, and ensure that the generated images conform to the cognition of the corresponding language and culture.
**The English part can directly reuse the existing open-source model, thereby achieving seamless compatibility with the open source community. **Note that BDM only needs to input one language when using it. For example, when inputting Chinese, the English branch uses empty text as input.
BDM v1 version uses 1 billion Chinese image–text data for training and is compatible with the SD1.5 community ecology.
The figure below shows the ability of BDM to generate Chinese-specific concepts and the generation effect in Chinese-English ambiguous situations that translation cannot handle:
The picture below shows the ecological compatibility of BDM in the SD1.5 community. It can be seen that BDM has good compatibility with different SD1.5 style fine-tuning models, especially BDM while maintaining the output capability of Chinese images. For more cases, please see the AAAI paper.
About 360 Artificial Intelligence Research Institute
All in 360 Group In the context of AI, 360 Artificial Intelligence Research Institute has given full play to its intellectual advantages and undertaken the strategic research and development tasks of understanding and generating large models (commonly known as pictures and text and text-to-image (T2I)). It has continued to exert efforts in both directions and has successively developed large multimodal models, Bridge Diffusion Model (BDM), the layout-controllable HiCo model, and Qihoo-T2X based on a new Diffusion Transformer architecture.
Recently, the institute's work on multimodal understanding direction IAA and multimodal generation direction BDM were respectively accepted by the top conference in the field of AI AAAI. The R&D leader of these two works is Dr. Leng Dawei.
It is reported that this AAAI 2025 conference received nearly 13,000 submissions and accepted 3,032 jobs, with an acceptance rate of only 23.4%.
Arxiv: https://arxiv.org/abs/2309.00952 Github: https://github.com/360CVGroup/Bridge_Diffusion_Model