Bridge Diffusion Model: Native Chinese Text-to-Image Generation
The problem of worldview bias in AI** painting models**
The breakthrough in image generation technology based on diffusion model in 2022 quickly triggered a global image AIGC research and development boom and application revolution. One of the most noteworthy is the Stable Diffusion[1] model developed and open sourced by Stability Company, which allows ordinary people to quickly experience the practical changes and promotion of real-world productivity caused by AI technology. Around the open source Stable Diffusion model, many open source developers and AI painting enthusiasts have formed many large AI painting communities, such as Civitai[2], Stable Diffusion Online[3], etc., and have continuously launched various derivative tools and models in these communities, forming an important force in promoting the advancement of AI painting technology.
The development of Chinese AI image-generation models lags behind English AI image-generation models as a whole. AI image-generation models belong to the category of CV large models. Training an AI image-generation model requires massive training data and requires high training computing power. Taking Stable Diffusion 2.1 as an example, according to public information [4]: SD2.1 only uses 256 A100 GPUs for the training of the base model. When the training is equivalent to 200,000 cards, a total of 2.87 billion image–text sample training volumes are required. There are only a handful of domestic R&D institutions that can meet both computing power and data requirements. This has resulted in a large number of Chinese AI painting products actually using open source English SD models and their fine-tuning models as their capability base.
However, English AI image-generation models represented by SD, including but not limited to SD1.4/1.5/2.1, DALLE-2[5], Midjourney[6], etc., generally have obvious bias towards the English world. As pointed out by work [7], the characters generated by the current English model are more biased towards white people and males. In addition to human images, as shown in Figure 1 below, items, buildings, vehicles, clothing, logos, etc. all have common prejudices in the English world. In addition, phenomena such as "busy traffic" and "braised lion's head" that have been ridiculed by netizens are essentially because Chinese concepts cannot be accurately generated by English AI image-generation models.
Figure 1 Example of the worldview bias of the English AI image-generation model. The generated vehicles, buildings, characters, flags, signs, etc. all have obvious bias towards the English world. From left to right: SDXL, Midjourney, domestic friend B*, domestic friend V*
Route selection for Chinese AI image-generation model
There are currently several ways to develop Chinese AI image-generation models from easy to difficult:
English model + translation. This method is simple and crude, with almost no cost except translation. It can be said to be the lowest-cost "Chinese culture" option, and has been widely used in a large number of AI painting products. However, this method can only solve the superficial Chinese input problem, and cannot solve the problem that the English model cannot accurately generate a cognitive image consistent with the Chinese world due to model bias.
English model + implicit translation. Different from the explicit method of calling the translation service, this method replaces the text encoder of the English model with the Chinese text encoder, and uses the Chinese and English parallel corpus used in the translation model training to train the Chinese text encoder so that its output embedding space is aligned with the original English text encoder. It is essentially an implicit translation.
English model + implicit translation + fine-tuning. Based on the above method, the model aligned with the text encoder is further fine-tuned as a whole using Chinese image–text data to improve the output capability of the AI painting model for Chinese images. This method can alleviate the model bias problem caused by the English base model to a certain extent, but due to the nature of its implicit translation, there are always loopholes in the ability to understand and draw polysemy words. Representative works in this area include IDEA Taiyi[9], Tiangong[10], etc. Wen Xinyiyan, which was opened to the public on August 31, also has this problem[11].
Figure 2 The latest version of Wen Xin Yi Yan still has the problem of confusing English polysemy words caused by implicit translation[11]
The structure of the English model is trained from scratch using Chinese data. This is a very thorough Chinese culture solution, because the model is completely trained from scratch using Chinese data, so it can be guaranteed that the final trained model has complete native Chinese capabilities: it understands Chinese input and can give image output results that are consistent with Chinese world cognition. Representative works in this area include Alibaba Damo Academy[12] and Huawei's Wukong Painting[13]. The advantage of this method is that it has the best Chinese ability and does not have the English model bias problem of the aforementioned routes. However, this method also has very big shortcomings in actual application: in addition to high training costs, the most fatal problem is that it cannot reuse the existing results of the English AI painting community, such as various styles of fine-tuning models, LoRA, ControlNet, etc.
Table 1 Comparison of different route options and their advantages and disadvantages of current Chinese AI image-generation models
BDMChinese native AI image-generation model
As summarized in Table 1, in the current Chinese AI image-generation model route, Chinese training from scratch can provide Chinese users with the most complete native Chinese capabilities, but the cost is that the Chinese model is incompatible with the English SD model. Therefore, a large number of community resources derived from the English SD model such as fine-tuned models, LoRA, Dreambooth, ControlNet, etc. cannot be used directly. In theory, these models need to be re-adapted and trained for the Chinese model. This makes it difficult for the Chinese model to form a community effect and may continue to lag behind the progress of the English community.
Whether we can further open up the compatibility issue between the Chinese native model and the English SD community on top of the native Chinese capabilities has become a key problem for us to overcome. After nearly half a year of technical research, we have proposed a new diffusion model structure called "Bridge Diffusion Model" (BDM) to solve the above dilemma. BDM can not only accurately generate Chinese semantic images, solving the world bias problem of English models, but also maintains interoperability with the English community and is seamlessly compatible with various English SD community plug-ins. This is also the origin of the "Bridge" in the name. BDM combines the original English model through the trunk-side branch architecture, and uses pure Chinese data for training to create a Chinese native AI image-generation model.
Paper link: https://arxiv.org/abs/2309.00952
1. Model framework
BDM adopts a trunk-side branch network structure similar to ControlNet[14], as shown in Figure 3(b). The backbone network adopts the structure of Stable Diffusion 1.5 and is initialized using its pre-training parameters, while the side network is composed of learnable copies derived from the backbone network. Compared with ControlNet, the structural difference of BDM is that there is no conditional image convolution layer in the side branch. This is because in BDM, the Chinese prompt is processed through the side branch network instead of the backbone network. We chose Chinese CLIP text encoder[15] as the Chinese text encoder. The English text encoder of the backbone network can be removed and only supports Chinese prompts through the side network, or it can be retained to achieve bilingual support in Chinese and English. In our implementation, we choose to retain the English text encoder, so BDM has the ability to draw in both Chinese and English. A key advantage of the BDM network structure is that the backbone network contains the complete English SD structure and is frozen during training. Therefore, the latent space of BDM is consistent with the English SD model, making it seamlessly compatible with various community plug-ins adapted to the English SD model.
Figure 3 Network structure diagram of ControlNet and BDM. The left picture is ControlNet and the right picture is BDM
2. Training strategy
The overall training loss of BDM is shown in Figure 4. The diffusion model[16] algorithm learns a network εθ to predict the noise added to the noisy image zt based on a set of conditions, including time step t, text input cent for the main stem, and text input cnlt for the side branch.
However, only having such a network structure and training goals is not enough, because it is not feasible to inject semantic information into the backbone and side branches at the same time during the training process. This is because the pre-trained backbone already contains powerful English semantic information, which will hinder the Chinese semantic learning of the side branches. Therefore, BDM uses a key training strategy - the text input of the backbone during the training phase is always an empty string, that is, cent is always "". This is because during the SD1.5 training process, the text input has a 10% probability of being empty, so it can be considered that the latent space corresponding to the empty string is the average latent space of the entire image generated by SD1.5. For BDM, the backbone provides the average latent space of the English model. At the same time, the side branches learn Chinese semantics in this space and find the offset of Chinese semantics in the English space. This truly organically combines the BDM Chinese native model with the English community, allowing BDM to seamlessly access the English community.
Figure 4 BDM training loss
3. Reasoning strategy
During the training phase, the backbone parameters are always locked and the backbone text input is always an empty string, but during the inference phase there are many different options.
First, we can set the positive/negative text input of the backbone to empty strings, which is consistent with the training stage, and only use Chinese positive/negative input to inject semantics. We observed that for the BDM model in the early stage of training, using universal positive/negative text input in the backbone during the inference stage significantly improved the image quality; however, for the BDM model at the end of training, the Chinese side branch has been fully trained, and the English positive/negative input has little impact on image quality. Of course, in order to achieve better generation effects, both positive/negative input in Chinese and English can be adaptively adjusted according to user needs.
BDM can also be seamlessly integrated with various plug-ins in the English community. When combined with LoRA[17], just embed the LoRA model into the BDM backbone structure in the same way as the conventional English model is embedded into LoRA, and then input the required Chinese prompts from the side branch. If LoRA contains trigger words, the trigger words need to be input into the backbone during inference. Similarly, when combined with ControlNet, the ControlNet branch can be embedded into the BDM trunk, thus forming a trunk-double branch structure. When combined with checkpoint or Dreambooth[18], just switch the BDM trunk from SD1.5 to the corresponding base mold. When combined with Textual Inversion[19], the corresponding embedding can be directly loaded into the main text input. The above operations can be combined as needed.
4. Effect display
As shown in Figure 5, BDM uses SD1.5 and realisticVisionV51[20] as the backbone to generate Chinese concepts. It can be seen that both Chinese unique concepts and English polysemy concepts are generated very reasonably.
Figure 5 Chinese concept generation
As shown in Figure 6, BDM uses different styles of checkpoint[21][22][23][24] as the backbone for generation. Since different models require different conditions to generate specific styles, such as Trigger words are required, and some require style words. During inference, the Chinese/English positive/negative input will be fine-tuned according to the style conditions to achieve better results; but it is certain that the fine-tuned text only involves trigger words or style words, and the specific image content will only be input from Chinese text.
Figure 6 Checkpoint effects of different styles
As shown in Figure 7, BDM is generated by combining different LoRA[25][26][27]
Figure 7 LoRA effects of different styles
As shown in Figure 8, BDM is generated by combining ControlNet[28]
Figure 8 Combined with ControlNet effect
As shown in Figure 9, BDM is generated by combining different Dreambooths, using the base molds of 6 celebrities[29].
Figure 9 Different Dreambooth effects
As shown in Figure 10, BDM is generated in combination with Textual Inversion[30], using age-adjusted Textual Inversion.
Figure 10 Different Textual Inversion effects
Future of WorkThe BDM1.0 model was trained using 1.2 billion Chinese Internet image–text data collected internally by 360. As shown previously, BDM has very good Chinese native AI painting capabilities and is seamlessly compatible with various models and plug-ins in the current English SD community. The Chinese AI painting product "360 Hongtu" developed based on BDM1.0 capabilities will also be opened to the public in the near future. The experience entrance is: https://ai.360.cn/。
The structure of BDM is very flexible. In addition to being combined with SD1.5, based on the same principle, BDM can also be combined with SDXL, DALLE-2, Imagen and other model structures to further enhance the scale and capabilities of the Chinese native AI image-generation model.
In addition, it is well known that the current AI image-generation model still has very significant flaws in its ability to understand text prompts. In order to obtain good generation results, prompt instructions are more of a pile of various keywords/magic words, which are still far from the natural language used in human communication. Improving the AI painting model's ability to follow prompt instructions is also the focus of our current efforts in combining multimodal LLM (SEEChat: https://github.com/360CVGroup/SEEChat) with AIGC.
About the author
Leng Dawei: Head of the visual direction of 360 Artificial Intelligence Research Institute. Currently leading the visual team of the institute to carry out cutting-edge exploration and industrial implementation work in large multimodal model (LMM), visual AIGC, cross-modal graphic learning, open-world object detection, open vocabulary video analysis, AIoT and other directions.
Liu Shanyuan: Algorithm expert of the Visual Engine Department of 360 Artificial Intelligence Research Institute. Currently conducting cutting-edge exploration in AIGC generation, editing, and multiple concepts.