HiCo: 360’s Layout-Controllable Image-Generation Model
360 AI Research released HiCo, a layout-controllable image-generation model. In addition to ordinary text prompts, HiCo lets users specify different content for different regions of an image. The model supports controllable layouts with more than eight regions and is designed to reduce visual fragmentation and viewpoint inconsistencies between regions, producing more coherent multi-region compositions. HiCo is compatible with both the Stable Diffusion ecosystem and Bridge Diffusion Model (https://arxiv.org/abs/2309.00952), allowing it to be integrated into existing workflows as a plug-in.
Background introduction
In the field of AI painting, many researchers are committed to enhancing the controllability of AI image-generation models. They not only focus on the model's grasp of details and style, but also actively explore how to let the model better understand and simulate human creative thinking, so as to improve the practicality and personalization of paintings while maintaining artistry.
The current controllable capabilities of AI painting mainly guide information such as text descriptions (Prompt-to-Prompt, etc.), specific types of images (Controlnet, etc.), and model structures (CrossAttention, etc.), which can achieve controllable generation of images in terms of shape, color, style, layout, etc.
The image generation models currently on the market have the following characteristics: 1. They can only generate limited coarse-grained categories of images; 2. They are not effectively compatible with open source community capabilities, including different base models, different LoRAs, etc.; 3. The controllable generation capabilities cannot be effectively combined with concept injection capabilities. The controllable layoutAI image-generation modelHiCo (Hierarchical Controllable difffusion model for layout-to-image generation) self-developed by 360 Artificial Intelligence Research Institute can realize fine-grainedlayout-controllable__ of different rough text descriptions, and can seamlessly migrate various capabilities of the open source community, and also has the position controllable generation of concepts.
Method Overview
Based on diffusion model's classic papers on the direction of controllable image generation, Controlnet, IP-Adapter, etc., it is proposed that the controllable generation of the final image can be achieved through external guidance conditions such as Canny, Depth, OpenPose and other different condition maps.
The controllable layout generation model HiCo independently developed by the Artificial Intelligence Research Institute draws on the idea of external control condition generation in the above-mentioned article, and realizes layout-controllable generation through the spatial position and text description of the image layout, combined with the principle of image layering. The generation guidance conditions of our model are the sub-boundingbox and corresponding sub-caption input by the user. It can also provide global-caption of the image background description. Our model cleverly avoids training the UNet part of the text-to-image (T2I) basic model. It only trains a lightweight pluggable module to achieve the purpose of image generation layout-controllable. At the same time, it can be seamlessly compatible with the open source Stable Diffusion 1.5 and XL model community ecology, including different base models, different LoRA, LCM/LCM-LoRA rapid generation and other capabilities.
The working principle of the HiCocontrollable layout model uses the trainable HiCo Net to generate content in different designated areas and the global background. The backbone UNet part fuses the output results and backgrounds of different HiCo Net to generate coordinated images.
Figure 1: Hierarchical layout-controllable generated model HiCo structure
The objective function of the original Stable Diffusion model is:
diffusion model training training objective of external control conditions, given the input image Z0, noise is added to Zt through progressive diffusion, where t is the number of noise adding steps, Ct is the text control condition, Cf is the specific control condition, єθ is the learnable network to predict the noise at different stages, and its objective function can be expressed as:
It can be seen from the HiCo schematic that compared to the native Stable Diffusion model, an additional HiCo Net is added to generate the global background and content of different layout areas. Due to the addition of multiple different control conditions, the training goals are:
Ability Application
We will show the effect of HiCo model combined with different models and plug-ins respectively. The following is a display of the multi-objective controllable generation effect of controllable layout generation model HiCo.
Figure 2-1: 3 targets layout-controllable generated
The left picture shows the three different areas specified by the user and the generated content specified in the different areas. The right picture shows the final generation result. The middle picture shows the effect of overlaying the user's layout input onto the generated image to verify the layout control.
Figure 2-2: 5 targets layout-controllable generated
The following is a demonstration of the effect of multi-objective controllable generation of controllable layout generation model HiCo combined with different basic model checkpoints. The corresponding different basic models are midjourneyPapercut, realisticVision, disneyPixarCartoon, and flat2DAnimerge. In order to easily verify the effect of layout control, we overlay the user-input layout (region box + region prompt) onto the final generated image for display.
Figure 3: Different base model controllable layout generation effects
The following is a display of the multi-concept controllable generation effect of controllable layout generation model HiCo combined with different LoRAs. The corresponding different LoRAs are Blindbox_v3 and Shrek.
Figure 4: Concept embedded layout-controllable generation effect
In addition, HiCo is also compatible with LCM/LCM-lora, etc. to achieve rapid generation. The following is a display of the controllable rapid generation of controllable layout generation model HiCo combined with LCM/LCM-LoRA. The corresponding number of LCM steps are: 4-steps, 6-steps, 8-steps.
Figure 5: controllable layoutLCM rapid generation effect of different steps
Indicator Verification
Based on GRIT-20M, we constructed the GRIT-VAL evaluation set after manual revision to quantitatively evaluate the controllable layout generation ability of HiCo. GRIT-VAL contains 300 pictures mainly of natural scenes, and carries rich target location annotations and detailed text description annotations.
On the GRIT-VAL evaluation set, comparing the English SD basic model, English model + HiCo, Chinese BDM basic model, Chinese BDM+ HiCo, we evaluated dimensions including the number of targets, target semantics and location. The evaluation of the Chinese model uses a translator to translate the English prompt into Chinese. The error introduced by the translation causes the layout control score of the Chinese BDM+HiCo to be slightly lower than the English version.
Note 1: The generation conditions of this evaluation dataset are fine-grained text description and corresponding position box, and the generated text has no description restrictions.
Figure 6: GRITfine-grained unlimited text description picture effect
As a further demonstration, the following also gives the generation effect obtained by using COCO-based regional annotation and coarse-grained text description (only category names) using the HiCo model (English):
Figure 7: COCO coarse-grained limited text description image effect
Summary and Outlook
The controllable layout generation model HiCo released by 360 Artificial Intelligence Research Institute this time provides more powerful screen layout control capabilities on top of ordinary text input. HiCo is mainly trained on different coarse fine-grained data in natural scenes. The evaluation of relevant indicators reflects the advancement of the HiCo model. There is still room for further improvement in the model, especially in capabilities such as image content editing and multi-style concept injection. Based on the HiCo model, the playability of AI drawing can be greatly improved. You can actually experience it in 360 Intelligence (aigc.360.com) later.