The AI ​​painting model has always been summarized as the "text-to-image (T2I)" model. The reason is that the current mainstream image generation models basically only provide the ability to generate images based on text conditions, and the AI ​​painting products of various companies also compete mainly on the quality of generated images and text understanding capabilities. However, for professional users, the screen control capabilities provided by text are very limited, resulting in the current AI painting products not meeting the standards for use as professional productivity tools.

In order to solve this problem, 360 Artificial Intelligence Research Institute proposed layout-controllableAI image-generation modelHiCo at the top artificial intelligence conference NeurIPS2024, and will be open sourced in the near future. Based on the HiCo model, users can freely control and adjust the layout of different subjects in the generated screen to achieve the generation effect of "hit wherever you point". Without further ado, let’s take a look at a set of generation effects (users control the generation of specified content at different locations on the screen based on different rectangular frames and corresponding text descriptions):

**The following is a detailed explanation of the working principle of HiCo: **

• Paper title: HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

• Paper link: https://arxiv.org/abs/2410.14324

• Project home page: https://360cvgroup.github.io/HiCo_T2I/

1. Summary

The task of layout-to-image generation is an important research direction in the field of AIGC, which usually refers to synthesizing target images based on the text description of instance objects and their spatial positions.

Existing methods are still difficult to generate complex layouts, and common undesirable situations include object loss, light and shadow inconsistency, perspective conflicts, and target interactions in overlapping areas.

In order to effectively solve these problems, we propose a hierarchically controllable (HiCo)diffusion model, conditional branching structure with object separation.

Our main idea is to achieve spatial separation through hierarchical modeling of layout. We use multi-branch structures to represent hierarchical content and aggregate them into fusion modules.

2． motivation

The mainstream layout-controllable generation uses a newly designed network structure or a special cross-attention mechanism to achieve controllable target position. However, in complex scenarios, these methods have problems such as target loss, reduced command following ability, generated image distortion, excessive consumption of reasoning resources, and open source community ecological adaptation.

How to introduce the position controllability of layout sub-areas while ensuring that the original diffusion model capabilities remain unchanged is the primary challenge we face. Typical methods of introducing external controllable conditions for image generation, such as ControlNet, IP-Adapter, etc., whose external controllable conditions are not limited to human posture, key points, and reference images.

In order to solve the above challenges, this paper proposes hierarchical controllablediffusion model. This method extracts hierarchical layout features through weight-sharing branches, and uses a fusion network for fine aggregation.

3. method

3.1 Overall model architecture

In response to the above problems and challenges, combined with the current common methods of introducing external control conditions, this paper proposes a hierarchical controllable Diffusion Model named HiCo, which is used to generate high-quality and realistic images from layouts. The overall architecture of HiCo is shown in Figure 1, including the backbone SD basic model, the weight-sharing side branch network HiCo, and the fusion module FuseNet.

Figure 1: Hierarchical layout-controllable generated model HiCo structure

3.2 Hierarchical modeling and fusion module

In order to combine the capabilities of diffusion model and introduce the capabilities of layout-controllable, the researcher proposed the HiCo method. This method decouples the spatial layout of different targets through hierarchical modeling, and dynamically integrates background information and the content and interaction of different foreground targets.

The objective function of simple diffusion model is as follows:

The diffusion model objective function that introduces external control conditions is as follows. Given an input image, noise is added through progressive diffusion to, where is the number of noise adding steps, is the text control condition, is the specific control condition, and θ is the learnable network to predict the noise at different stages.

Compared with diffusion model that contains conventional control conditions, this method adds an additional weight-sharing HiCo Net to generate the global background and content of different layout areas. Due to the addition of multiple different control conditions, the training goals are:

Among them, , represents the text description and spatial location of the th instance, and F represents the fusion module.

The fusion module mainly implements effective fusion of the foreground and background, and can select appropriate fusion methods according to specific scenarios, including averaging, summation, mask weighting, etc. This article adopts the mask-based fusion method as follows, where represents the mask information of the th instance and represents the mask information of the background area.

3.3 Hierarchical feature visualization

HiCo adopts an innovative weight sharing mechanism in the overall structure, which can decouple different foreground instances and background images to generate independent features based on text description and spatial layout information. These features are strategically integrated in the upsampling stage. Figure 2 shows the generation process of the HiCo model of four instance layouts.

Figure 2: Visualization of hierarchical features of the HiCo model

3.4 Training data and strategies

The HiCo model can be trained using Grounding data of different types of scenarios to obtain the corresponding layout-controllable generation capabilities. This paper conducts experimental verification on both the fine-grained description data of the open set (GRIT-20M) and the coarse-grained category data of the closed set (COCO). Among them, we cleaned and screened out 1.2M training data from the subset data of GRIT-20M, and constructed the evaluation set HiCo-7K of fine-grained.The HiCo structure supports different versions of diffusion model, including but not limited to SD1.5, SDXL, SD3, Flux, etc. At the same time, we support the rapid generation of plug-ins or base models for SD models, such as LoRA, LCM, SDXL-Lighting, etc. For detailed training and inference details, please refer to the original text.

4. Experimental evaluation

4.1 Effect evaluation

fine-grainedDescribe quantitative assessment. On the fine-grained test setHiCo-7K, the image generation quality and layout-controllable were quantitatively evaluated in two dimensions. The HiCo model has achieved remarkable results, and the generation effect of complex scene images with different number of targets is also very ideal.

Figure 3: Qualitative comparison of HiCo-7K. The HiCo method can generate high-quality images of simple and complex layout information.

Human evaluation. We use a multi-round, multi-participant cross-evaluation method to evaluate human preferences from the aspects of target number, spatial location, and global image quality. The following table shows the manual evaluation indicators of different comparison methods. The results show that the HiCo method is better than other models in terms of spatial location and semantic dimensions. Furthermore, its performance is close to the RealisticVisionV51 model (SDReal) in the fine-grained dimension of global image quality, indicating that despite the enhanced controllability, the generative ability of our model is still robust and effective.

4.2 ablation study

This article conducts a series of ablation study on the HiCo structure, fusion strategy, etc. The specific results can be seen in the table below.

4.3 Reasoning efficiency

For inference running time and memory usage, we performed a two-dimensional comparison, detailed data in Figure 4. To compare the performance differences of different methods horizontally, we evaluated the inference time and GPU memory usage of directly generating 512×512 resolution images on HiCo-7K using a 24GB VRAM 3090 GPU.

The multi-branch of HiCo has two reasoning modes: "parallel mode" and "serial mode". In order to verify the performance advantage of HiCo when the number of objects increases and longitudinally compare the performance differences of different inference modes of HiCo, we evaluated the inference time and GPU memory usage of generating 512×512 resolution images on HiCo-7K. The results show that the HiComodel inference method is flexible and has obvious advantages in terms of time-consuming inference and memory usage.

Figure 4: Horizontal and vertical comparison of performance of HiComodel inference

5. Summary and Outlook

Complex target interactions and occlusion sequences in overlapping areas are important challenges for image generation in the HiCo model and even in the layout-controllable field. HiCo achieves hierarchical generation by decoupling each object's position and appearance information into distinct branches while controlling the overall interaction between them through a background branch with global cues and a fusion network.

HiCo is able to handle complex interactions of overlapping regions through the fusion module. The order of occlusion of overlapping objects is also specified via global hints described in text. However, due to the lack of corresponding occlusion sequence training data, the success rate is far from optimal. For the current HiCo, there is indeed a lack of a clearer occlusion sequence control mechanism.

HiCo is a multi-branch structure-oriented controllable layout generation model based on diffusion model. The superiority of this method is proven by training and testing data of different granularities in natural scenes, and conducting algorithm metric evaluation and subjective human evaluation. However, there is still potential for further improvements, especially in image content editing and integrating multiple styling concepts. By combining the current controllable generation capabilities, the overall playability of AI-generated artwork can be improved.