ICLR 2025 | Qihoo-T2X Cuts DiT Compute with a Unified Text-to-Any Architecture

Due to the global self-attention, the computational complexity of the Diffusion Transformer model is proportional to the square of the sequence length, causing it to face the problem of high computational cost in high-resolution image and long-term video generation tasks.

To this end, researchers from Sun Yat-sen University and 360 AI Research proposed an efficient Diffusion Transformer based on Proxy token, namely PT-DiT, which can be applied to various generation tasks such as text generation, image, video and Multi-View. Based on PT-DiT, the author further built the Qihoo-T2X series including Qihoo-T2I, Qihoo-T2V and Qihoo-T2MV models to achieve arbitrary tasks of text generation.

Under the same parameter scale, the calculation amount of PT-DiT is only that of Lumina-Next, and that of DiT. While reducing the amount of calculation, Qihoo-T2X shows similar performance to PixArt-, Flux, Easyanimate and CogVideo, capable of generating realistic high-quality images and videos according to given instructions.

The research is now open source.

Paper address: https://arxiv.org/pdf/2409.04005

Project homepage: https://360cvgroup.github.io/Qihoo-T2X

Code repository: https://github.com/360CVGroup/Qihoo-T2X

Research motivation

Currently, models based on Diffusion Transformer (Sora, Vidu, Flux, etc.) can generate high-fidelity images or videos with strong consistency with text instructions, greatly promoting the progress of visual generation. However, the quadratic complexity of global self-attention with respect to sequence length increases the computational overhead of Diffusion Transformer, resulting in longer generation time and higher training cost in practical applications.

This problem also hinders the application of Diffusion Transformer in high-quality and long-term video generation.

For example, 3D full attention, which is better than 2D spatial attention + 1D temporal attention, is difficult to explore for higher resolution and longer video generation due to limitations in computational overhead.          

Some research in the field of visual understanding and recognition has found that global self-attention has a certain redundancy due to the sparse and repetitive nature of visual information. By visualizing the attention map, the research team found that different tokens in the same window pay similar attention to tokens that are far away in space, but pay different attention to tokens that are close in space.

As shown in the figure above, the research team analyzed the attention map in the self-attention of PixArt- at the resolution of , and then combined the attention maps located in the same spatial window token together, as shown on the right side of the figure (the vertical axis represents different tokens in the window, and the horizontal axis represents the degree of association between the token in the window and all tokens). It is obvious that the attention of different tokens within the same window is almost consistent for tokens that are spatially distant (i.e., at the same horizontal position, the vertical values ​​are almost the same). And spatially adjacent tokens show different concerns.   

This shows that computing attention for all tokens is redundant, while computing attention for spatially adjacent tokens is crucial. 

Therefore, the research team proposed a sparse attention strategy based on proxy tokens, sampling limited proxy tokens from each window to execute self-attention, thereby reducing redundancy and reducing complexity.

Qihoo-T2XMethod

As shown in the figure, PT-DiT proposed by the research team introduces proxy tokens to reduce the number of tokens involved in calculating global self-attention and efficiently establish the association of global visual information. PT-DiT contains two core modules: Global Information Interaction Module (GIIM) and Texture Complement Module (TCM).   

Among them, GIIM uses a sparse proxy token mechanism to promote efficient interaction between all potential codes, and the correlation between spatially adjacent tokens cannot be ignored, especially for image generation tasks with high requirements for detailed textures. For this purpose, the research team designed a TCM, which further refines local details through window attention and shift window attention.

These two parts will be introduced in detail below:

Global Information Interaction Module

Given a series of latent tokens, a series of proxy tokens are first obtained by calculating the average token according to the spatial and temporal priors (that is, located in the same spatial window).

Each proxy token represents the information of a local area within the image or video, and interacts with proxy tokens in other local areas through self-attention to establish global visual association.

Subsequently, the information contained in proxy tokens is propagated to all latent tokens through cross-attention with latent tokens, thereby achieving efficient global visual information interaction.

Texture Complement Module

Due to the interactive characteristics of sparse proxy tokens and the lack of correlation between spatially adjacent tokens, the generation model has limited ability to model detailed textures and is difficult to meet the high-quality requirements of the generation task.  In order to solve this problem, the research team introduced local window attention to supplement the model's detailed modeling and smoothing capabilities.          

Only window attention will lead to a lack of connection between tokens between windows, resulting in an obvious image grid phenomenon, as shown in the figure. Therefore, shift window attention is also introduced in TCM to alleviate the problems caused by local window attention.

Since the number of tokens involved in calculating window attention is small, the computational complexity of the model does not increase on a large scale.

Compression ratio

For the image generation task, the research team found that maintaining the same number of windows at different resolutions is crucial to ensure a consistent semantic hierarchy, which facilitates the training process from low to high resolution.

At the same time, the number of windows should be maintained to prevent the semantic information within the window from being too rich and causing a single token to be insufficient to represent the local area to complete global information modeling.          

Therefore, the research team set the compression ratio to , , and at 256, 512, 1024 and 2048 respectively. resolution.

When the input is an image, and will be set to 1.

For the video generation task, the research team set = 4 at different resolutions to maintain consistent time compression. Due to token compression in the frame, height and width dimensions, PT-DiT can train generators for longer videos.

Computational complexity analysis

PT-DiT uses only a small amount of representative token attention, reducing the computational redundancy of the original full token self-attention. The research team further theoretically analyzed the advantages of PT-DiT in terms of computational complexity.          

The computational complexity of self-attention is, which is calculated as follows:

represents the length of the potential tag, represents the feature dimension.

Similarly, the computational complexity of GIIM and TCM is calculated as follows:

is larger and the sequence length () is longer. When is , , and and the image resolution is 256 , 512 , 1024 and 2048, the computational complexity of PT-DiT is only that of global self-attention , , and . Furthermore, PT-DiT provides greater benefits for video generation tasks with longer sequence lengths.          

Experiment

The authors conducted qualitative and quantitative experiments on T2I T2V and T2MV tasks to evaluate Qihoo-T2X.

Qualitative AnalysisText-to-Image: As shown in the figure. Qihoo-T2I demonstrated competitive performance, producing photorealistic images that perfectly matched the provided text prompts.   

Text-to-Video: The researchers compared Qihoo-T2V with recently released open source text-to-video models, namely EasyAnimateV4 and CogVideoX, at 512 resolution and achieved better results, as shown in the figure.   

Text-to-MultiView: The authors further explore the effectiveness of PT-DiT in text-to-multiview (T2MV) tasks.

The trained Qihoo-T2MV is able to generate 512 \times 512 \times 24$ images from various viewpoints based on the provided text instructions, showing strong spatial consistency, as shown in the figure.   

Quantitative Analysis

The research team quantitatively evaluated Qihoo-T2I and Qihoo-T2V on benchmarks such as MS-COCO FID-30K, UCF-101 and MSR-VTT. The results showed that both Qihoo-T2I and Qihoo-T2V can achieve competitive performance, proving the effectiveness of PT-DiT.          

Computational complexity analysis

As shown in the figure, whether it is an image or video generation task, under the same parameter scale, PT-DiT has a substantial computational complexity advantage over the existing Diffusion Transformer method. At the same time, comparing EasyanimateV4 with 3D full attention modeling, its training memory explodes as the number of frames increases, while the memory of PT-DiT only increases slightly, indicating that PT-DiT has the potential to complete longer video generation tasks.

ablation study