FancyVideo: Dynamic and Consistent Video Generation with Cross-Frame Textual Guidance
Article link: https://arxiv.org/pdf/2408.08189 Project link: https://360cvgroup.github.io/FancyVideo/
Highlights
This article introduces FancyVideo, the first known pioneering attempt to explore cross-frame text guidance in T2V tasks. This approach provides a new perspective for enhancing current text control methods.
This paper proposes a cross-frame text guidance module (CTGM), which constructs cross-frame text conditions and subsequently guides the modeling of latent features with strong temporal rationality. It can effectively enhance the movement and consistency of the video.
This paper demonstrates that incorporating cross-frame text guidance is an effective way to achieve high-quality video generation. Experiments show that this method achieves state-of-the-art results in both quantitative and qualitative evaluations.
Synthesizing action-rich and temporally consistent videos remains a challenge in the field of artificial intelligence, especially when dealing with longer duration videos. Existing text-to-video (T2V) models typically use spatial cross-attention to implement text control, which is equivalent to guiding the generation of different frames without frame-specific text guidance. Therefore, the model is limited in its ability to understand the temporal logic conveyed in the cues and generate videos with coherent actions.
To address this limitation, this paper introduces FancyVideo, an innovative video generator that improves the existing text control mechanism through a carefully designed cross-frame text guidance module (CTGM). Specifically, CTGM adds a temporal information injector (TII), a temporal correlation refiner (TAR), and a temporal feature enhancer (TFB) at the beginning, middle, and end of cross-attention respectively to achieve frame-specific text guidance. First, TII injects frame-specific information from latent features into text conditions, thereby obtaining cross-frame text conditions. Then, TAR optimizes the correlation matrix between text conditions and latent features across frames in the temporal dimension. Finally, TFB enhances the temporal consistency of latent features. Extensive experiments, including quantitative and qualitative evaluations, demonstrate the effectiveness of FancyVideo. Our approach achieves state-of-the-art T2V generation results on the EvalCrafter benchmark and facilitates the synthesis of dynamic and consistent videos.
method
Preliminary knowledge
Latentdiffusion model: Latentdiffusion model (LDMs) are a class of efficient diffusion model that transform the denoising process into compressed latent space instead of pixel space. Specifically, LDMs use VAE’s encoder to compress images into latent codes and learn the data distribution by performing forward and backward diffusion processes on the latent codes. It assumes a forward process that gradually introduces Gaussian noise (ϵ ∼ N(0, I)) into the latent code (z), giving:
where represents a noisy scheduler with time steps. In the reverse process, it trains a denoising model () with the following objectives:
where represents the condition and the target can be noise ϵ, denoised input or (ϵ). This paper uses prediction as supervision.
Zero-Terminal SNR Noise Scheduling: Previous research proposed zero-terminal Signal-to-Noise Ratio (SNR) to address the SNR difference between the testing phase and the training phase, which affects the generation quality. During the training process, the SNR is still non-zero at the terminal time step T due to the residual signal from the noisy scheduler. However, at test time, when sampling from random Gaussian noise, the sampler lacked real data, resulting in an SNR of zero. This training-test difference is unreasonable and an obstacle to generating high-quality videos. Therefore, referring to the past, researchers have expanded the scope of noise scheduling and set up to solve this problem.
Model architecture
Figure 3 below shows the overall architecture of FancyVideo. The model structure is pseudo-3D UNet, which integrates the frozen space module derived from the text-to-image model, as well as the cross-frame text guidance module (CTGM) and the temporal attention block. The model accepts three features as input: noise latent variable, where and represent the height and width of the latent variable, represent the number of frames, and represent the number of channels of the latent variable; mask indicator, where the element of the first frame is set to 1 and other frames are set to 0; image indicator, where the first frame is the initial image and other frames are set to 0. The denoising input Z is formed by concatenating Zn, M and I along the channel dimension, denoted as . In each spatial block, CTGM is used to capture the complex dynamics described in the text prompts. Afterwards, temporal attention blocks are applied to enhance the temporal relationships between different segments.
Cross-frame text guidance module
CTGM improves the existing text control method through three sub-modules: Temporal Information Injector (TII), Temporal Affinity Refiner (TAR) and Temporal Feature Booster (TFB), as shown in Figure (III). Before entering cross-attention, TII first extracts temporal latent features, and then integrates temporal information into text embedding based on Zt to obtain cross-frame text conditions. Subsequently, TAR refines the affinities between texts along the timeline, enhancing the temporal coherence of text guidance. Finally, TFB further enhances the temporal continuity of features. The calculation process of CTGM can be formalized as:
Among them, , and represent the linear layer of the query, key and value in the original cross-attention respectively. Hyperparameters are obtained from the dimensions of the query. TII(·,·), TAR(·) and TFB(·) represent TII, TAR and TFB respectively. Finally, the optimized noise potential characteristics are obtained. Below is a detailed description of these three modules.
Temporal information injector: In previous work, the text embedding was repeated evenly f times, and the result was, where n represents the length of the embedding vector. Before the execution space cross-attention, time information is injected into the embedding, so that the focus on the text is different in different frames. In the temporal information injector (TII), the noise latent feature Z is first adjusted from to , and the temporal self-attention mechanism is applied to obtain. Then, spatial cross-attention is performed using repeated text embeddings as queries and noise latent features as keys and values, resulting in text embeddings with frame-specific temporal information. The formal expression of the TII module is as follows:
Among them, represents time self-attention and represents space cross-attention. Through TII, noise latent features with temporal information and text embedding aligned with the latent features are obtained.
Temporal Affinity Refiner: In order to dynamically allocate the attention of text embedding in different frames, the researchers designed the Temporal Affinity Refiner (TAR) to optimize the attention map of the space cross-attention. In the space cross-attention, noise latent features serve as queries, and text embeddings serve as both keys and values. The attention map is calculated as,reflecting the affinity between text and image patches. Subsequently, TAR applies time self-attention to the attention map to obtain the refined attention map, which is represented as follows:
With TAR, a more reasonable temporal connection is established in the affinity matrix. It can perform more dynamic movements while ensuring that no additional video distortion occurs. Finally, the cross-attention process is completed with the refined attention map, i.e.
Temporal feature enhancer: In order to further improve the temporal consistency of the features, it is processed through the temporal feature enhancer (TFB). This creates a closer connection over time. Specifically, TFB includes a simple but effective temporal self-attention layer to refine the noise latent features along the temporal dimension, expressed as:
experiment
Quantitative experiment: In the quantitative experiment, FancyVideo uses the T2I basic model to generate the image as the first frame. In qualitative experiments, an external model is used to generate beautiful first frames for aesthetic purposes and watermark removal.
Qualitative evaluation: The researchers selected AnimateDiff, DynamiCrafter and two commercial products, Pika and Gen2, for comprehensive qualitative analysis. It is worth noting that in the quantitative experiments, the first frame of FancyVideo was generated by SDXL to obtain more aesthetic results and minimize the appearance of watermarks (although subsequent frames may still show watermarks). As shown in Figure 4 below, our method demonstrates excellent performance, surpassing previous methods in terms of temporal consistency and action richness. In comparison, AnimateDiff, DynamiCrafter and Gen2 generated videos with less motion. Pika has difficulty generating object-consistent and high-quality video frames. It is worth noting that our method can accurately understand the action instructions in text prompts (for example, the cases of "a teddy bear walking...beautiful sunset" and "a teddy bear running...city").
Quantitative Evaluation: In order to conduct a comprehensive comparison with state-of-the-art methods, three popular benchmarks (e.g., EvalCrafter, UCF-101, and MSR-VTT) and manual evaluation are adopted to evaluate the quality of video generation. Among them, EvalCrafter is currently a relatively comprehensive video generation benchmark. UCF-101 and MSR-VTT are commonly used benchmarks in previous methods. At the same time, manual evaluation can make up for the inaccuracy in existing text-conditional video generation evaluation systems.
EvalCrafter Benchmark: EvalCrafter quantitatively evaluates the quality of text-to-video generation from four aspects including video quality, text-to-video alignment, motion quality, and temporal consistency. Each dimension contains indicators for multiple subcategories, as shown in Table 1 below. As mentioned in the community discussion, the authors acknowledge that the original composite indicator calculation was inappropriate. For a more intuitive comparison, a comprehensive indicator of each aspect is introduced by considering the numerical scale and positive and negative attributes of each sub-indicator. Detailed information on the composite metrics can be found in the supplementary material.
Specifically, the performance of previous SOTA methods for video generation (e.g., Pika, Gen2, Show-1, ModelScope, DynamiCrafter, and AnimateDiff) was compared and presented in Table 1 above. Our method outperforms existing methods in terms of video quality and text-to-video alignment. Although Show-1 performed best in terms of action quality (81.56), its video quality was inferior (only 85.08). This shows that it is unable to generate high-quality videos with reasonable motion. However, our method ranks second in terms of action quality (72.99) and performs best in terms of video quality (177.72), achieving a quality-action trade-off. The above results demonstrate the superiority of FancyVideo and its ability to generate time-consistent and motion-accurate videos.
UCF-101 & MSR-VTT: Based on previous work, researchers evaluated the zero-shot generation performance on UCF-101 and MSR-VTT, as shown in Table 2 below. Frechet Video Distance (FVD), Inception Score (IS), Frechet Inception Distance (FID) and CLIP Similarity (CLIPSIM) are used as evaluation metrics and compared with some current SOTA methods. FancyVideo achieved competitive results, especially on IS and CLIPSIM, which were 43.66 and 0.3076 respectively. Furthermore, previous studies have pointed out that these metrics do not accurately reflect human perception and are affected by gaps between training and test data distributions as well as low-level details of images.
Human Evaluation: Inspired by EvalCrafter, a multi-candidate ranking protocol is introduced covering four aspects: video quality, text-video alignment, action quality and temporal consistency. In this protocol, participants rank the results of multiple candidate models for each aspect. Each candidate model receives a score based on its ranking. For example, if there are N candidate models ranked by video quality, the first model gets N-1 points, the second model gets N-2 points, and so on, with the last model getting 0 points. Following this protocol, 108 samples were selected from the EvalCrafter validation set and 100 individual judges were collected. As shown in Figure 5 below, our method significantly outperforms text-to-video conversion methods including AnimateDiff, Pika, and Gen2 in all four aspects. FancyVideo demonstrates superior motion quality while maintaining superior text-to-video consistency. In addition, a similar comparison was performed on four image-to-video methods, including DynamiCrafter, Pika, and Gen2, as shown in Figure 5 below.
Ablation Study: Researchers conducted experiments and demonstrated visual comparisons on EvalCrafter to explore the effects of key designs in CTGM. Specifically, the ablation component includes three core modules (TII, TAR, and TFB). As shown in Table 3 below, TAR can effectively improve the performance of the two indicators, indicating that the temporal refinement attention map operation is beneficial to video generation. Continuous insertion of TFB and TII further improves the performance of the generator due to refined latent features and frame-level personalized text conditions. Also, the qualitative analysis is included in the appendix.
Conclusion
This paper proposes a new video generation method named FancyVideo, which optimizes common text control mechanisms (such as spatial cross-attention) through cross-frame text guidance. It improves cross-attention with a carefully designed cross-frame text guidance module (CTGM), enabling time-specific text conditional guidance for video generation. Comprehensive qualitative and quantitative analysis shows that this method is able to generate more dynamic and consistent videos. As the number of frames increases, this characteristic becomes more obvious. Our method achieves state-of-the-art results on the EvalCrafter benchmark and human evaluation.
References
[1] FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance