FancyVideo: Open-Source Video Generation on Consumer GPUs
** One of the authors of the paper, Ao Ma, holds a master's degree from the Institute of Computing Technology, Chinese Academy of Sciences. He has worked in the MSRA Visual Computing Group and Alibaba Tongyi Laboratory for academic research and algorithm implementation. Currently, he is the person in charge of the video generation direction of the Qihoo 360-AIGC team. He has long been committed to the research and implementation of visual generation, as well as the construction of open source communities. **
Recently, the open source community has ushered in a powerful "video generation" work that can generate videos of any resolution, any aspect ratio, different styles, and different motion ranges on consumer graphics cards (such as GeForce RTX 3090). Its derived model can also complete the functions of video expansion and video retrieval... This is FancyVideo, a video generation model based on the UNet architecture jointly developed by the 360AI team and Sun Yat-sen University.
The author is based on the 61-frame model that has been open sourced, and the measured results are as follows.
First adapt to different resolutions and aspect ratios:
, duration 00:03
Secondly, different styles are supported:
, duration 00:04
Finally, different motility is generated:
, duration 00:04
Paper address: https://arxiv.org/abs/2408.08189
Project homepage: https://fancyvideo.github.io/
Code repository: https://github.com/360CVGroup/FancyVideo
Paper title: FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance
Cross-frame text guidance module
During the research on video generation, the author found that existing text-to-video (T2V) work usually uses spatial cross-attention (Spatial Cross Attention) to guide text equivalently into the generation process of different frames, and lacks the flexibility of text guidance for different frames (as shown in the left figure below). This results in limitations in the model's ability to understand the temporal logic conveyed by the cue words and generate videos with continuous motion.
FancyVideo It is from this perspective that the cross-frame text guidance module (Cross-frame Textual Guidance Module, CTGM, as shown in the right figure below) is specially designed to improve the existing text control mechanism.
Specifically, CTGM contains 3 submodules:
Temporal Information Injector (TII)--Inject frame-specific information from latent features into text conditions to obtain cross-frame text conditions;
Temporal Affinity Refiner (TAR)--refines the correlation matrix between cross-frame text conditions and potential features along the time dimension;
Temporal Feature Booster (TFB)-- Enhances the temporal consistency of latent features.
FancyVideo Training process
FancyVideo The overall training Pipeline is as follows. In terms of model structure, FancyVideo chose to construct the T2V model by inserting the timing layer and the mobility module based on CTGM on the basis of the 2D T2I model. When generating a video, the T2I operation is first performed to generate the first frame, and then the I2V is performed. This not only preserves the capabilities of the T2I model, improves the overall video quality, but also greatly reduces the training cost.
In addition, in order to achieve motion control capabilities, FancyVideo will inject video motion information extracted based on RAFT and time embedding into the network during the training phase.
Experimental results
The author evaluates the model effect through both quantitative and qualitative aspects. They first compared FancyVideo with other T2V models on the EvalCrafter Benchmark, and it can be seen that FancyVideo is in the leading position in terms of video generation quality, text consistency, motion, and timing consistency.
The paper also conducted zero-shot evaluations on UCF-101 and MSR-VTT Benchmark, and achieved SOTA results in both the IS index that measures the richness of the generated video and the CLIPSIM index that measures text consistency.
In addition, the paper also conducts manual evaluation based on the T2V and I2V capabilities of the FancyVideo model and previous methods. The results show that FancyVideo is in the lead in terms of video generation quality, text consistency, motion and timing consistency.
Finally, the paper uses ablation study to explore the impact of different sub-modules of CTGM on the video generation results to verify the rationality and effectiveness of each sub-module.
Application Scenario
Based on this training pipline and strategy, FancyVideo can complete T2V and I2V functions at the same time, and can also perform frame insertion operations based on generating key key frames:
Video expansion and video backtracking operations:
Less than a week after FancyVideo was launched in the open source community, some fast students have already spontaneously built the ComfyUI plug-in of FancyVideo, so that everyone can have fun on their own machines.
In addition, according to the author's understanding, in addition to releasing longer and better models to the open source community, the FancyVideo team also plans to launch a web version for everyone to use [free of charge]. In the AIGC era, everyone is an artist who can "poetry and paint".
Conclusion
Compared with the development of SORA video generation "products", the updates and iterations of video generation models in the open source community are slightly slower. The release of FancyVideo also gives ordinary users more choices. I believe that with the joint efforts of community partners, video generation, which currently seems to be a time-consuming and laborious task, can become a tool for more ordinary friends in their daily life and work.