“Stitch it in Time: GAN-Based Facial Editing of Real Videos” by Tzaban, Mokady, Gal, Bermano and Cohen-Or
Conference:
Type(s):
Title:
- Stitch it in Time: GAN-Based Facial Editing of Real Videos
Session/Category Title:
- Technical Papers Fast-Forward
Presenter(s)/Author(s):
Abstract:
The ability of Generative Adversarial Networks to encode rich semantics within their latent space has been widely adopted for facial image editing. However, replicating their success with videos has proven challenging. Applying StyleGAN editing over real videos introduces two main challenges: (i) StyleGAN operates over aligned crops. When editing videos, these crops need to be pasted back into the frame, resulting in a spatial inconsistency. (ii) Videos introduce a fundamental barrier to overcome – temporal coherency. To address the first challenge, we propose a novel stitching-tuning procedure. The generator is carefully tuned to overcome the spatial artifacts at crop borders, resulting in a smooth transition even when difficult backgrounds are involved. Turning to temporal coherence, we propose that this challenge is largely artificial. The source video is already temporally coherent, and deviations arise in part due to the careless treatment of individual components in the editing pipeline. We leverage the natural alignment of StyleGAN and the tendency of neural networks to learn low-frequency functions, and demonstrate that they provide a strongly consistent prior. These components are combined in an end-to-end framework for semantic editing of facial videos. We compare our pipeline to the current state-of-the-art and demonstrate significant improvements. Our method produces various meaningful manipulations and maintain greater spatial and temporal consistency, even in challenging talking head videos which current methods struggle with. Our code will be made publicly available.
