“Production-Ready Face Re-Aging for Visual Effects” by Zoss, Chandran, Sifakis, Gross, Gotardo, et al. …
Conference:
Type(s):
Title:
- Production-Ready Face Re-Aging for Visual Effects
Session/Category Title: Image Editing and Manipulation
Presenter(s)/Author(s):
Abstract:
Photorealistic digital re-aging of faces in video is becoming increasingly common in entertainment and advertising. But the predominant 2D painting workflow often requires frame-by-frame manual work that can take days to accomplish, even by skilled artists. Although research on facial image re-aging has attempted to automate and solve this problem, current techniques are of little practical use as they typically suffer from facial identity loss, poor resolution, and unstable results across subsequent video frames. In this paper, we present the first practical, fully-automatic and production-ready method for re-aging faces in video images. Our first key insight is in addressing the problem of collecting longitudinal training data for learning to re-age faces over extended periods of time, a task that is nearly impossible to accomplish for a large number of real people. We show how such a longitudinal dataset can be constructed by leveraging the current state-of-the-art in facial re-aging that, although failing on real images, does provide photoreal re-aging results on synthetic faces. Our second key insight is then to leverage such synthetic data and formulate facial re-aging as a practical image-to-image translation task that can be performed by training a well-understood U-Net architecture, without the need for more complex network designs. We demonstrate how the simple U-Net, surprisingly, allows us to advance the state of the art for re-aging real faces on video, with unprecedented temporal stability and preservation of facial identity across variable expressions, viewpoints, and lighting conditions. Finally, our new face re-aging network (FRAN) incorporates simple and intuitive mechanisms that provides artists with localized control and creative freedom to direct and fine-tune the re-aging effect, a feature that is largely important in real production pipelines and often overlooked in related research work.
References:
1. Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. 2021. StyleFlow: Attribute-Conditioned Exploration of StyleGAN-Generated Images Using Conditional Continuous Normalizing Flows. ACM TOG 40, 3, Article 21 (2021).
2. Adobe. 2022. Photoshop. https://www.adobe.com/products/photoshop.html. [Online; accessed 20-May-2022].
3. Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. 2021. Only a Matter of Style: Age Transformation Using a Style-Based Regression Model. ACM TOG 40, 4, Article 45 (2021).
4. Raphael Angulu, Jules R. Tapamo, and Aderemi O. Adewumi. 2018. Age estimation via face images: a survey. EURASIP J. Image and Video Proc. 1, 42 (2018).
5. Grigory Antipov, Moez Baccouche, and Jean-Luc Dugelay. 2017. Face aging with conditional generative adversarial networks. In ICIP. 2089–2093.
6. Netflix Film Club. 2020. How The Irishman’s Groundbreaking VFX Took Anti-Aging To the Next Level | Netflix. Retrieved May 13, 2022 from https://www.youtube.com/watch?v=OF-lElIlZM0
7. Julien Despois, Frederic Flament, and Matthieu Perrot. 2020. AgingMapGAN (AMGAN): High-Resolution Controllable Face Aging with Spatially-Aware Conditional GANs. In ECCV.
8. Malaria Must Die. 2020. A World Without Malaria. Retrieved May 13, 2022 from www.youtube.com/watch?v=0l4eTfpIsKw
9. C N Duong, K Luu, K G Quach, N Nguyen, E Patterson, T D Bui, and N Le. 2019. Automatic Face Aging in Videos via Deep Reinforcement Learning. In IEEE CVPR. 10005–10014.
10. H Fang, W Deng, Y Zhong, and J Hu. 2020. Triple-GAN: Progressive Face Aging with Triple Translation Loss. In CVPR Workshops. 3500–3509.
11. Joe Fordham. 2019. The Power Within Her. Cinefex 164 (2019), 37–64.
12. Rinon Gal, Dana Cohen, Amit Bermano, and Daniel Cohen-Or. 2021. SWAGAN: A Style-based Wavelet-driven Generative Model. ACM TOG 40, 4, Article 134 (2021).
13. Markos Georgopoulos, Yannis Panagakis, and Maja Pantic. 2018. Modeling of facial aging and kinship: A survey. Image and Vis. Comp. 80 (Dec. 2018), 58–79.
14. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. NeurIPS 27 (2014), 2672–2680.
15. Sen He, Wentong Liao, Michael Ying Yang, Yi-Zhe Song, Bodo Rosenhahn, and Tao Xiang. 2021. Disentangled Lifespan Face Synthesis. In IEEE ICCV.
16. Gee-Sern Hsu, Rui-Cang Xie, and Zhi-Ting Chen. 2021. Wasserstein Divergence GAN With Cross-Age Identity Expert and Attribute Retainer for Facial Age Transformation. IEEE Access 9 (2021), 39695–39706.
17. Zhizhong Huang, Junping Zhang, and Hongming Shan. 2021. When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework. In IEEE CVPR.
18. Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. 2020. GANSpace: Discovering Interpretable GAN Controls. In NeurIPS.
19. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In IEEE CVPR.
20. Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2020a. Training Generative Adversarial Networks with Limited Data. In NeurIPS.
21. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In IEEE CVPR. 4401–4410.
22. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020b. Analyzing and improving the image quality of StyleGAN. In IEEE CVPR. 8110–8119.
23. Ira Kemelmacher-Shlizerman, Supasorn Suwajanakorn, and Steven M. Seitz. 2014. Illumination-Aware Age Progression. In IEEE CVPR. 3334–3341.
24. Zeqi Li, Ruowei Jiang, and Parham Aarabi. 2021. Continuous Face Aging via Self-estimated Residual Age Embedding. In IEEE CVPR.
25. FaceApp Technology Limited. 2022. FaceApp. https://www.faceapp.com. [Online; accessed 20-May-2022].
26. Yunfan Liu, Qi Li, Zhenan Sun, and Tieniu Tan. 2021. A3GAN: An Attribute-Aware Attentive Generative Adversarial Network for Face Aging. IEEE TIFS 16 (2021), 2776–2790.
27. Farkhod Makhmudkhujaev, Sungeun Hong, and In Kyu Park. 2021. Re-Aging GAN: Toward Personalized Face Age Transformation. In IEEE ICCV. 3908–3917.
28. Jacek Naruniec, Leonhard Helminger, Christopher Schroers, and Romann Weber. 2020. High-Resolution Neural Face Swapping for Visual Effects. Comput. Graph. Forum 39, 4 (2020), 173 — 184.
29. Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman. 2020. Lifespan Age Transformation Synthesis. In ECCV, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). 739–755.
30. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis with Spatially-Adaptive Normalization. In IEEE CVPR.
31. Narayanan Ramanathan, Rama Chellappa, and Soma Biswas. 2009. Computational Methods for Modeling Facial Aging: A Survey. J. Visual Lang. and Comp. 20, 3 (2009), 131–144.
32. Rasmus Rothe, Radu Timofte, and Luc Van Gool. 2018. Deep expectation of real and apparent age from a single image without facial landmarks. IJCV 126, 2–4 (2018), 144–157.
33. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A Unified Embedding for Face Recognition and Clustering. In IEEE CVPR.
34. Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. 2020. Interpreting the Latent Space of GANs for Semantic Face Editing. In IEEE CVPR.
35. Xiangbo Shu, Guo-Sen Xie, Zechao Li, and Jinhui Tang. 2016. Age progression: Current technologies and applications. Neurocomputing 208 (2016), 249–261.
36. Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR.
37. Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021. Designing an Encoder for StyleGAN Image Manipulation. 40, 4 (2021).
38. Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H. Bermano, and Daniel Cohen-Or. 2022. Stitch it in Time: GAN-Based Facial Editing of Real Videos. arXiv:2201.08361 [cs.CV]
39. Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. 2020. StyleGAN2 Distillation for Feed-Forward Image Manipulation. In ECCV. 170–186.
40. Hongyu Yang, Di Huang, Yunhong Wang, and Anil K. Jain. 2021. Learning Continuous Face Age Progression: A Pyramid of GANs. IEEE TPAMI 43, 2 (2021), 499–515.
41. Xu Yao, Gilles Puy, Alasdair Newson, Yann Gousseau, and Pierre Hellier. 2021. High Resolution Face Age Editing. In ICPR.
42. Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. 2021. BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation. IJCV 129 (2021), 3051–3068.
43. Richard Zhang. 2019. Making Convolutional Networks Shift-Invariant Again.
44. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In IEEE CVPR.
45. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In IEEE ICCV.
