“Deep Learned Face Swapping in Feature Film Production” by Swartz and Walker
Conference:
Type:
Entry Number: 35
Title:
- Deep Learned Face Swapping in Feature Film Production
Presenter(s)/Author(s):
Abstract:
In visual effects for film, replacement of stunt performers’ facial likeness for their doubled actor counterparts using traditional computer graphics methods is a multi-stage, labor intensive task. Recently, deep learning techniques have made a compelling argument to train neural networks to learn how to take an image of a person’s face and convincingly infer a rendered image of a second person’s face with a previously unseen perspective, pose and lighting environment.
A novel method is discussed for bringing deep neural network face swapping to feature film production which utilizes facial recognition for the discovery of training data. Our method further innovates in the area of utilizing traditional CG assets for informing some of the shortcomings of ML techniques. Connected with a technique for feature engineering during training dataset assembly, our Face Fabrication System enables Wētā FX to deliver final picture quality for use in production.
References:
Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z. Li, and Xudong Zou. 2018. Selective Refinement Network for High Performance Face Detection. arXiv:1809.02693 [cs] (Sept. 2018). http://arxiv.org/abs/1809.02693 arXiv:1809.02693.Google Scholar
Github Deepfakes. 2021. FaceSwap. https://github.com/deepfakes/faceswap Accessed 2021-02-21.Google Scholar
I. Korshunova, W. Shi, J. Dambre, and L. Theis. 2017. Fast Face-Swap Using Convolutional Neural Networks. In 2017 IEEE International Conference on Computer Vision (ICCV). 3697–3705. https://doi.org/10.1109/ICCV.2017.397 ISSN: 2380-7504.Google Scholar
Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, 2020. DeepFaceLab: A simple, flexible and extensible face swapping framework. arXiv:2005.05535 [cs, eess] (May 2020). http://arxiv.org/abs/2005.05535Google Scholar
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A Unified Embedding for Face Recognition and Clustering. 2015 IEEE Conference on Computer Vision and Pattern Recognition, 815–823. https://doi.org/10.1109/CVPR.2015.7298682Google Scholar
Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. 2020. FAN-Face: a Simple Orthogonal Improvement to Deep Face Recognition. Proceedings of the AAAI Conference on Artificial Intelligence 34, 07, 12621–12628. https://doi.org/10.1609/aaai.v34i07.6953Google ScholarCross Ref
