“Facial hair tracking for high fidelity performance capture” by Winberg, Zoss, Chandran, Gotardo and Bradley

  • ©Sebastian Winberg, Gaspard Zoss, Prashanth Chandran, Paulo Gotardo, and Derek Bradley

Conference:


Type:


Title:

    Facial hair tracking for high fidelity performance capture

Presenter(s)/Author(s):



Abstract:


    Facial hair is a largely overlooked topic in facial performance capture. Most production pipelines in the entertainment industry do not have a way to automatically capture facial hair or track the skin underneath it. Thus, actors are asked to shave clean before face capture, which is very often undesirable. Capturing the geometry of individual facial hairs is very challenging, and their presence makes it harder to capture the deforming shape of the underlying skin surface. Some attempts have already been made at automating this task, but only for static faces with relatively sparse 3D hair reconstructions. In particular, current methods lack the temporal correspondence needed when capturing a sequence of video frames depicting facial performance. The problem of robustly tracking the skin underneath also remains unaddressed. In this paper, we propose the first multiview reconstruction pipeline that tracks both the dense 3D facial hair, as well as the underlying 3D skin for entire performances. Our method operates with standard setups for face photogrammetry, without requiring dense camera arrays. For a given capture subject, our algorithm first reconstructs a dense, high-quality neutral 3D facial hairstyle by registering sparser hair reconstructions over multiple frames that depict a neutral face under quasi-rigid motion. This custom-built, reference facial hairstyle is then tracked throughout a variety of changing facial expressions in a captured performance, and the result is used to constrain the tracking of the 3D skin surface underneath. We demonstrate the proposed capture pipeline on a variety of different facial hairstyles and lengths, ranging from sparse and short to dense full-beards.

References:


    1. Sameer Agarwal, Keir Mierle, and Others. 2016. Ceres Solver. http://ceres-solver.org.Google Scholar
    2. Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, Article 40 (2010).Google ScholarDigital Library
    3. Thabo Beeler, Bernd Bickel, Gioacchino Noris, Paul Beardsley, Steve Marschner, Robert W. Sumner, and Markus Gross. 2012. Coupled 3D Reconstruction of Sparse Facial Hair and Skin. ACM Trans. Graphics (Proc. SIGGRAPH) 31, 4, Article 117 (2012).Google ScholarDigital Library
    4. Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, Article 75 (2011).Google ScholarDigital Library
    5. Pascal Bérard, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. Lightweight Eye Capture Using a Parametric Model. ACM Trans. Graphics (Proc. SIGGRAPH) 35, 4, Article 117 (2016).Google ScholarDigital Library
    6. Amit Bermano, Thabo Beeler, Yeara Kozlov, Derek Bradley, Bernd Bickel, and Markus Gross. 2015. Detailed Spatio-Temporal Reconstruction of Eyelids. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4, Article 44 (2015).Google ScholarDigital Library
    7. Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High Resolution Passive Facial Performance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, Article 41 (2010).Google ScholarDigital Library
    8. Menglei Chai, Lvdi Wang, Yanlin Weng, Xiaogang Jin, and Kun Zhou. 2013. Dynamic Hair Manipulation in Images and Videos. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, Article 75 (2013).Google ScholarDigital Library
    9. Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. 2020. Semantic Deep Face Models. In Int. Conf. on 3D Vision. 345–354.Google Scholar
    10. Graham Fyffe. 2012. High Fidelity Facial Hair Capture. In ACM SIGGRAPH 2012 Talks. Article 23.Google Scholar
    11. Graham Fyffe, Koki Nagano, Loc Huynh, Shunsuke Saito, Jay Busch, Andrew Jones, Hao Li, and Paul Debevec. 2017. Multi-View Stereo on Consistent Face Topology. Comp. Graphics Forum (Proc. Eurographics) 36, 2 (2017), 295–309.Google ScholarDigital Library
    12. Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Niessner. 2021. Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. In IEEE Computer Vision and Pattern Recognition (CVPR). 8649–8658.Google Scholar
    13. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6 (2011), 1–10.Google ScholarDigital Library
    14. Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical Dynamic Facial Appearance Modeling and Acquisition. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 37, 6, Article 232 (2018).Google Scholar
    15. Stéphane Grabli, François X. Sillion, Stephen R. Marschner, and Jerome E. Lengyel. 2002. Image-Based Hair Capture by Inverse Lighting. In Proc. of Graphics Interface (GI). 51–58.Google Scholar
    16. Tomas Lay Herrera, Arno Zinke, and Andreas Weber. 2012. Lighting Hair from the inside: A Thermal Approach to Hair Reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6, Article 146 (2012).Google Scholar
    17. Tomas Lay Herrera, Arno Zinke, Andreas Weber, and Thomas Vetter. 2010. Toward Image-Based Facial Hair Modeling. In Proc. of the 26th Spring Conf. on Computer Graphics. 93–100.Google ScholarDigital Library
    18. Osamu Hirose. 2021. A Bayesian Formulation of Coherent Point Drift. IEEE TPAMI 43, 7 (2021), 2269–2286.Google ScholarCross Ref
    19. Liwen Hu, Derek Bradley, Hao Li, and Thabo Beeler. 2017. Simulation-Ready Hair Capture. Comp. Graphics Forum (Proc. Eurographics) 36, 2 (2017), 281–294.Google ScholarDigital Library
    20. Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2014. Robust Hair Capture Using Simulated Examples. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, Article 126 (2014).Google ScholarDigital Library
    21. Takahito Ishikawa, Yosuke Kazama, Eiji Sugisaki, and Shigeo Morishima. 2007. Hair Motion Reconstruction Using Motion Capture System. In ACM SIGGRAPH 2007 Posters. 78–es.Google Scholar
    22. Wenzel Jakob, Jonathan T. Moon, and Steve Marschner. 2009. Capturing Hair Assemblies Fiber by Fiber. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 28, 5 (2009), 1–9.Google ScholarDigital Library
    23. Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shunsuke Saito, Ronald Yu, Hao Li, and Jaakko Lehtinen. 2017. Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks. In Proc. of Eurographics Symposium on Computer Animation. Article 10.Google ScholarDigital Library
    24. Chloe LeGendre, Loc Hyunh, Shanhe Wang, and Paul Debevec. 2017. Modeling Vellus Facial Hair from Asperity Scattering Silhouettes. In ACM SIGGRAPH 2017 Talks.Google ScholarDigital Library
    25. Tianye Li, Shichen Liu, Timo Bolkart, Jiayi Liu, Hao Li, and Yajie Zhao. 2021. Topologically Consistent Multi-View Face Inference Using Volumetric Sampling. In IEEE Int. Conf. on Computer Vision (ICCV). 3824–3834.Google Scholar
    26. Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda G. Shapiro, and Ira Kemelmacher-Shlizerman. 2018. Video to Fully Automatic 3D Hair Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 37, 6, Article 206 (2018).Google Scholar
    27. Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM Trans. Graphics (Proc. SIGGRAPH) 38, 4, Article 65 (2019).Google ScholarDigital Library
    28. Linjie Luo, Hao Li, and Szymon Rusinkiewicz. 2013. Structure-Aware Hair Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, Article 76 (2013).Google ScholarDigital Library
    29. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proc. Eurographics Conf. on Rendering Techniques. 183–194.Google Scholar
    30. Masayuki Nakajima, Kong Wai Ming, and Hiroki Takashi. 1997. Generation of 3d hair model from multiple pictures. IEEE Comp. Graphics and Applications (1997).Google Scholar
    31. Giljoo Nam, Chenglei Wu, Min H. Kim, and Yaser Sheikh. 2019. Strand-Accurate Multi-View Hair Capture. In IEEE Computer Vision and Pattern Recognition (CVPR). 155–164.Google Scholar
    32. Sylvain Paris, Hector M. Briceño, and François X. Sillion. 2004. Capture of Hair Geometry from Multiple Images. ACM Trans. Graphics (Proc. SIGGRAPH) 23, 3 (2004), 712–719.Google ScholarDigital Library
    33. Sylvain Paris, Will Chang, Oleg I. Kozhushnyan, Wojciech Jarosz, Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2008. Hair Photobooth: Geometric and Photometric Acquisition of Real Hairstyles. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3 (2008), 1–9.Google ScholarDigital Library
    34. Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. 2021. HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 40, 6, Article 238 (2021).Google Scholar
    35. Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 39, 4, Article 81 (2020).Google ScholarDigital Library
    36. Gemma Rotger, Francesc Moreno-Noguer, Felipe Lumbreras, and Antonio Agudo. 2019. Single View Facial Hair 3D Reconstruction. In Pattern Rec. and Image Anal. 423–436.Google Scholar
    37. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and Hans-Peter Seidel. 2004. Laplacian Surface Editing. In Proc. of the Symposium on Geometry Processing. 175–184.Google ScholarDigital Library
    38. Tiancheng Sun, Giljoo Nam, Carlos Aliaga, Christophe Hery, and Ravi Ramamoorthi. 2021. Human Hair Inverse Rendering using Multi-View Photometric data. In Eurographics Symposium on Rendering.Google Scholar
    39. Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. 2020. PIE: Portrait Image Embedding for Semantic Control. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 39, 6, Article 223 (2020).Google Scholar
    40. Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In Proc. ICCV Workshops.Google Scholar
    41. Chenglei Wu, Derek Bradley, Pablo Garrido, Michael Zollhöfer, Christian Theobalt, Markus Gross, and Thabo Beeler. 2016a. Model-Based Teeth Reconstruction. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 35, 6, Article 220 (2016).Google Scholar
    42. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016b. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 35, 4, Article 115 (2016).Google ScholarDigital Library
    43. Zexiang Xu, Hsiang-Tao Wu, Lvdi Wang, Changxi Zheng, Xin Tong, and Yue Qi. 2014. Dynamic Hair Capture Using Spacetime Optimization. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, Article 224 (2014).Google Scholar
    44. Tatsuhisa Yamaguchi, Bennett Wilburn, and Eyal Ofek. 2009. Video-Based Modeling of Dynamic Hair. In Adv. in Image and Video Technology. 585–596.Google Scholar
    45. Lingchen Yang, Zefeng Shi, Youyi Zheng, and Kun Zhou. 2019. Dynamic Hair Modeling from Monocular Videos Using Deep Neural Networks. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 38, 6, Article 235 (2019).Google Scholar
    46. Meng Zhang, Menglei Chai, Hongzhi Wu, Hao Yang, and Kun Zhou. 2017. A Data-Driven Approach to Four-View Image-Based Hair Modeling. ACM Trans. Graphics (Proc. SIGGRAPH) 36, 4, Article 156 (2017).Google ScholarDigital Library
    47. Gaspard Zoss, Thabo Beeler, Markus Gross, and Derek Bradley. 2019. Accurate Markerless Jaw Tracking for Facial Performance Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 38, 4, Article 50 (2019).Google ScholarDigital Library


ACM Digital Library Publication: