“Practical dynamic facial appearance modeling and acquisition”
Conference:
Type(s):
Title:
- Practical dynamic facial appearance modeling and acquisition
Session/Category Title: Faces, faces, faces
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a method to acquire dynamic properties of facial skin appearance, including dynamic diffuse albedo encoding blood flow, dynamic specular intensity, and per-frame high resolution normal maps for a facial performance sequence. The method reconstructs these maps from a purely passive multi-camera setup, without the need for polarization or requiring temporally multiplexed illumination. Hence, it is very well suited for integration with existing passive systems for facial performance capture. To solve this seemingly underconstrained problem, we demonstrate that albedo dynamics during a facial performance can be modeled as a combination of: (1) a static, high-resolution base albedo map, modeling full skin pigmentation; and (2) a dynamic, one-dimensional component in the CIE L*a*b* color space, which explains changes in hemoglobin concentration due to blood flow. We leverage this albedo subspace and additional constraints on appearance and surface geometry to also estimate specular reflection parameters and resolve high-resolution normal maps with unprecedented detail in a passive capture system. These constraints are built into an inverse rendering framework that minimizes the difference of the rendered face to the captured images, incorporating constraints from multiple views for every texel on the face. The presented method is the first system capable of capturing high-quality dynamic appearance maps at full resolution and video framerates, providing a major step forward in the area of facial appearance acquisition.
References:
1. Sameer Agarwal, Keir Mierle, and Others. 2016. Ceres Solver. http://ceres-solver.org.Google Scholar
2. Oleg Alexander, Mike Rogers, William Lambeth, Jen-Yuan Chiang, Wan-Chun Ma, Chuan-Chang Wang, and Paul Debevec. 2010. The digital emily project: Achieving a photorealistic digital actor. Computer Graphics and Applications, IEEE 30, 4 (2010), 20–31. Google ScholarDigital Library
3. R Rox Anderson and John A Parrish. 1981. The optics of human skin. Journal of investigative dermatology 77, 1 (1981), 13–19.Google ScholarCross Ref
4. Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-Quality Single-Shot Capture of Facial Geometry. ACM Transactions on Graphics (TOG) 29, 3 (2010), 40:1–40:9. Google ScholarDigital Library
5. Thabo Beeler and Derek Bradley. 2014. Rigid stabilization of facial expressions. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–9. Google ScholarDigital Library
6. Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-quality passive facial performance capture using anchor frames. ACM Transactions on Graphics (ACM) 30, Article 75 (August 2011), 10 pages. Issue 4. Google ScholarDigital Library
7. Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High resolution passive facial performance capture. ACM Transactions on Graphics (TOG) 29, 4 (2010), 41. Google ScholarDigital Library
8. Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. 25–36.Google Scholar
9. Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-time High-fidelity Facial Performance Capture. ACM Transactions on Graphics (TOG) 34, 4, Article 46 (July 2015), 9 pages. Google ScholarDigital Library
10. A Chardon, I Cretois, and C Hourseau. 1991. Skin colour typology and suntanning pathways. International journal of cosmetic science 13, 4 (1991), 191–208.Google ScholarCross Ref
11. Robert Cook and Kenneth E. Torrance. 1981. A reflectance model for computer graphics. Computer Graphics (SIGGRAPH ’81 Proceedings) 15, 3 (1981), 301–316. Google ScholarDigital Library
12. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 145–156. Google ScholarDigital Library
13. Eugene d’Eon, David Luebke, and Eric Enderton. 2007. Efficient Rendering of Human Skin. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). 147–157. Google ScholarDigital Library
14. Craig Donner and Henrik Wann Jensen. 2005. Light Diffusion in Multi-Layered Translucent Materials. ACM Transactions on Graphics (TOG) 24, 3 (2005), 1032–1039. Google ScholarDigital Library
15. Craig Donner and Henrik Wann Jensen. 2006. A Spectral BSSRDF for Shading Human Skin. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR ’06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 409–417. Google ScholarDigital Library
16. Craig Donner, Tim Weyrich, Eugene d’Eon, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2008. A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering Human Skin. ACM Transactions on Graphics (TOG) 27, 5 (Dec. 2008), 140:1–140:12. Google ScholarDigital Library
17. Martin Fuchs, Volker Blanz, Hendrik Lensch, and Hans-Peter Seidel. 2005. Reflectance from Images: A Model-Based Approach for Human Faces. IEEE Transactions on Visualization and Computer Graphics 11, 3 (May 2005), 296–305. Google ScholarDigital Library
18. Graham Fyffe and Paul Debevec. 2015. Single-Shot Reflectance Measurement from Polarized Color Gradient Illumination. In International Conference on Computational Photography (ICCP).Google ScholarCross Ref
19. Graham Fyffe, Paull Graham, Borom Tunwattanapong, Abhijeet Ghosh, and Paul Debevec. 2016. Near-Instant Capture of High-Resolution Facial Geometry and Reflectance. Computer Graphics Forum (CGF) 35, 2 (2016), 353–363.Google ScholarCross Ref
20. Graham Fyffe, Tim Hawkins, Chris Watts, Wan-Chun Ma, and Paul Debevec. 2011. Comprehensive Facial Performance Capture. Computer Graphics Forum (CGF) 30, 2 (2011).Google Scholar
21. Graham Fyffe, Andrew Jones, Oleg Alexander, Ryosuke Ichikari, and Paul Debevec. 2014. Driving High-Resolution Facial Scans with Video Performance Capture. ACM Transactions on Graphics (TOG) 34, 1, Article 8 (Dec. 2014), 14 pages. Google ScholarDigital Library
22. Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing Detailed Dynamic Face Geometry from Monocular Video. ACM Transactions on Graphics (TOG) 32, 6 (November 2013), 158:1–158:10. Google ScholarDigital Library
23. Sergio Garrido-Jurado, Rafael Mu noz Salinas, F.J. Madrid-Cuevas, and Manuel Jesus Marín-Jiménez. 2014. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition 47, 6 (2014), 2280 — 2292. Google ScholarDigital Library
24. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011), 129. Google ScholarDigital Library
25. Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical Modeling and Acquisition of Layered Facial Reflectance. ACM Trans. Graph. 27, 5 (Dec. 2008), 139:1–139:10. Google ScholarDigital Library
26. Paulo Gotardo, Tomas Simon, Yaser Sheikh, and Iain Matthews. 2015. Photogeometric Scene Flow for High-Detail Dynamic 3D Reconstruction. In IEEE International Journal of Computer Vision. 846–854. Google ScholarDigital Library
27. Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec, and Abhijeet Ghosh. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum (CGF) 32, 2 (2013), 335–344.Google ScholarCross Ref
28. Tim Hawkins, Andreas Wenger, Chris Tchou, Andrew Gardner, Fredrik Göransson, and Paul Debevec. 2004. Animatable Facial Reflectance Fields. In Proceedings of the Fifteenth Eurographics Conference on Rendering Techniques (EGSR’04). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 309–319. Google ScholarDigital Library
29. Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Paul Debevec, and Hao Li. 2018. Mesoscopic Facial Geometry Inference Using Deep Neural Networks. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
30. Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015. Dynamic 3D Avatar Creation from Hand-held Video Input. ACM Transactions on Graphics (TOG) 34, 4 (2015), 45:1–45:14. Google ScholarDigital Library
31. Henrik Wann Jensen, Steve Marschner, Marc Levoy, and Pat Hanrahan. 2001. A practical model for subsurface light transport. In In Proceedings of ACM SIGGRAPH. 511–518. Google ScholarDigital Library
32. Jorge Jimenez, Timothy Scully, Nuno Barbosa, Craig Donner, Xenxo Alvarez, Teresa Vieira, Paul Matts, Verónica Orvalho, Diego Gutierrez, and Tim Weyrich. 2010. A Practical Appearance Model for Dynamic Facial Color. ACM Transactions on Graphics (TOG) 29, 6, Article 141 (Dec. 2010), 10 pages. Google ScholarDigital Library
33. Jorge Jimenez, Veronica Sundstedt, and Diego Gutierrez. 2009. Screen-space perceptual rendering of human skin. ACM Transactions on Applied Perception 6, 4 (2009), 23:1–23:15. Google ScholarDigital Library
34. Oliver Klehm, Fabrice Rousselle, Marios Papas, Derek Bradley, Christophe Hery, Bernd Bickel, Wojciech Jarosz, and Thabo Beeler. 2015. Recent Advances in Facial Appearance Capture. Computer Graphics Forum (CGF) 34, 2 (May 2015), 709–733. Google ScholarDigital Library
35. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR’07). Eurographics Association, 183–194. Google ScholarDigital Library
36. Wan-Chun Ma, Andrew Jones, Jen-Yuan Chiang, Tim Hawkins, Sune Frederiksen, Pieter Peers, Marko Vukovic, Ming Ouhyoung, and Paul Debevec. 2008. Facial Performance Synthesis Using Deformation-driven Polynomial Displacement Maps. ACM Transactions on Graphics (TOG) 27, 5, Article 121 (Dec. 2008), 10 pages. Google ScholarDigital Library
37. Stephen R. Marschner, Stephen H. Westin, Eric P. F. Lafortune, Kenneth E. Torrance, and Donald P. Greenberg. 1999. Image-based BRDF Measurement Including Human Skin. In Proceedings of the 10th Eurographics Conference on Rendering (EGWR’99). 131–144. Google ScholarDigital Library
38. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbic, Hao Li, Abhijeet Ghosh, and Paul E Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (TOG) 34, 4 (2015), 109. Google ScholarDigital Library
39. Ruth Onn and Alfred Bruckstein. 1990. Integrability disambiguates surface recovery in two-image photometric stereo. International Journal of Computer Vision 5, 1 (1990), 105–113. Google ScholarDigital Library
40. Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, and Hao Li. 2017. Photorealistic Facial Texture Inference Using Deep Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
41. Christoph Schlick. 1994. An inexpensive BRDF model for physically-based rendering. Computer Graphics Forum (CGF) 13, 3 (1994), 233–246.Google ScholarCross Ref
42. Scott Shafer. 1985. Using Color to Separate Reflection Components. Color Research & Application 10, 4 (1985), 210–218.Google ScholarDigital Library
43. Gaurav Sharma and Raja Bala. 2002. Digital color imaging handbook. CRC press. Google ScholarDigital Library
44. Fuhao Shi, Hsiang-Tao Wu, Xin Tong, and Jinxiang Chai. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Transactions on Graphics (TOG) 33, 6 (2014), 222. Google ScholarDigital Library
45. Levi Valgaerts, Chenglei Wu, Andrés Bruhn, Hans-Peter Seidel, and Christian Theobalt. 2012. Lightweight Binocular Facial Performance Capture under Uncontrolled Lighting. ACM Transactions on Graphics (TOG) 31, 6 (November 2012), 187:1–187:11. Google ScholarDigital Library
46. Javier von der Pahlen, Jorge Jimenez, Etienne Danvoye, Paul Debevec, Graham Fyffe, and Oleg Alexander. 2014. Digital Ira and Beyond: Creating Real-time Photoreal Digital Actors. In ACM SIGGRAPH 2014 Courses (SIGGRAPH ’14). ACM, New York, NY, USA, Article 1, 384 pages. Google ScholarDigital Library
47. Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. 24, 3 (2005), 756–764. Google ScholarDigital Library
48. Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found. Trends. Comput. Graph. Vis. 4, 2 (Feb. 2009), 75–191. Google ScholarDigital Library
49. Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of Human Faces Using a Measurement-based Skin Reflectance Model. ACM Transactions on Graphics (TOG) 25, 3 (July 2006), 1013–1024. Google ScholarDigital Library
50. Cyrus A Wilson, Abhijeet Ghosh, Pieter Peers, Jen-Yuan Chiang, Jay Busch, and Paul Debevec. 2010. Temporal upsampling of performance geometry using photometric alignment. ACM Transactions on Graphics (TOG) 29, 2 (2010), 17. Google ScholarDigital Library
51. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An anatomically-constrained local deformation model for monocular face capture. ACM Transactions on Graphics (TOG) 35, 4 (2016), 115. Google ScholarDigital Library
52. Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-Fidelity Facial Reflectance and Geometry Inference From an Unconstrained Image. ACM Transactions on Graphics (TOG) 37, 4 (2018). Google ScholarDigital Library
