“Detailed spatio-temporal reconstruction of eyelids” by Bermano, Beeler, Kozlov, Bradley, Bickel, et al. …
Conference:
Type:
Title:
- Detailed spatio-temporal reconstruction of eyelids
Session/Category Title: Face Reality
Presenter(s)/Author(s):
Moderator(s):
Abstract:
In recent years we have seen numerous improvements on 3D scanning and tracking of human faces, greatly advancing the creation of digital doubles for film and video games. However, despite the high-resolution quality of the reconstruction approaches available, current methods are unable to capture one of the most important regions of the face – the eye region. In this work we present the first method for detailed spatio-temporal reconstruction of eyelids. Tracking and reconstructing eyelids is extremely challenging, as this region exhibits very complex and unique skin deformation where skin is folded under while opening the eye. Furthermore, eyelids are often only partially visible and obstructed due to self-occlusion and eyelashes. Our approach is to combine a geometric deformation model with image data, leveraging multi-view stereo, optical flow, contour tracking and wrinkle detection from local skin appearance. Our deformation model serves as a prior that enables reconstruction of eyelids even under strong self-occlusions caused by rolling and folding skin as the eye opens and closes. The output is a person-specific, time-varying eyelid reconstruction with anatomically plausible deformations. Our high-resolution detailed eyelids couple naturally with current facial performance capture approaches. As a result, our method can largely increase the fidelity of facial capture and the creation of digital doubles.
References:
1. Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., et al. 2013. Digital ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. Google ScholarDigital Library
2. Bando, Y., Kuratate, T., and Nishita, T. 2002. A simple method for modeling wrinkles on human skin. In Proc. Pacific Graphics. Google ScholarDigital Library
3. Beeler, T., and Bradley, D. 2014. Rigid stabilization of facial expressions. ACM Trans. Graph. 33, 4 (July), 44:1–44:9. Google ScholarDigital Library
4. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
5. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 75:1–75:10. Google ScholarDigital Library
6. Beeler, T., Bickel, B., Noris, G., Beardsley, P., Marschner, S., Sumner, R. W., and Gross, M. 2012. Coupled 3d reconstruction of sparse facial hair and skin. ACM Trans. Graph. 31, 4 (July), 117:1–117:10. Google ScholarDigital Library
7. Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6. Google ScholarDigital Library
8. Bermano, A. H., Bradley, D., Beeler, T., Zund, F., Nowrouzezahrai, D., Baran, I., Sorkine-Hornung, O., Pfister, H., Sumner, R. W., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Trans. Graphics 33, 2. Google ScholarDigital Library
9. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33. Google ScholarDigital Library
10. Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proc. SCA, 57–66. Google ScholarDigital Library
11. Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. Visualization and Computer Graphics, IEEE Transactions on 14, 1, 213–230. Google ScholarDigital Library
12. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 40:1–40:10. Google ScholarDigital Library
13. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH), 99. Google ScholarDigital Library
14. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 41:1–41:10. Google ScholarDigital Library
15. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. Springer, 25–36.Google Scholar
16. Canny, J. 1986. A computational approach to edge detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 6, 679–698. Google ScholarDigital Library
17. Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. In IEEE CVPR, 2887–2894. Google ScholarDigital Library
18. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 41:1–41:10. Google ScholarDigital Library
19. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, 43:1–43:10. Google ScholarDigital Library
20. Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Transactions on pattern analysis and machine intelligence 23, 6, 681–685. Google ScholarDigital Library
21. Deng, Z., Lewis, J., and Neumann, U. 2005. Automated eye motion using texture synthesis. CGA 25, 2. Google ScholarDigital Library
22. Dutreve, L., Meyer, A., and Bouakaz, S. 2011. Easy acquisition and real-time animation of facial wrinkles. Comput. Animat. Virtual Worlds 22, 2-3, 169–176. Google ScholarDigital Library
23. Feng, W.-W., Yu, Y., and Kim, B.-U. 2010. A deformation transformer for real-time cloth animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 108:1–108:9. Google ScholarDigital Library
24. François, G., Gautron, P., Breton, G., and Bouatouch, K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815–827. Google ScholarDigital Library
25. Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. In ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 32, 158:1–158:10. Google ScholarDigital Library
26. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129:1–129:10. Google ScholarDigital Library
27. Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 74:1–74:10. Google ScholarDigital Library
28. Kähler, K., Haber, J., Yamauchi, H., and Seidel, H.-P. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. SCA, 55–63. Google ScholarDigital Library
29. Kim, D., Koh, W., Narain, R., Fatahalian, K., Treuille, A., and O’Brien, J. F. 2013. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 87:1–87:8. Google ScholarDigital Library
30. Klaudiny, M., and Hilton, A. 2012. High-detail 3d capture and non-sequential alignment of facial performance. In 3DIM-PVT. Google ScholarDigital Library
31. Larboulette, C., and Cani, M.-P. 2004. Real-time dynamic wrinkles. In Proc. Computer Graphics Int., 522–525. Google ScholarDigital Library
32. Le, B. H., Ma, X., and Deng, Z. 2012. Live speech driven head-and-eye motion generators. IEEE TVCG 18, 11, 1902–1914. Google ScholarDigital Library
33. Lee, S. P., Badler, J. B., and Badler, N. I. 2002. Eyes alive. ACM Trans. Graphics (Proc. SIGGRAPH) 21, 3, 637–644. Google ScholarDigital Library
34. Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 42:1–42:10. Google ScholarDigital Library
35. Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 0, 117–122.Google ScholarDigital Library
36. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183–194. Google ScholarDigital Library
37. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121. Google ScholarDigital Library
38. Magnenat-Thalmann, N., Kalra, P., Luc Leveque, J., Bazin, R., Batisse, D., and Querleux, B. 2002. A computational skin model: Fold and wrinkle formation. Trans. Info. Tech. Biomed. 6, 4, 317–323. Google ScholarDigital Library
39. Müller, M., and Chentanez, N. 2010. Wrinkle meshes. In Proc. SCA, 85–92. Google ScholarDigital Library
40. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 7 (Jul), 629–639. Google ScholarDigital Library
41. Peterson, M. F., and Eckstein, M. P. 2012. Looking just below the eyes is optimal across face recognition tasks. Proceedings of the National Academy of Sciences 109, 48.Google ScholarCross Ref
42. Popa, T., Zhou, Q., Bradley, D., Kraevoy, V., Fu, H., Sheffer, A., and Heidrich, W. 2009. Wrinkling captured garments using space-time data-driven deformation. Computer Graphics Forum (Proc. Eurographics) 28, 2, 427–435.Google ScholarCross Ref
43. Rhee, T., Hwang, Y., Kim, J. D., and Kim, C. 2011. Real-time facial animation from live video tracking. In Proc. SCA. Google ScholarDigital Library
44. Rohmer, D., Popa, T., Cani, M.-P., Hahmann, S., and Sheffer, A. 2010. Animation wrinkling: Augmenting coarse cloth simulations with realistic-looking wrinkles. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 29, 6, 157:1–157:8. Google ScholarDigital Library
45. Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69–91.Google Scholar
46. Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205–212. Google ScholarDigital Library
47. Seiler, M., Spillmann, J., and Harders, M. 2012. Enriching coarse interactive elastic objects with high-resolution data-driven deformations. In Proc. SCA, 9–17. Google ScholarDigital Library
48. Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33. Google ScholarDigital Library
49. Smith, M. L., Cottrell, G. W., Gosselin, F., and Schyns, P. G. 2005. Transmitting and decoding facial expressions. Psychological Science 16, 3, 184–189.Google ScholarCross Ref
50. Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Symposium on Geometry processing, vol. 4. Google ScholarDigital Library
51. Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In ECCV.Google Scholar
52. Trutoiu, L. C., Carter, E. J., Matthews, I., and Hodgins, J. K. 2011. Modeling and animating eye blinks. ACM Trans. Appl. Percept. 8, 3. Google ScholarDigital Library
53. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6. Google ScholarDigital Library
54. Wang, H., Hecht, F., Ramamoorthi, R., and O’Brien, J. F. 2010. Example-based wrinkle synthesis for clothing animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 107:1–107:8. Google ScholarDigital Library
55. Warburton, M., and Maddock, S. 2014. Physically-based forehead animation including wrinkles. Comput. Animat. Virtual Worlds. Google ScholarDigital Library
56. Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Real-time performance-based facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 77:1–77:10. Google ScholarDigital Library
57. Weissenfeld, A., Liu, K., and Ostermann, J. 2010. Video-realistic image-based eye animation via statistically driven state machines. The Visual Computer 26, 9, 1201–1216. Google ScholarDigital Library
58. Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In ICCV. Google ScholarDigital Library
59. Zhang, Y., Sim, T., and Tan, C. 2005. Simulating wrinkles in facial expressions on an anatomy-based face. In Proc. ICCS, 207–215. Google ScholarDigital Library
60. Zurdo, J. S., Brito, J. P., and Otaduy, M. A. 2013. Animating wrinkles by example on non-skinned cloth. IEEE TVCG 19, 1, 149–158. Google ScholarDigital Library
