“Detailed spatio-temporal reconstruction of eyelids” by Bermano, Beeler, Kozlov, Bradley, Bickel, et al. …

  • ©Amit Haim Bermano, Thabo Beeler, Yeara Kozlov, Derek Bradley, Bernd Bickel, and Markus Gross




    Detailed spatio-temporal reconstruction of eyelids


Session Title: Face Reality



    In recent years we have seen numerous improvements on 3D scanning and tracking of human faces, greatly advancing the creation of digital doubles for film and video games. However, despite the high-resolution quality of the reconstruction approaches available, current methods are unable to capture one of the most important regions of the face – the eye region. In this work we present the first method for detailed spatio-temporal reconstruction of eyelids. Tracking and reconstructing eyelids is extremely challenging, as this region exhibits very complex and unique skin deformation where skin is folded under while opening the eye. Furthermore, eyelids are often only partially visible and obstructed due to self-occlusion and eyelashes. Our approach is to combine a geometric deformation model with image data, leveraging multi-view stereo, optical flow, contour tracking and wrinkle detection from local skin appearance. Our deformation model serves as a prior that enables reconstruction of eyelids even under strong self-occlusions caused by rolling and folding skin as the eye opens and closes. The output is a person-specific, time-varying eyelid reconstruction with anatomically plausible deformations. Our high-resolution detailed eyelids couple naturally with current facial performance capture approaches. As a result, our method can largely increase the fidelity of facial capture and the creation of digital doubles.


    1. Alexander, O., Fyffe, G., Busch, J., Yu, X., Ichikari, R., Jones, A., Debevec, P., Jimenez, J., Danvoye, E., Antionazzi, B., et al. 2013. Digital ira: creating a real-time photoreal digital actor. In ACM SIGGRAPH 2013 Posters. Google ScholarDigital Library
    2. Bando, Y., Kuratate, T., and Nishita, T. 2002. A simple method for modeling wrinkles on human skin. In Proc. Pacific Graphics. Google ScholarDigital Library
    3. Beeler, T., and Bradley, D. 2014. Rigid stabilization of facial expressions. ACM Trans. Graph. 33, 4 (July), 44:1–44:9. Google ScholarDigital Library
    4. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
    5. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 75:1–75:10. Google ScholarDigital Library
    6. Beeler, T., Bickel, B., Noris, G., Beardsley, P., Marschner, S., Sumner, R. W., and Gross, M. 2012. Coupled 3d reconstruction of sparse facial hair and skin. ACM Trans. Graph. 31, 4 (July), 117:1–117:10. Google ScholarDigital Library
    7. Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6. Google ScholarDigital Library
    8. Bermano, A. H., Bradley, D., Beeler, T., Zund, F., Nowrouzezahrai, D., Baran, I., Sorkine-Hornung, O., Pfister, H., Sumner, R. W., Bickel, B., and Gross, M. 2014. Facial performance enhancement using dynamic shape space analysis. ACM Trans. Graphics 33, 2. Google ScholarDigital Library
    9. Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graphics (Proc. SIGGRAPH), 33. Google ScholarDigital Library
    10. Bickel, B., Lang, M., Botsch, M., Otaduy, M. A., and Gross, M. 2008. Pose-space animation and transfer of facial details. In Proc. SCA, 57–66. Google ScholarDigital Library
    11. Botsch, M., and Sorkine, O. 2008. On linear variational surface deformation methods. Visualization and Computer Graphics, IEEE Transactions on 14, 1, 213–230. Google ScholarDigital Library
    12. Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 40:1–40:10. Google ScholarDigital Library
    13. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graphics (Proc. SIGGRAPH), 99. Google ScholarDigital Library
    14. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 41:1–41:10. Google ScholarDigital Library
    15. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In ECCV. Springer, 25–36.Google Scholar
    16. Canny, J. 1986. A computational approach to edge detection. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 6, 679–698. Google ScholarDigital Library
    17. Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. In IEEE CVPR, 2887–2894. Google ScholarDigital Library
    18. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 41:1–41:10. Google ScholarDigital Library
    19. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graphics (Proc. SIGGRAPH) 33, 4, 43:1–43:10. Google ScholarDigital Library
    20. Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Transactions on pattern analysis and machine intelligence 23, 6, 681–685. Google ScholarDigital Library
    21. Deng, Z., Lewis, J., and Neumann, U. 2005. Automated eye motion using texture synthesis. CGA 25, 2. Google ScholarDigital Library
    22. Dutreve, L., Meyer, A., and Bouakaz, S. 2011. Easy acquisition and real-time animation of facial wrinkles. Comput. Animat. Virtual Worlds 22, 2-3, 169–176. Google ScholarDigital Library
    23. Feng, W.-W., Yu, Y., and Kim, B.-U. 2010. A deformation transformer for real-time cloth animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 108:1–108:9. Google ScholarDigital Library
    24. François, G., Gautron, P., Breton, G., and Bouatouch, K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815–827. Google ScholarDigital Library
    25. Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. In ACM Trans. Graphics (Proc. SIGGRAPH Asia), vol. 32, 158:1–158:10. Google ScholarDigital Library
    26. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129:1–129:10. Google ScholarDigital Library
    27. Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 74:1–74:10. Google ScholarDigital Library
    28. Kähler, K., Haber, J., Yamauchi, H., and Seidel, H.-P. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. SCA, 55–63. Google ScholarDigital Library
    29. Kim, D., Koh, W., Narain, R., Fatahalian, K., Treuille, A., and O’Brien, J. F. 2013. Near-exhaustive precomputation of secondary cloth effects. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 87:1–87:8. Google ScholarDigital Library
    30. Klaudiny, M., and Hilton, A. 2012. High-detail 3d capture and non-sequential alignment of facial performance. In 3DIM-PVT. Google ScholarDigital Library
    31. Larboulette, C., and Cani, M.-P. 2004. Real-time dynamic wrinkles. In Proc. Computer Graphics Int., 522–525. Google ScholarDigital Library
    32. Le, B. H., Ma, X., and Deng, Z. 2012. Live speech driven head-and-eye motion generators. IEEE TVCG 18, 11, 1902–1914. Google ScholarDigital Library
    33. Lee, S. P., Badler, J. B., and Badler, N. I. 2002. Eyes alive. ACM Trans. Graphics (Proc. SIGGRAPH) 21, 3, 637–644. Google ScholarDigital Library
    34. Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graphics (Proc. SIGGRAPH) 32, 4, 42:1–42:10. Google ScholarDigital Library
    35. Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 0, 117–122.Google ScholarDigital Library
    36. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Symposium on Rendering, 183–194. Google ScholarDigital Library
    37. Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 27, 5, 121. Google ScholarDigital Library
    38. Magnenat-Thalmann, N., Kalra, P., Luc Leveque, J., Bazin, R., Batisse, D., and Querleux, B. 2002. A computational skin model: Fold and wrinkle formation. Trans. Info. Tech. Biomed. 6, 4, 317–323. Google ScholarDigital Library
    39. Müller, M., and Chentanez, N. 2010. Wrinkle meshes. In Proc. SCA, 85–92. Google ScholarDigital Library
    40. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. Pattern Analysis and Machine Intelligence, IEEE Transactions on 12, 7 (Jul), 629–639. Google ScholarDigital Library
    41. Peterson, M. F., and Eckstein, M. P. 2012. Looking just below the eyes is optimal across face recognition tasks. Proceedings of the National Academy of Sciences 109, 48.Google ScholarCross Ref
    42. Popa, T., Zhou, Q., Bradley, D., Kraevoy, V., Fu, H., Sheffer, A., and Heidrich, W. 2009. Wrinkling captured garments using space-time data-driven deformation. Computer Graphics Forum (Proc. Eurographics) 28, 2, 427–435.Google ScholarCross Ref
    43. Rhee, T., Hwang, Y., Kim, J. D., and Kim, C. 2011. Real-time facial animation from live video tracking. In Proc. SCA. Google ScholarDigital Library
    44. Rohmer, D., Popa, T., Cani, M.-P., Hahmann, S., and Sheffer, A. 2010. Animation wrinkling: Augmenting coarse cloth simulations with realistic-looking wrinkles. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 29, 6, 157:1–157:8. Google ScholarDigital Library
    45. Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69–91.Google Scholar
    46. Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205–212. Google ScholarDigital Library
    47. Seiler, M., Spillmann, J., and Harders, M. 2012. Enriching coarse interactive elastic objects with high-resolution data-driven deformations. In Proc. SCA, 9–17. Google ScholarDigital Library
    48. Shi, F., Wu, H.-T., Tong, X., and Chai, J. 2014. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33. Google ScholarDigital Library
    49. Smith, M. L., Cottrell, G. W., Gosselin, F., and Schyns, P. G. 2005. Transmitting and decoding facial expressions. Psychological Science 16, 3, 184–189.Google ScholarCross Ref
    50. Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Symposium on Geometry processing, vol. 4. Google ScholarDigital Library
    51. Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In ECCV.Google Scholar
    52. Trutoiu, L. C., Carter, E. J., Matthews, I., and Hodgins, J. K. 2011. Modeling and animating eye blinks. ACM Trans. Appl. Percept. 8, 3. Google ScholarDigital Library
    53. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 31, 6. Google ScholarDigital Library
    54. Wang, H., Hecht, F., Ramamoorthi, R., and O’Brien, J. F. 2010. Example-based wrinkle synthesis for clothing animation. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 107:1–107:8. Google ScholarDigital Library
    55. Warburton, M., and Maddock, S. 2014. Physically-based forehead animation including wrinkles. Comput. Animat. Virtual Worlds. Google ScholarDigital Library
    56. Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Real-time performance-based facial animation. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 77:1–77:10. Google ScholarDigital Library
    57. Weissenfeld, A., Liu, K., and Ostermann, J. 2010. Video-realistic image-based eye animation via statistically driven state machines. The Visual Computer 26, 9, 1201–1216. Google ScholarDigital Library
    58. Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In ICCV. Google ScholarDigital Library
    59. Zhang, Y., Sim, T., and Tan, C. 2005. Simulating wrinkles in facial expressions on an anatomy-based face. In Proc. ICCS, 207–215. Google ScholarDigital Library
    60. Zurdo, J. S., Brito, J. P., and Otaduy, M. A. 2013. Animating wrinkles by example on non-skinned cloth. IEEE TVCG 19, 1, 149–158. Google ScholarDigital Library

ACM Digital Library Publication: