“Lightweight eye capture using a parametric model”

  • ©Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Perez, and Christian Theobalt

Conference:


Type:


Title:

    Lightweight eye capture using a parametric model

Session/Category Title: CAPTURING HUMANS

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Facial scanning has become ubiquitous in digital media, but so far most efforts have focused on reconstructing the skin. Eye reconstruction, on the other hand, has received only little attention, and the current state-of-the-art method is cumbersome for the actor, time-consuming, and requires carefully setup and calibrated hardware. These constraints currently make eye capture impractical for general use. We present the first approach for high-quality lightweight eye capture, which leverages a database of pre-captured eyes to guide the reconstruction of new eyes from much less constrained inputs, such as traditional single-shot face scanners or even a single photo from the internet. This is accomplished with a new parametric model of the eye built from the database, and a novel image-based model fitting algorithm. Our method provides both automatic reconstructions of real eyes, as well as artistic control over the parameters to generate user-specific eyes.

References:


    1. Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: reconstruction and parameterization from range scans. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 587–594. Google ScholarDigital Library
    2. Amberg, B., Romdhani, S., and Vetter, T. 2007. Optimal step nonrigid icp algorithms for surface registration. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, IEEE, 1-8.Google Scholar
    3. Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. Scape: shape completion and animation of people. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 408-416. Google ScholarDigital Library
    4. Beeler, T., Bickel, B., Sumner, R., Beardsley, P., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graphics (Proc. SIGGRAPH). Google ScholarDigital Library
    5. Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C, Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4, 75. Google ScholarDigital Library
    6. Bérard, P., Bradley, D., Nitti, M., Beeler, T., and Gross, M. 2014. High-quality capture of eyes. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 33, 6, 223:1–223:12. Google ScholarDigital Library
    7. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. of the 26th annual conference on Computer graphics and interactive techniques, 187–194. Google ScholarDigital Library
    8. Boykov, Y., and Kolmogorov, V. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 9, 1124–1137. Google ScholarDigital Library
    9. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graphics (Proc. SIGGRAPH) 29, 4, 41. Google ScholarDigital Library
    10. Brown, B. J., and Rusinkiewicz, S. 2004. Non-rigid range-scan alignment using thin-plate splines. In 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004. Proceedings. 2nd International Symposium on, IEEE, 759–765. Google ScholarDigital Library
    11. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33, 4, 43. Google ScholarDigital Library
    12. Chen, K., Johan, H., and Mueller-Wittig, W. 2013. Simple and efficient example-based texture synthesis using tiling and deformation. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, 145–152. Google ScholarDigital Library
    13. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, 341–346. Google ScholarDigital Library
    14. Flickr. 2006. Mohammed Alnaser – Mr. Falcon. Creative Commons – Attribution 2.0. https://www.flickr.com/photos/69er/324313066.Google Scholar
    15. Francois G. Gautron P. Breton G. and Bouatouch K. 2009. Image-based modeling of the human eye. IEEE TVCG 15, 5, 815–827. Google ScholarDigital Library
    16. Funkhouser T. Kazhdan M. Shilane P. Min P. Kiefer W. Tal A. Rusinkiewicz S. and Dobkin D. 2004. Modeling by example. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 652–663. Google ScholarDigital Library
    17. Fyffe G. Hawkins T. Watts C. Ma W.-C. and Debevec P. 2011. Comprehensive facial performance capture. In Eurographics.Google Scholar
    18. Fyffe, G., Jones, A., Alexander, O., Ichikari, R., and Debevec, P. 2014. Driving high-resolution facial scans with video performance capture. ACM Trans. Graphics 34, 1, 8:1–8:14. Google ScholarDigital Library
    19. Garrido, P., Valgaerts, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 32, 6, 158. Google ScholarDigital Library
    20. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6, 129. Google ScholarDigital Library
    21. Guo, B., Liang, L., Liu, C., Shum, H.-Y., and Xu, Y. 2001. Real-time texture synthesis by patch-based sampling.Google Scholar
    22. Haehnel, D., Thrun, S., and Burgard, W. 2003. An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In IJCAI, vol. 3, 915–920. Google ScholarDigital Library
    23. Ikemoto, L., Gelfand, N., and Levoy, M. 2003. A hierarchical method for aligning warped meshes. In 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings. Fourth International Conference on, IEEE, 434–441.Google Scholar
    24. Jacob, M., and Unser, M. 2004. Design of steerable filters for feature detection using canny-like criteria. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 8, 1007–1019. Google ScholarDigital Library
    25. Kazhdan, M., Funkhouser, T., and Rusinkiewicz, S. 2004. Shape matching and anisotropy. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 623–629. Google ScholarDigital Library
    26. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: image and video synthesis using graph cuts. In ACM Transactions on Graphics (ToG), vol. 22, ACM, 277–286. Google ScholarDigital Library
    27. Lefohn, A., Budge, B., Shirley, P., Caruso, R., and Reinhard, E. 2003. An ocularist’s approach to human iris synthesis. IEEE CG&A 23, 6, 70–75. Google ScholarDigital Library
    28. Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. In Computer graphics forum, vol. 27, Wiley Online Library, 1421–1430. Google ScholarDigital Library
    29. Li, J., Xu, W., Cheng, Z., Xu, K., and Klein, R. 2015. Lightweight wrinkle synthesis for 3d facial modeling and animation. Computer-Aided Design 58, 117–122. Google ScholarDigital Library
    30. Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Proc. Rendering Techniques, 183–194. Google ScholarDigital Library
    31. Mohammed, U., Prince, S. J., and Kautz, J. 2009. Visiolization: generating novel facial images. ACM Transactions on Graphics (TOG) 28, 3, 57. Google ScholarDigital Library
    32. Palubicki, W., Horel, K., Longay, S., Runions, A., Lane, B., Měch, R., and Prusinkiewicz, P. 2009. Self-organizing tree models for image synthesis. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 58. Google ScholarDigital Library
    33. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG), vol. 22, ACM, 313–318. Google ScholarDigital Library
    34. Praun, E., Finkelstein, A., and Hoppe, H. 2000. Lapped textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 465–470. Google ScholarDigital Library
    35. Ramanarayanan, G., and Bala, K. 2007. Constrained texture synthesis via energy minimization. IEEE TVCG 13, 1. Google ScholarDigital Library
    36. Rozenberg, G., and Salomaa, A. 1976. The mathematical theory of L systems. Springer. Google ScholarDigital Library
    37. Ruhland, K., Andrist, S., Badler, J., Peters, C., Badler, N., Gleicher, M., Mutlu, B., and McDonnell, R. 2014. Look me in the eyes: A survey of eye and gaze animation for virtual agents and artificial systems. In Eurographics State of the Art Reports, 69–91.Google Scholar
    38. Sagar, M. A., Bullivant, D., Mallinson, G. D., and Hunter, P. J. 1994. A virtual environment and model of the eye for surgical simulation. In Proceedings of Computer Graphics and Interactive Techniques, 205–212. Google ScholarDigital Library
    39. Shen, H.-L., and Cai, Q.-Y. 2009. Simple and efficient method for specularity removal in an image. Applied optics 48, 14, 2711–2719.Google Scholar
    40. Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, ACM, 175–184. Google ScholarDigital Library
    41. Suwajanakorn, S., Kemelmacher-Shlizerman, I., and Seitz, S. M. 2014. Total moving face reconstruction. In Computer Vision–ECCV 2014. Springer, 796–812.Google Scholar
    42. Tai, Y.-W., Liu, S., Brown, M. S., and Lin, S. 2010. Super resolution using edge prior and single image detail synthesis. In CVPR.Google Scholar
    43. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., and Theobalt, C. 2012. Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans. Graph. 31, 6, 187. Google ScholarDigital Library
    44. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), vol. 24, ACM, 426–433. Google ScholarDigital Library
    45. Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In Eurographics 2009, State of the Art Report, EG-STAR, Eurographics Association, 93–117.Google Scholar
    46. Wikimedia Commons. 1485. Sandro Botticelli – The Birth of the Venus. Public Domain. https://commons.wikimedia.org/wiki/File:Venus_botticelli_detail.jpg.Google Scholar
    47. Wikimedia Commons. 1887. Vincent Van Gogh – Self-Portrait. Public Domain. https://commons.wikimedia.org/wiki/File:VanGogh_1887_Selbstbildnis.jpg.Google Scholar
    48. Wikimedia Commons. 2006. Blue Eye Image. GNU Free Documentation License Version 1.2. https://commons.wikimedia.org/wiki/File:Blueye.JPG.Google Scholar

ACM Digital Library Publication:


Overview Page: