“Perspective-aware manipulation of portrait photos” by Fried, Shechtman, Goldman and Finkelstein

  • ©

Conference:


Type(s):


Title:

    Perspective-aware manipulation of portrait photos

Session/Category Title:   FACES & PORTRAITS

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    This paper introduces a method to modify the apparent relative pose and distance between camera and subject given a single portrait photo. Our approach fits a full perspective camera and a parametric 3D head model to the portrait, and then builds a 2D warp in the image plane to approximate the effect of a desired change in 3D. We show that this model is capable of correcting objectionable artifacts such as the large noses sometimes seen in “selfies,” or to deliberately bring a distant camera closer to the subject. This framework can also be used to re-pose the subject, as well as to create stereo pairs from an input portrait. We show convincing results on both an existing dataset as well as a new dataset we captured to validate our method.

References:


    1. Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: Photoreal facial modeling and animation. In ACM SIGGRAPH 2009 Courses. Google ScholarDigital Library
    2. Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’99, 187–194. Google ScholarDigital Library
    3. Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4 (July), 41:1–41:10. Google ScholarDigital Library
    4. Brox, T., and Malik, J. 2011. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 33, 3 (Mar.), 500–513. Google ScholarDigital Library
    5. Bryan, R., Perona, P., and Adolphs, R. 2012. Perspective distortion from interpersonal distance is an implicit visual cue for social judgments of faces. PLoS ONE 7, 9 (09).Google Scholar
    6. Burgos-Artizzu, X. P., Ronchi, M. R., and Perona, P. 2014. Distance estimation of an unknown person from a portrait. In European Conference on Computer Vision (ECCV). Springer, 313–327.Google Scholar
    7. Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1–41:10. Google ScholarDigital Library
    8. Cao, C., Hou, Q., and Zhou, K. 2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans. Graph. 33, 4 (July), 43:1–43:10. Google ScholarDigital Library
    9. Cao, C., Weng, Y., Zhou, S., Tong, Y., and Zhou, K. 2014. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3, 413–425. Google ScholarDigital Library
    10. Coleman, T. F., and Li, Y. 1996. An interior trust region approach for nonlinear minimization subject to bounds. SIAM Journal on Optimization 6, 2, 418–445.Google ScholarDigital Library
    11. Cooper, E. A., Piazza, E. A., and Banks, M. S. 2012. The perceptual basis of common photographic practice. Journal of Vision 12, 5, 8.Google ScholarCross Ref
    12. DeCarlo, D., Metaxas, D., and Stone, M. 1998. An anthropometric face model using variational techniques. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’98, 67–74. Google ScholarDigital Library
    13. Giger, D., Bazin, J.-C., Kuster, C., Popa, T., and Gross, M. 2014. Gaze correction with a single webcam. IEEE International Conference on Multimedia & Expo.Google Scholar
    14. Hassner, T., and Basri, R. 2006. Example based 3d reconstruction from single 2d images. In Beyond Patches Workshop at IEEE CVPR’06. Google ScholarDigital Library
    15. Hassner, T., Harel, S., Paz, E., and Enbar, R. 2015. Effective face frontalization in unconstrained images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
    16. Hassner, T. 2013. Viewing real-world faces in 3D. In International Conference on Computer Vision (ICCV). Google ScholarDigital Library
    17. Kemelmacher-Shlizerman, I., and Basri, R. 2011. 3d face reconstruction from a single image using a single reference face shape. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 33, 2 (Feb), 394–405. Google ScholarDigital Library
    18. Kemelmacher-Shlizerman, I., and Seitz, S. M. 2011. Face reconstruction in the wild. In International Conference on Computer Vision (ICCV). Google ScholarDigital Library
    19. Kemelmacher-Shlizerman, I., Shechtman, E., Garg, R., and Seitz, S. M. 2011. Exploring photobios. ACM Trans. Graph. 30, 4 (July), 61:1–61:10. Google ScholarDigital Library
    20. Orlov, A., 2016. Selecting a portrait lens with correct focal length. Accessed 2016-01-15: http://petapixel.com/2016/01/04/selecting-a-portrait-lens-with-correct-focal-length/.Google Scholar
    21. Perona, P. 2007. A new perspective on portraiture. Journal of Vision 7, 992–992.Google ScholarCross Ref
    22. Perona, P. 2013. Far and yet close: Multiple viewpoints for the perfect portrait. Art & Perception 1, 1-2, 105–120.Google ScholarCross Ref
    23. Saragih, J. M., Lucey, S., and Cohn, J. 2009. Face alignment through subspace constrained mean-shifts. In International Conference on Computer Vision (ICCV).Google Scholar
    24. Tucker, L. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 3, 279–311.Google ScholarCross Ref
    25. Valind, E. 2014. Portrait Photography: From Snapshots to Great Shots. Pearson Education. Google ScholarDigital Library
    26. Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (July), 426–433. Google ScholarDigital Library
    27. Weise, T., Leibe, B., and Van Gool, L. 2007. Fast 3d scanning with automatic motion compensation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 861–868.Google ScholarCross Ref
    28. Yang, F., Wang, J., Shechtman, E., Bourdev, L., and Metaxas, D. 2011. Expression flow for 3d-aware face component transfer. ACM Trans. Graph. 30, 4 (July), 60:1–60:10. Google ScholarDigital Library
    29. Yang, F., Bourdev, L., Shechtman, E., Wang, J., and Metaxas, D. 2012. Facial expression editing in video using a temporally-smooth factorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 861–868. Google ScholarDigital Library
    30. Yang, F., Shechtman, E., Wang, J., Bourdev, L., and Metaxas, D. 2012. Face morphing using 3d-aware appearance optimization. In Proceedings of Graphics Interface (GI’12), 93–99. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: