“Portrait Lighting Transfer Using a Mass Transport Approach”
Conference:
Type(s):
Title:
- Portrait Lighting Transfer Using a Mass Transport Approach
Session/Category Title: Image and Light Field Manipulation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Lighting is a critical element of portrait photography. However, good lighting design typically requires complex equipment and significant time and expertise. Our work simplifies this task using a relighting technique that transfers the desired illumination of one portrait onto another. The novelty in our approach to this challenging problem is our formulation of relighting as a mass transport problem. We start from standard color histogram matching that only captures the overall tone of the illumination, and we show how to use the mass-transport formulation to make it dependent on facial geometry. We fit a three-dimensional (3D) morphable face model to the portrait, and for each pixel, we combine the color value with the corresponding 3D position and normal. We then solve a mass-transport problem in this augmented space to generate a color remapping that achieves localized, geometry-aware relighting. Our technique is robust to variations in facial appearance and small errors in face reconstruction. As we demonstrate, this allows our technique to handle a variety of portraits and illumination conditions, including scenarios that are challenging for previous methods.
References:
1. Yael Adini, Yael Moses, and Shimon Ullman. 1997. Face recognition: The problem of compensating for changes in illumination direction. IEEE Trans. PAMI 19, 7 (1997), 721.Google ScholarDigital Library
2. Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The digital emily project: Photoreal facial modeling and animation. In Proceedings of the ACM SIGGRAPH 2009 Courses. 12. Google ScholarDigital Library
3. Soonmin Bae, Sylvain Paris, and Frédo Durand. 2006. Two-scale tone management for photographic look. ACM Trans. Graph. 25, 3 (2006), 637. Google ScholarDigital Library
4. Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter Belhumeur, and Shree K. Nayar. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. 27, 3 (2008), 39. Google ScholarDigital Library
5. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH. 187. Google ScholarDigital Library
6. Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015a. Sliced and radon wasserstein barycenters of measures. J. Math. Imag. Vision 51, 1 (2015), 22.Google ScholarDigital Library
7. Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015b. Blind video temporal consistency. ACM Trans. Graph. 34, 6 (2015). Google ScholarDigital Library
8. Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. 2011. Displacement interpolation using lagrangian mass transport. ACM Trans. Graph. 30, 6 (2011), 158. Google ScholarDigital Library
9. Xiaowu Chen, Mengmeng Chen, Xin Jin, and Qinping Zhao. 2011. Face illumination transfer through edge-preserving filters. In Proceedings of CVPR. IEEE, 281–287. Google ScholarDigital Library
10. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Proceedings of NIPS. 2292–2300.Google Scholar
11. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of SIGGRAPH. 145. Google ScholarDigital Library
12. Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol. 2014. Regularized discrete optimal transport. SIAM J. Imag. Sci. 7, 3 (2014), 1853. Google ScholarCross Ref
13. Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. PAMI 23, 6 (2001), 643.Google ScholarDigital Library
14. John Haddon and David Forsyth. 1998. Shading primitives: Finding folds and shallow grooves. In Proceedings of ICCV. IEEE, 236–241. Google ScholarCross Ref
15. Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided image filtering. IEEE Trans. PAMI 35, 6 (2013), 1397.Google ScholarDigital Library
16. David W. Jacobs, Peter N. Belhumeur, and Ronen Basri. 1998. Comparing images under variable illumination. In Proceedings of CVPR. IEEE, 610–617. Google ScholarCross Ref
17. Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33, 4 (2014), 1. Google ScholarDigital Library
18. Zicheng Liu, Ying Shan, and Zhengyou Zhang. 2001. Expressive expression mapping with ratio images. In Proceedings of SIGGRAPH. 271–276. Google ScholarDigital Library
19. Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. 26, 3 (2007), 52. Google ScholarDigital Library
20. Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (2003), 313. Google ScholarDigital Library
21. Francois Pitie, C. Kokaram Anil, and Rozenn Dahyot. 2005. N-dimensional probability density function transfer and its application to color transfer. In Proceedings of ICCV. IEEE, 1434–1439. Google ScholarDigital Library
22. Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. 2012. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision. Springer, 435. Google ScholarDigital Library
23. E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. 2001. Color transfer between images. Comput. Graph. Appl. IEEE 21, 5 (2001), 34–41. Google ScholarDigital Library
24. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover’s distance as a metric for image retrieval. Int. J. Comput. Vis. 40, 2 (2000), 99. Google ScholarDigital Library
25. Amnon Shashua and Tammy Riklin-Raviv. 2001. The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE Trans. PAMI 23, 2 (2001), 129.Google ScholarDigital Library
26. Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian Price, Eli Shechtman, and Ian Sachs. 2016. Automatic portrait segmentation for image stylization. Comput. Graph. Forum 35, 2 (2016), 93.Google ScholarCross Ref
27. YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. 33, 4 (2014), 148. Google ScholarDigital Library
28. Yichang Shih, Sylvain Paris, Frédo Durand, and William T. Freeman. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32, 6 (2013), 200. Google ScholarDigital Library
29. Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. 2015. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. ACM Trans. Graph. 34, 4 (2015), 66. Google ScholarDigital Library
30. Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, and Hanspeter Pfister. 2010. Multi-scale image harmonization. ACM Trans. Graph. 29, 4 (2010), 125. Google ScholarDigital Library
31. Cédric Villani. 2003. Topics in Optimal Transportation (Graduate Studies in Mathematics, Vol. 58). American Mathematical Society.Google Scholar
32. Cédric Villani. 2008. Optimal Transport: Old and New (1st ed.).Google Scholar
33. Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. 2009. Face relighting from a single image under arbitrary unknown lighting conditions. IEEE Trans. PAMI 31, 11 (2009), 1968.Google ScholarDigital Library
34. Zhen Wen, Zicheng Liu, and Thomas S. Huang. 2003. Face relighting with radiance environment maps. In Proceedings of CVPR, Vol. 2. IEEE, II–158.Google Scholar
35. Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25, 3 (2006), 1013. Google ScholarDigital Library
36. Fei Yang, Jue Wang, Eli Shechtman, Lubomir Bourdev, and Dimitri Metaxas. 2011. Expression flow for 3D-aware face component transfer. ACM Trans. Graph. 30, 4 (2011), 60. Google ScholarDigital Library
