“Single image portrait relighting” by Sun, Barron, Tsai, Xu, Yu, et al. …
Conference:
Type:
Title:
- Single image portrait relighting
Session/Category Title: Relighting and View Synthesis
Presenter(s)/Author(s):
Abstract:
Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset’s validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 × 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.
References:
1. Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, et al. 2016. TensorFlow: A system for large-scale machine learning. OSDI (2016). Google ScholarDigital Library
2. Apple. 2017. Use Portrait mode on your iPhone. https://support.apple.com/en-us/HT208118.Google Scholar
3. Jonathan T. Barron, Andrew Adams, YiChang Shih, and Carlos Hernández. 2015. Fast Bilateral-Space Stereo for Synthetic Defocus. CVPR (2015).Google Scholar
4. Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. TPAMI (2015).Google Scholar
5. Peter N. Belhumeur, David J. Kriegman, and Alan L. Yuille. 1999. The Bas-Relief Ambiguity. IJCV (1999). Google ScholarDigital Library
6. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. SIGGRAPH (1999). Google ScholarDigital Library
7. Dan Andrei Calian, Jean-François Lalonde, Paulo F. U. Gotardo, Tomas Simon, Iain A. Matthews, and Kenny Mitchell. 2018. From Faces to Outdoor Light Probes. Computer Graphics Forum (2018).Google Scholar
8. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. SIGGRAPH (2000). Google ScholarDigital Library
9. Craig Donner, Tim Weyrich, Eugene d’Eon, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2008. A Layered, Heterogeneous Reflectance Model for Acquiring and Rendering Human Skin. SIGGRAPH (2008). Google ScholarDigital Library
10. Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. 2000. Incorporating Second-order Functional Knowledge for Better Option Pricing. NIPS (2000). Google ScholarDigital Library
11. Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. 2008. Edge-Preserving Decompositions for Multi-Scale Tone and Detail Manipulation. SIGGRAPH (2008). Google ScholarDigital Library
12. Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to Predict Indoor Illumination from a Single Image. SIGGRAPH Asia (2017). Google ScholarDigital Library
13. Michaël Gharbi, Jiawen Chen, Jonathan T. Barron, Samuel W. Hasinoff, and Frédo Durand. 2017. Deep Bilateral Learning for Real-Time Image Enhancement. SIGGRAPH (2017). Google ScholarDigital Library
14. Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-Light Imaging on Mobile Cameras. SIGGRAPH Asia (2016). Google ScholarDigital Library
15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CVPR (2015).Google Scholar
16. Derek Hoiem, Alexei A. Efros, and Martial Hebert. 2007. Recovering Surface Layout from an Image. IJCV (2007). Google ScholarDigital Library
17. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. CVPR (2017).Google Scholar
18. Berthold K. P. Horn. 1970. Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. Technical Report. Google Scholar
19. Yuanming Hu, Baoyuan Wang, and Stephen Lin. 2017. FC4: Fully convolutional color constancy with confidence-weighted pooling. CVPR (2017).Google Scholar
20. Yoshihiro Kanamori and Yuki Endo. 2018. Relighting humans: occlusion-aware inverse rendering for fullbody human images. SIGGRAPH Asia (2018). Google ScholarDigital Library
21. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR (2014).Google Scholar
22. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. NIPS (2012). Google ScholarDigital Library
23. Jean-François Lalonde, Alexei A. Efros, and Srinivasa G. Narasimhan. 2009. Estimating Natural Illumination from a Single Outdoor Image. ICCV (2009).Google Scholar
24. Edwin H. Land and John J. McCann. 1971. Lightness and Retinex Theory. Journal of the Optical Society of America (1971).Google Scholar
25. Yann Lecun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L.D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation (1989). Google ScholarDigital Library
26. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to Reconstruct Shape and Spatially-varying Reflectance from a Single Image. SIGGRAPH Asia (2018). Google ScholarDigital Library
27. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbiç, Hao Li, Abhijeet Ghosh, and Paul Debevec. 2015. Skin Microstructure Deformation with Displacement Map Convolution. SIGGRAPH (2015). Google ScholarDigital Library
28. Sylvain Paris, Samuel W. Hasinoff, and Jan Kautz. 2011. Local Laplacian Filters: Edgeaware Image Processing with a Laplacian Pyramid. SIGGRAPH (2011). Google ScholarDigital Library
29. Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production Facial Performance Relighting Using Reflectance Transfer. SIGGRAPH (2007). Google ScholarDigital Library
30. Georg Petschnigg, Richard Szeliski, Maneesh Agrawala, Michael Cohen, Hugues Hoppe, and Kentaro Toyama. 2004. Digital Photography with Flash and No-flash Image Pairs. SIGGRAPH (2004). Google ScholarDigital Library
31. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. MICCAI (2015).Google Scholar
32. Ashutosh Saxena, Min Sun, and Andrew Y. Ng. 2009. Make3D: Learning 3D Scene Structure from a Single Still Image. TPAMI (2009). Google ScholarDigital Library
33. J.B. Schriever. 1909. Complete Self-instructing Library of Practical Photography. American school of art and photography.Google Scholar
34. Sebastian Schütze. 2015. Caravaggio: The Complete Works. Taschen.Google Scholar
35. Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild. CVPR (2018).Google Scholar
36. YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style Transfer for Headshot Portraits. SIGGRAPH (2014). Google ScholarDigital Library
37. Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2018. Portrait lighting transfer using a mass transport approach. SIGGRAPH (2018). Google ScholarDigital Library
38. Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic Depth-of-Field with a Single-Camera Mobile Phone. SIGGRAPH (2018). Google ScholarDigital Library
39. Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. TIP (2004). Google ScholarDigital Library
40. Henrique Weber, Donald Prévost, and Jean-François Lalonde. 2018. Learning to Estimate Indoor Lighting from 3D Objects. 3DV (2018).Google Scholar
41. Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. 2005. Performance Relighting and Reflectance Transformation with Time-multiplexed Illumination. SIGGRAPH (2005). Google ScholarDigital Library
42. Yuxin Wu and Kaiming He. 2018. Group Normalization. ECCV (2018).Google Scholar
43. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. SIGGRAPH (2018). Google ScholarDigital Library
