“Total relighting: learning to relight portraits for background replacement”
Conference:
Title:
- Total relighting: learning to relight portraits for background replacement
Abstract:
We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject’s appearance as lit by novel illumination, thereby producing realistic composite images for any desired scene. Our technique includes foreground estimation via alpha matting, relighting, and compositing. We demonstrate that each of these stages can be tackled in a sequential pipeline without the use of priors (e.g. known background or known illumination) and with no specialized acquisition techniques, using only a single RGB portrait image and a novel, target HDR lighting environment as inputs. We train our model using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes. To perform realistic relighting for compositing, we introduce a novel per-pixel lighting representation in a deep learning framework, which explicitly models the diffuse and the specular components of appearance, producing relit portraits with convincingly rendered non-Lambertian effects like specular highlights. Multiple experiments and comparisons show the effectiveness of the proposed approach when applied to in-the-wild images.
References:
1. Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (2015).Google ScholarDigital Library
2. Richard Bluff, Landis Fields, Abby Keller, Hayden Jones, and Rachel Rose. 2020. ILM Presents “This is the Way” — the Making of Mandalorian. In ACM SIGGRAPH 2020 Production Sessions (SIGGRAPH 2020). Association for Computing Machinery, New York, NY, USA, Article 6, 1 pages. Google ScholarDigital Library
3. Brian Cabral, Marc Olano, and Philip Nemec. 1999. Reflection Space Image Based Rendering. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., USA, 165–170. Google ScholarDigital Library
4. Shaofan Cai, Xiaoshuai Zhang, Haoqiang Fan, Haibin Huang, Jiangyu Liu, Jiaming Liu, Jiaying Liu, Jue Wang, and Jian Sun. 2019. Disentangled Image Matting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).Google ScholarCross Ref
5. Guanying Chen, Kai Han, and Kwan-Yee K. Wong. 2018a. TOM-Net: Learning Transparent Object Matting From a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
6. Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018b. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV). 801–818.Google ScholarDigital Library
7. Paul Debevec. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). Association for Computing Machinery, New York, NY, USA, 189–198. Google ScholarDigital Library
8. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the Reflectance Field of a Human Face. In Proceedings of SIGGRAPH 2000 (SIGGRAPH ’00).Google ScholarDigital Library
9. Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, and Tim Hawkins. 2002. A lighting reproduction approach to live-action compositing. ACM Transactions on Graphics (TOG) 21, 3 (2002), 547–556.Google ScholarDigital Library
10. Julie Dorsey, James Arvo, and Donald Greenberg. 1995. Interactive Design of Complex Time-Dependent Lighting. IEEE Comput. Graph. Appl. 15, 2 (March 1995), 26–36. Google ScholarDigital Library
11. Per Einarsson, Charles-Felix Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond, Tim Hawkins, Mark Bolas, Sebastian Sylwan, and Paul Debevec. 2006. Relighting Human Locomotion with Flowed Reflectance Fields. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR).Google ScholarDigital Library
12. Marco Forte and François Pitié. 2020. F, B, Alpha Matting. arXiv:cs.CV/2003.07711Google Scholar
13. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A Wilson, and Paul Debevec. 2010. Circularly polarized spherical illumination reflectometry. In ACM SIGGRAPH Asia 2010 papers. 1–12.Google Scholar
14. Ned Greene. 1986. Environment mapping and other applications of world projections. IEEE Computer Graphics and Applications 6, 11 (1986), 21–29.Google ScholarDigital Library
15. Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, Sean Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul Debevec, and Shahram Izadi. 2019. The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting. In ACM TOG.Google ScholarDigital Library
16. Pierre-Loïc Hamon, James Harmer, Stuart Penn, and Nicolas Scapel. 2014. Gravity: Motion Control and Face Integration. In ACM SIGGRAPH 2014 Talks (SIGGRAPH ’14). Association for Computing Machinery, New York, NY, USA, Article 35, 1 pages. Google ScholarDigital Library
17. Qiqi Hou and Feng Liu. 2019. Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 – November 2, 2019. IEEE, 4129–4138. Google ScholarCross Ref
18. Tingbo Hou and Tyler Mullen. 2020. Background Features in Google Meet, Powered by Web ML. https://ai.googleblog.com/2020/10/background-features-in-google-meet.htmlGoogle Scholar
19. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.Google ScholarCross Ref
20. Yoshihiro Kanamori and Yuki Endo. 2018. Relighting Humans: Occlusion-aware Inverse Rendering for Full-body Human Images. In SIGGRAPH Asia. ACM.Google ScholarDigital Library
21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).Google Scholar
22. Chloe LeGendre, Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul E. Debevec. 2019. DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality. CVPR (2019).Google Scholar
23. Chloe LeGendre, Wan-Chun Ma, Rohit Pandey, Sean Fanello, Christoph Rhemann, Jason Dourgarian, Jay Busch, and Paul Debevec. 2020. Learning Illumination from Diverse Portraits. In SIGGRAPH Asia 2020 Technical Communications.Google Scholar
24. Anat Levin, Dani Lischinski, and Yair Weiss. 2007. A closed-form solution to natural image matting. IEEE transactions on pattern analysis and machine intelligence 30, 2 (2007), 228–242.Google Scholar
25. Yaoyi Li and Hongtao Lu. 2020. Natural Image Matting via Guided Contextual Attention. arXiv:cs.CV/2001.04069Google Scholar
26. Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman. 2020. Real-Time High-Resolution Background Matting. arXiv (2020), arXiv-2012.Google Scholar
27. Sebastian Lutz, Konstantinos Amplianitis, and Aljosa Smolic. 2018. AlphaGAN: Generative adversarial networks for natural image matting. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018. BMVA Press, 259. http://bmvc2018.org/contents/papers/0915.pdfGoogle Scholar
28. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR’07).Google ScholarDigital Library
29. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2794–2802.Google ScholarCross Ref
30. Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, and Sean Fanello. 2018. LookinGood: Enhancing Performance Capture with Real-time NeuralRe-Rendering. In SIGGRAPH Asia.Google Scholar
31. Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhoefer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, Peter Denny, Sofien Bouaziz, Peter Lincoln, Matt Whalen, Geoff Harvey, Jonathan Taylor, Shahram Izadi, Andrea Tagliasacchi, Paul Debevec, Christian Theobalt, Julien Valentin, and Christoph Rhemann. 2019. Deep Reflectance Fields – High-Quality Facial Reflectance Field Inference From Color Gradient Illumination. ACM Transactions on Graphics (Proceedings SIGGRAPH).Google Scholar
32. Abhimitra Meka, Maxim Maximov, Michael Zollhoefer, Avishek Chatterjee, Hans-Peter Seidel, Christian Richardt, and Christian Theobalt. 2018. LIME: Live Intrinsic Material Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR). 11.Google ScholarCross Ref
33. Abhimitra Meka, Rohit Pandey, Christian Häne, Sergio Orts-Escolano, Peter Barnum, Philip David-Son, Daniel Erickson, Yinda Zhang, Jonathan Taylor, Sofien Bouaziz, Chloe LeGendre, Wan-Chun Ma, Ryan Overbeck, Thabo Beeler, Paul Debevec, Shahram Izadi, Christian Theobalt, Christoph Rhemann, and Sean Fanello. 2020. Deep Relightable Textures: Volumetric Performance Capture with Neural Rendering. ACM Transactions on Graphics (2020).Google ScholarDigital Library
34. Gene S Miller and CD Hoffman. 1984. Illumination and reflection maps. Course Notes for Advanced Computer Graphics Animation, SIGGRAPH 84 (1984).Google Scholar
35. Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, and Tobias Ritschel. 2017. Deep Shading: Convolutional Neural Networks for Screen-Space Shading. 36, 4 (2017).Google Scholar
36. Thomas Nestmeyer, Jean-François Lalonde, Iain Matthews, and Andreas M. Lehrmann. 2020. Learning Physics-guided Face Relighting under Directional Light. In CVPR.Google Scholar
37. Rohit Pandey, Anastasia Tkach, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Ricardo Martin-Brualla, Andrea Tagliasacchi, George Papandreou, Philip Davidson, Cem Keskin, Shahram Izadi, and Sean Fanello. 2019. Volumetric Capture of Humans with a Single RGBD Camera via Semi-Parametric Learning. In CVPR.Google Scholar
38. Bui Tuong Phong. 1975. Illumination for computer generated pictures. Commun. ACM 18, 6 (1975), 311–317.Google ScholarDigital Library
39. Thomas Porter and Tom Duff. 1984. Compositing digital images. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques. 253–259.Google ScholarDigital Library
40. Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 497–500.Google ScholarDigital Library
41. Erik Reinhard, Michael Ashikhmin, Bruce Gooch, and Peter Shirley. 2001. Color Transfer between Images. IEEE Computer Graphics and Applications 21, 5 (2001), 34–41. Google ScholarDigital Library
42. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image Based Relighting Using Neural Networks. ACM Transactions on Graphics 34, 4 (July 2015).Google ScholarDigital Library
43. Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A perceptually motivated online benchmark for image matting. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA. IEEE Computer Society, 1826–1833. Google ScholarCross Ref
44. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI (2015).Google Scholar
45. Mark Sagar. 2005. Reflectance field rendering of human faces for” spider-man 2″. In ACM SIGGRAPH 2005 Courses. 14–es.Google ScholarDigital Library
46. Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. 2020. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
47. Shen Sang and M. Chandraker. 2020. Single-Shot Neural Relighting and SVBRDF Estimation. In ECCV.Google Scholar
48. Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steve Seitz, and Ira Kemelmacher-Shlizerman. 2020. Background Matting: The World is Your Green Screen. In Computer Vision and Pattern Recognition (CVPR).Google Scholar
49. Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017. Portrait lighting transfer using a mass transport approach. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1.Google ScholarDigital Library
50. Xiao Song, Guorun Yang, Xinge Zhu, Hui Zhou, Zhe Wang, and Jianping Shi. 2020. AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching. CoRR abs/2004.04627 (2020).Google Scholar
51. Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019. Single image portrait relighting. ACM Transactions on Graphics (TOG) 38, 4 (2019), 79.Google ScholarDigital Library
52. Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Fanello, Christoph Rhemann, Paul Debevec, Yun-Ta Tsai, Jonathan T Barron, and Ravi Ramamoorthi. 2020. Light stage super-resolution: continuous high-frequency relighting. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–12.Google ScholarDigital Library
53. Vladimir Tankovich, Christian Häne, Yinda Zhang, Adarsh Kowdle, Sean Fanello, and Sofien Bouaziz. 2021. HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching. CVPR (2021).Google Scholar
54. Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-Hsuan Yang. 2017. Deep Image Harmonization. CVPR (2017).Google Scholar
55. Yun-Ta Tsai and Rohit Pandey. 2020. Portrait Light: Enhancing Portrait Lighting with Machine Learning. https://ai.googleblog.com/2020/12/portrait-light-enhancing-portrait.htmlGoogle Scholar
56. Jue Wang and Michael F. Cohen. 2006. Simultaneous Matting and Compositing. In ACM SIGGRAPH.Google Scholar
57. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
58. Zhibo Wang, Xin Yu, Ming Lu, Quan Wang, Chen Qian, and Feng Xu. 2020. Single Image Portrait Relighting via Explicit Multiple Reflectance Channel Modeling. ACM SIGGRAPH Asia and Transactions on Graphics (2020).Google Scholar
59. Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. 2005. Performance Relighting and Reflectance Transformation with Time-Multiplexed Illumination. In SIGGRAPH.Google Scholar
60. Robert J. Woodham. 1989. Photometric Method for Determining Surface Orientation from Multiple Images. MIT Press, Cambridge, MA, USA.Google Scholar
61. Steve Wright. 2013. Digital compositing for film and video. Taylor & Francis.Google Scholar
62. Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. 2017. Deep image matting. In CVPR 2017 (Proceedings – 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017). United States, 311–320. Google ScholarCross Ref
63. Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. 2019. Deep View Synthesis from Sparse Photometric Images. SIGGRAPH (2019).Google Scholar
64. Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. 2018. Deep image-based relighting from optimal sparse samples. ACM Trans. on Graphics (2018).Google Scholar
65. Greg Zaal, Sergej Majboroda, and Andreas Mischok. 2020. HDRI Haven. https://www.hdrihaven.com/. Accessed: 2021-01-23.Google Scholar
66. He Zhang, Jianming Zhang, Federico Perazzi, Zhe Lin, and Vishal M Patel. 2020b. Deep Image Compositing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 365–374.Google Scholar
67. Richard Zhang. 2019. Making convolutional networks shift-invariant again. In International Conference on Machine Learning. PMLR, 7324–7334.Google Scholar
68. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. IEEE conference on computer vision and pattern recognition (CVPR) (2018).Google ScholarCross Ref
69. Xuaner Zhang, Jonathan T. Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, and David E. Jacobs. 2020a. Portrait Shadow Manipulation. ACM Transactions on Graphics (TOG) 39, 4.Google ScholarDigital Library
70. Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David Jacobs. 2019. Deep Single Image Portrait Relighting. In ICCV.Google Scholar
71. Douglas E Zongker, Dawn M Werner, Brian Curless, and David H Salesin. 1999. Environment matting and compositing. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 205–214.Google ScholarDigital Library
