“Relighting humans: occlusion-aware inverse rendering for full-body human images”
Conference:
Type(s):
Title:
- Relighting humans: occlusion-aware inverse rendering for full-body human images
Session/Category Title: Advanced SVBRDF
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Relighting of human images has various applications in image synthesis. For relighting, we must infer albedo, shape, and illumination from a human portrait. Previous techniques rely on human faces for this inference, based on spherical harmonics (SH) lighting. However, because they often ignore light occlusion, inferred shapes are biased and relit images are unnaturally bright particularly at hollowed regions such as armpits, crotches, or garment wrinkles. This paper introduces the first attempt to infer light occlusion in the SH formulation directly. Based on supervised learning using convolutional neural networks (CNNs), we infer not only an albedo map, illumination but also a light transport map that encodes occlusion as nine SH coefficients per pixel. The main difficulty in this inference is the lack of training datasets compared to unlimited variations of human portraits. Surprisingly, geometric information including occlusion can be inferred plausibly even with a small dataset of synthesized human figures, by carefully preparing the dataset so that the CNNs can exploit the data coherency. Our method accomplishes more realistic relighting than the occlusion-ignored formulation.
References:
1. Miika Aittala, Timo Aila, and Jaakko Lehtinen. 2016. Reflectance modeling by neural texture synthesis. ACM Trans. Graph. 35, 4 (2016), 65:1–65:13. Google ScholarDigital Library
2. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. ACM Trans. Graph. 24, 3 (2005), 408–416. Google ScholarDigital Library
3. Alexandru O. Balan, Leonid Sigal, Michael J. Black, James E. Davis, and Horst W. Haussecker. 2007. Detailed Human Shape and Pose from Images. In 2007 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007).Google Scholar
4. Jonathan T. Barron and Jitendra Malik. 2015. Shape, Illumination, and Reflectance from Shading. IEEE Trans. Pattern Anal. Mach. Intell. 37, 8 (2015), 1670–1687.Google ScholarDigital Library
5. H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics from images. Comp. Vis. Sys. (1978).Google Scholar
6. Anil S. Baslamisli, Hoang-An Le, and Theo Gevers. 2018. CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).Google ScholarCross Ref
7. R. Basri and D. W. Jacobs. 2003. Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 2 (Feb 2003), 218–233. Google ScholarDigital Library
8. Sean Bell, Kavita Bala, and Noah Snavely. 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4 (2014), 159:1–159:12. Google ScholarDigital Library
9. Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999). 187–194. Google ScholarDigital Library
10. Nicolas Bonneel, Balazs Kovacs, Sylvain Paris, and Kavita Bala. 2017. Intrinsic Decompositions for Image Editing. Comput. Graph. Forum 36, 2 (2017), 593–609. Google ScholarDigital Library
11. Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-quality hair modeling from a single portrait photo. ACM Trans. Graph. 34, 6 (2015), 204:1–204:10. Google ScholarDigital Library
12. Manmohan Krishna Chandraker and Ravi Ramamoorthi. 2011. What an image reveals about material reflectance. In IEEE International Conference on Computer Vision (ICCV 2011). 1076–1083. Google ScholarDigital Library
13. R. Danerek, Endri Dibra, A. Cengiz Öztireli, Remo Ziegler, and Markus H. Gross. 2017. DeepGarment: 3D Garment Shape Estimation from a Single Image. Comput. Graph. Forum 36, 2 (2017), 269–280. Google ScholarDigital Library
14. Paul Debevec. 2004. Light Probe Image Gallery. (2004). http://www.pauldebevec.com/Probes/.Google Scholar
15. Paul E. Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2000). 145–156. Google ScholarDigital Library
16. Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Trans. Graph. 36, 6 (2017), 177:1–177:10. Google ScholarDigital Library
17. Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36, 6 (2017), 176:1–176:14. Google ScholarDigital Library
18. Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6 (2013), 158:1–158:10. Google ScholarDigital Library
19. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Efstratios Gavves, Mario Fritz, Luc Van Gool, and Tinne Tuytelaars. 2018. Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. IEEE Trans. Pattern Anal. Mach. Intell. 40, 8 (2018), 1932–1947.Google ScholarCross Ref
20. Peng Guan, Alexander Weiss, Alexandru O. Balan, and Michael J. Black. 2009. Estimating human shape and pose from a single image. In IEEE 12th International Conference on Computer Vision (ICCV 2009). 1381–1388.Google Scholar
21. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 2373–2382.Google Scholar
22. Berthold K. P. Horn. 1989. Shape from Shading. MIT Press, Chapter Obtaining Shape from Shading Information, 123–171. Google ScholarDigital Library
23. Carlo Innamorati, Tobias Ritschel, Tim Weyrich, and Niloy J. Mitra. 2017. Decomposing Single Images for Layered Photo Retouching. Comput. Graph. Forum 36, 4 (2017), 15–25. Google ScholarDigital Library
24. Micah K. Johnson and Edward H. Adelson. 2011. Shape estimation in natural illumination. In The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). 2553–2560. Google ScholarDigital Library
25. Ira Kemelmacher-Shlizerman and Ronen Basri. 2011. 3D Face Reconstruction from a Single Image Using a Single Reference Face Shape. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2 (2011), 394–405. Google ScholarDigital Library
26. Natasha Kholgade, Tomas Simon, Alexei A. Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Trans. Graph. 33, 4 (2014), 127:1–127:12. Google ScholarDigital Library
27. Edwin H. Land and John J. McCann. 1971. Lightness and Retinex Theory. J. Opt. Soc. Am. 61, 1 (Jan 1971), 1–11.Google ScholarCross Ref
28. Guannan Li, Chenglei Wu, Carsten Stoll, Yebin Liu, Kiran Varanasi, Qionghai Dai, and Christian Theobalt. 2013. Capturing Relightable Human Performances under General Uncontrolled Illumination. Comput. Graph. Forum 32, 2 (2013), 275–284.Google ScholarCross Ref
29. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. 36, 4 (2017), 45:1–45:11. Google ScholarDigital Library
30. Jorge Lopez-Moreno, Elena Garces, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2013. Multiple Light Source Estimation in a Single Image. Comput. Graph. Forum 32, 8 (2013), 170–182.Google ScholarCross Ref
31. Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, and Rui Wang. 2017. 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. In 2017 International Conference on 3D Vision (3DV 2017).Google ScholarCross Ref
32. Takuya Narihira, Michael Maire, and Stella X. Yu. 2015. Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression. In 2015 IEEE International Conference on Computer Vision (ICCV 2015). 2992. Google ScholarDigital Library
33. Geoffrey Oxholm and Ko Nishino. 2012. Shape and Reflectance from Natural Illumination. In 12th European Conference on Computer Vision (ECCV 2012), Proceedings, Part I. 528–541. Google ScholarDigital Library
34. Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS 2009). 296–301. Google ScholarDigital Library
35. Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001. 497–500. Google ScholarDigital Library
36. Andreas Schneider, Sandro Schönborn, Bernhard Egger, Lavrenti Frobeen, and Thomas Vetter. 2017. Efficient Global Illumination for Morphable Models. In IEEE International Conference on Computer Vision (ICCV 2017). 3885–3893.Google Scholar
37. Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, and David W. Jacobs. 2018. SfSNet: Learning Shape, Reflectance and Illuminance of Faces ‘in the Wild’. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018).Google Scholar
38. Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. 2017. Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5844–5853.Google Scholar
39. Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017a. Portrait Lighting Transfer Using a Mass Transport Approach. ACM Trans. Graph. 36, 4, Article 145a (Oct. 2017).Google ScholarDigital Library
40. Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017b. Neural Face Editing with Intrinsic Image Disentangling. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5444–5453.Google Scholar
41. Peter-Pike J. Sloan, Jan Kautz, and John Snyder. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Trans. Graph. 21, 3 (2002), 527–536. Google ScholarDigital Library
42. Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, and Christian Theobalt. 2017. MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In IEEE International Conference on Computer Vision (ICCV 2017). 3735–3744.Google Scholar
43. Su Xue, Aseem Agarwala, Julie Dorsey, and Holly E. Rushmeier. 2012. Understanding and improving the realism of image composites. ACM Trans. Graph. 31, 4 (2012), 84:1–84:10. Google ScholarDigital Library
44. Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, and Hao Li. 2018. High-fidelity Facial Reflectance and Geometry Inference from an Unconstrained Image. ACM Trans. Graph. 37, 4, Article 162 (July 2018), 162:1–162:14 pages. Google ScholarDigital Library
45. Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll. 2017. Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 5484–5493.Google ScholarCross Ref
46. Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment Modeling from a Single Image. Comput. Graph. Forum 32, 7 (2013), 85–91.Google ScholarCross Ref
47. Sergey Zhukov, Andrei Iones, and Grigorij Kronin. 1998. An Ambient Light Illumination Model. In Rendering Techniques ’98, Proceedings of the Eurographics Workshop. 45–56.Google Scholar
