“Deep image-based relighting from optimal sparse samples” by Xu, Sunkavalli, Hadap and Ramamoorthi
Conference:
Type:
Entry Number: 126
Title:
- Deep image-based relighting from optimal sparse samples
Session/Category Title: Learning for Rendering and Material Acquisition
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present an image-based relighting method that can synthesize scene appearance under novel, distant illumination from the visible hemisphere, from only five images captured under pre-defined directional lights. Our method uses a deep convolutional neural network to regress the relit image from these five images; this relighting network is trained on a large synthetic dataset comprised of procedurally generated shapes with real-world reflectances. We show that by combining a custom-designed sampling network with the relighting network, we can jointly learn both the optimal input light directions and the relighting function. We present an extensive evaluation of our network, including an empirical analysis of reconstruction quality, optimal lighting configurations for different scenarios, and alternative network architectures. We demonstrate, on both synthetic and real scenes, that our method is able to reproduce complex, high-frequency lighting effects like specularities and cast shadows, and outperforms other image-based relighting methods that require an order of magnitude more images.
References:
1. Aayush Bansal, Bryan Russell, and Abhinav Gupta. 2016. Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. CVPR (2016).Google Scholar
2. Ronen Basri and David W. Jacobs. 2003. Lambertian Reflectance and Linear Subspaces. IEEE Trans. Pattern Anal Mach. Intell. 25, 2 (Feb. 2003), 218–233. Google ScholarDigital Library
3. Peter N. Belhumeur and David J. Kriegman. 1998. What Is the Set of Images of an Object Under All Possible Illumination Conditions? International Journal of Computer Vision 28, 3 (01 Jul 1998), 245–260. Google ScholarDigital Library
4. Brett Burley. 2012. Physically-based shading at Disney. In ACM SIGGRAPH 2012 Courses.Google Scholar
5. Ayan Chakrabarti. 2016. Learning sensor multiplexing design through back-propagation. In Advances in Neural Information Processing Systems. 3081–3089. Google ScholarDigital Library
6. Manmohan Chandraker. 2016. The information available to a moving observer on shape with unknown, isotropic BRDFs. IEEE transactions on pattern analysis and machine intelligence 38, 7 (2016), 1283–1297.Google ScholarCross Ref
7. Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 145–156. Google ScholarDigital Library
8. David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. ICCV (2015). Google ScholarDigital Library
9. Per Einarsson, Charles-Felix Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond, Tim Hawkins, Mark T Bolas, Sebastian Sylwan, and Paul E Debevec. 2006. Relighting Human Locomotion with Flowed Reflectance Fields. Rendering techniques 2006 (2006), 17th. Google ScholarDigital Library
10. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World’s Imagery. In CVPR.Google Scholar
11. Martin Fuchs, Volker Blanz, Hendrik Lensch, and Hans-Peter Seidel. 2007. Adaptive sampling of reflectance fields. ACM Transactions on Graphics (TOG) 26, 2 (2007), 10. Google ScholarDigital Library
12. Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. 2017. Learning to Predict Indoor Illumination from a Single Image. ACM Transactions on Graphics (SIGGRAPH Asia) 9, 4 (2017). Google ScholarDigital Library
13. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Tinne Tuytelaars, and Luc Van Gool. 2017. What Is Around The Camera?. In ICCV.Google Scholar
14. Dan B Goldman, Brian Curless, Aaron Hertzmann, and Steven M Seitz. 2010. Shape and spatially-varying brdfs from photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 6 (2010), 1060–1071. Google ScholarDigital Library
15. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In CVPR.Google Scholar
16. Z. Hui and A. C. Sankaranarayanan. 2017. Shape and Spatially-Varying Reflectance Estimation from Virtual Exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 10 (Oct 2017), 2060–2073.Google ScholarDigital Library
17. Wenzel Jakob. 2010. Mitsuba renderer. (2010). http://www.mitsuba-renderer.org.Google Scholar
18. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 193. Google ScholarDigital Library
19. Eric P Lafortune and Yves D Willems. 1993. Bi-directional path tracing. (1993).Google Scholar
20. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Transactions on Graphics (TOG) 36, 4 (2017), 45. Google ScholarDigital Library
21. Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. 2017. Material Editing Using a Physically Based Rendering Network. In ICCV. 2261–2269.Google Scholar
22. Dhruv Mahajan, Ira Kemelmacher Shlizerman, Ravi Ramamoorthi, and Peter Belhumeur. 2007. A Theory of Locally Low Dimensional Light Transport. ACM Trans. Graph. 26, 3, Article 62 (July 2007). Google ScholarDigital Library
23. Tom Malzbender, Dan Gelb, and Hans Wolters. 2001. Polynomial Texture Maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). 519–528. Google ScholarDigital Library
24. Wojciech Matusik, Matthew Loper, and Hanspeter Pfister. 2004. Progressively-Refined Reflectance Functions from Natural Illumination. In Eurographics Workshop on Rendering, Alexander Keller and Henrik Wann Jensen (Eds.). Google ScholarDigital Library
25. Shree K. Nayar, Peter N. Belhumeur, and Terry E. Boult. 2004. Lighting Sensitive Display. ACM Trans. Graph. 23, 4 (Oct. 2004), 963–979. Google ScholarDigital Library
26. Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2003. All-frequency shadows using non-linear wavelet lighting approximation. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 376–381. Google ScholarDigital Library
27. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics (TOG) 34, 6(2015), 186. Google ScholarDigital Library
28. Geoffrey Oxholm and Ko Nishino. 2016. Shape and reflectance estimation in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 38, 2 (2016), 376–389. Google ScholarDigital Library
29. Pieter Peers and Philip Dutré. 2005. Inferring reflectance functions from wavelet noise. In Proceedings of the Sixteenth Eurographics conference on Rendering Techniques. Eurographics Association, 173–182. Google ScholarDigital Library
30. Pieter Peers, Dhruv K Mahajan, Bruce Lamond, Abhijeet Ghosh, Wojciech Matusik, Ravi Ramamoorthi, and Paul Debevec. 2009. Compressive light transport sensing. ACM Transactions on Graphics (TOG) 28, 1 (2009), 3. Google ScholarDigital Library
31. Ravi Ramamoorthi and Pat Hanrahan. 2001. On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object. J. Opt. Soc. Am. A 18, 10 (Oct 2001), 2448–2459.Google ScholarCross Ref
32. Dikpal Reddy, Ravi Ramamoorthi, and Brian Curless. 2012. Frequency-space Decomposition and Acquisition of Light Transport Under Spatially Varying Illumination. In Proceedings of the 12th European Conference on Computer Vision – Volume Part VI (ECCV’12). Springer-Verlag, Berlin, Heidelberg, 596–610. Google ScholarDigital Library
33. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep reflectance maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4508–4516.Google ScholarCross Ref
34. Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. 2015. Image based relighting using neural networks. ACM Transactions on Graphics (TOG) 34, 4 (2015), 111. Google ScholarDigital Library
35. O. Ronneberger, P.Fischer, and T. Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI) (LNCS), Vol. 9351. Springer, 234–241.Google Scholar
36. Christopher Schwartz, Michael Weinmann, Roland Ruiters, and Reinhard Klein. 2011. Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage. In VAST. Eurographics Association, 25–32. Google ScholarDigital Library
37. Amnon Shashua. 1997. On Photometric Issues in 3D Visual Recognition from a Single 2D Image. International Journal of Computer Vision 21, 1 (01 Jan 1997), 99–122. Google ScholarDigital Library
38. Peter-Pike Sloan, Jesse Hall, John Hart, and John Snyder. 2003. Clustered Principal Components for Precomputed Radiance Transfer. ACM Trans. Graph. 22, 3 (July 2003), 382–391. Google ScholarDigital Library
39. Kalyan Sunkavalli, Todd Zickler, and Hanspeter Pfister. 2010. Visibility Subspaces: Uncalibrated Photometric Stereo with Shadows. In ECCV. 251–264. Google ScholarDigital Library
40. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 195–206. Google ScholarDigital Library
41. Jiaping Wang, Yue Dong, Xin Tong, Zhouchen Lin, and Baining Guo. 2009. Kernel Nyström method for light transport. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 29. Google ScholarDigital Library
42. Robert J. Woodham. 1980. Photometric Method For Determining Surface Orientation From Multiple Images. Optical Engineering 19 (1980), 19 – 19 – 6.Google ScholarCross Ref
43. Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF sampling for two-shot near-field reflectance acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 188. Google ScholarDigital Library
