“Learning to predict indoor illumination from a single image”
Conference:
Type(s):
Title:
- Learning to predict indoor illumination from a single image
Session/Category Title: HDR and Image Manipulation
Presenter(s)/Author(s):
Abstract:
We propose an automatic method to infer high dynamic range illumination from a single, limited field-of-view, low dynamic range photograph of an indoor scene. In contrast to previous work that relies on specialized image capture, user input, and/or simple scene models, we train an end-to-end deep neural network that directly regresses a limited field-of-view photo to HDR illumination, without strong assumptions on scene geometry, material properties, or lighting. We show that this can be accomplished in a three step process: 1) we train a robust lighting classifier to automatically annotate the location of light sources in a large dataset of LDR environment maps, 2) we use these annotations to train a deep neural network that predicts the location of lights in a scene from a single limited field-of-view photo, and 3) we fine-tune this network using a small dataset of HDR environment maps to predict light intensities. This allows us to automatically recover high-quality HDR illumination estimates that significantly outperform previous state-of-the-art methods. Consequently, using our illumination estimates for applications like 3D object insertion, produces photo-realistic results that we validate via a perceptual user study.
References:
1. Aayush Bansal, Bryan Russell, and Abhinav Gupta. 2016. Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. IEEE Conference on Computer Vision and Pattern Recognition (2016). Cross Ref
2. Francesco Banterle, Marco Callieri, Matteo Dellepiane, Massimiliano Corsini, Fabio Pellacini, and Roberto Scopigno. 2013. EnvyDepth: An interface for recovering local natural illumination from environment maps. Computer Graphics Forum 32, 7 (2013), 411–420. Cross Ref
3. Jonathan Barron and Jitendra Malik. 2013a. Intrinsic Scene Properties from a Single RGB-D Image. IEEE Conference on Computer Vision and Pattern Recognition (2013).
4. Jonathan Barron and Jitendra Malik. 2013b. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 8 (2013), 1670–1687. Cross Ref
5. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. 2015. Material Recognition in the Wild with the Materials in Context Database. IEEE Conference on Computer Vision and Pattern Recognition (2015).
6. Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter Belhumeur, and Shree K. Nayar. 2008. Face Swapping: Automatically Replacing Faces in Photographs. ACM Transactions on Graphics 27, 3, Article 39 (Aug. 2008), 8 pages.
7. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2016. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In International Conference on Learning Representations.
8. Navneet Dalal and Bill Triggs. 2005. Histograms of Oriented Gradients for Human Detection. In IEEE Conference on Computer Vision and Pattern Recognition. 886–893.
9. Paul Debevec.1997. Recovering High Dynamic Range Radiance Maps from Photographs. In ACM SIGGRAPH 1997. 1–10.
10. Paul Debevec. 1998. Rendering Synthetic Objects into Real Scenes : Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography. In Proceedings of ACM SIGGRAPH.
11. Paul Debevec, Paul Graham, Jay Busch, and Mark Bolas. 2012. A Single-shot Light Probe. In ACM SIGGRAPH 2012 Talks. ACM, New York, NY, USA, 10:1–10:1.
12. David Eigen and Rob Fergus. 2015. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture. International Conference on Computer Vision (2015).
13. Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester, and Deva Ramanan. 2010. Object Detection with Discriminative Trained Part Based Models. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 9 (2010), 1627–1645.
14. Stamatios Georgoulis, Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Luc J. Van Gool, and Tinne Tuytelaars. 2016. DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination. CoRR abs/1603.08240 (2016).
15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition.
16. Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. 2017. Deep Outdoor Illumination Estimation. In IEEE Conference on Computer Vision and Pattern Recognition.
17. Lukáš Hošek and Alexander Wilkie. 2012. An analytic model for full spectral sky-dome radiance. ACM Transactions on Graphics 31, 4 (2012), 1–9. http://www.scopus.com/inward/record.url?eid=2-s2.0-84872254894
18. Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem. 2011. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics 30, 6 (2011), 1.
19. Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, and David Forsyth. 2014. Automatic Scene Inference for 3D Object Compositing. ACM Transactions on Graphics 3 (2014), 32:1–32:15.
20. Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff. 2006. Image-based material editing. ACM Transactions on Graphics 25, 3 (2006), 654.
21. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014).
22. Philipp Krähenbühl and Vladlen Koltun. 2012. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In Neural Information Processing Systems.
23. Jean-François Lalonde, Derek Hoiem, Alexei A. Efros, Carsten Rother, John Winn, and Antonio Criminisi. 2007. Photo Clip Art. ACM Transactions on Graphics 26, 3, Article 3 (July 2007).
24. Jean-François Lalonde, Srinivasa G. Narasimhan, and Alexei A. Efros. 2010. What do the sun and the sky tell us about the camera? International Journal on Computer Vision 88, 1 (May 2010), 24–51.
25. Stephen Lombardi and Ko Nishino. 2016. Reflectance and Illumination Recovery in the Wild. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 1 (2016), 129–141.
26. Jorge Lopez-Moreno, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2010. Compositing images through light source detection. Computers & Graphics 34, 6 (2010), 698–707.
27. Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-processing Framework for Inverse Rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). 117–128.
28. Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. 2010. High Dynamic Range Imaging (2 ed.). Morgan Kaufman.
29. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, Efstratios Gavves, and Tinne Tuytelaars. 2016. Deep Reflectance Maps. In IEEE Conference on Computer Vision and Pattern Recognition.
30. Levi Valgaerts, Chenglei Wu, Andrés Bruhn, Hans-Peter Seidel, and Christian Theobalt. 2012. Lightweight Binocular Facial Performance Capture Under Uncontrolled Lighting. ACM Transactions on Graphics 31, 6, Article 187 (Nov. 2012), 11 pages.
31. Chenglei Wu, B. Wilburn, Y. Matsushita, and C. Theobalt. 2011. High-quality Shape from Multi-view Stereo and Shading Under General Illumination. In IEEE Conference on Computer Vision and Pattern Recognition.
32. Jianxiong Xiao, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2012. Recognizing scene viewpoint using panoramic place representation. In IEEE Conference on Computer Vision and Pattern Recognition.
33. Edward Zhang, Michael F. Cohen, and Brian Curless. 2016. Emptying, Refurnishing, and Relighting Indoor Spaces. ACM Transactions on Graphics 35, 6 (2016).
34. Tinghui Zhou, Philipp Krähenbühl, and Alexei A. Efros. 2015. Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition. International Conference on Computer Vision (2015).
