“NeRFactor: neural factorization of shape and reflectance under an unknown illumination” by Zhang, Srinivasan, Deng, Debevec, Freeman, et al. …
Conference:
Type(s):
Title:
- NeRFactor: neural factorization of shape and reflectance under an unknown illumination
Session/Category Title: Neural Rendering
Presenter(s)/Author(s):
Abstract:
We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object’s material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.
References:
1. Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-Scale Data for Multiple-View Stereopsis. International Journal of Computer Vision (IJCV) 120, 2 (2016), 153–168.
2. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A System for Large-Scale Machine Learning. In USENIX Symposium on Operating Systems Design and Implementation (OSDI).
3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-Shot SVBRDF Capture for Stationary Materials. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–13.
4. Jonathan T Barron and Jitendra Malik. 2014. Shape, Illumination, and Reflectance From Shading. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 37, 8 (2014), 1670–1687.
5. Sai Bi, Zexiang Xu, Pratul P Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. 2020. Neural Reflectance Fields for Appearance Acquisition. arXiv (2020).
6. Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. 2018. Optimizing the Latent Space of Generative Networks. In International Conference on Machine Learning (ICML).
7. Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021. NeRD: Neural Reflectance Decomposition From Image Collections. In IEEE/CVF International Conference on Computer Vision (ICCV).
8. Gershon Buchsbaum. 1980. A Spatial Processor Model for Object Colour Perception. Journal of the Franklin Institute 310, 1 (1980), 1–26.
9. Zhang Chen, Anpei Chen, Guli Zhang, Chengyuan Wang, Yu Ji, Kiriakos N Kutulakos, and Jingyi Yu. 2020. A Neural Rendering Framework for Free-Viewpoint Relighting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
10. Paul Debevec. 1998. Rendering Synthetic Objects Into Real Scenes: Bridging Traditional and Image-Based Graphics With Global Illumination and High Dynamic Range Photography. In SIGGRAPH.
11. Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-From-Motion: Recovering Spatially Varying Surface Reflectance Under Unknown Lighting. ACM Transactions on Graphics (TOG) 33, 6 (2014), 1–12.
12. Sing Choong Foo. 2015. A Gonioreflectometer for Measuring the Bidirectional Reflectance of Material for Use in Illumination Computation. Master’s thesis. Cornell University.
13. Yasutaka Furukawa and Jean Ponce. 2009. Accurate, Dense, and Robust Multiview Stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 32, 8 (2009), 1362–1376.
14. Duan Gao, Guojun Chen, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2020. Deferred Neural Lighting: Free-Viewpoint Relighting From Unstructured Photographs. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.
15. Stamatios Georgoulis, Vincent Vanweddingen, Marc Proesmans, and Luc Van Gool. 2015. A Gaussian Process Latent Variable Model for BRDF Inference. In IEEE/CVF International Conference on Computer Vision (ICCV).
16. Clement Godard, Peter Hedman, Wenbin Li, and Gabriel J Brostow. 2015. Multi-View Reconstruction of Highly Specular Surfaces in Uncontrolled Environments. In International Conference on 3D Vision (3DV).
17. Purvi Goel, Loudon Cohen, James Guesman, Vikas Thamizharasan, James Tompkin, and Daniel Ritchie. 2020. Shape From Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material From Images via Differentiable Path Tracing. In International Conference on 3D Vision (3DV).
18. Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, et al. 2019. The Relightables: Volumetric Performance Capture of Humans With Realistic Relighting. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–19.
19. Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs. In IEEE/CVF International Conference on Computer Vision (ICCV).
20. Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. 2014. Large Scale Multi-View Stereopsis Evaluation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
21. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Symposium on Geometry Processing (SGP).
22. Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR).
23. Pierre-Yves Laffont, Adrien Bousseau, and George Drettakis. 2012. Rich Intrinsic Image Decomposition of Outdoor Scenes From Multiple Views. IEEE Transactions on Visualization and Computer Graphics (TVCG) 19, 2 (2012), 210–224.
24. Edwin H Land and John J McCann. 1971. Lightness and Retinex Theory. Journal of the Optical Society of America 61, 1 (1971), 1–11.
25. Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-Based Reconstruction of Spatial Appearance and Geometric Detail. ACM Transactions on Graphics (TOG) 22, 2 (2003), 234–257.
26. Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2020. Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
27. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to Reconstruct Shape and Spatially-Varying Reflectance From a Single Image. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–11.
28. Daniel Lichy, Jiaye Wu, Soumyadip Sengupta, and David W Jacobs. 2021. Shape and Material Capture at Home. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
29. Stephen Lombardi and Ko Nishino. 2012. Reflectance and Natural Illumination From a Single Image. In European Conference on Computer Vision (ECCV).
30. Stephen R Marschner. 1998. Inverse Rendering for Computer Graphics. Cornell University.
31. Ricardo Martin-Brualla, Noha Radwan, Mehdi S M Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
32. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on Graphics (TOG) 22, 3 (2003), 759–769.
33. Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy Networks: Learning 3D Reconstruction in Function Space. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
34. Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In European Conference on Computer Vision (ECCV).
35. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF Acquisition of 3D Objects With Unstructured Flash Photography. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–12.
36. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On Optimal, Minimal Brdf Sampling for Reflectance Acquisition. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1–11.
37. Ko Nishino. 2009. Directional Statistics BRDF Model. In IEEE/CVF International Conference on Computer Vision (ICCV).
38. Ko Nishino and Stephen Lombardi. 2011. Directional Statistics-Based Reflectance Model for Isotropic Bidirectional Reflectance Distribution Functions. Journal of the Optical Society of America A 28, 1 (2011), 8–18.
39. Michael Oechsle, Songyou Peng, and Andreas Geiger. 2021. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In IEEE/CVF International Conference on Computer Vision (ICCV).
40. Geoffrey Oxholm and Ko Nishino. 2014. Multiview Shape and Reflectance From Natural Illumination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
41. Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
42. Jeong Joon Park, Aleksander Holynski, and Steve Seitz. 2020. Seeing the World in a Bag of Chips. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
43. Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. 2019. Multi-View Relighting Using a Geometry-Aware Network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.
44. Ravi Ramamoorthi and Pat Hanrahan. 2001. A Signal-Processing Framework for Inverse Rendering. In SIGGRAPH.
45. Szymon M Rusinkiewicz. 1998. A New Change of Variables for Efficient BRDF Representation. In Eurographics Symposium on Rendering Techniques (EGSR).
46. Shen Sang and Manmohan Chandraker. 2020. Single-Shot Neural Relighting and SVBRDF Estimation. In European Conference on Computer Vision (ECCV).
47. Imari Sato, Yoichi Sato, and Katsushi Ikeuchi. 2003. Illumination From Shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 25, 3 (2003), 290–300.
48. Yoichi Sato, Mark D Wheeler, and Katsushi Ikeuchi. 1997. Object Shape and Reflectance Modeling From Observation. In SIGGRAPH.
49. Carolin Schmitt, Simon Donne, Gernot Riegler, Vladlen Koltun, and Andreas Geiger. 2020. On Joint Estimation of Pose, Geometry and SVBRDF From a Handheld Scanner. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
50. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-From-Motion Revisited. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
51. Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. 2019. Neural Inverse Rendering of an Indoor Scene From a Single Image. In IEEE/CVF International Conference on Computer Vision (ICCV).
52. Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. 2021. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
53. Jessi Stumpfel, Chris Tchou, Andrew Jones, Tim Hawkins, Andreas Wenger, and Paul Debevec. 2004. Direct HDR Capture of the Sun and Sky. In AFRIGRAPH.
54. Tiancheng Sun, Henrik Wann Jensen, and Ravi Ramamoorthi. 2018. Connecting Measured BRDFs to Analytic BRDFs by Data-Driven Diffuse-Specular Separation. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–15.
55. Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. 2020. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Advances in Neural Information Processing Systems (NeurIPS).
56. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet Models for Refraction Through Rough Surfaces. In Eurographics Symposium on Rendering Techniques (EGSR).
57. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing (TIP) 13, 4 (2004), 600–612.
58. Greg Ward and Rob Shakespeare. 1998. Rendering With Radiance: The Art and Science of Lighting Visualization. Morgan Kaufmann Publishers.
59. Xin Wei, Guojun Chen, Yue Dong, Stephen Lin, and Xin Tong. 2020. Object-Based Illumination Estimation With Rendering-Aware Neural Networks. In European Conference on Computer Vision (ECCV).
60. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-Varying Surface Reflectance Under Unknown Illumination. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–12.
61. Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. 2020. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Advances in Neural Information Processing Systems (NeurIPS).
62. Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999. Inverse Global Illumination: Recovering Reflectance Models of Real Scenes From Photographs. In SIGGRAPH.
63. Ye Yu and William A P Smith. 2019. InverseRenderNet: Learning Single Image Inverse Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
64. Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. 2021b. PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
65. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
66. Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T Barron, Ravi Ramamoorthi, and William T Freeman. 2021a. Neural Light Transport for Relighting and View Synthesis. ACM Transactions on Graphics (TOG) 40, 1 (2021), 1–17.
