“Learning efficient illumination multiplexing for joint capture of reflectance and shape” by Kang, Xie, He, Yi, Gu, et al. …
Conference:
Type(s):
Title:
- Learning efficient illumination multiplexing for joint capture of reflectance and shape
Session/Category Title: Photography in the Field
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We propose a novel framework that automatically learns the lighting patterns for efficient, joint acquisition of unknown reflectance and shape. The core of our framework is a deep neural network, with a shared linear encoder that directly corresponds to the lighting patterns used in physical acquisition, as well as non-linear decoders that output per-pixel normal and diffuse / specular information from photographs. We exploit the diffuse and normal information from multiple views to reconstruct a detailed 3D shape, and then fit BRDF parameters to the diffuse / specular information, producing texture maps as reflectance results. We demonstrate the effectiveness of the framework with physical objects that vary considerably in reflectance and shape, acquired with as few as 16 ~ 32 lighting patterns that correspond to 7 ~ 15 seconds of per-view acquisition time. Our framework is useful for optimizing the efficiency in both novel and existing setups, as it can automatically adapt to various factors, including the geometry / the lighting layout of the device and the properties of appearance.
References:
1. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Trans. Graph. 32, 4, Article 110 (July 2013), 12 pages.Google ScholarDigital Library
2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF Capture for Stationary Materials. ACM Trans. Graph. 34, 4, Article 110 (July 2015), 13 pages.Google ScholarDigital Library
3. Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017. Patch-based Optimization for Image-based Texture Mapping. ACM Trans. Graph. 36, 4, Article 106 (July 2017), 11 pages.Google ScholarDigital Library
4. Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance Scanning: Estimating Shading Frame and BRDF with Generalized Linear Light Sources. ACM Trans. Graph. 33, 4, Article 117 (July 2014), 11 pages.Google ScholarDigital Library
5. Kristin J. Dana, Bram van Ginneken, Shree K. Nayar, and Jan J. Koenderink. 1999. Reflectance and Texture of Real-world Surfaces. ACM Trans. Graph. 18, 1 (Jan. 1999), 1–34.Google ScholarDigital Library
6. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image SVBRDF Capture with a Rendering-aware Deep Network. ACM Trans. Graph. 37, 4, Article 128 (July 2018), 15 pages.Google ScholarDigital Library
7. Yue Dong. 2019. Deep appearance modeling: A survey. Visual Informatics (2019).Google Scholar
8. Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold Bootstrapping for SVBRDF Capture. ACM Trans. Graph. 29, 4, Article 98 (July 2010), 10 pages.Google ScholarDigital Library
9. Mark Fiala. 2005. ARTag, a fiducial marker system using digital techniques. In CVPR.Google Scholar
10. Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3 (2003), 749–758.Google ScholarDigital Library
11. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2009. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. Computer Graphics Forum 28, 4 (2009), 1161–1170.Google ScholarDigital Library
12. Darya Guarnera, Giuseppe C. Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35, 2 (2016), 625–650.Google ScholarCross Ref
13. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A Coaxial Optical Scanner for Synchronous Acquisition of 3D Geometry and Surface Reflectance. ACM Trans. Graph. 29, 4, Article 99 (July 2010), 12 pages.Google ScholarDigital Library
14. Satoshi Ikehata. 2018. CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In ECCV.Google Scholar
15. James T. Kajiya. 1986. The Rendering Equation (SIGGRAPH ’86). 143–150.Google Scholar
16. Kaizhang Kang, Zimin Chen, Jiaping Wang, Kun Zhou, and Hongzhi Wu. 2018. Efficient Reflectance Capture Using an Autoencoder. ACM Trans. Graph. 37, 4, Article 127 (July 2018), 10 pages.Google ScholarDigital Library
17. Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson Surface Reconstruction. ACM Trans. Graph. 32, 3, Article 29 (July 2013), 13 pages.Google ScholarDigital Library
18. Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. 2017. End-To-End Learning of Geometry and Context for Deep Stereo Regression. In ICCV.Google Scholar
19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.Google Scholar
20. Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. 2009. Procedural Noise Using Sparse Gabor Convolution. ACM Trans. Graph. 28, 3, Article 54 (July 2009), 10 pages.Google ScholarDigital Library
21. Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse Shade Trees for Non-parametric Material Representation and Editing. ACM Trans. Graph. 25, 3 (July 2006), 735–745.Google ScholarDigital Library
22. Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Trans. Graph. 22, 2 (April 2003), 234–257.Google ScholarDigital Library
23. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling Surface Appearance from a Single Photograph Using Self-augmented Convolutional Neural Networks. ACM Trans. Graph. 36, 4, Article 45 (July 2017), 11 pages.Google ScholarDigital Library
24. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018. Learning to Reconstruct Shape and Spatially-varying Reflectance from a Single Image. ACM Trans. Graph. 37, 6, Article 269 (Dec. 2018), 11 pages.Google ScholarDigital Library
25. Stephen McAuley, Stephen Hill, Naty Hoffman, Yoshiharu Gotanda, Brian Smits, Brent Burley, and Adam Martinez. 2012. Practical Physically-based Shading in Film and Game Production. In ACM SIGGRAPH 2012 Courses. Article 10, 7 pages.Google Scholar
26. José Luis Morales and Jorge Nocedal. 2011. Remark on “Algorithm 778: L-BFGS-B: Fortran Subroutines for Large-scale Bound Constrained Optimization”. ACM Trans. Math. Softw. 38, 1, Article 7 (Dec. 2011), 4 pages.Google ScholarDigital Library
27. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. In SIGGRAPH Asia Technical Papers. 267.Google Scholar
28. Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. 2005. Efficiently Combining Positions and Normals for Precise 3D Geometry. ACM Trans. Graph. 24, 3 (July 2005), 536–543.Google ScholarDigital Library
29. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On Optimal, Minimal BRDF Sampling for Reflectance Acquisition. ACM Trans. Graph. 34, 6, Article 186 (Oct. 2015), 11 pages.Google ScholarDigital Library
30. Daniel Scharstein and Richard Szeliski. 2003. High-accuracy stereo depth maps using structured light. In CVPR.Google Scholar
31. Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In ECCV.Google Scholar
32. Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Trans. Graph. 32, 4, Article 109 (July 2013), 12 pages.Google ScholarDigital Library
33. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques (Proc. EGWR).Google ScholarDigital Library
34. Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition. In SIGGRAPH Asia Courses. Article 1, 71 pages.Google Scholar
35. Tim Weyrich, Jason Lawrence, Hendrik P. A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Found. Trends. Comput. Graph. Vis. 4, 2 (2009), 75–191.Google ScholarDigital Library
36. Robert J Woodham. 1980. Photometric method for determining surface orientation from multiple images. Optical engineering 19, 1 (1980), 191139.Google Scholar
37. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera. IEEE TVCG 22, 8 (Aug 2016), 2012–2023.Google Scholar
38. Shihao Wu, Hui Huang, Tiziano Portenier, Matan Sela, Daniel Cohen-Or, Ron Kimmel, and Matthias Zwicker. 2018. Specular-to-Diffuse Translation for Multi-View Reconstruction. In ECCV.Google Scholar
39. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-varying Surface Reflectance Under Unknown Illumination. ACM Trans. Graph. 35, 6, Article 187 (Nov. 2016), 12 pages.Google ScholarDigital Library
40. Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018. MVSNet: Depth Inference for Unstructured Multi-view Stereo. In ECCV.Google Scholar
41. Kun Zhou, John Synder, Baining Guo, and Heung-Yeung Shum. 2004. Iso-charts: Stretch-driven Mesh Parameterization Using Spectral Analysis. In SGP.Google ScholarDigital Library
42. Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view photometric stereo with spatially varying isotropic materials. In CVPR.Google Scholar
