“Practical SVBRDF acquisition of 3D objects with unstructured flash photography”
Conference:
Type(s):
Title:
- Practical SVBRDF acquisition of 3D objects with unstructured flash photography
Session/Category Title: Advanced SVBRDF
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both.
References:
1. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF Capture in the Frequency Domain. ACM Transactions on Graphics (TOG) 32, 4, Article 110 (2013), 12 pages. Google ScholarDigital Library
2. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF capture for stationary materials. ACM Transactions on Graphics (TOG) 34, 4 (2015), 110:1–13. Google ScholarDigital Library
3. Neil Alldrin, Todd Zickler, and David Kriegman. 2008. Photometric stereo with non-parametric and spatially-varying reflectance. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2008. 1–8.Google ScholarCross Ref
4. Michael Ashikhmin, Simon Premoze, and Peter Shirley. 2000. A microfacet-based BRDF generator. In Proc. ACM SIGGRAPH 2000. 65–74. Google ScholarDigital Library
5. Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, and Min H. Kim. 2018. Simultaneous Acquisition of Polarimetric SVBRDF and Normals. ACM Transactions on Graphics (TOG) 37, 6 (2018). Google ScholarDigital Library
6. Mahdi M Bagher, John Snyder, and Derek Nowrouzezahrai. 2016. A Non-Parametric Factor Microfacet Model for Isotropic BRDFs. ACM Transactions on Graphics (TOG) 35, 5 (2016), 159. Google ScholarDigital Library
7. Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance scanning: estimating shading frame and BRDF with generalized linear light sources. ACM Transactions on Graphics (TOG) 33, 4 (2014), 117:1–11. Google ScholarDigital Library
8. Paolo Cignoni, Claudio Rocchini, and Roberto Scopigno. 1998. Metro: Measuring error on simplified surfaces. In Computer Graphics Forum, Vol. 17. Wiley Online Library, 167–174.Google Scholar
9. Robert L. Cook and Kenneth E. Torrance. 1982. A Reflectance Model for Computer Graphics. ACM Transactions on Graphics (TOG) 1, 1 (1982), 7–24. Google ScholarDigital Library
10. Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. ACM SIGGRAPH ’97. 369–378. Google ScholarDigital Library
11. Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold bootstrapping for SVBRDF capture. ACM Transactions on Graphics (TOG) 29, 4 (2010), 98. Google ScholarDigital Library
12. Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear Light Source Reflectometry. ACM Transactions on Graphics (TOG) 22, 3 (2003), 749–758. Google ScholarDigital Library
13. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2010. Circularly Polarized Spherical Illumination Reflectometry. ACM Transactions on Graphics (TOG) (Dec. 2010). Google ScholarDigital Library
14. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011), 129. Google ScholarDigital Library
15. Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical modeling and acquisition of layered facial reflectance. ACM Transactions on Graphics (TOG) 27, 5 (2008), 139. Google ScholarDigital Library
16. Paul Graham, Borom Tunwattanapong, Jay Busch, Xueming Yu, Andrew Jones, Paul Debevec, and Abhijeet Ghosh. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum 32 (2013), 335–344.Google ScholarCross Ref
17. Darya Guarnera, Giuseppe Guarnera, Abhijeet Ghosh, Cornelia Denk, and Mashhuda Glencross. 2016. BRDF Representation and Acquisition. Computer Graphics Forum 35 (2016), 625–650.Google ScholarCross Ref
18. Carlos Hernandez, George Vogiatzis, and Roberto Cipolla. 2008. Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 3 (2008), 548–554. Google ScholarDigital Library
19. Tomoaki Higo, Yasuyuki Matsushita, Neel Joshi, and Katsushi Ikeuchi. 2009. A handheld photometric stereo camera for 3-d modeling. In Computer Vision, 2009 IEEE 12th International Conference on. IEEE, 1234–1241.Google ScholarCross Ref
20. Michael Holroyd, Jason Lawrence, Greg Humphreys, and Todd Zickler. 2008. A photometric approach for estimating normals and tangents. ACM Transactions on Graphics (TOG) 27, 5 (2008), 133. Google ScholarDigital Library
21. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Transactions on Graphics (TOG) 29, 4 (2010), 99. Google ScholarDigital Library
22. Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5362–5370.Google ScholarCross Ref
23. Olaf Kahler, Victor Adrian Prisacariu, Carl Yuheng Ren, Xin Sun, Philip Torr, and David Murray. 2015. Very high frame rate volumetric integration of depth images on mobile devices. IEEE Transactions on Visualization and Computer Graphics 21, 11 (2015), 1241–1250. Google ScholarDigital Library
24. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013), 29. Google ScholarDigital Library
25. Kalin Kolev, Petri Tanskanen, Pablo Speciale, and Marc Pollefeys. 2014. Turning Mobile Phones into 3D Scanners. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Google ScholarDigital Library
26. Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics (TOG) 25, 3 (2006), 735–745. Google ScholarDigital Library
27. Joo Ho Lee, Adrian Jarabo, Daniel S. Jeon, Diego Gutierrez, and Min H. Kim. 2018. Practical Multiple Scattering for Rough Surfaces. ACM Transactions on Graphics (TOG) 37, 6 (2018).Google ScholarDigital Library
28. Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2001. Image-based reconstruction of spatially varying materials. In Rendering Techniques 2001. Springer, 103–114. Google ScholarDigital Library
29. Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Transactions on Graphics (TOG) 22, 2 (2003), 234–257. Google ScholarDigital Library
30. Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J Gortler, and Leonard McMillan. 2000. Image-based Visual Hulls. In Proc. ACM SIGGRAPH ’00. 369–374. Google ScholarDigital Library
31. Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Transactions on Graphics 22, 3 (July 2003), 759–769. Google ScholarDigital Library
32. NAG. 2015. The NAG Library, Numerical Algorithms Group. http://www.nag.com/.Google Scholar
33. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbic, Hao Li, Abhijeet Ghosh, and Paul E Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (TOG) 34, 4 (2015), 109–1. Google ScholarDigital Library
34. Giljoo Nam and Min H. Kim. 2014. Multispectral Photometric Stereo for Acquiring High-Fidelity Surface Normals. IEEE Computer Graphics and Applications 34, 6 (2014), 57–68.Google ScholarCross Ref
35. Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H. Kim. 2016. Simultaneous Acquisition of Microscale Reflectance and Normals. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2016) 35, 6 (2016). Google ScholarDigital Library
36. Diego Nehab, Szymon Rusinkiewicz, James Davis, and Ravi Ramamoorthi. 2005. Efficiently combining positions and normals for precise 3D geometry. ACM Trans. Graph 24, 3 (2005), 536–543. Google ScholarDigital Library
37. Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Proc. Int. Sym. Mixed and augmented reality (ISMAR). 127–136. Google ScholarDigital Library
38. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics 34, 6 (2015), 186. Google ScholarDigital Library
39. Peter Ondruska, Pushmeet Kohli, and Shahram Izadi. 2015. Mobilefusion: Real-time volumetric surface reconstruction and dense tracking on mobile phones. IEEE transactions on visualization and computer graphics 21, 11 (2015), 1251–1258. Google ScholarDigital Library
40. Geoffrey Oxholm and Ko Nishino. 2014. Multiview Shape and Reflectance from Natural Illumination. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2163–2170. Google ScholarDigital Library
41. Jaesik Park, Sudipta Sinha, Yasuyuki Matsushita, Yu Wing Tai, and In So Kweon. 2016. Robust Multiview Photometric Stereo using Planar Mesh Parameterization. IEEE Transactions on Pattern Analysis and Machine Intelligence (2016).Google Scholar
42. Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket reflectometry. ACM Transactions on Graphics (TOG) 30, 4 (2011), 45:1–10. Google ScholarDigital Library
43. Jérémy Riviere, Pieter Peers, and Abhijeet Ghosh. 2015. Mobile Surface Reflectometry. Computer Graphics Forum (2015). Google ScholarDigital Library
44. Jeremy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 206. Google ScholarDigital Library
45. Szymon Rusinkiewicz. 1998. A New Change of Variables for Efficient BRDF Representation. In Rendering Techniques ’98 (Proceedings of Eurographics Rendering Workshop ’98), G. Drettakis and N. Max (Eds.). Springer Wien, 11–22.Google ScholarCross Ref
46. Johannes L. Schönberger. 2016. COLMAP. https://colmap.github.io.Google Scholar
47. Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision. Springer, 501–518.Google ScholarCross Ref
48. Thomas Schöps, Torsten Sattler, Christian Häne, and Marc Pollefeys. 2015. 3d modeling on the go: Interactive 3d reconstruction of large-scale scenes on mobile devices. In 3D Vision (3DV), 2015 International Conference on. IEEE, 291–299. Google ScholarDigital Library
49. Christopher Schwartz, Ralf Sarlette, Michael Weinmann, and Reinhard Klein. 2013. Dome ii: A parallelized btf acquisition system. In Proceedings of the Eurographics 2013 Workshop on Material Appearance Modeling: Issues and Acquisition. Eurographics Association, 25–31. Google ScholarDigital Library
50. Niranjan Thanikachalam, Loïc Baboulaz, Damien Firmenich, Sabine Süsstrunk, and Martin Vetterli. 2017. Handheld Reflectance Acquisition of Paintings. IEEE Transactions on Computational Imaging 3, 4 (2017), 580–591.Google ScholarCross Ref
51. Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Transactions on Graphics (TOG) 32, 4 (2013), 109:1–12. Google ScholarDigital Library
52. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proc. Eurographics. 195–206. Google ScholarDigital Library
53. Jiaping Wang, Shuang Zhao, Xin Tong, John Snyder, and Baining Guo. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Transactions on Graphics (TOG) 27, 3 (2008), 41:1–10. Google ScholarDigital Library
54. Michael Weinmann and Reinhard Klein. 2015. Advances in Geometry and Reflectance Acquisition (Course Notes). In SIGGRAPH Asia 2015 Courses. ACM, Article 1, 71 pages. Google ScholarDigital Library
55. Tim Weyrich, Jason Lawrence, Hendrik P.A. Lensch, Szymon Rusinkiewicz, and Todd Zickler. 2009. Principles of Appearance Acquisition and Representation. Foundations and Trends in Computer Graphics and Vision 4, 2 (2009), 75–191. Google ScholarDigital Library
56. Jae Hyun Won, Man Hee Lee, and In Kyu Park. 2012. Active 3D shape acquisition using smartphones. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on. IEEE, 29–34.Google ScholarCross Ref
57. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE transactions on visualization and computer graphics 22, 8 (2016), 2012–2023. Google ScholarDigital Library
58. Hongzhi Wu and Kun Zhou. 2015. Appfusion: Interactive appearance acquisition using a kinect sensor. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 289–298. Google ScholarDigital Library
59. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering Shape and Spatially-varying Surface Reflectance Under Unknown Illumination. ACM Transactions on Graphics (TOG) 35, 6, Article 187 (2016), 12 pages. Google ScholarDigital Library
60. Z. Y. Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 11 (Nov. 2000), 1330–1334. Google ScholarDigital Library
61. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-possible SVBRDF acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 189. Google ScholarDigital Library
62. Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view Photometric Stereo with Spatially Varying Isotropic Materials. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2013. 1482–1489. Google ScholarDigital Library
63. Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2015. Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics (TOG) 34, 4 (2015), 96. Google ScholarDigital Library
