“Sparse ellipsometry: portable acquisition of polarimetric SVBRDF and shape with unstructured flash photography” by Hwang, Jeon, Munoz, Gutierrez, Tong, et al. …

Ellipsometry techniques allow to measure polarization information of materials, requiring precise rotations of optical components with different configurations of lights and sensors. This results in cumbersome capture devices, carefully calibrated in lab conditions, and in very long acquisition times, usually in the order of a few days per object. Recent techniques allow to capture polarimetric spatially-varying reflectance information, but limited to a single view, or to cover all view directions, but limited to spherical objects made of a single homogeneous material. We present sparse ellipsometry, a portable polarimetric acquisition method that captures both polarimetric SVBRDF and 3D shape simultaneously. Our handheld device consists of off-the-shelf, fixed optical components. Instead of days, the total acquisition time varies between twenty and thirty minutes per object. We develop a complete polarimetric SVBRDF model that includes diffuse and specular components, as well as single scattering, and devise a novel polarimetric inverse rendering algorithm with data augmentation of specular reflection samples via generative modeling. Our results show a strong agreement with a recent ground-truth dataset of captured polarimetric BRDFs of real-world objects.

1. Neil Alldrin, Todd Zickler, and David Kriegman. 2008. Photometric stereo with non-parametric and spatially-varying reflectance. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) 2008. 1–8.Google ScholarCross Ref
2. Gary Atkinson and E.R. Hancock. 2006. Recovery of surface orientation from diffuse polarization. IEEE Transactions on Image Processing 15, 6 (jun 2006), 1653–1664. Google ScholarDigital Library
3. RMA Azzam. 1978. Photopolarimetric measurement of the Mueller matrix by Fourier analysis of a single detected signal. Optics Letters 2, 6 (1978), 148–150.Google ScholarCross Ref
4. RMA Azzam. 2016. Stokes-vector and Mueller-matrix polarimetry. JOSA A 33, 7 (2016), 1396–1408.Google ScholarCross Ref
5. Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, Rui Chen, Yiqin Wang, Lei Yan, Boxin Shi, and Achuta Kadambi. 2020. Deep Shape from Polarization. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12369 LNCS (2020), 554–571.Google ScholarDigital Library
6. Seung-Hwan Baek and Felix Heide. 2021. Polarimetric Spatio-Temporal Light Transport Probing. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 40, 6 (2021).Google Scholar
7. Seung-Hwan Baek, Daniel S. Jeon, Xin Tong, and Min H. Kim. 2018. Simultaneous Acquisition of Polarimetric SVBRDF and Normals. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2018) 36, 6 (2018), 268:1–15. Google ScholarDigital Library
8. Seung-Hwan Baek, Tizian Zeltner, Hyun Jin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, and Min H. Kim. 2020. Image-Based Acquisition and Modeling of Polarimetric Reflectance. ACM Transactions on Graphics (Proc. SIGGRAPH 2020) 39, 4 (2020).Google Scholar
9. Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12684–12694.Google ScholarCross Ref
10. Guojun Chen, Yue Dong, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Reflectance scanning: estimating shading frame and BRDF with generalized linear light sources. ACM Transactions on Graphics (TOG) 33, 4 (2014), 117:1–11.Google ScholarDigital Library
11. Edward Collett. 2005. Field guide to polarization. Spie Bellingham, WA.Google Scholar
12. Robert L. Cook and Kenneth E. Torrance. 1982. A Reflectance Model for Computer Graphics. ACM Transactions on Graphics (TOG) 1, 1 (1982), 7–24. Google ScholarDigital Library
13. Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, and Jan Kautz. 2017. Polarimetric Multi-View Stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1558–1567.Google ScholarCross Ref
14. Zhaopeng Cui, Viktor Larsson, and Marc Pollefeys. 2019. Polarimetric relative pose estimation. Proceedings of the IEEE International Conference on Computer Vision 2019-October (2019), 2671–2680. Google ScholarCross Ref
15. Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. ACM SIGGRAPH ’97. 369–378.Google Scholar
16. Fernando del Molino and Adolfo Muñoz. 2019. Polarization mapping. Computers & Graphics 83 (2019), 42 — 50. Google ScholarDigital Library
17. Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. 2021. Deep polarization imaging for 3D shape and SVBRDF acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref
18. Yuqi Ding, Yu Ji, Mingyuan Zhou, Sing Bing Kang, and Jinwei Ye. 2021. Polarimetric Helmholtz Stereopsis. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 5017–5026. Google ScholarCross Ref
19. Kenneth K Ellis. 1996. Polarimetric bidirectional reflectance distribution function of glossy coatings. JOSA A 13, 8 (1996), 1758–1762.Google ScholarCross Ref
20. Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino. 2021. Polarimetric Normal Stereo. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 682–690. Google ScholarCross Ref
21. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. 2010. Circularly polarized spherical illumination reflectometry. ACM Transactions on Graphics 29, 6 (2010). Google ScholarDigital Library
22. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. Proceedings of the 2011 SIGGRAPH Asia Conference on – SA ’11 30, 6 (2011), 1. Google ScholarDigital Library
23. Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical Modeling and Acquisition of Layered Facial Reflectance. ACM Trans. Graph. 27, 5, Article 139 (dec 2008), 10 pages. Google ScholarDigital Library
24. Paulo Gotardo, Jérémy Riviere, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2018. Practical Dynamic Facial Appearance Modeling and Acquisition. ACM Trans. Graph. 37, 6, Article 232 (dec 2018), 13 pages. Google ScholarDigital Library
25. Pat Hanrahan and Wolfgang Krueger. 1993. Reflection from layered surfaces due to subsurface scattering. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques. 165–174.Google ScholarDigital Library
26. Eric Heitz. 2014. Understanding the Masking-Shadowing Function in Microfacet-Based BRDFs. J. Comput. Graph. Techn. 3, 2 (2014), 48–107.Google Scholar
27. Louis G Henyey and Jesse Leonard Greenstein. 1941. Diffuse radiation in the galaxy. The Astrophysical Journal 93 (1941), 70–83.Google ScholarCross Ref
28. Cong Phuoc Huynh, Antonio Robles-Kelly, and Edwin R. Hancock. 2013. Shape and refractive index from single-view spectro-polarimetric images. Vol. 101. 64–94 pages. Google ScholarDigital Library
29. MW Hyde IV, JD Schmidt, and MJ Havrilla. 2009. A geometrical optics polarimetric bidirectional reflectance distribution function for dielectric and metallic surfaces. Optics express 17, 24 (2009), 22138–22153.Google Scholar
30. Adrian Jarabo and Victor Arellano. 2017. Bidirectional Rendering of Vector Light Transport. Computer Graphics Forum (12 2017).Google Scholar
31. Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. 2015. Polarized 3D: High-Quality Depth Sensing with Polarization Cues. In Proc. ICCV. IEEE Computer Society, 3370–3378.Google ScholarDigital Library
32. Christos Kampouris, Stefanos Zafeiriou, and Abhijeet Ghosh. 2018. Diffuse-Specular Separation using Binary Spherical Gradient Illumination.. In EGSR (EI&I). 1–10.Google Scholar
33. Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM Transactions on Graphics (TOG) 32, 3 (2013), 29.Google ScholarDigital Library
34. Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Transactions on Graphics (TOG) 25, 3 (2006), 735–745.Google ScholarDigital Library
35. Joo Ho Lee, Adrian Jarabo, Daniel S. Jeon, Diego Gutierrez, and Min H. Kim. 2018. Practical Multiple Scattering for Rough Surfaces. ACM Transactions on Graphics (TOG) 37, 6 (2018).Google ScholarDigital Library
36. Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen. 2021. Shape from Polarization for Complex Scenes in the Wild. (2021). arXiv:2112.11377 http://arxiv.org/abs/2112.11377Google Scholar
37. Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. 2003. Image-based Reconstruction of Spatial Appearance and Geometric Detail. ACM Transactions on Graphics (TOG) 22, 2 (2003), 234–257.Google ScholarDigital Library
38. Fei Liu, Lei Cao, Xiaopeng Shao, Pingli Han, and Xiangli Bin. 2015. Polarimetric dehazing utilizing spatial frequency segregation of images. Appl. Opt. 54, 27 (Sep 2015), 8116–8122. Google ScholarCross Ref
39. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (Grenoble, France) (EGSR’07). Eurographics Association, Goslar, DEU, 183–194.Google ScholarDigital Library
40. Xiaohe Ma, Kaizhang Kang, Ruisheng Zhu, Hongzhi Wu, and Kun Zhou. 2021. Free-Form Scanning of Non-Planar Appearance with Neural Trace Photography. ACM Trans. Graph. 40, 4, Article 124 (jul 2021), 13 pages. Google ScholarDigital Library
41. JR Maxwell, J Beard, S Weiner, Da Ladd, and S Ladd. 1973. Bidirectional Reflectance Model Validation and Utilization. Technical Report. Environmental Research Inst Of Michiganann Arbor Infrared And Optics Div.Google Scholar
42. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.Google Scholar
43. Daisuke Miyazaki, Robby T. Tan, Kenji Hara, and Katsushi Ikeuchi. 2003. Polarization-based inverse rendering from a single view. Proceedings of the IEEE International Conference on Computer Vision 2, Iccv (2003), 982–987. Google ScholarCross Ref
44. Michal Mojzik, Tomas Skrivan, Alexander Wilkie, and Jaroslav Krivanek. 2016. Bi-Directional Polarised Light Transport. In Eurographics Symposium on Rendering.Google Scholar
45. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H. Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Transactions on Graphics 37, 6 (2018), 1–12. Google ScholarDigital Library
46. Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H. Kim. 2016. Simultaneous Acquisition of Microscale Reflectance and Normals. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2016) 35, 6 (2016). Google ScholarDigital Library
47. Geoffrey Oxholm and Ko Nishino. 2014. Multiview shape and reflectance from natural illumination. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2014), 2163–2170. Google ScholarDigital Library
48. Richard G Priest and Thomas A Gerner. 2000. Polarimetric BRDF in the microfacet model: Theory and measurements. Technical Report. NAVAL RESEARCH LAB WASHINGTON DC.Google Scholar
49. Jérémy Riviere, Paulo Gotardo, Derek Bradley, Abhijeet Ghosh, and Thabo Beeler. 2020. Single-Shot High-Quality Facial Geometry and Skin Appearance Capture. 39, 4 (2020). Google ScholarDigital Library
50. Jérémy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics 36, 6 (2017). Google ScholarDigital Library
51. Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision. Springer, 501–518.Google ScholarCross Ref
52. Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. 2021. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In CVPR.Google Scholar
53. Yinlong Sun. 2007. Statistical ray method for deriving reflection models of rough surfaces. JOSA A 24, 3 (2007), 724–744.Google ScholarCross Ref
54. Kenneth E Torrance and Ephraim M Sparrow. 1967. Theory for off-specular reflection from roughened surfaces. JOSA 57, 9 (1967), 1105–1112.Google ScholarCross Ref
55. Silvia Tozza, William A.P. Smith, Dizhong Zhu, Ravi Ramamoorthi, and Edwin R. Hancock. 2017. Linear Differential Constraints for Photo-Polarimetric Height Estimation. Proceedings of the IEEE International Conference on Computer Vision 2017-October (2017), 2298–2306. Google ScholarCross Ref
56. Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring Reflectance and Shape from Continuous Spherical Harmonic Illumination. ACM Trans. Graph. 32, 4, Article 109 (jul 2013), 12 pages. Google ScholarDigital Library
57. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Eurographics conference on Rendering Techniques. Eurographics Association, 195–206.Google ScholarDigital Library
58. Alexander Wilkie and Andrea Weidlich. 2010. A standardised polarisation visualisation for images. In Proceedings of the 26th Spring Conference on Computer Graphics. 43–50.Google ScholarDigital Library
59. Alexander Wilkie and Andrea Weidlich. 2012. Polarised light in computer graphics. In SIGGRAPH Asia 2012 Courses. 1–87.Google Scholar
60. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera. IEEE Transactions on Visualization and Computer Graphics 22, 8 (2016), 2012–2023. Google ScholarDigital Library
61. Hongzhi Wu and Kun Zhou. 2015. Appfusion: Interactive appearance acquisition using a kinect sensor. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 289–298.Google Scholar
62. Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. 2016. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics 35, 6 (2016). Google ScholarDigital Library
63. Fanchao Yang, Xingjia Tang, Bingliang Hu, Ruyi Wei, Liang Kong, and Yong Li. 2016. A Method of Removing Reflected Highlight on Images Based on Polarimetric Imaging. Journal of Sensors 2016 (2016).Google Scholar
64. Z. Y. Zhang. 2000. A Flexible New Technique for Camera Calibration. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 11 (Nov. 2000), 1330–1334.Google ScholarDigital Library
65. Jinyu Zhao, Yusuke Monno, and Masatoshi Okutomi. 2020. Polarimetric Multi-view Inverse Rendering. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12369 LNCS (2020), 85–102. Google ScholarDigital Library
66. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-possible SVBRDF acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 189.Google ScholarDigital Library
67. Zhenglong Zhou, Zhe Wu, and Ping Tan. 2013. Multi-view photometric stereo with spatially varying isotropic materials. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2013), 1482–1489. Google ScholarDigital Library
68. Dizhong Zhu and William AP Smith. 2019. Depth from a polarisation + RGB stereo pair. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Google ScholarCross Ref