“Polarimetric spatio-temporal light transport probing” by Baek and Heide
Conference:
Type(s):
Title:
- Polarimetric spatio-temporal light transport probing
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Abstract:
Light emitted from a source into a scene can undergo complex interactions with multiple scene surfaces of different material types before being reflected towards a detector. During this transport, every surface reflection and propagation is encoded in the properties of the photons that ultimately reach the detector, including travel time, direction, intensity, wavelength and polarization. Conventional imaging systems capture intensity by integrating over all other dimensions of the incident light into a single quantity, hiding this rich scene information in these aggregate measurements. Existing methods are capable of untangling these measurements into their spatial and temporal dimensions, fueling geometric scene understanding tasks. However, examining polarimetric material properties jointly with geometric properties is an open challenge that could enable unprecedented capabilities beyond geometric scene understanding, allowing for material-dependent scene understanding and imaging through complex transport, such as macroscopic scattering.In this work, we close this gap, and propose a computational light transport imaging method that captures the spatially- and temporally-resolved complete polarimetric response of a scene, which encodes rich material properties. Our method hinges on a novel 7D tensor theory of light transport. We discover low-rank structure in the polarimetric tensor dimension and propose a data-driven rotating ellipsometry method that learns to exploit redundancy of polarimetric structure. We instantiate our theory with two imaging prototypes: spatio-polarimetric imaging and coaxial temporal-polarimetric imaging. This allows us, for the first time, to decompose scene light transport into temporal, spatial, and complete polarimetric dimensions that unveil scene properties hidden to conventional methods. We validate the applicability of our method on diverse tasks, including shape reconstruction with subsurface scattering, seeing through scattering media, untangling multi-bounce light transport, breaking metamerism with polarization, and spatio-polarimetric decomposition of crystals.
References:
1. Edward H Adelson, James R Bergen, et al. 1991. The plenoptic function and the elements of early vision. Vol. 2. MIT.
2. Gianluca Agresti and Pietro Zanuttigh. 2018. Combination of spatially-modulated ToF and structured light for MPI-free depth estimation. In European Conference on Computer Vision (ECCV) Workshops. 0–0.
3. Hans Arwin. 2011. Application of ellipsometry techniques to biological materials. Thin Solid Films 519, 9 (2011), 2589–2592.
4. Gary A Atkinson and Edwin R Hancock. 2006. Recovery of surface orientation from diffuse polarization. IEEE Transactions on Image Processing (TIP) 15, 6 (2006), 1653–1664.
5. RMA Azzam. 1978. Photopolarimetric measurement of the Mueller matrix by Fourier analysis of a single detected signal. Optics Letters 2, 6 (1978), 148–150.
6. RMA Azzam. 2016. Stokes-vector and Mueller-matrix polarimetry. Journal of the Optical Society of America A (JOSA A) 33, 7 (2016), 1396–1408.
7. Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, and Rui Chen. 2020. Deep Shape from Polarization. In European Conference on Computer Vision (ECCV).
8. Seung-Hwan Baek, Diego Gutierrez, and Min H Kim. 2016. Birefractive stereo imaging for single-shot depth acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 194.
9. Seung-Hwan Baek, Daniel S Jeon, Xin Tong, and Min H Kim. 2018. Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Transactions on Graphics (TOG) 37, 6 (2018), 268–1.
10. Seung-Hwan Baek, Tizian Zeltner, Hyun Jin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, and Min H Kim. 2020. Image-based acquisition and modeling of polarimetric reflectance. ACM Transactions on Graphics (TOG) 39, 4 (2020).
11. Kai Berger, Andrea Weidlich, Alexander Wilkie, and Marcus Magnor. 2011. Modeling and verifying the polarizing reflectance of real-world metallic surfaces. IEEE Computer Graphics and Applications (CGA) 32, 2 (2011), 24–33.
12. Tongbo Chen, Hendrik PA Lensch, Christian Fuchs, and Hans-Peter Seidel. 2007. Polarization and phase-shifting for 3D scanning of translucent objects. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1–8.
13. Edward Collett. 2005. Field guide to polarization. SPIE Bellingham, WA.
14. Aniket Dashpute, Chandani Anand, and Mukul Sarkar. 2018. Depth resolution enhancement in time-of-flight cameras using polarization state of the reflected light. IEEE Transactions on Instrumentation and Measurement 68, 1 (2018), 160–168.
15. Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. 2021. Deep Polarization Imaging for 3D Shape and SVBRDF Acquisition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 15567–15576.
16. Christian Fuchs, Michael Heinz, Marc Levoy, Hans-Peter Seidel, and Hendrik PA Lensch. 2008. Combining confocal imaging and descattering. In Computer Graphics Forum (CGF), Vol. 27. Wiley Online Library, 1245–1253.
17. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A Wilson, and Paul Debevec. 2010. Circularly polarized spherical illumination reflectometry. ACM Transactions on Graphics (TOG) 29, 6 (2010), 162.
18. Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. ACM Transactions on Graphics (TOG) 30, 6 (2011), 129.
19. Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical modeling and acquisition of layered facial reflectance. ACM Transactions on Graphics (TOG) 27, 5 (2008), 139.
20. Nirmalya Ghosh and Alex I Vitkin. 2011. Tissue polarimetry: concepts, challenges, applications, and outlook. Journal of Biomedical Optics 16, 11 (2011), 110801.
21. Ioannis Gkioulekas, Anat Levin, Frédo Durand, and Todd Zickler. 2015. Micron-scale light transport decomposition using interferometry. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–14.
22. Mohit Gupta, Amit Agrawal, Ashok Veeraraghavan, and Srinivasa G Narasimhan. 2011. Structured light 3D scanning in the presence of global illumination. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 713–720.
23. Mohit Gupta, Srinivasa G Narasimhan, and Yoav Y Schechner. 2008. On controlling light transport in poor visibility environments. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1–8.
24. Mohit Gupta, Shree K Nayar, Matthias B Hullin, and Jaime Martin. 2015. Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics (TOG) 34, 5 (2015), 1–18.
25. Felix Heide, Steven Diamond, David B Lindell, and Gordon Wetzstein. 2018. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Scientific Reports 8, 1 (2018), 1–8.
26. Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–10.
27. Felix Heide, Matthew O’Toole, Kai Zang, David B Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight imaging with partial occluders and surface normals. ACM Transactions on Graphics (TOG) 38, 3 (2019), 1–10.
28. Tomoki Ichikawa, Matthew Purri, Ryo Kawahara, Shohei Nobuhara, Kristin Dana, and Ko Nishino. 2021. Shape From Sky: Polarimetric Normal Recovery Under the Sky. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 14832–14841.
29. Mikhail Itskov. 2007. Tensor algebra and tensor analysis for engineers. Springer.
30. Adrian Jarabo, Julio Marco, Adolfo Munoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Transactions on Graphics (TOG) 33, 6 (2014), 1–10.
31. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65–79.
32. Gerald Earle Jellison Jr, Donovan N Leonard, Lawrence M Anovitz, Chad M Parish, Eliot D Specht, and TM Rosseel. 2018. Crystallographic orientation of uniaxial calcite and dolomite determined using reflection generalized ellipsometry. Journal of Applied Physics 124, 22 (2018), 223102.
33. Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. 2015. Polarized 3D: High-Quality Depth Sensing with Polarization Cues. In IEEE International Conference on Computer Vision (ICCV). 3370–3378.
34. Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1–10.
35. Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas. 2020. Interferometric transmission probing with coded mutual intensity. ACM Transactions on Graphics (TOG) 39, 4 (2020), 74–1.
36. Hiroyuki Kubo, Suren Jayasuriya, Takafumi Iwaguchi, Takuya Funatomi, Yasuhiro Mukaigawa, and Srinivasa G Narasimhan. 2018. Acquiring and characterizing plane-to-ray indirect light transport. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–10.
37. Dong C Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical programming 45, 1 (1989), 503–528.
38. Shih-Yau Lu and Russell A Chipman. 1996. Interpretation of Mueller matrices based on polar decomposition. Journal of the Optical Society of America A (JOSA A) 13, 5 (1996), 1106–1113.
39. Wan-Chun Ma, Tim Hawkins, Pieter Peers, Charles-Felix Chabert, Malte Weiss, and Paul Debevec. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Eurographics Conference on Rendering Techniques. 183–194.
40. Tomohiro Maeda, Achuta Kadambi, Yoav Y Schechner, and Ramesh Raskar. 2018. Dynamic heterodyne interferometry. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–11.
41. Alkhazur Manakov, John Restrepo, Oliver Klehm, Ramon Hegedus, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Transactions on Graphics (TOG) 32, 4 (2013), 47–1.
42. Andreas Meuleman, Seung-Hwan Baek, Felix Heide, and Min H Kim. 2020. Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2465–2474.
43. Kazuhiro Morimoto, Andrei Ardelean, Ming-Lo Wu, Arin Can Ulku, Ivan Michel Antolovic, Claudio Bruschini, and Edoardo Charbon. 2020. Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications. Optica 7, 4 (Apr 2020), 346–354.
44. Koki Nagano, Graham Fyffe, Oleg Alexander, Jernej Barbic, Hao Li, Abhijeet Ghosh, and Paul E Debevec. 2015. Skin microstructure deformation with displacement map convolution. ACM Transactions on Graphics (TOG) 34, 4 (2015), 109–1.
45. Nikhil Naik, Achuta Kadambi, Christoph Rhemann, Shahram Izadi, Ramesh Raskar, and Sing Bing Kang. 2015. A light transport model for mitigating multipath interference in time-of-flight sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 73–81.
46. Shree K Nayar, Gurunandan Krishnan, Michael D Grossberg, and Ramesh Raskar. 2006. Fast separation of direct and global components of a scene using high frequency illumination. In ACM Transactions on Graphics (TOG). 935–944.
47. Addy Ngan, Frédo Durand, and Wojciech Matusik. 2005. Experimental Analysis of BRDF Models. In Proceedings of the Eurographics Symposium on Rendering (Konstanz, Germany). Eurographics Association, 117–226.
48. Jannik Boll Nielsen, Henrik Wann Jensen, and Ravi Ramamoorthi. 2015. On optimal, minimal BRDF sampling for reflectance acquisition. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1–11.
49. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1–17.
50. Matthew O’Toole, Supreeth Achar, Srinivasa G Narasimhan, and Kiriakos N Kutulakos. 2015. Homogeneous codes for energy-efficient illumination and imaging. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1–13.
51. Matthew O’Toole, Felix Heide, David B Lindell, Kai Zang, Steven Diamond, and Gordon Wetzstein. 2017. Reconstructing transient images from single-photon sensors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1539–1547.
52. Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B Hullin, Wolfgang Heidrich, and Kiriakos N Kutulakos. 2014a. Temporal frequency probing for 5D transient analysis of global light transport. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–11.
53. Matthew O’Toole and Kiriakos N Kutulakos. 2010. Optical computing for fast light transport analysis. ACM Transactions on Graphics (TOG) 29, 6 (2010), 1–164.
54. Matthew O’Toole, David B Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696 (2018), 338–341.
55. Matthew O’Toole, John Mather, and Kiriakos N Kutulakos. 2014b. 3d shape and indirect appearance by structured light transport. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3246–3253.
56. Matthew O’Toole, Ramesh Raskar, and Kiriakos N Kutulakos. 2012. Primal-dual coding to probe light transport. ACM Transactions on Graphics (TOG) 31, 4 (2012), 39–1.
57. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).
58. Adithya Pediredla, Ashok Veeraraghavan, and Ioannis Gkioulekas. 2019. Ellipsoidal path connections for time-gated rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–12.
59. Christoph Peters, Jonathan Klein, Matthias B Hullin, and Reinhard Klein. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1–11.
60. David A Ramsey and Kenneth C Ludema. 1994. The influences of roughness on film thickness measurements by Mueller matrix ellipsometry. Review of Scientific Instruments 65, 9 (1994), 2874–2881.
61. Jérémy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization Imaging Reflectometry in the Wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 206:1–206:14.
62. Vishwanath Saragadam and Aswin C Sankaranarayanan. 2019. KRISM—Krylov subspace-based optical computing of hyperspectral images. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1–14.
63. Pradeep Sen, Billy Chen, Gaurav Garg, Stephen R Marschner, Mark Horowitz, Marc Levoy, and Hendrik PA Lensch. 2005. Dual photography. ACM Transactions on Graphics (TOG) (2005), 745–755.
64. Kenichiro Tanaka, Yasuhiro Mukaigawa, and Achuta Kadambi. 2020. Polarized non-line-of-sight imaging. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2136–2145.
65. Kenichiro Tanaka, Yasuhiro Mukaigawa, Yasuyuki Matsushita, and Yasushi Yagi. 2013. Descattering of transmissive observation using parallel high-frequency illumination. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1–8.
66. Tali Treibitz and Yoav Y Schechner. 2008. Active polarization descattering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 31, 3 (2008), 385–399.
67. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature communications 3, 1 (2012), 1–8.
68. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–8.
69. Gordon Wetzstein, Ivo Ihrke, Douglas Lanman, and Wolfgang Heidrich. 2011. Computational plenoptic imaging. In Computer Graphics Forum, Vol. 30. Wiley Online Library, 2397–2426.
70. Refael Whyte, Lee Streeter, Michael J Cree, and Adrian A Dorrington. 2015. Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods. Optical Engineering 54, 11 (2015), 113109.
71. Di Wu, Andreas Velten, Matthew O’Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. 2014. Decomposing global light transport using time of flight imaging. International Journal of Computer Vision (IJCV) 107, 2 (2014), 123–138.
72. Rihui Wu, Adrian Jarabo, Jinli Suo, Feng Dai, Yongdong Zhang, Qionghai Dai, and Diego Gutierrez. 2018. Adaptive polarization-difference transient imaging for depth estimation in scattering media. Optics Letters 43, 6 (2018), 1299–1302.
73. Rihui Wu, Jinli Suo, Feng Dai, Yongdong Zhang, and Qionghai Dai. 2016. Scattering robust 3D reconstruction via polarized transient imaging. Optics letters 41, 17 (2016), 3948–3951.
74. Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal brdf sampling for two-shot near-field reflectance acquisition. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–12.
