“Neural Spectro-polarimetric Fields” by Kim, Jin, Cho and Baek
Conference:
Type(s):
Title:
- Neural Spectro-polarimetric Fields
Session/Category Title: Computer Vision
Presenter(s)/Author(s):
Abstract:
The spatial radiance distribution modeling of any light ray within a scene has been extensively explored for applications, including view synthesis. Spectrum and polarization — the wave properties of light — are often neglected due to their integration into three RGB spectral bands and their non-perceptibility to human vision. Despite this, these properties encompass substantial material and geometric information about a scene. In this work, we propose to model spectro-polarimetric fields, the spatial Stokes-vector distribution of any light ray at an arbitrary wavelength. To realize this, we present Neural Spectro-polarimetric Fields (NeSpoF) — a neural representation that models the physically-valid Stokes vector at given continuous variables of position, direction, and wavelength. NeSpoF manages inherently noisy raw measurements, showcases memory efficiency, and preserves physically vital signals — factors that are crucial for representing the high-dimensional signal of a spectro-polarimetric field. To validate NeSpoF, we introduce the first multi-view hyperspectral-polarimetric image dataset, comprised of both synthetic and real-world scenes. These were captured using our compact hyperspectral-polarimetric imaging system, which has been calibrated for robustness against system imperfections. We demonstrate the capabilities of NeSpoF on diverse scenes.
References:
[1]
Edward H Adelson and John YA Wang. 1992. Single lens stereo with a plenoptic camera. IEEE transactions on pattern analysis and machine intelligence 14, 2 (1992), 99–106.
[2]
Eleni Aloupogianni, Takaya Ichimura, Mei Hamada, Masahiro Ishikawa, Takuo Murakami, Atsushi Sasaki, Koichiro Nakamura, Naoki Kobayashi, and Takashi Obi. 2022. Hyperspectral imaging for tumor segmentation on pigmented skin lesions. Journal of Biomedical Optics 27, 10 (2022), 106007.
[3]
Ali Altaqui, Pratik Sen, Harry Schrickx, Jeromy Rech, Jin-Woo Lee, Michael Escuti, Wei You, Bumjoon J Kim, Robert Kolbas, Brendan T O’Connor, 2021. Mantis shrimp–inspired organic photodetector for simultaneous hyperspectral and polarimetric imaging. Science Advances 7, 10 (2021), eabe3196.
[4]
Amrit Ambirajan and Dwight C Look Jr. 1995. Optimum angles for a polarimeter: part I. Optical Engineering 34, 6 (1995), 1651–1655.
[5]
Dongsheng An, Jinli Suo, Haoqian Wang, and Qionghai Dai. 2015. Illumination estimation from specular highlight in a multi-spectral image. Optics Express 23, 13 (2015), 17008–17023.
[6]
Yunhao Ba, Alex Gilbert, Franklin Wang, Jinfa Yang, Rui Chen, Yiqin Wang, Lei Yan, Boxin Shi, and Achuta Kadambi. 2020. Deep shape from polarization. In European Conference on Computer Vision. Springer, 554–571.
[7]
Seung-Hwan Baek and Felix Heide. 2021. Polarimetric spatio-temporal light transport probing. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–18.
[8]
Seung-Hwan Baek, Daniel S Jeon, Xin Tong, and Min H Kim. 2018. Simultaneous acquisition of polarimetric SVBRDF and normals.ACM Trans. Graph. 37, 6 (2018), 268–1.
[9]
Seung-Hwan Baek, Tizian Zeltner, Hyunjin Ku, Inseung Hwang, Xin Tong, Wenzel Jakob, and Min H Kim. 2020. Image-based acquisition and modeling of polarimetric reflectance.ACM Trans. Graph. 39, 4 (2020), 139.
[10]
Caixun Bai, Jianxin Li, Wenfei Zhang, Yixuan Xu, and Yutao Feng. 2021. Static full-Stokes Fourier transform imaging spectropolarimeter capturing spectral, polarization, and spatial characteristics. Opt. Express 29, 23 (Nov 2021), 38623–38645. https://doi.org/10.1364/OE.443350
[11]
James R Bergen and Edward H Adelson. 1991. The plenoptic function and the elements of early vision. Computational models of visual processing 1 (1991), 8.
[12]
Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
[13]
Anthony Boccaletti, Jean Schneider, Wes Traub, Pierre-Olivier Lagage, Daphne Stam, Raffaele Gratton, John Trauger, Kerri Cahoy, Frans Snik, Pierre Baudoz, 2012. SPICES: spectro-polarimetric imaging and characterization of exoplanetary systems. Experimental Astronomy 34, 2 (2012), 355–384.
[14]
Zhengyi Chen, Chunmin Zhang, Tingkui Mu, and Yifan He. 2022. Tensorial multi-view subspace clustering for polarimetric hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing (2022).
[15]
Li-Jen Cheng and George F Reyes. 1995. AOTF polarimetric hyperspectral imaging for mine detection. In Detection Technologies for Mines and Minelike Targets, Vol. 2496. SPIE, 305–311.
[16]
Edward Collett. 2005. Field guide to polarization. Spie Bellingham, WA.
[17]
Qi Cui, Jongchan Park, Yayao Ma, and Liang Gao. 2021. Snapshot hyperspectral light field tomography. Optica 8, 12 (2021), 1552–1558.
[18]
Akshat Dave, Yongyi Zhao, and Ashok Veeraraghavan. 2022. PANDORA: Polarization-Aided Neural Decomposition Of Radiance. arXiv preprint arXiv:2203.13458 (2022).
[19]
Abe Davis, Marc Levoy, and Fredo Durand. 2012. Unstructured light fields. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 305–314.
[20]
Fernando del Molino and Adolfo Muñoz. 2019. Polarization mapping. Computers & Graphics 83 (2019), 42–50.
[21]
Louis J Denes, Milton S Gottlieb, Boris Kaminsky, and Daniel F Huber. 1998. Spectropolarimetric imaging for object recognition. In 26th AIPR Workshop: Exploiting New Image Sources and Sensors, Vol. 3240. SPIE, 8–18.
[22]
Valentin Deschaintre, Yiming Lin, and Abhijeet Ghosh. 2021. Deep polarization imaging for 3D shape and SVBRDF acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15567–15576.
[23]
Yuqi Ding, Yu Ji, Mingyuan Zhou, Sing Bing Kang, and Jinwei Ye. 2021. Polarimetric helmholtz stereopsis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5037–5046.
[24]
Tom Duff, James Burgess, Per Christensen, Christophe Hery, Andrew Kensler, Max Liani, and Ryusuke Villemin. 2017. Building an orthonormal basis, revisited. JCGT 6, 1 (2017).
[25]
Axin Fan, Tingfa Xu, Jianan Li, Geer Teng, Xi Wang, Yuhan Zhang, and Chang Xu. 2023. Compressive full-Stokes polarization and flexible hyperspectral imaging with efficient reconstruction. Optics and Lasers in Engineering 160 (2023), 107256.
[26]
Axin Fan, Tingfa Xu, Xu Ma, Jianan Li, Xi Wang, Yuhan Zhang, and Chang Xu. 2022. Four-dimensional compressed spectropolarimetric imaging. Signal Processing (2022).
[27]
Axin Fan, Tingfa Xu, Geer Teng, Xi Wang, Yuhan Zhang, and Chenguang Pan. 2020. Hyperspectral polarization-compressed imaging and reconstruction with sparse basis optimized by particle swarm optimization. Chemometrics and Intelligent Laboratory Systems 206 (2020), 104163.
[28]
Yoshiki Fukao, Ryo Kawahara, Shohei Nobuhara, and Ko Nishino. 2021. Polarimetric normal stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 682–690.
[29]
Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Debevec. 2011. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 SIGGRAPH Asia Conference. 1–10.
[30]
Joseph W Goodman and P Sutton. 1996. Introduction to Fourier optics. Quantum and Semiclassical Optics-Journal of the European Optical Society Part B 8, 5 (1996), 1095.
[31]
Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 43–54.
[32]
Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M Susskind, and Qi Shan. 2022. Fast and explicit neural view synthesis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3791–3800.
[33]
Xia Hua, Yujie Wang, Shuming Wang, Xiujuan Zou, You Zhou, Lin Li, Feng Yan, Xun Cao, Shumin Xiao, Din Ping Tsai, 2022. Ultra-compact snapshot spectral light-field imaging. Nature communications 13, 1 (2022), 2732.
[34]
Inseung Hwang, Daniel S Jeon, Adolfo Munoz, Diego Gutierrez, Xin Tong, and Min H Kim. 2022. Sparse ellipsometry: portable acquisition of polarimetric SVBRDF and shape with unstructured flash photography. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–14.
[35]
Nevrez Imamoglu, Yu Oishi, Xiaoqiang Zhang, Guanqun Ding, Yuming Fang, Toru Kouyama, and Ryosuke Nakamura. 2018. Hyperspectral image dataset for benchmarking on salient object detection. In 2018 Tenth international conference on quality of multimedia experience (qoMEX). IEEE, 1–3.
[36]
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022. Mitsuba 3 renderer. https://mitsuba-renderer.org.
[37]
Hailin Jin, Stefano Soatto, and Anthony J Yezzi. 2005. Multi-view stereo reconstruction of dense shape and complex appearance. International Journal of Computer Vision 63, 3 (2005), 175–189.
[38]
Achuta Kadambi, Vage Taamazyan, Boxin Shi, and Ramesh Raskar. 2015. Polarized 3d: High-quality depth sensing with polarization cues. In Proceedings of the IEEE International Conference on Computer Vision. 3370–3378.
[39]
James T Kajiya and Brian P Von Herzen. 1984. Ray tracing volume densities. ACM SIGGRAPH computer graphics 18, 3 (1984), 165–174.
[40]
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–10.
[41]
Agastya Kalra, Vage Taamazyan, Supreeth Krishna Rao, Kartik Venkataraman, Ramesh Raskar, and Achuta Kadambi. 2020. Deep Polarization Cues for Transparent Object Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[42]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–14.
[43]
Haris Ahmad Khan, Jean-Baptiste Thomas, Jon Yngve Hardeberg, and Olivier Laligant. 2017. Illuminant estimation in multispectral imaging. JOSA A 34, 7 (2017), 1085–1098.
[44]
Min H Kim, Todd Alan Harvey, David S Kittle, Holly Rushmeier, Julie Dorsey, Richard O Prum, and David J Brady. 2012. 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1–11.
[45]
Yuhi Kondo, Taishi Ono, Legong Sun, Yasutaka Hirasawa, and Jun Murayama. 2020. Accurate polarimetric BRDF for real polarization scene rendering. In European Conference on Computer Vision. Springer, 220–236.
[46]
Chenyang Lei, Chenyang Qi, Jiaxin Xie, Na Fan, Vladlen Koltun, and Qifeng Chen. 2022. Shape from polarization for complex scenes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12632–12641.
[47]
Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. 31–42.
[48]
Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. 2006. Light field microscopy. In ACM SIGGRAPH 2006 Papers. 924–934.
[49]
Yuqi Li, Qiang Fu, and Wolfgang Heidrich. 2021. Multispectral illumination estimation using deep unrolling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2672–2681.
[50]
Fei Liu, Lei Cao, Xiaopeng Shao, Pingli Han, and Xiangli Bin. 2015. Polarimetric dehazing utilizing spatial frequency segregation of images. Applied Optics 54, 27 (2015), 8116–8122.
[51]
Jiachen Liu, Pan Ji, Nitin Bansal, Changjiang Cai, Qingan Yan, Xiaolei Huang, and Yi Xu. 2022. PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8665–8675.
[52]
Xiaobo Lv, Yiwei Li, Shuaishuai Zhu, Xinmin Guo, Jianlong Zhang, Jie Lin, and Peng Jin. 2020. Snapshot spectral polarimetric light field imaging using a single detector. Optics Letters 45, 23 (2020), 6522–6525.
[53]
Alkhazur Manakov, John Restrepo, Oliver Klehm, Ramon Hegedus, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Transactions on Graphics 32, 4 (2013), 47–1.
[54]
Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, and Xin Yang. 2022. Glass Segmentation Using Intensity and Spectral Polarization Cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12622–12631.
[55]
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. 2022. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16190–16199.
[56]
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–14.
[57]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision. Springer, 405–421.
[58]
Shree K Nayar, Xi-Sheng Fang, and Terrance Boult. 1997. Separation of reflection components using color and polarization. International Journal of Computer Vision 21, 3 (1997), 163–186.
[59]
Eric Penner and Li Zhang. 2017. Soft 3D reconstruction for view synthesis. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–11.
[60]
Henry Peters, Yunhao Ba, and Achuta Kadambi. 2023. pCON: Polarimetric Coordinate Networks for Neural Scene Representations. In Proc. CVPR.
[61]
Jérémy Riviere, Ilya Reshetouski, Luka Filipi, and Abhijeet Ghosh. 2017. Polarization imaging reflectometry in the wild. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1–14.
[62]
Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).
[63]
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. 2019. DeepVoxels: Learning Persistent 3D Feature Embeddings. In Proc. CVPR.
[64]
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. 2022. Light Field Neural Rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8269–8279.
[65]
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33 (2020), 7537–7547.
[66]
Stojan Trajanovski, Caifeng Shan, Pim JC Weijtmans, Susan G Brouwer de Koning, and Theo JM Ruers. 2020. Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE transactions on biomedical engineering 68, 4 (2020), 1330–1340.
[67]
Zhixin Wang, Peng Xu, Bohan Liu, Yankun Cao, Zhi Liu, and Zhaojun Liu. 2021. Hyperspectral imaging for underwater object detection. Sensor Review 41, 2 (2021), 176–191.
[68]
Sijia Wen, Yinqiang Zheng, and Feng Lu. 2021. Polarization guided specular reflection separation. IEEE Transactions on Image Processing 30 (2021), 7280–7291.
[69]
Gordon Wetzstein, Douglas R Lanman, Matthew Waggener Hirsch, and Ramesh Raskar. 2012. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. (2012).
[70]
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc Levoy, and Mark Horowitz. 2004. High-speed videography using a dense camera array. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 2. IEEE, II–II.
[71]
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy. 2005. High performance imaging using large camera arrays. In ACM SIGGRAPH 2005 Papers. 765–776.
[72]
Alexander Wilkie and Andrea Weidlich. 2010. A standardised polarisation visualisation for images. In Proceedings of the 26th Spring Conference on Computer Graphics. 43–50.
[73]
Pu Xia and Xuebin Liu. 2016. Image dehazing technique based on polarimetric spectral analysis. Optik 127, 18 (2016), 7350–7358.
[74]
Fanchao Yang, Xingjia Tang, Bingliang Hu, Ruyi Wei, Liang Kong, and Yong Li. 2016. A method of removing reflected highlight on images based on polarimetric imaging. Journal of Sensors 2016 (2016).
[75]
Jason C Yang, Matthew Everett, Chris Buehler, and Leonard McMillan. 2002. A real-time distributed light field camera.Rendering Techniques 2002 (2002), 77–86.
[76]
Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide. 2022. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia 2022 Conference Papers. 1–12.
[77]
Yongqiang Zhao, Lei Zhang, and Quan Pan. 2009. Spectropolarimetric imaging for pathological analysis of skin. Applied optics 48, 10 (2009), D236–D246.
[78]
Chu Zhou, Minggui Teng, Yufei Han, Chao Xu, and Boxin Shi. 2021. Learning to dehaze with polarization. Advances in Neural Information Processing Systems 34 (2021), 11487–11500.
[79]
Shihao Zou, Xinxin Zuo, Yiming Qian, Sen Wang, Chi Xu, Minglun Gong, and Li Cheng. 2020. 3d human shape reconstruction from a polarization image. In European Conference on Computer Vision. Springer, 351–368.
