“Virtual Mirrors: Non-Line-of-Sight Imaging Beyond the Third Bounce” by Royo, Sultan, Munoz, Masumnia-Bisheh, Brandt, et al. …

  • ©Diego Royo, Talha Sultan, Adolfo Munoz, Khadijeh Masumnia-Bisheh, Eric Brandt, Diego Gutierrez, Andreas Velten, and Julio Marco

Conference:


Type:


Title:

    Virtual Mirrors: Non-Line-of-Sight Imaging Beyond the Third Bounce

Session/Category Title: Environmental Rendering: Nerves On Earth

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Non-line-of-sight (NLOS) imaging methods are capable of reconstructing complex scenes that are not visible to an observer using indirect illumination. However, they assume only third-bounce illumination, so they are currently limited to single-corner configurations, and present limited visibility when imaging surfaces at certain orientations. To reason about and tackle these limitations, we make the key observation that planar diffuse surfaces behave specularly at wavelengths used in the computational wave-based NLOS imaging domain. We call such surfaces virtual mirrors. We leverage this observation to expand the capabilities of NLOS imaging using illumination beyond the third bounce, addressing two problems: imaging single-corner objects at limited visibility angles, and imaging objects hidden behind two corners. To image objects at limited visibility angles, we first analyze the reflections of the known illuminated point on surfaces of the scene as an estimator of the position and orientation of objects with limited visibility. We then image those limited visibility objects by computationally building secondary apertures at other surfaces that observe the target object from a direct visibility perspective. Beyond single-corner NLOS imaging, we exploit the specular behavior of virtual mirrors to image objects hidden behind a second corner by imaging the space behind such virtual mirrors, where the mirror image of objects hidden around two corners is formed. No specular surfaces were involved in the making of this paper.

References:


    1. Byeongjoo Ahn, Akshat Dave, Ashok Veeraraghavan, Ioannis Gkioulekas, and Aswin C Sankaranarayanan. 2019. Convolutional approximations to the general non-line-of-sight imaging operator. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7889–7899.
    2. Martin Benning, Christoph Brune, Marinus Jan Lagerwerf, and Carola-Bibliane Schönlieb. 2015. TGV sinogram inpainting for limited angle tomography. Proceedings of the Royal Society A (2015).
    3. Katherine L Bouman, Vickie Ye, Adam B Yedidia, Frédo Durand, Gregory W Wornell, Antonio Torralba, and William T Freeman. 2017. Turning corners into cameras: Principles and methods. In Proceedings of the IEEE International Conference on Computer Vision. 2270–2278.
    4. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics Express 23, 16 (2015), 20997–21011.
    5. Ruizhi Cao, Frederic de Goumoens, Baptiste Blochet, Jian Xu, and Changhuei Yang. 2022. High-resolution non-line-of-sight imaging employing active focusing. Nature Photonics 16, 6 (2022), 462–468.
    6. Alexander H Delaney and Yoram Bresler. 1998. Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography. IEEE Transactions on Image Processing 7, 2 (1998), 204–221.
    7. Justin Dove and Jeffrey H Shapiro. 2019. Paraxial theory of phasor-field imaging. Optics Express 27, 13 (2019), 18016–18037.
    8. Justin Dove and Jeffrey H Shapiro. 2020a. Nonparaxial phasor-field propagation. Optics Express 28, 20 (2020), 29212–29229.
    9. Justin Dove and Jeffrey H Shapiro. 2020b. Paraxial phasor-field physical optics. Optics Express 28, 14 (2020), 21095–21109.
    10. Justin Dove and Jeffrey H Shapiro. 2020c. Speckled speckled speckle. Optics Express 28, 15 (2020), 22105–22120.
    11. Daniele Faccio, Andreas Velten, and Gordon Wetzstein. 2020. Non-line-of-sight imaging. Nature Reviews Physics 2, 6 (2020), 318–327.
    12. Genevieve Gariepy, Francesco Tonolini, Robert Henderson, Jonathan Leach, and Daniele Faccio. 2016. Detection and tracking of moving objects hidden from view. Nature Photonics 10, 1 (2016), 23–26.
    13. Ibón Guillén, Xiaochun Liu, Andreas Velten, Diego Gutierrez, and Adrian Jarabo. 2020. On the Effect of Reflectance on Phasor Field Non-Line-of-Sight Imaging. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 9269–9273.
    14. Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics 32, 4 (2013), 1–10.
    15. Adrian Jarabo, Julio Marco, Adolfo Munoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Transactions on Graphics 33, 6 (2014), 1–10.
    16. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65–79.
    17. Masaki Kaga, Takahiro Kushida, Tsuyoshi Takatani, Kenichiro Tanaka, Takuya Funatomi, and Yasuhiro Mukaigawa. 2019. Thermal non-line-of-sight imaging from specular and diffuse reflections. IPSJ Transactions on Computer Vision and Applications 11, 1 (2019), 1–6.
    18. Ori Katz, Pierre Heidmann, Mathias Fink, and Sylvain Gigan. 2014. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nature Photonics 8, 10 (2014), 784–790.
    19. William Krska, Sheila W Seidel, Charles Saunders, Robinson Czajkowski, Christopher Yu, John Murray-Bruce, and Vivek Goyal. 2022. Double your corners, double your fun: the doorway camera. In Proceedings of the IEEE International Conference on Computational Photography. IEEE, 1–12.
    20. Marco La Manna, Ji-Hyun Nam, Syed Azer Reza, and Andreas Velten. 2020. Non-line-of-sight-imaging using dynamic relay surfaces. Optics Express 28, 4 (2020), 5331–5339.
    21. Martin Laurenzis and Frank Christnacher. 2022. Time domain analysis of photon scattering and Huygens-Fresnel back projection. Optics Express 30, 17 (2022), 30441–30454.
    22. Zhengpeng Liao, Deyang Jiang, Xiaochun Liu, Andreas Velten, Yajun Ha, and Xin Lou. 2021. FPGA Accelerator for Real-Time Non-Line-of-Sight Imaging. IEEE Transactions on Circuits and Systems I: Regular Papers 69, 2 (2021), 721–734.
    23. David B Lindell, Gordon Wetzstein, and Vladlen Koltun. 2019a. Acoustic non-line-of-sight imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6780–6789.
    24. David B Lindell, Gordon Wetzstein, and Matthew O’Toole. 2019b. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Transactions on Graphics 38, 4 (2019), 1–13.
    25. Xiaochun Liu, Sebastian Bauer, and Andreas Velten. 2019a. Analysis of feature visibility in non-line-of-sight measurements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10140–10148.
    26. Xiaochun Liu, Sebastian Bauer, and Andreas Velten. 2020. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nature Communications 11, 1 (2020), 1645.
    27. Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. 2019b. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572, 7771 (2019), 620–623.
    28. Pablo Luesia, Miguel Crespo, Adrian Jarabo, and Albert Redo-Sanchez. 2022. Non-line-of-sight imaging in the presence of scattering media using phasor fields. Optics Letters 47, 15 (2022), 3796–3799.
    29. Pablo Luesia-Lahoz, Diego Gutierrez, and Adolfo Muñoz. 2023. Zone Plate Virtual Lenses for Memory-Constrained NLOS Imaging. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE.
    30. Tomohiro Maeda, Guy Satat, Tristan Swedish, Lagnojita Sinha, and Ramesh Raskar. 2019a. Recent Advances in Imaging Around Corners. arXiv preprint arXiv:1910.05613 (2019).
    31. Tomohiro Maeda, Yiqin Wang, Ramesh Raskar, and Achuta Kadambi. 2019b. Thermal non-line-of-sight imaging. In Proceedings of the IEEE International Conference on Computational Photography. IEEE, 1–11.
    32. Julio Marco, Adrian Jarabo, Ji Hyun Nam, Xiaochun Liu, Miguel Ángel Cosculluela, Andreas Velten, and Diego Gutierrez. 2021. Virtual light transport matrices for non-line-of-sight imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2440–2449.
    33. Ji Hyun Nam, Eric Brandt, Sebastian Bauer, Xiaochun Liu, Marco Renna, Alberto Tosi, Eftychios Sifakis, and Andreas Velten. 2021. Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second. Nature Communications 12, 1 (2021), 1–10.
    34. Ji Hyun Nam, Eric Brandt, Sebastian Bauer, Xiaochun Liu, Eftychios Sifakis, and Andreas Velten. 2020. Real-time Non-line-of-Sight imaging of dynamic scenes. arXiv preprint arXiv:2010.12737 (2020).
    35. Matthew O’Toole, David B Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696 (2018), 338.
    36. Adithya Pediredla, Akshat Dave, and Ashok Veeraraghavan. 2019. Snlos: Non-line-of-sight scanning through temporal focusing. In Proceedings of the IEEE International Conference on Computational Photography. IEEE, 1–13.
    37. Syed Azer Reza, Marco La Manna, Sebastian Bauer, and Andreas Velten. 2019a. Phasor field waves: A Huygens-like light transport model for non-line-of-sight imaging applications. Optics Express 27, 20 (2019), 29380–29400.
    38. Syed Azer Reza, Marco La Manna, Sebastian Bauer, and Andreas Velten. 2019b. Phasor field waves: experimental demonstrations of wave-like properties. Optics Express 27, 22 (2019), 32587–32608.
    39. Simone Riccardo, Enrico Conca, Vincenzo Sesta, Andreas Velten, and Alberto Tosi. 2022. Fast-Gated 16 × 16 SPAD Array With 16 on-Chip 6 ps Time-to-Digital Converters for Non-Line-of-Sight Imaging. IEEE Sensors Journal 22, 17 (2022), 16874–16885.
    40. Diego Royo, Jorge García, Adolfo Muñoz, and Adrian Jarabo. 2022. Non-line-of-sight transient rendering. Computers & Graphics 107 (2022), 84–92.
    41. Charles Saunders, John Murray-Bruce, and Vivek K Goyal. 2019. Computational periscopy with an ordinary digital camera. Nature 565, 7740 (2019), 472–475.
    42. Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Nils Appenrodt, Werner Ritter, Jurgen Dickmann, Klaus Dietmayer, Bernhard Sick, et al. 2020. Seeing around street corners: Non-line-of-sight detection and tracking in-the-wild using doppler radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2068–2077.
    43. Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa, Vivek K Goyal, Franco NC Wong, and Jeffrey H Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nature Communications 7, 1 (2016), 12046.
    44. Jeremy A Teichman. 2019. Phasor field waves: a mathematical treatment. Optics Express 27, 20 (2019), 27500–27506.
    45. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. 2012a. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3, 1 (2012), 745.
    46. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-Photography: Capturing and Visualizing the Propagation of Light. ACM Transactions on Graphics 32, 4 (2013), 1–8.
    47. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Everett Lawson, Chinmaya Joshi, Diego Gutierrez, Moungi G. Bawendi, and Ramesh Raskar. 2012b. Relativistic ultrafast rendering using time-of-flight imaging. In ACM SIGGRAPH 2012 Talks.
    48. Cheng Wu, Jianjiang Liu, Xin Huang, Zheng-Ping Li, Chao Yu, Jun-Tian Ye, Jun Zhang, Qiang Zhang, Xiankang Dou, Vivek K Goyal, et al. 2021. Non-line-of-sight imaging over 1.43 km. Proceedings of the National Academy of Sciences 118, 10 (2021).
    49. Shumian Xin, Sotiris Nousias, Kiriakos N Kutulakos, Aswin C Sankaranarayanan, Srinivasa G Narasimhan, and Ioannis Gkioulekas. 2019. A theory of Fermat paths for non-line-of-sight shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6800–6809.
    50. Shinyoung Yi, Donggun Kim, Kiseok Choi, Adrian Jarabo, Diego Gutierrez, and Min H Kim. 2021. Differentiable Transient Rendering. ACM Transactions on Graphics 40, 6 (2021), 1–11.

ACM Digital Library Publication:


Overview Page: