“Computational imaging with multi-camera time-of-flight systems”

  • ©Shikhar Shrestha, Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein

Conference:


Type:


Title:

    Computational imaging with multi-camera time-of-flight systems

Session/Category Title: COMPUTATIONAL CAMERAS

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Depth cameras are a ubiquitous technology used in a wide range of applications, including robotic and machine vision, human-computer interaction, autonomous vehicles as well as augmented and virtual reality. In this paper, we explore the design and applications of phased multi-camera time-of-flight (ToF) systems. We develop a reproducible hardware system that allows for the exposure times and waveforms of up to three cameras to be synchronized. Using this system, we analyze waveform interference between multiple light sources in ToF applications and propose simple solutions to this problem. Building on the concept of orthogonal frequency design, we demonstrate state-of-the-art results for instantaneous radial velocity capture via Doppler time-of-flight imaging and we explore new directions for optically probing global illumination, for example by de-scattering dynamic scenes and by non-line-of-sight motion detection via frequency gating.

References:


    1. Bamji, C., O’Connor, P., Elkhatib, T., Mehta, S., Thompson, B., Prather, L., Snow, D., Akkaya, O., Daniel, A., Payne, A., amd M. Fenton, T. P., and Chan, V. 2015. A 0.13 um CMOS System-on-Chip for a 512 x 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC. IEEE Journal of Solid-State Circuits 50, 1, 303–319.Google ScholarCross Ref
    2. Bhandari, A., Kadambi, A., Whyte, R., Barsi, C., Feigin, M., Dorrington, A., and Raskar, R. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Optics Letters 39, 1705–1708.Google ScholarCross Ref
    3. Buades, A., Coll, B., and Morel, J.-M. 2005. A non-local algorithm for image denoising. In Proc. IEEE CVPR, vol. 2. Google ScholarDigital Library
    4. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proc. SIGGRAPH, 425–432. Google ScholarDigital Library
    5. Butler, D., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., and Kim, D. 2012. Shake’n’sense: reducing interference for overlapping structured light depth cameras. In Proc. ACM UIST, 1933–1936. Google ScholarDigital Library
    6. Büttgen, B., and Seitz, P. 2008. Robust optical time-of-flight range imaging based on smart pixel structures. IEEE Trans. Circuits and Systems 55, 6, 1512–1525.Google ScholarCross Ref
    7. Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. ACM Trans. Graph. (SIGGRAPH) 22, 3, 569–577. Google ScholarDigital Library
    8. Castaneda, V., Mateus, D., and Navab, N. 2014. Stereo time-of-flight with constructive interference. IEEE Trans. PAMI 36, 7, 1402–1413. Google ScholarDigital Library
    9. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proc. SIGGRAPH, 145–156. Google ScholarDigital Library
    10. Dong, S., Horstmeyer, R., Shiradkar, R., Guo, K., Ou, X., Bian, Z., Xin, H., and Zheng, G. 2014. Aperture-scanning fourier ptychography for 3d refocusing and super-resolution macroscopic imaging. Optics Express 22, 11, 13586–99.Google ScholarCross Ref
    11. Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. Electronic Imaging.Google Scholar
    12. Freedman, D., Krupka, E., Smolin, Y., Leichter, I., and Schmidt, M. 2014. Sra: fast removal of general multipath for tof sensors. In Proc. ECCV.Google Scholar
    13. Fuchs, S. 2010. Multipath interference compensation in time-of-flight camera images. In Proc. ICPR. Google ScholarDigital Library
    14. Gall, J., Ho, H., Izadi, S., Kohli, P., Ren, X., and Yang, R. 2014. Towards solving real-world vision problems with rgb-d cameras. In CVPR Tutorial.Google Scholar
    15. Gariepy, G., Tonolini, F., and Jonathan Leach, R. H., and Faccio, D. 2016. Detection and tracking of moving objects hidden from view. Nature Photonics Letters 10, 23–26.Google ScholarCross Ref
    16. Gokturk, S., Yalcin, H., and Bamji, C. 2004. A time-of-flight depth sensor – system description, issues and solutions. In Proc. CVPR, 35–35. Google ScholarDigital Library
    17. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In Proc. SIGGRAPH. Google ScholarDigital Library
    18. Hansard, M., Lee, S., Choi, O., and Horaud, R. 2012. Time of Flight Cameras: Principles, Methods, and Applications. Springer. Google ScholarDigital Library
    19. Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. (SIGGRAPH) 32, 4, 45:1–45:10. Google ScholarDigital Library
    20. Heide, F., Xiao, L., Heidrich, W., and Hullin, M. B. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR. Google ScholarDigital Library
    21. Heide, F., Xiao, L., Kolb, A., Hullin, M. B., and Heidrich, W. 2014. Imaging in scattering media using correlation image sensors and sparse convolutional coding. Optics Express 22, 21, 26338–26350.Google ScholarCross Ref
    22. Heide, F., Heidrich, W., Hullin, M., and Wetzstein, G. 2015. Doppler Time-of-Flight Imaging. ACM Trans. Graph. (SIGGRAPH), 4. Google ScholarDigital Library
    23. Holloway, J., Salman Asif, M., Sharma, M. K., Matsuda, N., Horstmeyer, R., Cossairt, O., and Veeraraghavan, A. 2015. Toward Long Distance, Sub-diffraction Imaging Using Coherent Camera Arrays. ArXiv 1510.08470.Google Scholar
    24. Jayasuriya, S., Pediredla, A., Sivaramakrishnan, S., Molnar, A., and Veeraraghavan, A. 2015. Depth fields: Extending light field techniques to time-of-flight imaging. In Proc. 3DV, 1–9. Google ScholarDigital Library
    25. Jimenez, D., Pizarro, D., Mazo, M., and Palazuelos, S. 2012. Modelling and correction of multipath interference in time of flight cameras. In Proc. CVPR. Google ScholarDigital Library
    26. Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6. Google ScholarDigital Library
    27. Kadambi, A., Bhandari, A., Whyte, R., Dorrington, A., and Raskar, R. 2014. Demultiplexing Illumination via Low Cost Sensing and Nanosecond Coding. In Proc. ICCP.Google Scholar
    28. Kim, S. K., Kang, B., Heo, J., Jung, S.-W., and Choi, O. 2014. Photometric stereo-based single time-of-flight camera. Optics Letters 39, 1, 166–169.Google ScholarCross Ref
    29. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2009. Looking around the corner using transient imaging. In Proc. ICCV, 159–166.Google Scholar
    30. Lange, R., and Seitz, P. 2001. Solid-state time-of-flight range camera. IEEE J. Quantum Electronics 37, 3, 390–397.Google ScholarCross Ref
    31. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proc. SIGGRAPH, 31–42. Google ScholarDigital Library
    32. Li, L., Xiang, S., Yang, Y., and Yu, L. 2015. Multi-camera interference cancellation of time-of-flight (tof) cameras. In Proc. IEEE ICIP, 556–560.Google Scholar
    33. Maimone, A., and Fuchs, H. 2012. Reducing interference between multiple structured light depth sensors using motion. In Proc. VR. Google ScholarDigital Library
    34. Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. In Proc. SIGGRAPH, 369–374. Google ScholarDigital Library
    35. McCandless, S. W., and Jackson, C. R. 2004. Principles of synthetic aperture radar. In AR Marine Users Manual, J. Fagerberg, D. C. Mowery, and R. R. Nelson, Eds. NOAA, ch. 1, 11.Google Scholar
    36. Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) 30, 6, 171:1–171:10. Google ScholarDigital Library
    37. Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., and Kang, S. 2015. A light transport model for mitigating multipath interference in tof sensors. In Proc. CVPR.Google Scholar
    38. O’Toole, M., Heide, F., Xiao, L., Hullin, M. B., Heidrich, W., and Kutulakos, K. N. 2014. Temporal frequency probing for 5d transient analysis of global light transport. ACM Trans. Graph. (SIGGRAPH) 33, 4, 87:1–87:11. Google ScholarDigital Library
    39. Payne, A., Jongenelen, A., Dorrington, A., Cree, M., and Carnegie, D. 2009. Multiple Frequency Range Imaging to Remove Measurement Ambiguity. In Proc. Optical 3-D measurement techniques IX.Google Scholar
    40. Peters, C., Klein, J., Hullin, M. B., and Klein, R. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Trans. Graph. (SIGGRAPH Asia) 34, 6. Google ScholarDigital Library
    41. Rander, P., Narayanan, P. J., and Kanade, T. 1997. Virtualized reality: Constructing time-varying virtual worlds from real events. In Proc. IEEE Visualization, 277–283. Google ScholarDigital Library
    42. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. 2011. Real-time human pose recognition in parts from single depth images. In Proc. CVPR. Google ScholarDigital Library
    43. Tadano, R., Pediredla, A. K., and Veeraraghavan, A. 2015. Depth selective camera: A direct, on-chip, programmable technique for depth selectivity in photography. In Proc. IEEE ICCV. Google ScholarDigital Library
    44. Ti, C., Yang, R., Davis, J., and Pan, Z. 2015. Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor. In Proc. CVPR.Google Scholar
    45. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat Commun 745, 3.Google Scholar
    46. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. (SIGGRAPH) 32, 4, 44:1–44:8. Google ScholarDigital Library
    47. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. (SIGGRAPH) 24, 3, 765–776. Google ScholarDigital Library
    48. Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1.Google ScholarCross Ref
    49. Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., O’Toole, M., Naik, N., Dai, Q., Kutulakos, K., and Raskar, R. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proc. ECCV. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: