“Computational imaging with multi-camera time-of-flight systems”
Conference:
Type(s):
Title:
- Computational imaging with multi-camera time-of-flight systems
Session/Category Title: COMPUTATIONAL CAMERAS
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Depth cameras are a ubiquitous technology used in a wide range of applications, including robotic and machine vision, human-computer interaction, autonomous vehicles as well as augmented and virtual reality. In this paper, we explore the design and applications of phased multi-camera time-of-flight (ToF) systems. We develop a reproducible hardware system that allows for the exposure times and waveforms of up to three cameras to be synchronized. Using this system, we analyze waveform interference between multiple light sources in ToF applications and propose simple solutions to this problem. Building on the concept of orthogonal frequency design, we demonstrate state-of-the-art results for instantaneous radial velocity capture via Doppler time-of-flight imaging and we explore new directions for optically probing global illumination, for example by de-scattering dynamic scenes and by non-line-of-sight motion detection via frequency gating.
References:
1. Bamji, C., O’Connor, P., Elkhatib, T., Mehta, S., Thompson, B., Prather, L., Snow, D., Akkaya, O., Daniel, A., Payne, A., amd M. Fenton, T. P., and Chan, V. 2015. A 0.13 um CMOS System-on-Chip for a 512 x 424 Time-of-Flight Image Sensor With Multi-Frequency Photo-Demodulation up to 130 MHz and 2 GS/s ADC. IEEE Journal of Solid-State Circuits 50, 1, 303–319.Google ScholarCross Ref
2. Bhandari, A., Kadambi, A., Whyte, R., Barsi, C., Feigin, M., Dorrington, A., and Raskar, R. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Optics Letters 39, 1705–1708.Google ScholarCross Ref
3. Buades, A., Coll, B., and Morel, J.-M. 2005. A non-local algorithm for image denoising. In Proc. IEEE CVPR, vol. 2. Google ScholarDigital Library
4. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proc. SIGGRAPH, 425–432. Google ScholarDigital Library
5. Butler, D., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., and Kim, D. 2012. Shake’n’sense: reducing interference for overlapping structured light depth cameras. In Proc. ACM UIST, 1933–1936. Google ScholarDigital Library
6. Büttgen, B., and Seitz, P. 2008. Robust optical time-of-flight range imaging based on smart pixel structures. IEEE Trans. Circuits and Systems 55, 6, 1512–1525.Google ScholarCross Ref
7. Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. ACM Trans. Graph. (SIGGRAPH) 22, 3, 569–577. Google ScholarDigital Library
8. Castaneda, V., Mateus, D., and Navab, N. 2014. Stereo time-of-flight with constructive interference. IEEE Trans. PAMI 36, 7, 1402–1413. Google ScholarDigital Library
9. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proc. SIGGRAPH, 145–156. Google ScholarDigital Library
10. Dong, S., Horstmeyer, R., Shiradkar, R., Guo, K., Ou, X., Bian, Z., Xin, H., and Zheng, G. 2014. Aperture-scanning fourier ptychography for 3d refocusing and super-resolution macroscopic imaging. Optics Express 22, 11, 13586–99.Google ScholarCross Ref
11. Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. Electronic Imaging.Google Scholar
12. Freedman, D., Krupka, E., Smolin, Y., Leichter, I., and Schmidt, M. 2014. Sra: fast removal of general multipath for tof sensors. In Proc. ECCV.Google Scholar
13. Fuchs, S. 2010. Multipath interference compensation in time-of-flight camera images. In Proc. ICPR. Google ScholarDigital Library
14. Gall, J., Ho, H., Izadi, S., Kohli, P., Ren, X., and Yang, R. 2014. Towards solving real-world vision problems with rgb-d cameras. In CVPR Tutorial.Google Scholar
15. Gariepy, G., Tonolini, F., and Jonathan Leach, R. H., and Faccio, D. 2016. Detection and tracking of moving objects hidden from view. Nature Photonics Letters 10, 23–26.Google ScholarCross Ref
16. Gokturk, S., Yalcin, H., and Bamji, C. 2004. A time-of-flight depth sensor – system description, issues and solutions. In Proc. CVPR, 35–35. Google ScholarDigital Library
17. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In Proc. SIGGRAPH. Google ScholarDigital Library
18. Hansard, M., Lee, S., Choi, O., and Horaud, R. 2012. Time of Flight Cameras: Principles, Methods, and Applications. Springer. Google ScholarDigital Library
19. Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. (SIGGRAPH) 32, 4, 45:1–45:10. Google ScholarDigital Library
20. Heide, F., Xiao, L., Heidrich, W., and Hullin, M. B. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR. Google ScholarDigital Library
21. Heide, F., Xiao, L., Kolb, A., Hullin, M. B., and Heidrich, W. 2014. Imaging in scattering media using correlation image sensors and sparse convolutional coding. Optics Express 22, 21, 26338–26350.Google ScholarCross Ref
22. Heide, F., Heidrich, W., Hullin, M., and Wetzstein, G. 2015. Doppler Time-of-Flight Imaging. ACM Trans. Graph. (SIGGRAPH), 4. Google ScholarDigital Library
23. Holloway, J., Salman Asif, M., Sharma, M. K., Matsuda, N., Horstmeyer, R., Cossairt, O., and Veeraraghavan, A. 2015. Toward Long Distance, Sub-diffraction Imaging Using Coherent Camera Arrays. ArXiv 1510.08470.Google Scholar
24. Jayasuriya, S., Pediredla, A., Sivaramakrishnan, S., Molnar, A., and Veeraraghavan, A. 2015. Depth fields: Extending light field techniques to time-of-flight imaging. In Proc. 3DV, 1–9. Google ScholarDigital Library
25. Jimenez, D., Pizarro, D., Mazo, M., and Palazuelos, S. 2012. Modelling and correction of multipath interference in time of flight cameras. In Proc. CVPR. Google ScholarDigital Library
26. Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6. Google ScholarDigital Library
27. Kadambi, A., Bhandari, A., Whyte, R., Dorrington, A., and Raskar, R. 2014. Demultiplexing Illumination via Low Cost Sensing and Nanosecond Coding. In Proc. ICCP.Google Scholar
28. Kim, S. K., Kang, B., Heo, J., Jung, S.-W., and Choi, O. 2014. Photometric stereo-based single time-of-flight camera. Optics Letters 39, 1, 166–169.Google ScholarCross Ref
29. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2009. Looking around the corner using transient imaging. In Proc. ICCV, 159–166.Google Scholar
30. Lange, R., and Seitz, P. 2001. Solid-state time-of-flight range camera. IEEE J. Quantum Electronics 37, 3, 390–397.Google ScholarCross Ref
31. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proc. SIGGRAPH, 31–42. Google ScholarDigital Library
32. Li, L., Xiang, S., Yang, Y., and Yu, L. 2015. Multi-camera interference cancellation of time-of-flight (tof) cameras. In Proc. IEEE ICIP, 556–560.Google Scholar
33. Maimone, A., and Fuchs, H. 2012. Reducing interference between multiple structured light depth sensors using motion. In Proc. VR. Google ScholarDigital Library
34. Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. In Proc. SIGGRAPH, 369–374. Google ScholarDigital Library
35. McCandless, S. W., and Jackson, C. R. 2004. Principles of synthetic aperture radar. In AR Marine Users Manual, J. Fagerberg, D. C. Mowery, and R. R. Nelson, Eds. NOAA, ch. 1, 11.Google Scholar
36. Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia) 30, 6, 171:1–171:10. Google ScholarDigital Library
37. Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., and Kang, S. 2015. A light transport model for mitigating multipath interference in tof sensors. In Proc. CVPR.Google Scholar
38. O’Toole, M., Heide, F., Xiao, L., Hullin, M. B., Heidrich, W., and Kutulakos, K. N. 2014. Temporal frequency probing for 5d transient analysis of global light transport. ACM Trans. Graph. (SIGGRAPH) 33, 4, 87:1–87:11. Google ScholarDigital Library
39. Payne, A., Jongenelen, A., Dorrington, A., Cree, M., and Carnegie, D. 2009. Multiple Frequency Range Imaging to Remove Measurement Ambiguity. In Proc. Optical 3-D measurement techniques IX.Google Scholar
40. Peters, C., Klein, J., Hullin, M. B., and Klein, R. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Trans. Graph. (SIGGRAPH Asia) 34, 6. Google ScholarDigital Library
41. Rander, P., Narayanan, P. J., and Kanade, T. 1997. Virtualized reality: Constructing time-varying virtual worlds from real events. In Proc. IEEE Visualization, 277–283. Google ScholarDigital Library
42. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. 2011. Real-time human pose recognition in parts from single depth images. In Proc. CVPR. Google ScholarDigital Library
43. Tadano, R., Pediredla, A. K., and Veeraraghavan, A. 2015. Depth selective camera: A direct, on-chip, programmable technique for depth selectivity in photography. In Proc. IEEE ICCV. Google ScholarDigital Library
44. Ti, C., Yang, R., Davis, J., and Pan, Z. 2015. Simultaneous Time-of-Flight Sensing and Photometric Stereo With a Single ToF Sensor. In Proc. CVPR.Google Scholar
45. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat Commun 745, 3.Google Scholar
46. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. (SIGGRAPH) 32, 4, 44:1–44:8. Google ScholarDigital Library
47. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. (SIGGRAPH) 24, 3, 765–776. Google ScholarDigital Library
48. Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1.Google ScholarCross Ref
49. Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., O’Toole, M., Naik, N., Dai, Q., Kutulakos, K., and Raskar, R. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proc. ECCV. Google ScholarDigital Library
