“Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles” by Kadambi, Whyte, Bhandari, Streeter, Barsi, et al. …
Conference:
Type(s):
Title:
- Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles
Session/Category Title: Computational Photography
Presenter(s)/Author(s):
Abstract:
Time of flight cameras produce real-time range maps at a relatively low cost using continuous wave amplitude modulation and demodulation. However, they are geared to measure range (or phase) for a single reflected bounce of light and suffer from systematic errors due to multipath interference.We re-purpose the conventional time of flight device for a new goal: to recover per-pixel sparse time profiles expressed as a sequence of impulses. With this modification, we show that we can not only address multipath interference but also enable new applications such as recovering depth of near-transparent surfaces, looking through diffusers and creating time-profile movies of sweeping light.Our key idea is to formulate the forward amplitude modulated light propagation as a convolution with custom codes, record samples by introducing a simple sequence of electronic time delays, and perform sparse deconvolution to recover sequences of Diracs that correspond to multipath returns. Applications to computer vision include ranging of near-transparent objects and subsurface imaging through diffusers. Our low cost prototype may lead to new insights regarding forward and inverse problems in light transport.
References:
1. Abramson, N. 1980. Light-in-flight recording by holography. In 1980 Los Angeles Technical Symposium, International Society for Optics and Photonics, 140–143.
2. Bamji, C., et al., 2008. Method and system to avoid inter-system interference for phase-based time-of-flight systems, July 29. US Patent 7,405,812.
3. Bhandari, A., Kadambi, A., Whyte, R., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Multifrequency time of flight in the context of transient renderings. In ACM SIGGRAPH 2013 Posters, ACM, 46.
4. Bruckstein, A. M., Elad, M., and Zibulevsky, M. 2008. On the uniqueness of nonnegative sparse solutions to underdetermined systems of equations. Information Theory, IEEE Transactions on 54, 11, 4813–4820.
5. Buttgen, B., and Seitz, P. 2008. Robust optical time-of-flight range imaging based on smart pixel structures. Circuits and Systems I: Regular Papers, IEEE Transactions on 55, 6, 1512–1525.
6. Carnegie, D. A., McClymont, J., Jongenelen, A. P., Drayton, B., Dorrington, A. A., and Payne, A. D. 2011. Design and construction of a configurable full-field range imaging system for mobile robotic applications. In New Developments and Applications in Sensing Technology. Springer, 133–155.
7. Claerbout, J. F., and F, M. 1973. Robust modelling with erratic data. Geophysics 38, 1, 826–844.
8. 2008. Special issue on compressed sensing. IEEE Signal Processing Magazine 25, 2.
9. Darche, G. 1989. Iterative l1 deconvolution. SEP Annual Report 61, 281–301.
10. Daubechies, I., Defrise, M., and De Mol, C. 2004. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on pure and applied mathematics 57, 11, 1413–1457.
11. Dorrington, A. A., Godbaz, J. P., Cree, M. J., Payne, A. D., and Streeter, L. V. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 786404–786404.
12. Godbaz, J. P., Cree, M. J., and Dorrington, A. A. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in amcw lidar. In IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 829618–829618.
13. Godbaz, J. P., Dorrington, A. A., and Cree, M. J. 2013. Understanding and ameliorating mixed pixels and multipath interference in amcw lidar. In TOF Range-Imaging Cameras. Springer, 91–116.
14. Gupta, M., Agrawal, A., Veeraraghavan, A., and Narasimhan, S. G. 2013. A practical approach to 3d scanning in the presence of interreflections, subsurface scattering and defocus. International Journal of Computer Vision, 1–23.
15. Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics 32, 4 (July).
16. Hodrick, R., and Prescott, E. 1997. Postwar u. s. business cycles: An empirical investigation. Journal of Money, Credit, and Banking 29, 1–16.
17. Jensen, H. W., Marschner, S. R., Levoy, M., and Hanrahan, P. 2001. A practical model for subsurface light transport. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, 511–518.
18. Kadambi, A., Ikoma, H., Lin, X., Wetzstein, G., and Raskar, R. 2013. Subsurface enhancement through sparse representations of multispectral direct/global decomposition. In Computational Optical Sensing and Imaging, Optical Society of America.
19. Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. In ACM Transactions on Graphics (TOG), vol. 30, ACM, 171.
20. Naik, N., Barsi, C., Velten, A., and Raskar, R. 2013. Time-resolved reconstruction of scene reflectance hidden by a diffuser. In CLEO: Science and Innovations, Optical Society of America.
21. Narasimhan, S. G., Nayar, S. K., Sun, B., and Koppal, S. J. 2005. Structured light in scattering media. In Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, vol. 1, IEEE, 420–427.
22. O’Brien, M. S., Sinclair, A. N., and Kramer, S. M. 1994. Recovery of a sparse spike time series by l1 norm deconvolution. IEEE Trans. Signal Proc. 42, 12, 3353–3365.
23. Raskar, R., and Davis, J. 2008. 5d time-light transport matrix: What can we reason about scene properties. Int. Memo07 2.
24. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. In ACM Transactions on Graphics (TOG), vol. 25, ACM, 795–804.
25. Santosa, F., and Symes, W. W. 1986. Linear inversion of band-limited reflection seismograms. SIAM Journal on Scientific and Statistical Computing 7, 4, 1307–1330.
26. Taylor, H. L., Banks, S. C., and McCoy, J. F. 1979. Deconvolution with the l1 norm. Geophysics 44, 1, 39–52.
27. Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. Pattern Analysis and Machine Intelligence, IEEE Transactions on 28, 2, 290–301.
28. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M. G., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3, 745.
29. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Transactions on Graphics 32, 4 (July).
30. Whyte, R. Z., Payne, A. D., Dorrington, A. A., and Cree, M. J. 2010. Multiple range imaging camera operation with minimal performance impact. In IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 75380I-75380I.
31. Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., OToole, M., Naik, N., Dai, Q., Kutulakos, K., and Raskar, R. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Computer Vision–ECCV 2012. Springer, 542–555.
