“Temporal frequency probing for 5D transient analysis of global light transport” by O’Toole, Heide, Xiao, Hullin, Heidrich, et al. …
Conference:
Type(s):
Title:
- Temporal frequency probing for 5D transient analysis of global light transport
Session/Category Title: Computational Sensing & Display
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We analyze light propagation in an unknown scene using projectors and cameras that operate at transient timescales. In this new photography regime, the projector emits a spatio-temporal 3D signal and the camera receives a transformed version of it, determined by the set of all light transport paths through the scene and the time delays they induce. The underlying 3D-to-3D transformation encodes scene geometry and global transport in great detail, but individual transport components (e.g., direct reflections, inter-reflections, caustics, etc.) are coupled nontrivially in both space and time.To overcome this complexity, we observe that transient light transport is always separable in the temporal frequency domain. This makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport. We use this idea in a prototype that offers three never-seen-before abilities: (1) acquiring time-of-flight depth images that are robust to general indirect transport, such as interreflections and caustics; (2) distinguishing between direct views of objects and their mirror reflection; and (3) using a photonic mixer device to capture sharp, evolving wavefronts of “light-in-flight”.
References:
1. Bai, J., Chandraker, M., Ng, T.-T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In Proc. ECCV, 294–307. Google ScholarDigital Library
2. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. ACM SIGGRAPH, 145–156. Google ScholarDigital Library
3. Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. SPIE, vol. 7864.Google Scholar
4. Fuchs, S. 2010. Multipath interference compensation in time-of-flight camera images. In Proc. ICPR, 3583–3586. Google ScholarDigital Library
5. Garg, G., Talvala, E.-V., Levoy, M., and Lensch, H. P. 2006. Symmetric photography: exploiting data-sparseness in reflectance fields. In Proc. EGSR, 251–262. Google ScholarDigital Library
6. Godbaz, J. P., Cree, M. J., and Dorrington, A. A. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in amcw lidar. In Proc. SPIE, vol. 8296.Google Scholar
7. Goral, C. M., Torrance, K. E., Greenberg, D. P., and Battaile, B. 1984. Modeling the interaction of light between diffuse surfaces. ACM SIGGRAPH, 213–222. Google ScholarDigital Library
8. Gupta, M., and Nayar, S. K. 2012. Micro phase shifting. In Proc. ICCV, 813–820. Google ScholarDigital Library
9. Heide, F., Hullin, M. B., Gregson, J., and Heidrich, W. 2013. Low-budget transient imaging using photonic mixer devices. ACM SIGGRAPH, 45:1–45:10. Google ScholarDigital Library
10. Hlawitschka, M., Ebling, J., and Scheuermann, G. 2004. Convolution and fourier transform of second order tensor fields. In Proc. IASTED VIIP, 78–83.Google Scholar
11. Jiménez, D., Pizarro, D., Mazo, M., and Palazuelos, S. 2014. Modeling and correction of multipath interference in time of flight cameras. Image and Vision Computing 32, 1, 1–13. Google ScholarDigital Library
12. Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A., and Raskar, R. 2013. Coded time of flight cameras: sparse deconvolution to address multi-path interference and recover time profiles. ACM Trans. Graph. 32, 6, 167:1–167:10. Google ScholarDigital Library
13. Kajiya, J. T. 1986. The rendering equation. ACM SIGGRAPH, 143–150. Google ScholarDigital Library
14. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2011. Looking around the corner using ultrafast transient imaging. Int. J. of Computer Vision 95, 13–28. Google ScholarDigital Library
15. Kirmani, A., Colaco, A., Wong, F. N. C., and Goyal, V. K. 2012. Codac: a compressive depth acquisition camera framework. In Proc. ICASSP, 5425–5428.Google Scholar
16. Kirmani, A., Venkatraman, D., Shin, D., Colaco, A., Wong, F. N. C., Shapiro, J. H., and Goyal, V. K. 2013. First-photon imaging. Science.Google Scholar
17. Mei, J., Kirmani, A., Colaco, A., and Goyal, V. 2013. Phase unwrapping and denoising for time-of-flight imaging using generalized approximate message passing. In Proc. ICIP, 364–368.Google Scholar
18. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM SIGGRAPH, 935–944. Google ScholarDigital Library
19. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM SIGGRAPH, 376–381. Google ScholarDigital Library
20. O’Toole, M., and Kutulakos, K. N. 2010. Optical computing for fast light transport analysis. ACM Trans. Graph., 164:1–164:12. Google ScholarDigital Library
21. O’Toole, M., Raskar, R., and Kutulakos, K. N. 2012. Primal-dual coding to probe light transport.Google Scholar
22. O’Toole, M., Mather, J., and Kutulakos, K. N. 2014. 3D shape and indirect appearance by structured light transport. In Proc. CVPR.Google Scholar
23. Peers, P., Mahajan, D. K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R., and Debevec, P. 2009. Compressive light transport sensing. ACM Trans. Graph. 28, 1. Google ScholarDigital Library
24. Reddy, D., Ramamoorthi, R., and Curless, B. 2012. Frequency-space decomposition and acquisition of light transport under spatially varying illumination. In Proc. ECCV, 596–610. Google ScholarDigital Library
25. Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. ICCV, 1440–1447. Google ScholarDigital Library
26. Sen, P., and Darabi, S. 2009. Compressive dual photography. Computer Graphics Forum 28, 2, 609–618.Google ScholarCross Ref
27. Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. ACM SIGGRAPH, 745–755. Google ScholarDigital Library
28. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M. G., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications.Google Scholar
29. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4. Google ScholarDigital Library
30. Wetzstein, G., and Bimber, O. 2007. Radiometric compensation through inverse light transport. 391–399. Google ScholarDigital Library
31. Wu, D., O’Toole, M., Velten, A., Agrawal, A., and Raskar, R. 2012. Decomposing global light transport using time of flight imaging. In Proc. CVPR, 366–373. Google ScholarDigital Library
32. Wu, D., Wetzstein, G., Barsi, C., Willwacher, T., Dai, Q., and Raskar, R. 2013. Ultra-fast lensless computational imaging through 5d frequency analysis of time-resolved light transport. Int. J. of Computer Vision, 1–13.Google Scholar
