“Low-budget transient imaging using photonic mixer devices” by Heide, Hullin, Gregson and Heidrich

  • ©Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich



Session Title:

    Computational Light Capture


    Low-budget transient imaging using photonic mixer devices




    Transient imaging is an exciting a new imaging modality that can be used to understand light propagation in complex environments, and to capture and analyze scene properties such as the shape of hidden objects or the reflectance properties of surfaces.Unfortunately, research in transient imaging has so far been hindered by the high cost of the required instrumentation, as well as the fragility and difficulty to operate and calibrate devices such as femtosecond lasers and streak cameras.In this paper, we explore the use of photonic mixer devices (PMD), commonly used in inexpensive time-of-flight cameras, as alternative instrumentation for transient imaging. We obtain a sequence of differently modulated images with a PMD sensor, impose a model for local light/object interaction, and use an optimization procedure to infer transient images given the measurements and model. The resulting method produces transient images at a cost several orders of magnitude below existing methods, while simultaneously simplifying and speeding up the capture process.


    1. Abramson, N. 1978. Light-in-flight recording by holography. Optics Letters 3, 4, 121–123.Google ScholarCross Ref
    2. Abramson, N. 1983. Light-in-flight recording: high-speed holographic motion pictures of ultrafast phenomena. Applied optics 22, 2, 215–232.Google Scholar
    3. Audet, C., and Dennis Jr, J. 2006. Mesh adaptive direct search algorithms for constrained optimization. SIAM Journal on optimization 17, 1, 188–217. Google ScholarDigital Library
    4. Beck, A., and Teboulle, M. 2009. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2, 183–202. Google ScholarDigital Library
    5. Chambolle, A., and Pock, T. 2011. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145. Google ScholarDigital Library
    6. Conroy, R., Dorrington, A., Künnemeyer, R., and Cree, M. 2009. Range imager performance comparison in homo-dyne and heterodyne operating modes. In Proc. SPIE, vol. 7239, 723905.Google Scholar
    7. Dorrington, A., Cree, M., Payne, A., Conroy, R., and Carnegie, D. 2007. Achieving sub-millimetre precision with a solid-state full-field heterodyning range imaging camera. Measurement Science and Technology 18, 9, 2809.Google ScholarCross Ref
    8. Dorrington, A., Godbaz, J., Cree, M., Payne, A., and Streeter, L. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proc. SPIE, vol. 7864.Google Scholar
    9. Hassan, A., Künnemeyer, R., Dorrington, A., and Payne, A. 2010. Proof of concept of diffuse optical tomography using time-of-flight range imaging cameras. In Proc. Electronics New Zealand Conference, 115–120.Google Scholar
    10. Huber, P. 1973. Robust regression: asymptotics, conjectures and monte carlo. The Annals of Statistics 1, 5, 799–821.Google ScholarCross Ref
    11. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2009. Looking around the corner using transient imaging. In Proc. ICCV, 159–166.Google Scholar
    12. Lange, R., and Seitz, P. 2001. Solid-state time-of-flight range camera. IEEE J. Quantum Electronics 37, 3, 390–397.Google ScholarCross Ref
    13. Lange, R., Seitz, P., Biber, A., and Lauxtermann, S. 2000. Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging. Sensors and camera systems for scientific, industrial, and digital photography applications, 177–188.Google Scholar
    14. Naik, N., Zhao, S., Velten, A., Raskar, R., and Bala, K. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. 30, 6, 171. Google ScholarDigital Library
    15. Nilsson, B., and Carlsson, T. E. 1998. Direct three-dimensional shape measurement by digital light-in-flight holography. Applied Optics 37, 34, 7954–7959.Google ScholarCross Ref
    16. Pandharkar, R., Velten, A., Bardagjy, A., Lawson, E., Bawendi, M., and Raskar, R. 2011. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In Proc. CVPR, 265–272. Google ScholarDigital Library
    17. Schmidt, M., 2013. minConf (retrieved Jan 7, 2013). http://www.di.ens.fr/~mschmidt/Software/minConf.html.Google Scholar
    18. Schwarte, R., Xu, Z., Heinol, H., Olk, J., Klein, R., Buxbaum, B., Fischer, H., and Schulte, J. 1997. New electro-optical mixing and correlating sensor: facilities and applications of the photonic mixer device. In Proc. SPIE, vol. 3100, 245–253.Google Scholar
    19. Schwarte, R., 1997. Verfahren und Vorrichtung zur Bestimmung der Phasen und/oder Amplitudeninformation einer elektromagnetischen Welle. German Patent DE 19704496.Google Scholar
    20. Smith, A., Skorupski, J., and Davis, J. 2008. Transient rendering. Tech. Rep. UCSC-SOE-08-26, University of California Santa Cruz, School of Engineering.Google Scholar
    21. Velten, A., Raskar, R., and Bawendi, M. 2011. Picosecond camera for time-of-flight imaging. In OSA Imaging Systems and Applications.Google Scholar
    22. Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M., and Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3, 745.Google ScholarCross Ref
    23. Velten, A., Wu, D., Jarabo, A., Masia, B., Barsi, C., Joshi, C., Lawson, E., Bawendi, M., Gutierrez, D., and Raskar, R. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. 32. Google ScholarDigital Library
    24. Wan, G., Li, X., Agranov, G., Levoy, M., and Horowitz, W. 2012. CMOS image sensors with multi-bucket pixels for computational photography. IEEE J. Solid-State Circuits 47, 4, 1031–1042.Google ScholarCross Ref
    25. Wu, D., O’Toole, M., Velten, A., Agrawal, A., and Raskar, R. 2012. Decomposing global light transport using time of flight imaging. In Prov. CVPR, 366–373. Google ScholarDigital Library

ACM Digital Library Publication: