“Primal-dual coding to probe light transport” by O’Toole, Raskar and Kutulakos

We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene’s transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.

1. Bekas, C., Kokiopoulou, E., and Saad, Y. 2007. An estimator for the diagonal of a matrix. Appl. Numer. Math. 57, 11–12, 1214–1229. Google ScholarDigital Library
2. Chandraker, M., Ng, T., and Ramamoorthi, R. 2010. A dual theory of inverse and forward light transport. In Proc. ECCV. Google ScholarDigital Library
3. Clark, R. N. 2007. Canon 1D Mark II analysis. In http://wwwclarkvisioncom/articles/evaluation-1d2/.Google Scholar
4. Corle, T. R., and Kino, G. S. 1996. Confocal scanning optical microscopy and related imaging systems. Academic Press.Google Scholar
5. Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. ACM SIGGRAPH, 145–156. Google ScholarDigital Library
6. Fuchs, C., Heinz, M., Levoy, M., Scidel, H.-P., and Lensch, H. P. A. 2008. Combining confocal imaging and descattering. Computer Graphics Forum 27, 4, 1245–1253. Google ScholarDigital Library
7. Garg, G., Talvala, E.-V., Levoy, M., and Lensch, H. P. A. 2006. Symmetric photography: Exploiting data-sparseness in reflectance fields. In Proc. EGSR, 251–262. Google ScholarDigital Library
8. Ghosh, A., Chen, T., Peers, P., Wilson, C. A., and Debevec, P. 2010. Circularly polarized spherical illumination reflectometry. ACM SIGGRAPH Asia. Google ScholarDigital Library
9. Gupta, M., Agrawal, A., Veeraraghavan, A., and Narasimhan, S. 2011. Structured light 3D scanning in the presence of global illumination. In Proc. CVPR, 713–720. Google ScholarDigital Library
10. Heintzmann, R., Hanley, Q., Arndt-Jovin, D., and Jovin, T. 2001. A dual path programmable array microscope (PAM): simultaneous acquisition of conjugate and non-conjugate images. J. Microscopy 204, 119–135.Google ScholarCross Ref
11. Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., and Nayar, S. K. 2011. Video from a single coded exposure photograph using a learned over-complete dictionary. In Proc. ICCV. Google ScholarDigital Library
12. Iso 2721:1982. Photography—Cameras—Automatic controls of exposure.Google Scholar
13. Kirmani, A., Hutchison, T., Davis, J., and Raskar, R. 2011. Looking around the corner using ultrafast transient imaging. Int. J. Computer Vision 95, 1, 13–28. Google ScholarDigital Library
14. Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM SIGGRAPH, 825–834. Google ScholarDigital Library
15. Levoy, M., Ng, R., Adams, A., Footer, M., and Horowitz, M. 2006. Light field microscopy. ACM SIGGRAPH, 924–934. Google ScholarDigital Library
16. Mertz, J. 2011. Optical sectioning microscopy with planar or structured illumination. Nat. Meth. 8, 10, 811–819.Google ScholarCross Ref
17. Nayar, S. K., Branzoi, V., and Boult, T. 2004. Programmable imaging using a digital micromirror array. In Proc. CVPR, 436–443.Google Scholar
18. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM SIGGRAPH, 935–944. Google ScholarDigital Library
19. Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM SIGGRAPH, 376–381. Google ScholarDigital Library
20. O’Toole, M., and Kutulakos, K. N. 2010. Optical computing for fast light transport analysis. ACM SIGGRAPH Asia. Google ScholarDigital Library
21. Peers, P., Mahajan, D. K., Lamond, B., Ghosh, A., Matusik, W., Ramamoorthi, R., and Debevec, P. 2009. Compressive light transport sensing. ACM Trans. on Graphics 28, 1. Google ScholarDigital Library
22. Popoff, S. M., Lerosey, G., Carminati, R., Fink, M., Boccara, A. C., and Gigan, S. 2010. Measuring the transmission matrix in optics. Phys. Rev. Lett. 104, 10.Google ScholarCross Ref
23. Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. N. 2007. Multiplexing for optimal lighting. IEEE T-PAMI 29, 8, 1339–1354. Google ScholarDigital Library
24. Seitz, S. M., Matsushita, Y., and Kutulakos, K. N. 2005. A theory of inverse light transport. In Proc. ICCV, 1440–1447. Google ScholarDigital Library
25. Sen, P., and Darabi, S. 2009. Compressive dual photography. Computer Graphics Forum 28, 2, 609–618.Google ScholarCross Ref
26. Sen, P., Chen, B., Garg, G., Marschner, S., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. ACM SIGGRAPH, 745–755. Google ScholarDigital Library
27. Sloan, P.-P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM SIGGRAPH, 527–536. Google ScholarDigital Library
28. Tang, J. M., and Saad, Y. 2012. A probing method for computing the diagonal of a matrix inverse. Numer. Linear Algebra Appl. 19, 3, 485–501.Google ScholarCross Ref
29. Veeraraghavan, A., Reddy, D., and Raskar, R. 2011. Coded strobing photography: compressive sensing of high speed periodic videos. IEEE T-PAMI 33, 4, 671–686. Google ScholarDigital Library
30. Wang, J., Dong, Y., Tong, X., Lin, Z., and Guo, B. 2009. Kernel Nyström method for light transport. ACM SIGGRAPH. Google ScholarDigital Library
31. Wetzstein, G., Heidrich, W., and Luebke, D. 2010. Optical image processing using light modulation displays. Computer Graphics Forum 29, 6, 1934–1944.Google ScholarCross Ref
32. Wilson, T., Juškaitis, R., and Neil, M. 1996. Confocal microscopy by aperture correlation. Optics Letters 21, 23.Google ScholarCross Ref
33. Zhang, L., and Nayar, S. K. 2006. Projection defocus analysis for scene capture and image display. ACM SIGGRAPH, 907–915. Google ScholarDigital Library