“Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing” by Veeraraghavan, Raskar, Agrawal, Mohan and Tumblin

  • ©Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan, and Jack Tumblin




    Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing



    We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene.We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.


    1. Accorsi, R., Gasparini, F., and Lanza, R. C. 2001. Optimal coded aperture patterns for improved SNR in nuclear medicine imaging. Nuclear Instruments and Methods in Physics Research A 474 (Dec.), 273–284.Google ScholarCross Ref
    2. Adelson, T., and Wang, J. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Machine Intell. 14, 99–106. Google ScholarDigital Library
    3. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294–302. Google ScholarDigital Library
    4. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization using graph cuts. IEEE Trans. Pattern Anal. Machine Intell. 23, 1222–1239. Google ScholarDigital Library
    5. Chaudhuri, S., and Rajagopalan, A. 1999. Depth from Defocus: A Real Aperture Imaging Approach. Springer.Google ScholarCross Ref
    6. Dowski, E. R., and Cathey, W. 1995. Extended depth of field through wavefront coding. Appl. Optics 34, 11 (Apr.), 1859–1866.Google ScholarCross Ref
    7. Dowski, E. R., and Johnson, G. E. 1999. Wavefront coding: A modern method of achieving high performance and/or low cost imaging systems. In SPIE Annual Meeting.Google Scholar
    8. Farid, H., and Simoncelli, E. 1998. Range estimation by optical differentiation. J. Opt. Soc. of America A 15, 1, 1777–1786.Google ScholarCross Ref
    9. Fergus, R., Torralba, A., and Freeman, W. 2006. Random lens imaging. Tech. rep., MIT.Google Scholar
    10. Fessenden, R. 1908. Wireless telephony. Trans. American Institute of Electrical Engineers 27, 553–629.Google ScholarCross Ref
    11. Field, D. 1994. What is the goal of sensory coding? Neural Comput. 6, 559–601. Google ScholarDigital Library
    12. Georgiev, T., Zheng, C., Nayar, S., Curless, B., Salasin, D., and Intwala, C. 2006. Spatio-angular resolution trade-offs in integral photography. In Eurographics Symposium on Rendering, 263–272. Google ScholarCross Ref
    13. Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The lumigraph. In SIGGRAPH, 43–54. Google ScholarDigital Library
    14. Gottesman, S. R., and Fenimore, E. E. 1989. New family of binary arrays for coded aperture imaging. Appl. Optics 28, 20 (Oct), 4344–4352.Google ScholarCross Ref
    15. Haeberli, P., 1994. A multifocus method for controlling depth of field. GraficaObscura.Google Scholar
    16. Hiura, S., and Matsuyama, T. 1998. Depth measurement by the multi-focus camera. In Proc. Conf. Computer Vision and Pattern Recognition, 953–961. Google ScholarDigital Library
    17. Imatest. Image quality evaluation software. http://www.imatest.com/.Google Scholar
    18. Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. In SIGGRAPH, 297–306. Google ScholarDigital Library
    19. Ives, H. 1928. Camera for making parallax panoramagrams. J. Opt. Soc. Amer. 17, 435–439.Google ScholarCross Ref
    20. Javidi, B., and Okano, F., Eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag. Google ScholarDigital Library
    21. Johnson, G. E., Dowski, E. R., and Cathey, W. T. 2000. Passive ranging through wave-front coding: Information and application. Applied Optics 39, 1700–1710.Google ScholarCross Ref
    22. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH 96, 31–42. Google ScholarDigital Library
    23. Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, M., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 825–834. Google ScholarDigital Library
    24. Lippmann, G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821–825.Google Scholar
    25. Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745–754.Google ScholarCross Ref
    26. Martnez-Corral, M., Javidi, B., Martnez-Cuenca, R., and Saavedra, G. 2004. Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays. Applied Optics 43, 5806–5813.Google ScholarCross Ref
    27. Nayar, S., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, 472–479.Google Scholar
    28. Nayar, S. K., Branzoi, V., and Boult, T. E. 2006. Programmable imaging: Towards a flexible camera. Int’l J. Computer Vision 70, 1, 7–22. Google ScholarDigital Library
    29. Ng, R., Levoy, M., Brdif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Tech. rep., Stanford Univ.Google Scholar
    30. Ng, R. 2005. Fourier slice photography. ACM Trans. Graph. 24, 735–744. Google ScholarDigital Library
    31. Okano, F., Hoshino, H., and Yuyama, A. 1997. Real-time pickup method for a three-dimensional image based on integral photography. Applied Optics 36, 15981603.Google ScholarCross Ref
    32. Okano, F., Arai, J., Hoshino, H., and Yuyama, I. 1999. Three dimensional video system based on integral photography. Optical Engineering 38, 1072–1077.Google ScholarCross Ref
    33. Oppenheim, A. V., Schafer, R. W., and Buck, J. R. 1999. Discrete-Time Signal Processing. Prentice-Hall. Google ScholarDigital Library
    34. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795–804. Google ScholarDigital Library
    35. Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of Am. 62, 1, 55–59.Google ScholarCross Ref
    36. Skinner, G. K. 1988. X-Ray Imaging with Coded Masks. Scientific American 259 (Aug.), 84.Google ScholarCross Ref
    37. Sun, W., and Barbastathis, G. 2005. Rainbow volume holographic imaging. Opt. Lett. 30, 976–978.Google ScholarCross Ref
    38. Vaish, V., Wilburn, B., Joshi, N., and Levoy, M. 2004. Using plane + parallax for calibrating dense camera arrays. In Proc. Conf. Computer Vision and Pattern Recognition, 2–9.Google Scholar
    39. van der Gracht, J., Dowski, E., Taylor, M., and Deaver, D. 1996. Broadband behavior of an optical-digital focus-invariant system. Optics Letters 21, 13 (July), 919–921.Google ScholarCross Ref
    40. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Non-refractive modulators for coding and capturing scene appearance. Tech. Rep. UMIACS-TR-2007-21, Univ. of Maryland.Google Scholar
    41. Wang, S., and Heidrich, W. 2004. The design of an inexpensive very high resolution scan camera system. Eurographics 23, 441–450.Google ScholarCross Ref
    42. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3, 765–776. Google ScholarDigital Library
    43. Yang, J. C. 2000. A Light Field Camera For Image Based Rendering. Master’s thesis, Massachussettes Institute of Technology.Google Scholar
    44. Zomet, A., and Nayar, S. 2006. Lensless imaging with a controllable aperture. In Proc. Conf. Computer Vision and Pattern Recognition, 339–346. Google ScholarDigital Library

ACM Digital Library Publication: