“Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing” by Veeraraghavan, Raskar, Agrawal, Mohan and Tumblin

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene.We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.

1. Accorsi, R., Gasparini, F., and Lanza, R. C. 2001. Optimal coded aperture patterns for improved SNR in nuclear medicine imaging. Nuclear Instruments and Methods in Physics Research A 474 (Dec.), 273–284.Google ScholarCross Ref
2. Adelson, T., and Wang, J. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Machine Intell. 14, 99–106. Google ScholarDigital Library
3. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294–302. Google ScholarDigital Library
4. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization using graph cuts. IEEE Trans. Pattern Anal. Machine Intell. 23, 1222–1239. Google ScholarDigital Library
5. Chaudhuri, S., and Rajagopalan, A. 1999. Depth from Defocus: A Real Aperture Imaging Approach. Springer.Google ScholarCross Ref
6. Dowski, E. R., and Cathey, W. 1995. Extended depth of field through wavefront coding. Appl. Optics 34, 11 (Apr.), 1859–1866.Google ScholarCross Ref
7. Dowski, E. R., and Johnson, G. E. 1999. Wavefront coding: A modern method of achieving high performance and/or low cost imaging systems. In SPIE Annual Meeting.Google Scholar
8. Farid, H., and Simoncelli, E. 1998. Range estimation by optical differentiation. J. Opt. Soc. of America A 15, 1, 1777–1786.Google ScholarCross Ref
9. Fergus, R., Torralba, A., and Freeman, W. 2006. Random lens imaging. Tech. rep., MIT.Google Scholar
10. Fessenden, R. 1908. Wireless telephony. Trans. American Institute of Electrical Engineers 27, 553–629.Google ScholarCross Ref
11. Field, D. 1994. What is the goal of sensory coding? Neural Comput. 6, 559–601. Google ScholarDigital Library
12. Georgiev, T., Zheng, C., Nayar, S., Curless, B., Salasin, D., and Intwala, C. 2006. Spatio-angular resolution trade-offs in integral photography. In Eurographics Symposium on Rendering, 263–272. Google ScholarCross Ref
13. Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The lumigraph. In SIGGRAPH, 43–54. Google ScholarDigital Library
14. Gottesman, S. R., and Fenimore, E. E. 1989. New family of binary arrays for coded aperture imaging. Appl. Optics 28, 20 (Oct), 4344–4352.Google ScholarCross Ref
15. Haeberli, P., 1994. A multifocus method for controlling depth of field. GraficaObscura.Google Scholar
16. Hiura, S., and Matsuyama, T. 1998. Depth measurement by the multi-focus camera. In Proc. Conf. Computer Vision and Pattern Recognition, 953–961. Google ScholarDigital Library
17. Imatest. Image quality evaluation software. http://www.imatest.com/.Google Scholar
18. Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. In SIGGRAPH, 297–306. Google ScholarDigital Library
19. Ives, H. 1928. Camera for making parallax panoramagrams. J. Opt. Soc. Amer. 17, 435–439.Google ScholarCross Ref
20. Javidi, B., and Okano, F., Eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag. Google ScholarDigital Library
21. Johnson, G. E., Dowski, E. R., and Cathey, W. T. 2000. Passive ranging through wave-front coding: Information and application. Applied Optics 39, 1700–1710.Google ScholarCross Ref
22. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH 96, 31–42. Google ScholarDigital Library
23. Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, M., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 825–834. Google ScholarDigital Library
24. Lippmann, G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821–825.Google Scholar
25. Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745–754.Google ScholarCross Ref
26. Martnez-Corral, M., Javidi, B., Martnez-Cuenca, R., and Saavedra, G. 2004. Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays. Applied Optics 43, 5806–5813.Google ScholarCross Ref
27. Nayar, S., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, 472–479.Google Scholar
28. Nayar, S. K., Branzoi, V., and Boult, T. E. 2006. Programmable imaging: Towards a flexible camera. Int’l J. Computer Vision 70, 1, 7–22. Google ScholarDigital Library
29. Ng, R., Levoy, M., Brdif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Tech. rep., Stanford Univ.Google Scholar
30. Ng, R. 2005. Fourier slice photography. ACM Trans. Graph. 24, 735–744. Google ScholarDigital Library
31. Okano, F., Hoshino, H., and Yuyama, A. 1997. Real-time pickup method for a three-dimensional image based on integral photography. Applied Optics 36, 15981603.Google ScholarCross Ref
32. Okano, F., Arai, J., Hoshino, H., and Yuyama, I. 1999. Three dimensional video system based on integral photography. Optical Engineering 38, 1072–1077.Google ScholarCross Ref
33. Oppenheim, A. V., Schafer, R. W., and Buck, J. R. 1999. Discrete-Time Signal Processing. Prentice-Hall. Google ScholarDigital Library
34. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795–804. Google ScholarDigital Library
35. Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of Am. 62, 1, 55–59.Google ScholarCross Ref
36. Skinner, G. K. 1988. X-Ray Imaging with Coded Masks. Scientific American 259 (Aug.), 84.Google ScholarCross Ref
37. Sun, W., and Barbastathis, G. 2005. Rainbow volume holographic imaging. Opt. Lett. 30, 976–978.Google ScholarCross Ref
38. Vaish, V., Wilburn, B., Joshi, N., and Levoy, M. 2004. Using plane + parallax for calibrating dense camera arrays. In Proc. Conf. Computer Vision and Pattern Recognition, 2–9.Google Scholar
39. van der Gracht, J., Dowski, E., Taylor, M., and Deaver, D. 1996. Broadband behavior of an optical-digital focus-invariant system. Optics Letters 21, 13 (July), 919–921.Google ScholarCross Ref
40. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Non-refractive modulators for coding and capturing scene appearance. Tech. Rep. UMIACS-TR-2007-21, Univ. of Maryland.Google Scholar
41. Wang, S., and Heidrich, W. 2004. The design of an inexpensive very high resolution scan camera system. Eurographics 23, 441–450.Google ScholarCross Ref
42. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3, 765–776. Google ScholarDigital Library
43. Yang, J. C. 2000. A Light Field Camera For Image Based Rendering. Master’s thesis, Massachussettes Institute of Technology.Google Scholar
44. Zomet, A., and Nayar, S. 2006. Lensless imaging with a controllable aperture. In Proc. Conf. Computer Vision and Pattern Recognition, 339–346. Google ScholarDigital Library