“Programmable aperture photography: multiplexed light field acquisition” by Liang, Lin, Wong, Liu and Chen

In this paper, we present a system including a novel component called programmable aperture and two associated post-processing algorithms for high-quality light field acquisition. The shape of the programmable aperture can be adjusted and used to capture light field at full sensor resolution through multiple exposures without any additional optics and without moving the camera. High acquisition efficiency is achieved by employing an optimal multiplexing scheme, and quality data is obtained by using the two post-processing algorithms designed for self calibration of photometric distortion and for multi-view depth estimation. View-dependent depth maps thus generated help boost the angular resolution of light field. Various post-exposure photographic effects are given to demonstrate the effectiveness of the system and the quality of the captured light field.

1. Adelson, E. H., and Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 99–106. Google ScholarDigital Library
2. Aggarwal, M., Hua, H., and Ahuja, N. 2001. On cosinefourth and vignetting effects in real lenses. In Proc. ICCV ’01: Proc. the Eighth IEEE International Conference on Computer Vision, vol. 1, 472–479.Google Scholar
3. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222–1239. Google ScholarDigital Library
4. Chai, J.-X., Chan, S.-C., and Tong, H.-Y. S. X. 2000. Plenoptic sampling. In SIGGRAPH ’00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 307–318. Google ScholarDigital Library
5. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, New York, NY, USA, 11–20. Google ScholarDigital Library
6. Donoho, D. 2006. Compressed Sensing. IEEE Trans. Information Theory 52, 4, 1289–1306. Google ScholarDigital Library
7. Durand, F., Holzschuch, N., Soler, C., Chan, E., and Sillion, F. X. 2005. A frequency analysis of light transport. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM Press, New York, NY, USA, 1115–1126. Google ScholarDigital Library
8. Farid, H., and Simoncelli, E. P. 1998. Range estimation by optical differentiation. Journal of the Optical Society of America A 15, 7, 1777–1786.Google ScholarCross Ref
9. Georgiev, T., Zheng, K. C., Curless, B., Salesin, D., Nayar, S., and Intwala, C. 2006. Spatio-angular resolution tradeoff in integral photography. In EGRW ’06: Proc. the 17th Eurographics workshop on Rendering. Google ScholarCross Ref
10. Georgiev, T., Intwala, C., and Babacan, D. 2007. Lightfield capture by multiplexing in the frequency domain. Adobe technical report, Adobe Systems Incorporated.Google Scholar
11. Goldman, D. B., and Chen, J.-H. 2005. Vignette and exposure calibration and compensation. In Proc. ICCV ’05: Proc. the 10th IEEE International Conference on Computer Vision, 899–906. Google ScholarDigital Library
12. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 43–54. Google ScholarDigital Library
13. Green, P., Sun, W., Matusik, W., and Durand, F. 2007. Multi-aperture photography. ACM Trans. Graph. 26, 3, 68. Google ScholarDigital Library
14. Harwit, M., and Sloane, N. J. 1979. Hadamard Transform Optics. Academic Press, New York.Google Scholar
15. HP components group. 1998. Noise sources in CMOS image sensors. Technical report, Hewlett-Packard Company.Google Scholar
16. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In SIGGRAPH ’00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 297–306. Google ScholarDigital Library
17. Ive, H. E. 1930. Parallax panoramagrams made with a large diameter lens. Journal of the Optical Society of America 20, 6 (June), 332–342.Google Scholar
18. Joshi, N., Matusik, W., and Avidan, S. 2006. Natural video matting using camera arrays. ACM Trans. Graph. 25, 3, 779–786. Google ScholarDigital Library
19. Kang, S. B., and Szeliski, R. 2004. Extracting view-dependent depth maps from a collection of images. International Journal of Computer Vision 58, 2, 139–163. Google ScholarDigital Library
20. Kolmogorov, V., and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. Proc. European Conference on Computer Vision 3, 82–96. Google ScholarDigital Library
21. Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28, 10, 1568–1583. Google ScholarDigital Library
22. Levin, A., Fergus, R., Durand, F., and Freeman, W. T. 2007. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26, 3, 70. Google ScholarDigital Library
23. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH ’96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 31–42. Google ScholarDigital Library
24. Liang, C.-K., Liu, G., and Chen, H. H. 2007. Light field acquisition using programmable aperture camera. In Proc. IEEE International Conference on Image Processing, vol. 5, 233–236.Google Scholar
25. Lippmann, M. G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821–825.Google Scholar
26. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91–110. Google ScholarDigital Library
27. Nayar, S. K., and Branzoi, V. 2003. Adaptive dynamic range imaging: Optical control of pixel exposures over space and time. In Proc. ICCV ’03: Proc. the Ninth IEEE International Conference on Computer Vision, vol. 2, 1168–1175. Google ScholarDigital Library
28. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. CSTR 2005-02, Stanford University, April.Google Scholar
29. Ng, R. 2005. Fourier slice photography. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 735–744. Google ScholarDigital Library
30. Okoshi, T. 1976. Three-dimensional imaging techniques. Academic Press New York.Google Scholar
31. Raskar, R., Tan, K.-H., Feris, R., Yu, J., and Turk, M. 2004. Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. In SIGGRAPH ’04: ACM SIGGRAPH 2004 Papers, ACM Press, New York, NY, USA, 679–688. Google ScholarDigital Library
32. Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795–804. Google ScholarDigital Library
33. Ratner, N., and Schechner, Y. Y. 2007. Illumination multiplexing within fundamental limits. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1–8.Google Scholar
34. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 1–3, 7–42. Google ScholarDigital Library
35. Schechner, Y. Y., and Nayar, S. K. 2004. Uncontrolled modulation imaging. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 197–204. Google ScholarDigital Library
36. Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. N. 2003. A theory of multiplexed illumination. In Proc. ICCV ’03: Proc. the Ninth IEEE International Conference on Computer Vision, vol. 2, 808–815. Google ScholarDigital Library
37. Senkichi, C., Toshio, M., Toshinori, H., Yuichi, M., and Hidetoshi, K., 2003. Device and method for correcting camera-shake and device for detecting camera shake. JP Patent No. 2003-138436.Google Scholar
38. Sun, J., Li, Y., Kang, S. B., and Shum, H.-Y. 2005. Symmetric stereo matching for occlusion handling. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, 399–406. Google ScholarDigital Library
39. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., and Rother, C. 2006. A comparative study of energy minimization methods for markov random fields. In Proc. European Conference on Computer Vision, vol. 2, 16–29. Google ScholarDigital Library
40. Tsin, Y., Ramesh, V., and Kanade, T. 2001. Statistical calibration of CCD imaging process. In Proc. ICCV ’01: Proc. the Eighth IEEE International Conference on Computer Vision, 480–487.Google Scholar
41. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3, 69. Google ScholarDigital Library
42. Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. In ACM SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM Press, New York, NY, USA, 756–764. Google ScholarDigital Library
43. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 765–776. Google ScholarDigital Library
44. Yang, J. C., Everett, M., Buehler, C., and McMillan, L. 2002. A real-time distributed light field camera. In EGRW ’02: Proc. the 13th Eurographics workshop on Rendering, 77–86. Google ScholarDigital Library
45. Yang, Q., Yang, R., Davis, J., and Nister, D. 2007. Spatial-depth super resolution for range images. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 1–8.Google Scholar
46. Zomet, A., and Nayar, S. K. 2006. Lensless imaging with a controllable aperture. Proc. IEEE Conference on Computer Vision and Pattern Recognition, 339–346. Google ScholarDigital Library
47. Zwicker, M., Matusik, W., Durand, F., and Pfister, H. 2006. Antialiasing for automultiscopic 3d displays. In EGSR’06: Proc. the 17th Eurographics Symposium on Rendering. Google ScholarCross Ref