“Light field microscopy” by Levoy, Ng, Adams, Footer and Horowitz

By inserting a microlens array into the optical train of a conventional microscope, one can capture light fields of biological specimens in a single photograph. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective views and focal stacks from them. Since microscopes are inherently orthographic devices, perspective views represent a new way to look at microscopic specimens. The ability to create focal stacks from a single photograph allows moving or light-sensitive specimens to be recorded. Applying 3D deconvolution to these focal stacks, we can produce a set of cross sections, which can be visualized using volume rendering. In this paper, we demonstrate a prototype light field microscope (LFM), analyze its optical performance, and show perspective views, focal stacks, and reconstructed volumes for a variety of biological specimens. We also show that synthetic focusing followed by 3D deconvolution is equivalent to applying limited-angle tomography directly to the 4D light field.

1. Adelson, T., Wang, J. Y. A. 1992. Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2, 99–106. Google ScholarDigital Library
2. Agard, D. A. 1984. Optical sectioning microscopy: Cellular architecture in three dimensions. Ann. Rev. Biophys. Bioeng 13, 191–219.Google ScholarCross Ref
3. Andersen, A. H., Kak, A. C., 1984. Simultaneous algebraic reconstruction technique (SART): A superior implementation of the ART algorithm. Ultrasonic Imaging 6, 81–94.Google ScholarCross Ref
4. Arridge, S. R. 2001. Methods for the inverse problem in optical tomography. Proc. Waves and Imaging Through Complex Media. Kluwer, 307–329.Google ScholarCross Ref
5. Castleman, K. R. 1979. Digital Image Processing. Prentice Hall. Google ScholarDigital Library
6. Chamgoulov, R. O., Lane, P. M., Macaulay, C. E. 2004. Optical computed-tomography microscope using digital spatial light modulation. Proc. SPIE 5324, 182–190.Google Scholar
7. Colsher, J. G. 1980. Fully three-dimensional positron emission tomography. Phys. Med. Biol. 25, 1, 103–115.Google ScholarCross Ref
8. Corle, T. R., Kino, G. S. 1996. Confocal Scanning Optical Microscopy and Related Imaging Systems. Academic Press.Google Scholar
9. Ellis, G. W. 1966. Holomicrography: transformation of image during reconstruction a posteriori. Science 143, 1195–1196.Google ScholarCross Ref
10. Goldberg, N. 1992. Camera technology: the dark side of the lens. Academic Press.Google Scholar
11. Goodman, J. 1996. Introduction to Fourier optics. 2nd edition, McGraw-Hill.Google Scholar
12. Gustafsson, M. G. L. 2005. Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution. Proc. National Academy of Sciences 102, 37.Google ScholarCross Ref
13. Holmes, T. J., Bhattacharyya, S., et al. 1995. Light microscopic images reconstructed by maximum likelihood deconvolution. In Handbook of Biological Confocal Microscopy, ed. J. B. Pawley, Plenum Press, 389–402.Google Scholar
14. Inoue, S., Oldenbourg, R. 1995. Microscopes. In Handbook of Optics, 2nd edition, McGraw-Hill.Google Scholar
15. Inoue, S. and Spring, K. R. 1997. Video Microscopy. 2nd edition, Plenum Press.Google Scholar
16. Gabor, D. 1948. A new microscopic principle. Nature 161, 777–778.Google ScholarCross Ref
17. Isaksen, A., Mcmillan, L., Gortler, S. J. 2000. Dynamically reparameterized light fields. Proc. SIGGRAPH 2000. Google ScholarDigital Library
18. Javidi, B., Okano, F., eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag. Google ScholarDigital Library
19. Kak, A. C., Slaney, M. 1988. Principles of Computerized Tomographic Imaging. IEEE Press.Google Scholar
20. Kawata, S., Nakamura, 0., Minami, S. 1987. Optical microscope tomography. I. Support constraint. J. Opt. Soc. Am. A 4, 1, 292–297.Google ScholarCross Ref
21. Kingslake, R. 1983. Optical system design. Academic Press.Google Scholar
22. Levoy, M., Hanrahan, P. 1996. Light field rendering. Proc. SIGGRAPH 1996. Google ScholarDigital Library
23. Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 825–834. Google ScholarDigital Library
24. Markham, J., Conchello, J.-A. 2001. Artefacts in restored images due to intensity loss in three-dimensional fluorescence microscopy. J. Microscopy 204, 2, 93–98.Google ScholarCross Ref
25. Mcnally, J. G., Preza, C., Conchello, J. A., Thomas, L. J. Jr. 1994. Artifacts in computational optical-sectioning microscopy. J. Opt. Soc. Am. A 11, 3, 1056–67.Google ScholarCross Ref
26. Nayar, S. K., Nakagawa, Y. 1990. Shape from focus: An effective approach for rough surfaces. Proc. International Conference on Robotics and Automation (ICRA), Vol. 2, 218–225.Google ScholarCross Ref
27. Nayar, S. K., Narasimhan, S. G. 2002. Assorted pixels: Multi-sampled imaging with structural models. Proc. ECCV. Google ScholarDigital Library
28. Ng, R. 2005. Fourier slice photography. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 735–744. Google ScholarDigital Library
29. Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M., Hanrahan, P. 2005. Light Field Photography with a Hand-Held Plenoptic Camera. Stanford Tech Report CTSR 2005-02.Google Scholar
30. Ng, R. 2006. Digital Light Field Photography. PhD dissertation, Stanford University. Google ScholarDigital Library
31. Noguchi, M., Nayar, S. 1994. Microscopic shape from focus using active illumination. Proc. IAPR International Conference on Pattern Recognition (ICPR), Vol. A, 147–152.Google ScholarCross Ref
32. Okoshi, T. 1976. Three-Dimensional Imaging Techniques. Academic Press.Google Scholar
33. Piller, H. 1977. Microscope Photometry. Springer-Verlag.Google Scholar
34. Pluta, M. 1988. Advanced Light Microscopy (in 3 volumes). Elsevier.Google Scholar
35. Schechner, Y., Kiryati, N. 1999. The optimal axial interval for estimating depth from defocus. Proc. ICCV. Google ScholarDigital Library
36. Schechner, Y., Kiryati, N. 2000. Depth from defocus vs. stereo: How different really are they? IJCV 39, 2, 141–162. Google ScholarDigital Library
37. Schechner, Y. Y., Kiryati, N., Basri, R. 2000. Separation of transparent layers using focus. IJCV 39, 1, 25–39. Google ScholarDigital Library
38. Schechner, Y., Nayar, S. 2001. Generalized Mosaicing. Proc. ICCV.Google Scholar
39. Shah, U. B., Nayar, S. K. 1992. Extracting 3-D structure and focused images using an optical microscope. Proc. IEEE Symposium on Computer-Based Medical Systems.Google ScholarCross Ref
40. Streibl, N. 1984. Depth transfer by an imaging system. Optica Acta 31, 11, 1233–1241.Google ScholarCross Ref
41. Streibl, N. 1985. Three-dimensional imaging by a microscope. J. Opt. Soc. Am. A 2, 2, 121–127.Google ScholarCross Ref
42. Swedlow, J. R., Sedat, J. W., Agard, D. A. 1997. Deconvolution in optical microscopy. In Deconvolution of Images and Spectra, ed. P. A. Jansson, Academic Press, 284–309. Google ScholarDigital Library
43. Vaish, V., Garg, G., Talvala, E., Antunez, E., Wilburn, B., Horowitz, M., Levoy, M., Synthetic aperture focusing using a shear-warp factorization of the viewing transform. Proc. Workshop on Advanced 3D Imaging for Safety and Security, in conjunction with CVPR 2005. Google ScholarDigital Library
44. Weinstein, R. S., Descour, M. R., et al. 2004. An array microscope for ultrarapid virtual slide processing and telepathology. Design, fabrication, and validation study. Human Pathology 35, 11, 1303–1314.Google ScholarCross Ref