“Light field rendering” by Levoy and Hanrahan

A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function – the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction.

1. Adelson, E.H., Bergen, J.R., “The Plenoptic Function and the Elements of Early Vision,” In Computation Models of Visual Processing, M. Landy and J.A. Movshon, eds., MIT Press, Cambridge, 1991.
2. Ashdown, I., “Near-Field Photometry: A New Approach,” Journal of the Illuminating Engineering Society, Vol. 22, No. 1, Winter, 1993, pp. 163-180.
3. Beers, A., Agrawala, M., Chaddha, N., “Rendering from Compressed Textures.” In these proceedings.
4. Benton, S., “Survey of Holographic Stereograms,” Processing and Display of Three-Dimensional Data, Proc. SPIE, Vol. 367, 1983.
5. Blinn, J.F., Newell, M.E., “Texture and Reflection in Computer Generated Images,” CACM, Vol. 19, No. 10, October, 1976, pp. 542-547.
6. Bolles, R., Baker, H., Marimont, D., “Epipolar-Plane Image Analysis: An Approach to Determining Structure from Motion,” International Journal of Computer Vision, Vol. 1, No. 1, 1987, pp. 7-55.
7. Chen, S.E., Williams, L., “View Interpolation for Image Synthesis,” Proc. SIGGRAPH ’93 (Anaheim, California, August 1-6, 1993). In Computer Graphics Proceedings, Annual Conference Series, 1993, ACM SIGGRAPH, pp. 279-288.
8. Chen, S.E., “QuickTime VR- An Image-Based Approach to Virtual Environment Navigation,” Proc. SIGGRAPH ’95 (Los Angeles, CA, August 6-11, 1995). In Computer Graphics Proceedings, Annual Conference Series, 1995, ACM SIGGRAPH, pp. 29-38.
9. Fuchs, H., Bishop, G., Arthur, K., McMillan, L., Bajcsy, R., Lee, S.W., Farid, H., Kanade, T., “Virtual Space Teleconferencing Using a Sea of Cameras,” Proc. First International Conference on Medical Robotics and Computer Assisted Surgery, 1994, pp. 161-167.
10. Gersho, A., Gray, R.M., Vector Quantization and Signal Compression, Kluwer Academic Publishers, 1992.
11. Gershun, A., “The Light Field,” Moscow, 1936. Translated by E Moon and G. Timoshenko in Journal of Mathematics and Physics, Vol. XVIII, MIT, 1939, pp. 51-151.
12. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M., “The Lumigraph.” In these proceedings.
13. Greene, N., “Environment Mapping and Other Applications of World Projections,” IEEE Computer Graphics and Applications, Vol. 6, No. 11, November, 1986, pp. 21-29.
14. Greene, N. and Kass, M., “Approximating Visibility with Environment Maps,” Apple Technical Report No. 41, November, 1994.
15. Halle, M., “Holographic Stereograms as Discrete Imaging Systems.” Practical Holography, Proc. SPIE, Vol. 2176, February, 1994.
16. Katayama, A., Tanaka, K., Oshino, T., Tamura, H., “Viewpoint-Dependent Stereoscopic Display Using Interpolation of Multiviewpoint Images,” Stereoscopic Displays and Virtual Reality Systems H, Proc. SPIE, Vol. 2409, S. Fisher, J. Merritt, B. Bolas eds. 1995, pp. 11-20.
17. Laveau, S., Faugeras, O.D., “3-D Scene Representation as a Collection of Images and Fundamental Matrices,” INRIA Technical Report No. 2205, 1994.
18. Levin, R., “Photometric Characteristics of Light Controlling Apparatus,” Illuminating Engineering, Vol. 66, No. 4, 1971, pp. 205-215.
19. McMillan, L., Bishop, G., “Head-Tracked Stereoscopic Display Using Image Warping,” Stereoscopic Displays and Virtual Reality Systems II, Proc. SPIE, Vol. 2409, S. Fisher, J. Merritt, B. Bolas eds. 1995, pp. 21-30.
20. McMillan, L., Bishop, G., Plenoptic Modeling: An Image-Based Rendering System, Proc. SIGGRAPH ’95 (Los Angeles, CA, August 6-11, 1995). In Computer Graphics Proceedings, Annual Conference Series, 1995, ACM SIGGRAPH, pp. 39-46.
21. Miller, G., “Volumetric Hyper-Reality: A Computer Graphics Holy Grail for the 21 st Century?,” Proc. Graphics Interface ’95, W. Davis and E Prusinkiewicz eds., Canadian Information Processing Society, 1995, pp. 56-64.
22. Moon, E, Spencer, D.E., The Photic Field, MIT Press, 1981.
23. Narayanan, EJ., “Virtualized Reality: Concepts and Early Results,” Proc. IEEE Workshop on the Representation of Visual Scenes, IEEE, 1995.
24. Nimeroff, J., Simoncelli, E., Dorsey, J., “Efficient Rerendering of Naturally Illuminated Scenes,” Proc. Fifth Eurographics Rendering Workshop, 1994, pp. 359-373.
25. Sbert, A.M., “An Integral Geometry Based Method for Form- Factor Computation,” Computer Graphics Forum, Vol. 13, No. 3, 1993, pp. 409-420.
26. Seitz, S., Dyer, C., “Physically-Valid View Synthesis by Image Interpolation,” Proc. IEEE Workshop on the Representation of Visual Scenes, IEEE, 1995.
27. Williams, L., “Pyramidal Parametrics,” Computer Graphics (Proc. Siggraph ’83), Vol. 17, No. 3, July, 1983, pp. 1-11.
28. Ziv, J., Lempel, A., “A universal algorithm for sequential data compression,” IEEE Transactions on Information Theory, IT-23:337-343, 1977.