“Eyes for relighting” by Nishino and Nayar

  • ©Ko Nishino and Shree K. Nayar




    Eyes for relighting



    The combination of the cornea of an eye and a camera viewing the eye form a catadioptric (mirror + lens) imaging system with a very wide field of view. We present a detailed analysis of the characteristics of this corneal imaging system. Anatomical studies have shown that the shape of a normal cornea (without major defects) can be approximated with an ellipsoid of fixed eccentricity and size. Using this shape model, we can determine the geometric parameters of the corneal imaging system from the image. Then, an environment map of the scene with a large field of view can be computed from the image. The environment map represents the illumination of the scene with respect to the eye. This use of an eye as a natural light probe is advantageous in many relighting scenarios. For instance, it enables us to insert virtual objects into an image such that they appear consistent with the illumination of the scene. The eye is a particularly useful probe when relighting faces. It allows us to reconstruct the geometry of a face by simply waving a light source in front of the face. Finally, in the case of an already captured image, eyes could be the only direct means for obtaining illumination information. We show how illumination computed from eyes can be used to replace a face in an image with another one. We believe that the eye not only serves as a useful tool for relighting but also makes relighting possible in situations where current approaches are hard to use.


    1. AGARWAL, S., RAMAMOORTHI, R., BELONGIE, S., AND JENSEN, H. 2003. Structured Importance Sampling of Environment Maps. In Proc. of ACM SIGGRAPH 2003, 605–612. Google ScholarDigital Library
    2. BAKER, S., AND NAYAR, S. 1999. A Theory of Single-Viewpoint Catadioptric Image Formation. IJCV 35, 2 (Nov.), 1–22. Google ScholarDigital Library
    3. BAKER, T. 1943. Ray tracing through non-spherical surfaces. Proc. of The Royal Society of London 55, 361–364.Google ScholarCross Ref
    4. BARSKY, B., BARGTEIL, A., GARCIA, D., AND KLEIN, S. 2002. Introducing Vision-Realistic Rendering. In Proc. of EGWR 2002, 1–7.Google Scholar
    5. BARSKY, B. 2003. Geometry for Analysis of Corneal Shape. In Computational Geometry: Lectures at Morningside Center of Mathematics. Amer Mathematical Society, 33–56.Google Scholar
    6. BLANZ, V., AND VETTER, T. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proc. of ACM SIGGRAPH 99. Google ScholarDigital Library
    7. BOIVIN, S., AND GAGALOWICZ, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. of ACM SIGGRAPH 2001, 197–116. Google ScholarDigital Library
    8. BOLT, R. 1982. Eyes at the Interface. In Proc. of ACM CHI 82, 360–362. Google ScholarDigital Library
    9. BURKHARD, D., AND SHEALY, D. 1973. Flux Density for Ray Propoagation in Gemoetrical Optics. JOSA 63, 3 (Mar.), 299–304.Google ScholarCross Ref
    10. CORNBLEET, S. 1984. Microwave and Optical Ray Geometry. John Wiley and Sons.Google Scholar
    11. DAUGMAN, J. 1993. High Confidence Visual Recognition of Persons by a Test of Statistical Independence. IEEE TPAMI 15, 11 (Nov.), 1148–1161. Google ScholarDigital Library
    12. DEBEVEC, P., AND MALIK, J. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proc. of ACM SIGGRAPH 97, 369–378. Google ScholarDigital Library
    13. DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., AND SAROKIN, W. 2000. Acquiring the Reflectance Field of a Human Face. In Proc. of ACM SIGGRAPH 2000, 145–156. Google ScholarDigital Library
    14. DEBEVEC, P. 1998. Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography. In Proc. of ACM SIGGRAPH 98, 189–198. Google ScholarDigital Library
    15. FLOM, L., AND SAFIR, A., 1987. Iris Recognition System. US patent 4,641,349.Google Scholar
    16. FRANKOT, R., AND CHELLAPPA, R. 1988. A Method for Enforcing Integrability in Shape from Shading Algorithms. IEEE TPAMI 10, 4, 439–451. Google ScholarDigital Library
    17. GEORGHIADES, A. 2003. Recovering 3-D Shape and Reflectance From a Small Number of Photographs. In Proc. of EGSR 2003, 230–240. Google ScholarDigital Library
    18. HALSTEAD, M., BARSKY, B., KLEIN, S., AND MANDELL, R. 1996. Reconstructing Curved Surfaces From Specular Reflection Patterns Using Spline Surface Fitting of Normals. In Proc. of ACM SIGGRAPH 96, 335–342. Google ScholarDigital Library
    19. HUTCHINSON, T., WHITE, K., REICHERT, K., AND FREY, L. 1989. Human-computer Interaction using Eye-gaze Input. IEEE TSMC 19 (Nov./Dec.), 1527–1533.Google Scholar
    20. JACOB, R. 1990. What You Look At is What You Get: Eye Movement-Based Interaction Techniques. In Proc. of ACM CHI 90, 11–18. Google ScholarDigital Library
    21. KAUFMAN, P., AND ALM, A., Eds. 2003. Adler’s Physiology of the Eye: Clinical Application, 10th ed. Mosby.Google Scholar
    22. LEFOHN, A., CARUSO, R., REINHARD, E., BUDGE, B., AND SHIRLEY, P. 2003. An Ocularist’s Approach to Human Iris Synthesis. IEEE CGA 23, 6 (Nov./Dec.), 70–75. Google ScholarDigital Library
    23. LENSCH, H., KAUTZ, J., GOESELE, M., HEIDRICH, W., AND SEIDEL, H.-P. 2001. Image-Based Reconstruction of Spatially Varying Materials. In Proc. of EGWR 2001, 104–115. Google ScholarDigital Library
    24. LIN, Z., WONG, T.-T., AND SHUM, H.-Y. 2001. Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction. In Proc. of IEEE CVPR 2001, vol. 1, 561–567.Google Scholar
    25. MAGDA, S., KRIEGMAN, D., ZICKLER, T., AND BELHUMEUR, P. 2001. Beyond Lambert: Reconstructing Surfaces with Arbitrary BRDFs. In Proc. of IEEE ICCV 01, vol. II, 391–398.Google ScholarCross Ref
    26. MARSCHNER, S., AND GREENBERG, D. 1997. Inverse Lighting for Photography. In Proc. of IS&T/SID CIC, 262–265.Google Scholar
    27. MATUSIK, W., PFISTER, H., NGAN, A., BEARDSLEY, P., ZIEGLER, R., AND MCMILLAN, L. 2002. Image-based 3D Photography using Opacity Hulls. In Proc. of ACM SIGGRAPH 2002, 427–437. Google ScholarDigital Library
    28. MITSUNAGA, T., AND NAYAR, S. 1999. Radiometric Self Calibration. In Proc. of IEEE CVPR 99, vol. 1, 1374–1380.Google ScholarCross Ref
    29. MOSTAFAWY, S., KERMANI, O., AND LUBATSCHOWSKI, H. 1997. Virtual Eye: Retinal Image Visualization of the Human Eye. IEEE CGA 17, 1 (Jan./Feb.), 8–12. Google ScholarDigital Library
    30. NAYAR, S., AND MITSUNAGA, T. 2000. High Dynamic Range Imaging: Spatially Varying Pixel Exposures. In Proc. of IEEE CVPR 00, 1472–1479.Google Scholar
    31. NISHINO, K., AND NAYAR, S. 2004. Corneal imaging system: Environment from eyes. Tech. rep., Dept. of Computer Science, Columbia University. In preparation.Google Scholar
    32. RAMAMOORTHI, R., AND HANRAHAN, P. 2001. A Signal-Processing Framework for Inverse Rendering. In Proc. of ACM SIGGRAPH 2001. Google ScholarDigital Library
    33. SAGAR, M., BULLIVANT, D., MALLINSON, G., AND HUNTER, P. 1994. A Virtual Environment and Model of the Eye for Surgical Simulation. In Proc. of ACM SIGGRAPH 94, 205–212. Google ScholarDigital Library
    34. SATO, Y., WHEELER, M., AND IKEUCHI, K. 1997. Object shape and reflectance modeling from observation. In Proc. of ACM SIGGRAPH 97, 379–387. Google ScholarDigital Library
    35. SATO, I., SATO, Y., AND IKEUCHI, K. 2003. Illumination from Shadows. IEEE TPAMI 25, 3, 290–300. Google ScholarDigital Library
    36. STEIN, C. 1995. Accurate Internal Camera Calibration using Rotation, with Analysis of Sources of Errors. In Proc. of ICCV 95, 230–236. Google ScholarDigital Library
    37. SZELISKI, R., AND SHUM, H.-Y. 1997. Creating Full View Panoramic Image Mosaics and Environment Maps. In Proc. of ACM SIGGRAPH 97, 251–258. Google ScholarDigital Library
    38. TELLER, S., ANTONE, M., BODNER, Z., BOSSE, M., COORG, S., JETHWA, M., AND MASTER, N. 2003. Calibrated, Registered Images of an Extended Urban Area. IJCV 53, 1, 93–107. Google ScholarDigital Library
    39. TORRANCE, K., AND SPARROW, E. 1967. Theory for off-specular reflection from roughened surfaces. JOSA, 57, 1105–1114.Google ScholarCross Ref
    40. TSUMURA, N., DANG, M., MAKINO, T., AND MIYAKE, Y. 2003. Estimating the Directions to Light Sources Using Images of Eye for Reconstructing 3D Human Face. In Proc. of IS&T/SID’s CIC, 77–81.Google Scholar
    41. VON HELMHOLTZ, H. 1909. Physiologic Optics, third ed., vol. 1 and 2. Voss, Hamburg, Germany.Google Scholar
    42. WOODHAM, R. 1978. Photometric Stereo: A Reflectance Map Technique for Determining Surface Orientation from a Single View. In Proc. of SPIE, vol. 155, 136–143.Google Scholar
    43. YAGI, Y., AND YACHIDA, M. 1991. Real-time Generation of Environmental Map and Obstacle Avoidance using Omnidirectional Image Sensor with Conic Mirror. In Proc. of IEEE CVPR 91, 160–165.Google Scholar
    44. YU, Y., DEBEVEC, P., MALIK, J., AND HAWKINS, T. 1999. Inverse Globall Illumination: Recovering Reflectance Models of Real Scenes From Photographs. In Proc. of ACM SIGGRAPH 99, 215–224. Google ScholarDigital Library

ACM Digital Library Publication: