“Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography” by Debevec

We present a method that uses measured scene radiance and global illumination in order to add new objects to light-based models with correct lighting. The method uses a high dynamic range image-based model of the scene, rather than synthetic light sources, to illuminate the new objects. To compute the illumination, the scene is considered as three components: the distant scene, the local scene, and the synthetic objects. The distant scene is assumed to be photometrically unaffected by the objects, obviating the need for reflectance model information. The local scene is endowed with estimated reflectance model information so that it can catch shadows and receive reflected light from the new objects. Renderings are created with a standard global illumination method by simulating the interaction of light amongst the three components. A differential rendering technique allows for good results to be obtained when only an estimate of the local scene reflectance properties is known. We apply the general method to the problem of rendering synthetic objects into real scenes. The light-based model is constructed from an approximate geometric model of the scene and by using a light probe to measure the incident illumination at the location of the synthetic objects. The global illumination solution is then composited into a photograph of the scene using the differential rendering technique. We conclude by discussing the relevance of the technique to recovering surface reflectance properties in uncontrolled lighting situations. Applications of the method include visual effects, interior design, and architectural visualization.

1. ADELSON,E.H.,ANDBERGEN,J.R.ComputationalModelsofVisualPro-cessing. MITPress,Cambridge,Mass.,1991,ch.1.ThePlenopticFunctionand theElementsofEarlyVision.
2. AZARMI,M.OpticalEffectsCinematography:ItsDevelopment,Methods,and Techniques.UniversityMicrofilmsInternational,AnnArbor,Michigan,1973.
3. BLINN,J.F.Textureandreflectionincomputergeneratedimages.Communica-tionsoftheACM19, 10(October1976),542-547.
4. CHEN,E.QuickTimeVR-animage-basedapproachtovirtualenvironmentnav-igation. InSIGGRAPH’95(1995).
5. CHEN,S.E.Incrementalradiosity:Anextensionofprogressiveradiositytoan interactivesynthesissystem.InSIGGRAPH’90(1990),pp.135-144.
6. COHEN,M.F.,CHEN,S.E.,WALLACE,J.R.,ANDGREENBERG,D.P.Apro-gressiverefinementapproachtofastradiosityimagegeneration. InSIGGRAPH ’88(1988),pp.75-84.
7. CURLESS,B.,ANDLEVOY,M.Avolumetricmethodforbuildingcomplexmod-elsfromrangeimages. InSIGGRAPH’96(1996),pp.303-312.
8. DANA,K.J.,GINNEKEN,B.,NAYAR,S.K.,ANDKOENDERINK,J.J.Re-flectanceandtextureofrealworldsurfaces.InProc.IEEEConf.onComp.Vision andPatt.Recog.(1997),pp.151-157.
9. DEBEVEC,P.E.,ANDMALIK,J.Recoveringhighdynamicrangeradiancemaps fromphotographs.InSIGGRAPH’97(August1997),pp.369-378.
10. DEBEVEC,P.E.,TAYLOR,C.J.,ANDMALIK,J.Modelingandrenderingar-chitecturefromphotographs: Ahybridgeometry-andimage-basedapproach.In SIGGRAPH’96(August1996),pp.11-20.
11. DEBEVEC,P.E.,YU,Y.,ANDBORSHUKOV,G.D.Efficientview-dependent image-basedrenderingwithprojectivetexture-mapping.Tech.Rep.UCB//CSD-98- 1003,UniversityofCaliforniaatBerkeley,1998.
12. DRETTAKIS,G.,ROBERT,L.,ANDBOUGNOUX,S.Interactivecommonillu-minationforcomputeraugmentedreality. In8thEurographicsworkshoponRen-dering, St.Etienne,France(May1997),J.DorseyandP.Slusallek,Eds.,pp.45- 57
13. FIELDING,R.TheTechniqueofSpecialEffectsCinematography.Hastings House,NewYork,1968.
14. FOURNIER,A.,GUNAWAN,A.,ANDROMANZIN,C.Commonillumination betweenrealandcomputergeneratedscenes.InGraphicsInterface(May1993), pp.254-262.
15. GERSHBEIN,R.,SCHRODER,P.,ANDHANRAHAN,P.Texturesandradiosity: Controllingemissionandreflectionwithtexturemaps.InSIGGRAPH’94(1994).
16. GORAL,C.M.,TORRANCE,K.E.,GREENBERG,D.P.,ANDBATTAILE,B. Modelingtheinteractionoflightbetweendiffusesurfaces.InSIGGRAPH’84 (1984),pp.213-222.
17. GORTLER,S.J.,GRZESZCZUK,R.,SZELISKI,R.,ANDCOHEN,M.F.The Lumigraph.InSIGGRAPH’96(1996),pp.43-54.
18. HECKBERT,P.S.Surveyoftexturemapping.IEEEComputerGraphicsand Applications6,11(November1986),56-67.
19. KAJIYA,J.Therenderingequation.InSIGGRAPH’86(1986),pp.143-150.
20. KARNER,K.F.,MAYER,H.,ANDGERVAUTZ,M.Animagebasedmeasure-mentsystemforanisotropicreflection. InEUROGRAPHICSAnnualConference Proceedings(1996).
21. KOENDERINK,J.J.,ANDVANDOORN,A.J.Illuminancetextureduetosurface mesostructure.J.Opt.Soc.Am.13,3(1996).
22. LAVEAU,S.,ANDFAUGERAS,O.3-Dscenerepresentationasacollectionof images.InProceedingsof12thInternationalConferenceonPatternRecognition (1994),vol.1,pp.689-691.
23. LEVOY,M.,ANDHANRAHAN,P.Lightfieldrendering.InSIGGRAPH’96 (1996),pp.31-42.
24. MCMILLAN,L.,ANDBISHOP,G.PlenopticModeling:Animage-basedren-deringsystem. InSIGGRAPH’95(1995).
25. NAKAMAE,E.,HARADA,K.,ANDISHIZAKI,T.Amontagemethod:Theover-layingofthecomputergeneratedimagesontoabackgroundphotograph. InSIG-GRAPH’86( 1986),pp.207-214.
26. PORTER,T.,ANDDUFF,T.Compositingdigitalimages.InSIGGRAPH84(July 1984),pp.253-259.
27. SATO,Y.,WHEELER,M.D.,ANDIKEUCHI,K.Objectshapeandreflectance modelingfromobservation.InSIGGRAPH’97(1997),pp.379-387.
28. SMITH,T.G.IndustrialLightandMagic:TheArtofSpecialEffects.Ballantine Books,NewYork,1986.
29. SZELISKI,R.Imagemosaicingfortele-realityapplications.InIEEEComputer GraphicsandApplications(1996).
30. TURK,G.,ANDLEVOY,M.Zipperedpolygonmeshesfromrangeimages.In SIGGRAPH’94(1994),pp.311-318.
31. VEACH,E.,ANDGUIBAS,L.J.Metropolislighttransport.InSIGGRAPH’97 (August1997),pp.65-76.
32. WARD,G.J.Measuringandmodelinganisotropicreflection.InSIGGRAPH’92 (July1992),pp.265-272.
33. WARD,G.J.Theradiancelightingsimulationandrenderingsystem.InSIG-GRAPH’94( July1994),pp.459-472.
34. WATANABE,M.,ANDNAYAR,S.K.Telecentricopticsforcomputationalvi-sion. InProceedingsofImageUnderstandingWorkshop(IUW96)(February 1996).
35. Y.CHEN,ANDMEDIONI,G.Objectmodelingfrommultiplerangeimages.Im-ageandVisionComputing10, 3(April1992),145-155.