“Rendering synthetic objects into legacy photographs”
Conference:
Type(s):
Title:
- Rendering synthetic objects into legacy photographs
Session/Category Title: Image Mix and Match
Presenter(s)/Author(s):
Abstract:
We propose a method to realistically insert synthetic objects into existing photographs without requiring access to the scene or any additional scene measurements. With a single image and a small amount of annotation, our method creates a physical model of the scene that is suitable for realistically rendering synthetic objects with diffuse, specular, and even glowing materials while accounting for lighting interactions between the objects and the scene. We demonstrate in a user study that synthetic images produced by our method are confusable with real scenes, even for people who believe they are good at telling the difference. Further, our study shows that our method is competitive with other insertion methods while requiring less scene information. We also collected new illumination and reflectance datasets; renderings produced by our system compare well to ground truth. Our system has applications in the movie and gaming industry, as well as home decorating and user content creation, among others.
References:
1. Alnasser, M., and Foroosh, H. 2006. Image-based rendering of synthetic diffuse objects in natural scenes. In ICPR, 787–790. Google ScholarDigital Library
2. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. In Comp. Vision Sys., 3–26.Google Scholar
3. Blake, A. 1985. Boundary conditions for lightness computation in mondrian world. Computer Vision, Graphics and Image Processing 32, 314–327.Google ScholarCross Ref
4. Boivin, S., and Gagalowicz, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. ACM SIGGRAPH, 107–116. Google ScholarDigital Library
5. Brelstaff, G., and Blake, A. 1987. Computing lightness. Pattern Recognition Letters 5, 2, 129–138. Google ScholarDigital Library
6. Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. 30 (August), 43:1–43:10. Google ScholarDigital Library
7. Cossairt, O., Nayar, S., and Ramamoorthi, R. 2008. Light field transfer: global illumination between real and synthetic objects. ACM Trans. Graph. 27 (August), 57:1–57:6. Google ScholarDigital Library
8. Criminisi, A., Reid, I., and Zisserman, A. 2000. Single view metrology. Int. J. Comput. Vision 40 (November), 123–148. Google ScholarDigital Library
9. Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’98, 189–198. Google ScholarDigital Library
10. Farenzena, M., and Fusiello, A. 2007. Recovering intrinsic images using an illumination invariant image. In ICIP, 485–488.Google Scholar
11. Fournier, A., Gunawan, A. S., and Romanzin, C. 1993. Common illumination between real and computer generated scenes. In Proceedings of Graphics Interface ’93, 254–262.Google Scholar
12. Funt, B. V., Drew, M. S., and Brockington, M. 1992. Recovering shading from color images. In ECCV, 124–132. Google ScholarDigital Library
13. Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32 (August), 1362–1376. Google ScholarDigital Library
14. Greger, G., Shirley, P., Hubbard, P. M., and Greenberg, D. P. 1998. The irradiance volume. IEEE Computer Graphics and Applications 18, 32–43. Google ScholarDigital Library
15. Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2335–2342.Google Scholar
16. Guo, R., Dai, Q., and Hoiem, D. 2011. Single-image shadow detection and removal using paired regions. In CVPR, 2033–2040. Google ScholarDigital Library
17. Hartley, R., and Zisserman, A. 2003. Multiple View Geometry in Computer Vision, 2 ed. Cambridge University Press, New York, NY, USA. Google ScholarDigital Library
18. Hedau, V., Hoiem, D., and Forsyth, D. 2009. Recovering the spatial layout of cluttered rooms. In ICCV, 1849–1856.Google Scholar
19. Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM Trans. Graph. 24 (July), 577–584. Google ScholarDigital Library
20. Horn, B. K. P. 1974. Determining lightness from an image. Computer Vision, Graphics and Image Processing 3, 277–299.Google ScholarCross Ref
21. Horry, Y., Anjyo, K.-I., and Arai, K. 1997. Tour into the picture: using a spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’97, 225–232. Google ScholarDigital Library
22. Kang, H. W., Pyo, S. H., Anjyo, K., and Shin, S. Y. 2001. Tour into the picture using a vanishing line and its extension to panoramic images. Computer Graphics Forum 20, 3, 132–141.Google ScholarCross Ref
23. Kee, E., and Farid, H. 2010. Exposing digital forgeries from 3-d lighting environments. In WIFS, 1–6.Google Scholar
24. Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. ACM Trans. Graph. 25 (July), 654–663. Google ScholarDigital Library
25. Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In ICCV, 1–8.Google Scholar
26. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. Graph. 26 (July). Google ScholarDigital Library
27. Lalonde, J.-F., Efros, A. A., and Narasimhan, S. G. 2009. Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. 28 (December), 131:1–131:10. Google ScholarDigital Library
28. Land, E., and McCann, J. 1971. Lightness and retinex theory. J. Opt. Soc. Am. 61, 1, 1–11.Google ScholarCross Ref
29. Lee, D. C., Hebert, M., and Kanade, T. 2009. Geometric reasoning for single image structure recovery. In CVPR, 2136–2143.Google Scholar
30. Lee, D. C., Gupta, A., Hebert, M., and Kanade, T. 2010. Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. Advances in Neural Information Processing Systems (NIPS) 24 (November), 1288–1296.Google Scholar
31. Levin, A., Rav-Acha, A., and Lischinski, D. 2008. Spectral matting. IEEE PAMI 30 (October), 1699–1712. Google ScholarDigital Library
32. Liebowitz, D., Criminisi, A., and Zisserman, A. 1999. Creating architectural models from images. In Eurographics, vol. 18, 39–50.Google ScholarCross Ref
33. Lopez-Moreno, J., Hadap, S., Reinhard, E., and Gutierrez, D. 2010. Compositing images through light source detection. Computers & Graphics 34, 6, 698–707. Google ScholarDigital Library
34. Merrell, P., Schkufza, E., Li, Z., Agrawala, M., and Koltun, V. 2011. Interactive furniture layout using interior design guidelines. ACM Trans. Graph. 30 (August), 87:1–87:10. Google ScholarDigital Library
35. Mury, A. A., Pont, S. C., and Koenderink, J. J. 2009. Representing the light field in finite three-dimensional spaces from sparse discrete samples. Applied Optics 48, 3 (Jan), 450–457.Google ScholarCross Ref
36. Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’01, 433–442. Google ScholarDigital Library
37. Rother, C. 2002. A new approach to vanishing point detection in architectural environments. IVC 20, 9–10 (August), 647–655.Google ScholarCross Ref
38. Sato, I., Sato, Y., and Ikeuchi, K. 2003. Illumination from shadows. IEEE PAMI 25, 3, 290–300. Google ScholarDigital Library
39. Saxena, A., Sun, M., and Ng, A. Y. 2008. Make3d: depth perception from a single still image. In Proceedings of the 23rd national conference on Artificial intelligence – Volume 3, AAAI Press, 1571–1576. Google ScholarDigital Library
40. Sinha, S. N., Steedly, D., Szeliski, R., Agrawala, M., and Pollefeys, M. 2008. Interactive 3d architectural modeling from unordered photo collections. ACM Trans. Graph. 27 (December), 159:1–159:10. Google ScholarDigital Library
41. Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27 (September), 1459–1472. Google ScholarDigital Library
42. Tappen, M. F., Adelson, E. H., and Freeman, W. T. 2006. Estimating intrinsic component images using non-linear regression. In CVPR, vol. 2, 1992–1999. Google ScholarDigital Library
43. Wang, Y., and Samaras, D. 2003. Estimation of multiple directional light sources for synthesis of augmented reality images. Graphical Models 65, 4, 185–205. Google ScholarDigital Library
44. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV, II: 68–75.Google Scholar
45. Yeung, S.-K., Tang, C.-K., Brown, M. S., and Kang, S. B. 2011. Matting and compositing of transparent and refractive objects. ACM Trans. Graph. 30 (February), 2:1–2:13. Google ScholarDigital Library
46. Yu, Y., Debevec, P., Malik, J., and Hawkins, T. 1999. Inverse global illumination: recovering reflectance models of real scenes from photographs. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’99, 215–224. Google ScholarDigital Library
47. Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., and Osher, S. J. 2011. Make it home: automatic optimization of furniture arrangement. ACM Trans. Graph. 30 (August), 86:1–86:12. Google ScholarDigital Library
48. Zhang, L., Dugas-Phocion, G., Samson, J., and Seitz, S. 2001. Single view modeling of free-form scenes. In CVPR, 990–997.Google Scholar
