“Image-based rendering for scenes with reflections” by Sinha, Kopf, Goesele, Scharstein and Szeliski

We present a system for image-based modeling and rendering of real-world scenes containing reflective and glossy surfaces. Previous approaches to image-based rendering assume that the scene can be approximated by 3D proxies that enable view interpolation using traditional back-to-front or z-buffer compositing. In this work, we show how these can be generalized to multiple layers that are combined in an additive fashion to model the reflection and transmission of light that occurs at specular surfaces such as glass and glossy materials. To simplify the analysis and rendering stages, we model the world using piecewise-planar layers combined using both additive and opaque mixing of light. We also introduce novel techniques for estimating multiple depths in the scene and separating the reflection and transmission components into different layers. We then use our system to model and render a variety of real-world scenes with reflections.

1. Beery, E., and Yeredor, A. 2008. Blind separation of superimposed shifted images using parameterized joint diagonalization. IEEE Transactions on Image Processing 17, 3 (March), 340–353. Google ScholarDigital Library
2. Bergen, J. R., Burt, P. J., Hingorani, R., and Peleg, S. 1992. A three-frame algorithm for estimating two-component image motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9 (September), 886–896. Google ScholarDigital Library
3. Bhat, D. N., and Nayar, S. K. 1998. Stereo and specular reflection. International Journal of Computer Vision 26, 2 (February), 91–106. Google ScholarDigital Library
4. Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering, 327–338. Google ScholarDigital Library
5. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (November), 1222–1239. Google ScholarDigital Library
6. Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured Lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH ’01, 425–432. Google ScholarDigital Library
7. Carceroni, R. L., and Kutulakos, K. N. 2002. Multi-view scene capture by surfel sampling: From video streams to nonrigid 3d motion, shape and reflectance. International Journal of Computer Vision 49, 2/3, 175–214. Google ScholarDigital Library
8. Chen, S., and Williams, L. 1993. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’93, 279–288. Google ScholarDigital Library
9. Cohen-Steiner, D., Alliez, P., and Desbrun, M. 2004. Variational shape approximation. ACM Trans. Graph. 23 (August), 905–914. Google ScholarDigital Library
10. Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. Computer Vision and Image Understanding 97, 1 (January), 51–85. Google ScholarDigital Library
11. Darrell, T., and Pentland, A. 1995. Cooperative robust estimation using layers of support. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 5 (May), 474–487. Google ScholarDigital Library
12. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In ACM SIGGRAPH 1996 Conference Proceedings, 11–20. Google ScholarDigital Library
13. Diamant, Y., and Schechner, Y. Y. 2008. Overcoming visual reverberations. In Computer Vision and Pattern Recognition (CVPR’08), 1–8.Google Scholar
14. Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. 2009. Manhattan-world stereo. In Computer Vision and Pattern Recognition (CVPR 2009), 1422–1429.Google Scholar
15. Gallup, D., Frahm, J.-M., and Pollefeys, M. 2010. Piece-wise planar and non-planar stereo for urban scene reconstruction. In Computer Vision and Pattern Recognition (CVPR’10), 1418–1425.Google Scholar
16. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH ’96, 43–54. Google ScholarDigital Library
17. Hirschmüller, H. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (February), 328–341. Google ScholarDigital Library
18. Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions. International Journal of Computer Vision 12, 1 (January), 5–16. Google ScholarDigital Library
19. Ju, S. X., Black, M. J., and Jepson, A. D. 1996. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In Computer Vision and Pattern Recognition (CVPR’96), 307–314. Google ScholarDigital Library
20. Levin, A., Zomet, A., and Weiss, Y. 2002. Learning to perceive transparency from the statistics of natural scenes. In (NIPS), MIT Press, 1247–1254.Google Scholar
21. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH ’96, 31–42. Google ScholarDigital Library
22. Nocedal, J., and Wright, S. J. 2006. Numerical Optimization, second ed. Springer, New York.Google Scholar
23. Popescu, V., Mei, C., Dauble, J., and Sacks, E. 2006. Reflected-scene impostors for realistic reflections at interactive rates. Computer Graphics Forum 25, 3 (Sept.), 313–322.Google ScholarCross Ref
24. Schechner, Y. Y., Shamir, J., and Kiryati, N. 1999. Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface. In International Conference on Computer Vision (ICCV’99), 814–819. Google ScholarDigital Library
25. Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In ACM SIGGRAPH 1998 Conference Proceedings, 231–242. Google ScholarDigital Library
26. Shizawa, M., and Mase, K. 1991. A unified computational theory of motion transparency and motion boundaries based on eigenenergy analysis. In Computer Vision and Pattern Recognition (CVPR), 289–295.Google Scholar
27. Sinha, S. N., Steedly, D., and Szeliski, R. 2009. Piece-wise planar stereo for image-based rendering. In International Conference on Computer Vision (ICCV 2009), 1881–1888.Google Scholar
28. Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics (Proc. SIGGRAPH 2006) 25, 3 (August), 835–846. Google ScholarDigital Library
29. Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Computer Vision and Pattern Recognition (CVPR’2000), vol. 1, 246–253.Google Scholar
30. Szeliski, R. 2010. Computer Vision: Algorithms and Applications. Springer, New York. Google ScholarDigital Library
31. Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 2 (February), 290–301. Google ScholarDigital Library
32. Weingarten, J. W., Gruener, G., and Dorf, A. 2004. Probabilistic plane fitting in 3D and an application to robotic mapping. In International Conference on Robotics and Automation (ICRA), 927–932.Google Scholar
33. Zebedin, L., Bauer, J., Karner, K. F., and Bischof, H. 2008. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In European Conference on Computer Vision (ECCV’08), 873–886. Google ScholarDigital Library
34. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics (Proc. SIGGRAPH 2004) 23, 3 (August), 600–608. Google ScholarDigital Library