“Multi-View Intrinsic Images of Outdoors Scenes With an Application to Relighting” by Duchene, Riant, Chaurasia, Lopez-Moreno, Laffont, et al. …

  • ©Sylvain Duchene, Clement Riant, Gaurav Chaurasia, Jorge Lopez-Moreno, Pierre-Yves Laffont, Stefan Popov, Adrien Bousseau, and George Drettakis

Conference:


Type:


Title:

    Multi-View Intrinsic Images of Outdoors Scenes With an Application to Relighting

Session/Category Title: INTRINSIC IMAGES

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We introduce a method to compute intrinsic images for a multiview set of outdoor photos with cast shadows, taken under the same lighting. We use an automatic 3D reconstruction from these photos and the sun direction as input and decompose each image into reflectance and shading layers, despite the inaccuracies and missing data of the 3D model. Our approach is based on two key ideas. First, we progressively improve the accuracy of the parameters of our image formation model by performing iterative estimation and combining 3D lighting simulation with 2D image optimization methods. Second, we use the image formation model to express reflectance as a function of discrete visibility values for shadow and light, which allows to introduce a robust visibility classifier for pairs of points in a scene. This classifier is used for shadow labeling, allowing to compute high-quality reflectance and shading layers. Our multiview intrinsic decomposition is of sufficient quality to allow relighting of the input images. We create shadow-caster geometry which preserves shadow silhouettes and, using the intrinsic layers, we can perform multiview relighting with moving cast shadows. We present results on several multiview datasets, and show how it is now possible to perform image-based rendering with changing illumination conditions.

References:


    1. J. Barron and J. Malik. 2013a. Shape, illumination, and reflectance from shading. Tech. rep., Berkeley. http://www.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-117.pdf.
    2. J. T. Barron and J. Malik. 2013b. Intrinsic scene properties from a single RGB-D image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). 17–24. 
    3. H. G. Barrow and J. M. Tenenbaum. 1978. Recovering intrinsic scene characteristics from images. Comput. Vis. Syst. 3, 3–26.
    4. S. Bell, K. Bala, and N. Snavely. 2014. Intrinsic images in the wild. ACM Trans. Graph. 33, 4. 
    5. A. Bousseau, S. Paris, and F. Durand. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 5, 1–10. 
    6. G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis. 2013. Depth synthesis and local warps for plausible imagebased navigation. ACM Trans. Graph. 32, 3. 
    7. Q. Chen and V. Koltun. 2013. A simple model for intrinsic image decomposition with depth cues. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). 241–248. 
    8. D. Comaniciu and P. Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24, 5, 603–619. 
    9. P. Debevec, C. Tchou, A. Gardner, T. Hawkins, C. Poullis, J. Stumpfel, A. Jones, N. Yun, P. Einarsson, T. Lundgren, M. Fajardo, and P. Martinez. 2004. Estimating surface reflectance properties of a complex scene under captured natural illumination. Tech. rep., USC Institute for Creative Technologies. http://ict.usc.edu/pubs/ICT-TR-06.2004.pdf.
    10. G. D. Finlayson, M. S. Drew, and C. Lu. 2004. Intrinsic images by entropy minimization. In Proceedings of the 8th European Conference on Computer Vision (ECCV’04). 582–595.
    11. Y. Furukawa and J. Ponce. 2007. Accurate, dense, and robust multiview stereopsis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). 1–8.
    12. E. Garces, A. Munoz, J. Lopez-Moreno, and D. Gutierrez. 2012. Intrinsic images by clustering. Comput. Graph. Forum 31, 4. 
    13. M. Goesele, J. Ackermann, S. Fuhrmann, C. Haubold, and R. Klowsky. 2010. Ambient point clouds for view interpolation. ACM Trans. Graph. 29, 4. 
    14. M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. M. Seitz. 2007. Multi-view stereo for community photo collections. In Proceedings of the 11th International Conference on Computer Vision (ICCV’07). 1–8.
    15. R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In Proceedings of the 11th International Conference on Compter Vision (ICCV’07).
    16. R. Guo, Q. Dai, and D. Hoiem. 2011. Single-image shadow detection and removal using paired regions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 2033–2040. 
    17. R. Guo, Q. Dai, and D. Hoiem. 2012. Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35, 12, 2956–2967. 
    18. T. Haber, C. Fuchs, P. Bekaer, H.-P. Seidel, M. Goesele, and H. Lensch. 2009. Relighting objects from image collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 627–634.
    19. K. Jacobs and C. Loscos. 2006. Classification of illumination methods for mixed reality. Comput. Graph. Forum 25, 29–51.
    20. K. Karsch, V. Hedau, D. Forsyth, and D. Hoiem. 2011. Rendering synthetic objects into legacy photographs. ACM Trans. Graph. 30, 6. 
    21. V. Kolmogorov. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE Trans. Pattern Anal. Mach. Intell. 28, 10, 1568–1583. 
    22. P.-Y. Laffont. 2012. Intrinsic image decomposition from multiple photographs. Ph.D. thesis, University of Nice Sophia-Antipolis.
    23. P.-Y. Laffont, A. Bousseau, and G. Drettakis. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Visual. Comput. Graph. 19, 2, 210–224. 
    24. P.-Y. Laffont, A. Bousseau, S. Paris, F. Durand, and G. Drettakis. 2012. Coherent intrinsic images from photo collections. ACM Trans. Graph. 31, 6. 
    25. J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan. 2009. Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. 28, 5. 
    26. J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan. 2010. Detecting ground shadows in outdoor consumer photographs. In Proceedings of the 11th European Conference on Computer Vision (ECCV’10). 322–335. 
    27. J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan. 2011. Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. 98, 2, 123–145. 
    28. E. H. Land and J. J. Mccann. 1971. Lightness and retinex theory. J. Optic. Soc. Amer. 61, 1.
    29. K. J. Lee, Q. Zhao, X. Tong, M. Gong, S. Izadi, S. U. Lee, P. Tan, and S. Lin. 2012. Estimation of intrinsic image sequences from image+depth video. In Proceedings of the 12th European Conference on Computer Vision (ECCV’12). 327–340. 
    30. A. Levin, D. Lischinski, and Y. Weiss. 2008. A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2, 228–242. 
    31. X. Liu, L. Wan, Y. Qu, T.-T. Wong, S. Lin, C.-S. Leung, and P.-A. Heng. 2008. Intrinsic colorization. ACM Trans. Graph. 27, 152:1–152:9. 
    32. C. Loscos, M.-C. Frasson, G. Drettakis, B. Walter, X. Granier, and P. Poulin. 1999. Interactive virtual relighting and remodeling of real scenes. In Proceedings of the 10th Eurographics Conference on Rendering (EGWR’99). 329–340. 
    33. M. Niessner, M. Zollhöfer, S. Izadi, and M. Stamminger. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM Trans. Graph. 32, 6. 
    34. M. Okabe, G. Zeng, Y. Matsushita, T. Igarashi, L. Quan, and H. Yeung Shum. 2006. Single-view relighting with normal map painting. In Proceedings of Pacific Graphics Conference (PG’06). 27–34.
    35. I. Omer and M. Werman. 2004. Color lines: Image specific color representation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04). 946–953. 
    36. A. Panagopoulos, C. Wang, D. Samaras, and N. Paragios. 2013. Simultaneous cast shadows, illumination and geometry inference using hypergraphs. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2, 437–449. 
    37. R. Perez, R. Seals, and J. Michalsky. 1993. All-weather model for sky luminance distribution – Preliminary configuration and validation. Solar Energy 50, 3, 235–245.
    38. J.-P. Pons, R. Keriven, and O. Faugeras. 2007. Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. The Int. J. Comput. Vis. 72, 2, 179–193. 
    39. A. J. Preetham, P. Shirley, and B. Smits. 1999. A practical analytic model for daylight. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’99). 91–100. 
    40. A. Sanin, C. Sanderson, and B. C. Lovell. 2012. Shadow detection: A survey and comparative evaluation of recent methods. Pattern Recogn. 45, 4, 1684–1695. 
    41. S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini, and P. Perona. 2007. 3D reconstruction by shadow carving: Theory and practical evaluation. Int. J. Comput. Vis. 71, 3, 305–336. 
    42. Q. Shan, R. Adams, B. Curless, Y. Furukawa, and S. M. Seitz. 2013. The visual turing test for scene reconstruction. In Proceedings of the International Conference on 3D Vision (3DV’13). IEEE Computer Society, 25–32. 
    43. J. Shen, X. Yang, Y. Jia, and X. Li. 2011. Intrinsic images using optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 3481–3487. 
    44. L. Shen and C. Yeo. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 697–704. 
    45. Y. Shih, S. Paris, F. Durand, and W. T. Freeman. 2013. Datadriven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32, 6. 
    46. Y. Shor and D. Lischinski. 2008. The shadow meets the mask: Pyramid-based shadow removal. Comput. Graph. Forum 27, 2, 577–586.
    47. N. Snavely, S. M. Seitz, and R. Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. ACM Trans. Graph. 25, 3, 835–846. 
    48. K. Sunkavalli, W. Matusik, H. Pfister, and S. Rusinkiewicz. 2007. Factored time-lapse video. ACM Trans. Graph. 26, 3. 
    49. R. Szeliski. 2010. Computer Vision: Algorithms and Applications, 1st ed. Springer. 
    50. L. Tao, L. Yuan, and J. Sun. 2009. Skyfinder: Attribute-based sky image search. ACM Trans. Graph. 28, 3. 
    51. A. Troccoli and P. Allen. 2008. Building illumination coherent 3d models of large-scale outdoor scenes. Int. J. Comput. Vis. 78, 2–3, 261–280. 
    52. Y. Weiss. 2001. Deriving intrinsic images from image sequences. In Proceedings of the 8th International Conference on Computer Vision (ICCV’01). 68–75.
    53. T.-P. Wu, C.-K. Tang, M. S. Brown, and H.-Y. Shum. 2007. Natural shadow matting. ACM Trans. Graph. 26, 2. 
    54. G. Xing, X. Zhou, Q. Peng, Y. Liu, and X. Qin. 2013. Lighting simulation of augmented outdoor scene based on a legacy photograph. Comput. Graph. Forum 32, 7, 101–110.
    55. G. Ye, E. Garces, Y. Liu, Q. Dai, and D. Gutierrez. 2014. Intrinsic video and applications. ACM Trans. Graph. 33, 4. 
    56. Y. Yu, P. Debevec, J. Malik, and T. Hawkins. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’99). 215–224. 
    57. Y. Yu and J. Malik. 1998. Recovering photometric properties of architectural scenes from photographs. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’98). 207–217. 
    58. Q. Zhao, P. Tan, Q. Dai, L. Shen, E. Wu, and S. Lin. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7, 1437–1444. 
    59. J. Zhu, K. G. G. Samuel, S. Masood, and M. F. Tappen. 2010. Learning to recognize shadows in monochromatic natural images. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’10). 223–230.

ACM Digital Library Publication:


Overview Page: