“Intrinsic video and applications” by Ye, Garces, Liu, Dai and Gutierrez

  • ©Genzhi Ye, Elena Garces, Yebin Liu, Qionghai Dai, and Diego Gutierrez




    Intrinsic video and applications


Session Title: Video Applications



    We present a method to decompose a video into its intrinsic components of reflectance and shading, plus a number of related example applications in video editing such as segmentation, stylization, material editing, recolorization and color transfer. Intrinsic decomposition is an ill-posed problem, which becomes even more challenging in the case of video due to the need for temporal coherence and the potentially large memory requirements of a global approach. Additionally, user interaction should be kept to a minimum in order to ensure efficiency. We propose a probabilistic approach, formulating a Bayesian Maximum a Posteriori problem to drive the propagation of clustered reflectance values from the first frame, and defining additional constraints as priors on the reflectance and shading. We explicitly leverage temporal information in the video by building a causal-anticausal, coarse-to-fine iterative scheme, and by relying on optical flow information. We impose no restrictions on the input video, and show examples representing a varied range of difficult cases. Our method is the first one designed explicitly for video; moreover, it naturally ensures temporal consistency, and compares favorably against the state of the art in this regard.


    1. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graph. (SIGGRAPH) 28, 3. Google ScholarDigital Library
    2. Bhat, P., Zitnick, C. L., Cohen, M., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2. Google ScholarDigital Library
    3. Bonneel, N., Sunkavalli, K., Paris, S., and Pfister, H. 2013. Example-based video color grading. ACM Trans. Graph. (SIGGRAPH) 32, 4. Google ScholarDigital Library
    4. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. (SIGGRAPH Asia) 28, 5, 130. Google ScholarDigital Library
    5. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV, Springer, 25–36.Google Scholar
    6. Butler, D. J., Wulff, J., Stanley, G. B., and Black, M. J. 2012. A naturalistic open source movie for optical flow evaluation. In Proc. ECCV, Springer, 611–625. Google ScholarDigital Library
    7. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Trans. Graph. (SIGGRAPH) 30, 4. Google ScholarDigital Library
    8. Felzenszwalb, P. F., and Huttenlocher, D. P. 2004. Efficient graph-based image segmentation. International Journal of Computer Vision 59, 2, 167–181. Google ScholarDigital Library
    9. Funt, B. V., Drew, M. S., and Brockington, M. 1992. Recovering shading from color images. In Proc. ECCV, Springer, 124–132. Google ScholarDigital Library
    10. Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. In Computer Graphics Forum (EGSR), vol. 31, 1415–1424. Google ScholarDigital Library
    11. Gehler, P. V., Rother, C., Kiefel, M., Zhang, L., and Schölkopf, B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In Proc. NIPS, 765.Google Scholar
    12. Grundmann, M., Kwatra, V., Han, M., and Essa, I. 2010. Efficient hierarchical graph-based video segmentation. In Proc. CVPR, IEEE.Google Scholar
    13. Hauagge, D., Wehrwein, S., Baval, K., and Snavely, N. 2013. Photometric ambient occlusion. In Proc. CVPR, IEEE. Google ScholarDigital Library
    14. Jiang, X., Schofield, A. J., and Wyatt, J. L. 2010. Correlation-based intrinsic image extraction from a single image. In Proc. ECCV, Springer, 58–71. Google ScholarDigital Library
    15. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. Graph. (SIGGRAPH Asia) 31. Google ScholarDigital Library
    16. Land, E. H., and McCann, J. J. 1971. Lightness and retinex theory. Journal of the Optical Society of America 61, 1.Google ScholarCross Ref
    17. Lang, M., Wang, O., Aydin, T., Smolic, A., and Gross, M. 2012. Practical temporal consistency for image-based graphics applications. ACM Trans. Graph. (SIGGRAPH) 31, 4. Google ScholarDigital Library
    18. Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image + depth video. In Proc. ECCV, Springer, 327–340. Google ScholarDigital Library
    19. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. (SIGGRAPH) 23, 3. Google ScholarDigital Library
    20. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. (SIGGRAPH) 23, 3, 303–308. Google ScholarDigital Library
    21. Li, Y., Ju, T., and Hu, S.-M. 2010. Instant propagation of sparse edits on images and videos. In Computer Graphics Forum, vol. 29, Wiley Online Library, 2049–2054.Google Scholar
    22. Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM Trans. Graph. (SIGGRAPH Asia) 27, 5, 152:1–152:9. Google ScholarDigital Library
    23. Lombardi, S., and Nishino, K. 2012. Reflectance and natural illumination from a single image. In Proc. ECCV, Springer, 582–595. Google ScholarDigital Library
    24. Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M. 2004. Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Trans. Pattern Anal. Mach. Intell. 26, 10, 1336–1347. Google ScholarDigital Library
    25. Oskam, T., Hornung, A., Sumner, R., and Gross, M. 2012. Fast and stable color balancing for images and augmented reality. In Proc. 3DIMPVT, IEEE, 49–56. Google ScholarDigital Library
    26. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In Proc. ECCV, Springer. Google ScholarDigital Library
    27. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. (SIGGRAPH) 22, 3, 313–318. Google ScholarDigital Library
    28. Shen, L., and Yeo, C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In Proc. CVPR, IEEE, 697–704. Google ScholarDigital Library
    29. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. CVPR, IEEE, vol. 0, 1–7.Google Scholar
    30. Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. CVPR, IEEE, 3481–3487. Google ScholarDigital Library
    31. Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Trans. Graph. (SIGGRAPH) 26, 3, 101. Google ScholarDigital Library
    32. Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27, 9, 1459–1472. Google ScholarDigital Library
    33. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. ICCV, IEEE, vol. 2, 68–75.Google Scholar
    34. Xu, K., Li, Y., Ju, T., Hu, S.-M., and Liu, T.-Q. 2009. Efficient affinity-based edit propagation using kd tree. In ACM Trans. Graph. (SIGGRAPH Asia), vol. 28. Google ScholarDigital Library
    35. Yan, X., Shen, J., He, Y., and Mao, X. 2010. Re-texturing by intrinsic video. In Proc. DICTA, IEEE, 486–491. Google ScholarDigital Library
    36. Yatziv, L., and Sapiro, G. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. Image Processing, 15, 5, 1120–1129. Google ScholarDigital Library
    37. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7, 1437–1444. Google ScholarDigital Library

ACM Digital Library Publication: