“Intrinsic video and applications” by Ye, Garces, Liu, Dai and Gutierrez
Conference:
Type:
Title:
- Intrinsic video and applications
Session/Category Title: Video Applications
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a method to decompose a video into its intrinsic components of reflectance and shading, plus a number of related example applications in video editing such as segmentation, stylization, material editing, recolorization and color transfer. Intrinsic decomposition is an ill-posed problem, which becomes even more challenging in the case of video due to the need for temporal coherence and the potentially large memory requirements of a global approach. Additionally, user interaction should be kept to a minimum in order to ensure efficiency. We propose a probabilistic approach, formulating a Bayesian Maximum a Posteriori problem to drive the propagation of clustered reflectance values from the first frame, and defining additional constraints as priors on the reflectance and shading. We explicitly leverage temporal information in the video by building a causal-anticausal, coarse-to-fine iterative scheme, and by relying on optical flow information. We impose no restrictions on the input video, and show examples representing a varied range of difficult cases. Our method is the first one designed explicitly for video; moreover, it naturally ensures temporal consistency, and compares favorably against the state of the art in this regard.
References:
1. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graph. (SIGGRAPH) 28, 3. Google ScholarDigital Library
2. Bhat, P., Zitnick, C. L., Cohen, M., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2. Google ScholarDigital Library
3. Bonneel, N., Sunkavalli, K., Paris, S., and Pfister, H. 2013. Example-based video color grading. ACM Trans. Graph. (SIGGRAPH) 32, 4. Google ScholarDigital Library
4. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. (SIGGRAPH Asia) 28, 5, 130. Google ScholarDigital Library
5. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV, Springer, 25–36.Google Scholar
6. Butler, D. J., Wulff, J., Stanley, G. B., and Black, M. J. 2012. A naturalistic open source movie for optical flow evaluation. In Proc. ECCV, Springer, 611–625. Google ScholarDigital Library
7. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Trans. Graph. (SIGGRAPH) 30, 4. Google ScholarDigital Library
8. Felzenszwalb, P. F., and Huttenlocher, D. P. 2004. Efficient graph-based image segmentation. International Journal of Computer Vision 59, 2, 167–181. Google ScholarDigital Library
9. Funt, B. V., Drew, M. S., and Brockington, M. 1992. Recovering shading from color images. In Proc. ECCV, Springer, 124–132. Google ScholarDigital Library
10. Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. In Computer Graphics Forum (EGSR), vol. 31, 1415–1424. Google ScholarDigital Library
11. Gehler, P. V., Rother, C., Kiefel, M., Zhang, L., and Schölkopf, B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In Proc. NIPS, 765.Google Scholar
12. Grundmann, M., Kwatra, V., Han, M., and Essa, I. 2010. Efficient hierarchical graph-based video segmentation. In Proc. CVPR, IEEE.Google Scholar
13. Hauagge, D., Wehrwein, S., Baval, K., and Snavely, N. 2013. Photometric ambient occlusion. In Proc. CVPR, IEEE. Google ScholarDigital Library
14. Jiang, X., Schofield, A. J., and Wyatt, J. L. 2010. Correlation-based intrinsic image extraction from a single image. In Proc. ECCV, Springer, 58–71. Google ScholarDigital Library
15. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. Graph. (SIGGRAPH Asia) 31. Google ScholarDigital Library
16. Land, E. H., and McCann, J. J. 1971. Lightness and retinex theory. Journal of the Optical Society of America 61, 1.Google ScholarCross Ref
17. Lang, M., Wang, O., Aydin, T., Smolic, A., and Gross, M. 2012. Practical temporal consistency for image-based graphics applications. ACM Trans. Graph. (SIGGRAPH) 31, 4. Google ScholarDigital Library
18. Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image + depth video. In Proc. ECCV, Springer, 327–340. Google ScholarDigital Library
19. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. (SIGGRAPH) 23, 3. Google ScholarDigital Library
20. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Trans. Graph. (SIGGRAPH) 23, 3, 303–308. Google ScholarDigital Library
21. Li, Y., Ju, T., and Hu, S.-M. 2010. Instant propagation of sparse edits on images and videos. In Computer Graphics Forum, vol. 29, Wiley Online Library, 2049–2054.Google Scholar
22. Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM Trans. Graph. (SIGGRAPH Asia) 27, 5, 152:1–152:9. Google ScholarDigital Library
23. Lombardi, S., and Nishino, K. 2012. Reflectance and natural illumination from a single image. In Proc. ECCV, Springer, 582–595. Google ScholarDigital Library
24. Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M. 2004. Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Trans. Pattern Anal. Mach. Intell. 26, 10, 1336–1347. Google ScholarDigital Library
25. Oskam, T., Hornung, A., Sumner, R., and Gross, M. 2012. Fast and stable color balancing for images and augmented reality. In Proc. 3DIMPVT, IEEE, 49–56. Google ScholarDigital Library
26. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In Proc. ECCV, Springer. Google ScholarDigital Library
27. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. Graph. (SIGGRAPH) 22, 3, 313–318. Google ScholarDigital Library
28. Shen, L., and Yeo, C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In Proc. CVPR, IEEE, 697–704. Google ScholarDigital Library
29. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. CVPR, IEEE, vol. 0, 1–7.Google Scholar
30. Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. CVPR, IEEE, 3481–3487. Google ScholarDigital Library
31. Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Trans. Graph. (SIGGRAPH) 26, 3, 101. Google ScholarDigital Library
32. Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 27, 9, 1459–1472. Google ScholarDigital Library
33. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In Proc. ICCV, IEEE, vol. 2, 68–75.Google Scholar
34. Xu, K., Li, Y., Ju, T., Hu, S.-M., and Liu, T.-Q. 2009. Efficient affinity-based edit propagation using kd tree. In ACM Trans. Graph. (SIGGRAPH Asia), vol. 28. Google ScholarDigital Library
35. Yan, X., Shen, J., He, Y., and Mao, X. 2010. Re-texturing by intrinsic video. In Proc. DICTA, IEEE, 486–491. Google ScholarDigital Library
36. Yatziv, L., and Sapiro, G. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. Image Processing, 15, 5, 1120–1129. Google ScholarDigital Library
37. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7, 1437–1444. Google ScholarDigital Library
