“Interactive intrinsic video editing” by Bonneel, Sunkavalli, Tompkin, Sun, Paris, et al. …
Conference:
Type(s):
Title:
- Interactive intrinsic video editing
Session/Category Title: Moving Pictures
Presenter(s)/Author(s):
Abstract:
Separating a photograph into its reflectance and illumination intrinsic images is a fundamentally ambiguous problem, and state-of-the-art algorithms combine sophisticated reflectance and illumination priors with user annotations to create plausible results. However, these algorithms cannot be easily extended to videos for two reasons: first, näively applying algorithms designed for single images to videos produce results that are temporally incoherent; second, effectively specifying user annotations for a video requires interactive feedback, and current approaches are orders of magnitudes too slow to support this. We introduce a fast and temporally consistent algorithm to decompose video sequences into their reflectance and illumination components. Our algorithm uses a hybrid ℓ2ℓp formulation that separates image gradients into smooth illumination and sparse reflectance gradients using look-up tables. We use a multi-scale parallelized solver to reconstruct the reflectance and illumination from these gradients while enforcing spatial and temporal reflectance constraints and user annotations. We demonstrate that our algorithm automatically produces reasonable results, that can be interactively refined by users, at rates that are two orders of magnitude faster than existing tools, to produce high-quality decompositions for challenging real-world video sequences. We also show how these decompositions can be used for a number of video editing applications including recoloring, retexturing, illumination editing, and lighting-aware compositing.
References:
1. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: Robust video object cutout using localized classifiers. ACM Trans. on Graph. (SIGGRAPH) 28, 3.
2. Barron, J. T., and Malik, J. 2012. Color constancy, intrinsic images, and shape estimation. ECCV.
3. Barron, J. T., and Malik, J. 2013. Intrinsic scene properties from a single RGB-D image. In CVPR
4. Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems.
5. Bell, S., Bala, K., and Snavely, N. 2014. Intrinsic images in the wild. ACM Trans. Graph. (SIGGRAPH) 33, 4.
6. Björck, A. 1996. Numerical Methods for Least Squares Problems. SIAM.
7. Bonneel, N., Sunkavalli, K., Paris, S., and Pfister, H. 2013. Example-based video color grading. ACM Trans. Graph. (SIGGRAPH) 32, 4.
8. Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. (SIGGRAPH) 28, 5.
9. Carroll, R., Ramamoorthi, R., and Agrawala, M. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Trans. Graph. (SIGGRAPH) 30, 4.
10. Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In ICCV.
11. Dong, Y., Tong, X., Pellacini, F., and Guo, B. 2011. Appgen: Interactive material modeling from a single image. ACM Trans. Graph. (SIGGRAPH Asia) 30, 6.
12. Dong, W., 2014. LSHKIT: A C++ locality sensitive hashing library.
13. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Trans. Graph. (SIGGRAPH) 30, 4.
14. Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. Computer Graphics Forum (EGSR) 31, 4.
15. Gehler, P., Rother, C., Kiefel, M., Zhang, L., and Schlkopf, B. 2011. Recovering intrinsic images with a global sparsity prior on reflectance. In NIPS.
16. Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2335–2342.
17. Hsu, E., Mertens, T., Paris, S., Avidan, S., and Durand, F. 2008. Light mixture estimation for spatially varying white balance. ACM Trans. Graph. (SIGGRAPH).
18. Korman, S., and Avidan, S. 2011. Coherency sensitive hashing. In ECCV.
19. Krishnan, D., and Fergus., R. 2009. Fast image deconvolution using hyper-laplacian priors. In NIPS.
20. Laffont, P.-Y., Bousseau, A., Paris, S., Durand, F., and Drettakis, G. 2012. Coherent intrinsic images from photo collections. ACM Trans. Graph. (SIGGRAPH Asia) 31, 6.
21. Laffont, P.-Y., Bousseau, A., and Drettakis, G. 2013. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Vis. Comput. Graph. 19, 2, 210–224.
22. Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America 61, 1–11.Cross Ref
23. Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In ECCV.
24. Li, Y., and Brown, M. S. 2014. Single image layer separation using relative smoothness. In CVPR.
25. Liu, C. 2009. Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, MIT.
26. Matsushita, Y., Nishino, K., Ikeuchi, K., and Sakauchi, M. 2004. Illumination normalization with time-dependent intrinsic images for video surveillance. PAMI 26, 10.
27. Omer, I., and Werman, M. 2004. Color lines: Image specific color representation. In CVPR.
28. Paris, S. 2008. Edge-preserving smoothing and mean-shift segmentation of video streams. In ECCV.
29. Pharr, M., and Humphreys, G. 2010. Physically Based Rendering, Second Edition: From Theory To Implementation.
30. Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Amarasinghe, S. P., and Durand, F. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. ACM Trans. Graph. (SIGGRAPH) 31, 4.
31. Shen, L., and Yeo, C. 2011. Intrinsic images decomposition using a local and global sparse representation of reflectance. In CVPR.
32. Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In CVPR.
33. Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In CVPR.
34. Tappen, M., Freeman, W., and Adelson, E. 2005. Recovering intrinsic images from a single image. PAMI 27, 9, 1459–1472.
35. Weiss, Y. 2001. Deriving intrinsic images from image sequences. In ICCV.
36. Ye, G., Garces, E., Liu, Y., Dai, Q., and Gutierrez, D. 2014. Intrinsic Video and Applications. ACM Trans. Graph. (SIGGRAPH) 33, 4.
37. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. PAMI 34, 7, 1437–1444.
