“Interactive High-Quality Green-Screen Keying via Color Unmixing” by Lien, Tan and Gingold
Conference:
Type(s):
Title:
- Interactive High-Quality Green-Screen Keying via Color Unmixing
Session/Category Title: Color & Compositing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Due to the widespread use of compositing in contemporary feature films, green-screen keying has become an essential part of postproduction workflows. To comply with the ever-increasing quality requirements of the industry, specialized compositing artists spend countless hours using multiple commercial software tools, while eventually having to resort to manual painting because of the many shortcomings of these tools. Due to the sheer amount of manual labor involved in the process, new green-screen keying approaches that produce better keying results with less user interaction are welcome additions to the compositing artist’s arsenal. We found that—contrary to the common belief in the research community—production-quality green-screen keying is still an unresolved problem with its unique challenges. In this article, we propose a novel green-screen keying method utilizing a new energy minimization-based color unmixing algorithm. We present comprehensive comparisons with commercial software packages and relevant methods in literature, which show that the quality of our results is superior to any other currently available green-screen keying solution. It is important to note that, using the proposed method, these high-quality results can be generated using only one-tenth of the manual editing time that a professional compositing artist requires to process the same content having all previous state-of-the-art tools at one’s disposal.
References:
1. R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 11, 2274–2282. Google ScholarDigital Library
2. Xue Bai, Jue Wang, and David Simons. 2011. Towards temporally-coherent video matting. In Proceedings of MIRAGE. Google ScholarDigital Library
3. Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. 2009. Video SnapCut: Robust video object cutout using localized classifiers. ACM Transactions on Graphics 28, 3, 70:1–70:11. Google ScholarDigital Library
4. Dimitri P. Bertsekas. 1982. The method of multipliers for equality constrained problems. In Constrained Optimization and Lagrange Multiplier Methods. Academic Press, New York, NY, 96–157.Google Scholar
5. Y. Boykov and V. Kolmogorov. 2004. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 9, 1124–1137. Google ScholarDigital Library
6. Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11, 1222–1239. Google ScholarDigital Library
7. Robert Carroll, Ravi Ramamoorthi, and Maneesh Agrawala. 2011. Illumination decomposition for material recoloring with consistent interreflections. ACM Transactions on Graphics 30, 4, 43:1–43:10. Google ScholarDigital Library
8. Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Transactions on Graphics 34, 4, 139:1–139:11. Google ScholarDigital Library
9. Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. 2013a. KNN matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 9, 2175–2188. Google ScholarDigital Library
10. Xiaowu Chen, Dongqing Zou, S. Z. Zhou, Qinping Zhao, and Ping Tan. 2013b. Image matting with local and nonlocal smooth priors. In Proceedings of CVPR. Google ScholarDigital Library
11. Inchang Choi, Minhaeng Lee, and Yu-Wing Tai. 2012. Video matting using multi-frame nonlocal matting Laplacian. In Proceedings of ECCV. Google ScholarDigital Library
12. Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. 2002. Video matting of complex scenes. Transactions on Graphics 243–248. Google ScholarDigital Library
13. Yung-Yu Chuang, Brian Curless, David H. Salesin, and Richard Szeliski. 2001. A Bayesian approach to digital matting. In Proceedings of CVPR.Google Scholar
14. Jialue Fan, Xiaohui Shen, and Ying Wu. 2012. Scribble tracker: A matting-based approach for robust tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 8, 1633–1644. Google ScholarDigital Library
15. Eduardo S. L. Gastal and Manuel M. Oliveira. 2010. Shared sampling for real-time alpha matting. Computer Graphics Forum 29, 2, 575–584.Google ScholarCross Ref
16. Anselm Grundhöfer and Oliver Bimber. 2008. VirtualStudio2Go: Digital video composition for real environments. ACM Transactions on Graphics 27, 5, 151:1–151:8. Google ScholarDigital Library
17. Anselm Grundhöfer, Daniel Kurz, Sebastian Thiele, and Oliver Bimber. 2010. Color invariant chroma keying and color spill neutralization for dynamic scenes and cameras. The Visual Computer 26, 9, 1167–1176. Google ScholarDigital Library
18. Bei He, Guijin Wang, Chenbo Shi, Xuanwu Yin, Bo Liu, and Xinggang Lin. 2013. Iterative transductive learning for alpha matting. In Proceedings of ICIP.Google ScholarCross Ref
19. Kaiming He, C. Rhemann, C. Rother, Xiaoou Tang, and Jian Sun. 2011. A global sampling method for alpha matting. In Proceedings of CVPR. Google ScholarDigital Library
20. Meiguang Jin, Byoung-Kwang Kim, and Woo-Jin Song. 2014. Adaptive propagation-based color-sampling for alpha matting. IEEE Transactions on Circuits and Systems for Video Technology 24, 7, 1101–1110.Google ScholarCross Ref
21. Jubin Johnson, Deepu Rajan, and Hisham Cholakkal. 2014. Sparse codes as alpha matte. In Proceedings of BMVC.Google ScholarCross Ref
22. V. Kolmogorov and R. Zabih. 2004. What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 2, 147–159. Google ScholarDigital Library
23. P. Lee and Ying Wu. 2011. Nonlocal matting. In Proceedings of CVPR. Google ScholarDigital Library
24. Anat Levin, Dani Lischinski, and Yair Weiss. 2008a. A closed-form solution to natural image matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 228–242. Google ScholarDigital Library
25. Anat Levin, Alex Rav-Acha, and Dani Lischinski. 2008b. Spectral matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 10, 1699–1712. Google ScholarDigital Library
26. Dingzeyu Li, Qifeng Chen, and Chi-Keung Tang. 2013. Motion-aware KNN Laplacian for video matting. In Proceedings of ICCV. Google ScholarDigital Library
27. Yin Li, Jian Sun, and Heung-Yeung Shum. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595–600. Google ScholarDigital Library
28. Hai Ting Lin, Yu-Wing Tai, and M. S. Brown. 2011. Motion regularization for matting motion blurred objects. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 11, 2329–2336. Google ScholarDigital Library
29. Christoph Rhemann, Carsten Rother, and Margrit Gelautz. 2008. Improving color modeling for alpha matting. In Proceedings of BMVC.Google ScholarCross Ref
30. Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. 2009. A perceptually motivated online benchmark for image matting. In Proceedings of CVPR.Google ScholarCross Ref
31. Mark A. Ruzon and Carlo Tomasi. 2000. Alpha estimation in natural images. In Proceedings of CVPR.Google Scholar
32. E. Shahrian, B. Price, S. Cohen, and D. Rajan. 2014. Temporally coherent and spatially accurate video matting. Computer Graphics Forum 33, 2, 381–390. Google ScholarDigital Library
33. E. Shahrian and D. Rajan. 2012. Weighted color and texture sample selection for image matting. In Proceedings of CVPR. Google ScholarDigital Library
34. E. Shahrian, D. Rajan, B. Price, and S. Cohen. 2013. Improving image matting using comprehensive sampling sets. In Proceedings of CVPR. Google ScholarDigital Library
35. Yongfang Shi, O. C. Au, Jiahao Pang, K. Tang, Wenxiu Sun, Hong Zhang, Wenjing Zhu, and Luheng Jia. 2013. Color clustering matting. In Proceedings of ICME.Google Scholar
36. YiChang Shih, Dilip Krishnan, Fredo Durand, and William T. Freeman. 2015. Reflection removal using ghosting cues. In Proceedings of CVPR.Google Scholar
37. D. Singaraju, C. Rother, and C. Rhemann. 2009. New appearance models for natural image matting. In Proceedings of CVPR.Google Scholar
38. D. Singaraju and R. Vidal. 2011. Estimation of alpha mattes for multiple image layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 7, 1295–1309. Google ScholarDigital Library
39. Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics 31, 4, 100:1–100:10. Google ScholarDigital Library
40. Alvy Ray Smith and James F. Blinn. 1996. Blue screen matting. ACM Transactions on Graphics (1996), 259–268. Google ScholarDigital Library
41. Jian Sun, Jiaya Jia, Chi-Keung Tang, and Heung-Yeung Shum. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315–321. Google ScholarDigital Library
42. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proceedings of CVPR. Google ScholarDigital Library
43. Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2007. Soft color segmentation and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 9, 1520–1537. Google ScholarDigital Library
44. Zhen Tang, Zhenjiang Miao, Yanli Wan, and Dianyong Zhang. 2012. Video matting via opacity propagation. The Visual Computer 28, 1, 47–61. Google ScholarDigital Library
45. Jue Wang, Pravin Bhat, R. Alex Colburn, Maneesh Agrawala, and Michael F. Cohen. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585–594. Google ScholarDigital Library
46. Jue Wang and M. F. Cohen. 2005. An iterative optimization approach for unified image segmentation and matting. In Proceedings of ICCV. Google ScholarDigital Library
47. Jue Wang and M. F. Cohen. 2007. Optimized color sampling for robust matting. In Proceedings of CVPR.Google Scholar
48. Tai-Pang Wu, Chi-Keung Tang, Michael S. Brown, and Heung-Yeung Shum. 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2. Google ScholarDigital Library
49. Sai-Kit Yeung, Tai-Pang Wu, and Chi-Keung Tang. 2008. Extracting smooth and transparent layers from a single image. In Proceedings of CVPR.Google Scholar
50. Fan Zhong, Xueying Qin, Qunsheng Peng, and Xiangxu Meng. 2012. Discontinuity-aware video object cutout. ACM Transactions on Graphics 31, 6, 175:1–175:10. Google ScholarDigital Library
