“Natural video matting using camera arrays” by Joshi, Matusik and Avidan
Conference:
Type:
Title:
- Natural video matting using camera arrays
Presenter(s)/Author(s):
Abstract:
We present an algorithm and a system for high-quality natural video matting using a camera array. The system uses high frequencies present in natural scenes to compute mattes by creating a synthetic aperture image that is focused on the foreground object, which reduces the variance of pixels reprojected from the foreground while increasing the variance of pixels reprojected from the background. We modify the standard matting equation to work directly with variance measurements and show how these statistics can be used to construct a trimap that is later upgraded to an alpha matte. The entire process is completely automatic, including an automatic method for focusing the synthetic aperture image on the foreground object and an automatic method to compute the trimap and the alpha matte. The proposed algorithm is very efficient and has a per-pixel running time that is linear in the number of cameras. Our current system runs at several frames per second, and we believe that it is the first system capable of computing high-quality alpha mattes at near real-time rates without the use of active illumination or special backgrounds.
References:
1. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 2001, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 425–432. Google ScholarDigital Library
2. Chuang, Y.-Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D. H., and Szeliski, R. 2000. Environment matting extensions: towards higher accuracy and real-time capture. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 121–130. Google ScholarDigital Library
3. Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2001), IEEE Computer Society, vol. 2, 264–271.Google Scholar
4. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243–248. Google ScholarDigital Library
5. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 297–306. Google ScholarDigital Library
6. Kolomogrov, V., Criminisi, A., Blake, A., Cross, G., and Rother, C. 2005. Bi-layer segmentation of binocular stereo video. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, 407–414. Google ScholarDigital Library
7. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303–308. Google ScholarDigital Library
8. Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595–600. Google ScholarDigital Library
9. McGuire, M., Matusik, W., Pfister, H., Durand, F., and Hughes, J. 2005. Defocus Video Matting. ACM Transactions on Graphics 24, 3, 567–576. Google ScholarDigital Library
10. Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309–314. Google ScholarDigital Library
11. Ruzon, M. A., and Tomasi, C. 2000. Alpha estimation in natural images. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2000), vol. 1, 18–25.Google Scholar
12. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH 1996. ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 259–268. Google ScholarDigital Library
13. Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3 (August), 315–321. Google ScholarDigital Library
14. Vlahos, P., 1958. Composite photography utilizing sodium vapor illumination (u.s. patent 3,095,304), May.Google Scholar
15. Vlahos, P., 1971. Electronic composite photography (u.s. patent 3,595,987, July.Google Scholar
16. Vlahos, P., 1978. Comprehensive electronic compositing system (u.s. patent 4,100,569), July.Google Scholar
17. Wang, J., Bhat, P., Colburn, A., Agrawala, M., and Cohen, M. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585–594. Google ScholarDigital Library
18. Wexler, Y., Fitzgibbon, A., and Zisserman., A. 2002. Bayesian estimation of layers from multiple images. In Proceedings of 7th European Conference on Computer Vision (ECCV), vol. III, 487–501. Google ScholarDigital Library
19. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3, 765–776. Google ScholarDigital Library
20. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 23, 3, 600–608. Google ScholarDigital Library
21. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In Proceedings of ACM SIGGRAPH 1999, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 205–214. Google ScholarDigital Library