“Natural video matting using camera arrays” by Joshi, Matusik and Avidan

  • ©Neel Joshi, Wojciech Matusik, and Shai Avidan

Conference:


Type:


Title:

    Natural video matting using camera arrays

Presenter(s)/Author(s):



Abstract:


    We present an algorithm and a system for high-quality natural video matting using a camera array. The system uses high frequencies present in natural scenes to compute mattes by creating a synthetic aperture image that is focused on the foreground object, which reduces the variance of pixels reprojected from the foreground while increasing the variance of pixels reprojected from the background. We modify the standard matting equation to work directly with variance measurements and show how these statistics can be used to construct a trimap that is later upgraded to an alpha matte. The entire process is completely automatic, including an automatic method for focusing the synthetic aperture image on the foreground object and an automatic method to compute the trimap and the alpha matte. The proposed algorithm is very efficient and has a per-pixel running time that is linear in the number of cameras. Our current system runs at several frames per second, and we believe that it is the first system capable of computing high-quality alpha mattes at near real-time rates without the use of active illumination or special backgrounds.

References:


    1. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH 2001, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 425–432. Google ScholarDigital Library
    2. Chuang, Y.-Y., Zongker, D. E., Hindorff, J., Curless, B., Salesin, D. H., and Szeliski, R. 2000. Environment matting extensions: towards higher accuracy and real-time capture. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 121–130. Google ScholarDigital Library
    3. Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. 2001. A bayesian approach to digital matting. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2001), IEEE Computer Society, vol. 2, 264–271.Google Scholar
    4. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Transactions on Graphics 21, 3, 243–248. Google ScholarDigital Library
    5. Isaksen, A., McMillan, L., and Gortler, S. J. 2000. Dynamically reparameterized light fields. In Proceedings of ACM SIGGRAPH 2000, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 297–306. Google ScholarDigital Library
    6. Kolomogrov, V., Criminisi, A., Blake, A., Cross, G., and Rother, C. 2005. Bi-layer segmentation of binocular stereo video. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, 407–414. Google ScholarDigital Library
    7. Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. ACM Transactions on Graphics 23, 3, 303–308. Google ScholarDigital Library
    8. Li, Y., Sun, J., and Shum, H.-Y. 2005. Video object cut and paste. ACM Transactions on Graphics 24, 3, 595–600. Google ScholarDigital Library
    9. McGuire, M., Matusik, W., Pfister, H., Durand, F., and Hughes, J. 2005. Defocus Video Matting. ACM Transactions on Graphics 24, 3, 567–576. Google ScholarDigital Library
    10. Rother, C., Kolmogorov, V., and Blake, A. 2004. Grabcut: interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 3, 309–314. Google ScholarDigital Library
    11. Ruzon, M. A., and Tomasi, C. 2000. Alpha estimation in natural images. In Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR 2000), vol. 1, 18–25.Google Scholar
    12. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of ACM SIGGRAPH 1996. ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 259–268. Google ScholarDigital Library
    13. Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3 (August), 315–321. Google ScholarDigital Library
    14. Vlahos, P., 1958. Composite photography utilizing sodium vapor illumination (u.s. patent 3,095,304), May.Google Scholar
    15. Vlahos, P., 1971. Electronic composite photography (u.s. patent 3,595,987, July.Google Scholar
    16. Vlahos, P., 1978. Comprehensive electronic compositing system (u.s. patent 4,100,569), July.Google Scholar
    17. Wang, J., Bhat, P., Colburn, A., Agrawala, M., and Cohen, M. 2005. Interactive video cutout. ACM Transactions on Graphics 24, 3, 585–594. Google ScholarDigital Library
    18. Wexler, Y., Fitzgibbon, A., and Zisserman., A. 2002. Bayesian estimation of layers from multiple images. In Proceedings of 7th European Conference on Computer Vision (ECCV), vol. III, 487–501. Google ScholarDigital Library
    19. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3, 765–776. Google ScholarDigital Library
    20. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics 23, 3, 600–608. Google ScholarDigital Library
    21. Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. In Proceedings of ACM SIGGRAPH 1999, ACM Press/ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, 205–214. Google ScholarDigital Library


ACM Digital Library Publication: