“Video tapestries with continuous temporal zoom” by Barnes, Goldman, Shechtman and Finkelstein

  • ©Connelly Barnes, Daniel (Dan) B. Goldman, Eli Shechtman, and Adam Finkelstein




    Video tapestries with continuous temporal zoom



    We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.


    1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graphics 23, 3, 294–302. Google ScholarDigital Library
    2. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. In ACM Intl. Conference on Computer Graphics and Interactive Techniques, 667–676. Google ScholarDigital Library
    3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graphics 28, 3. Google ScholarDigital Library
    4. Berkhin, P. 2002. Grouping Multidimensional Data: A survey of clustering data mining techniques. Springer.Google Scholar
    5. Bernstein, S. 1994. Film Production, Second Edition. Focal Press.Google Scholar
    6. Boreczky, J., Girgensohn, A., Golovchinsky, G., and Uchihashi, S. 2000. An interactive comic book presentation for exploring video. In Proceedings of SIGCHI, ACM, 185–192. Google ScholarDigital Library
    7. Bourdev, L., and Brandt, J. 2005. Robust object detection via soft cascade. In IEEE CVPR 2005, vol. 2. Google ScholarDigital Library
    8. Chiu, P., Girgensohn, A., and Liu, Q. 2004. Stained-glass visualization for highly condensed video summaries. In IEEE ICME 2004.Google Scholar
    9. Christel, M., Hauptmann, A., Wactlar, H., and Ng, T. 2002. Collages as dynamic summaries for news video. In ACM Multimedia, 561–569. Google ScholarDigital Library
    10. Cockburn, A., Karlson, A., and Bederson, B. B. 2008. A review of overview+detail, zooming, and focus+context interfaces. ACM Comput. Surv. 41, 1, 1–31. Google ScholarDigital Library
    11. Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Trans. Graphics 29, 3. Google ScholarDigital Library
    12. Davis, M. 1995. Media streams: representing video for retrieval and repurposing. PhD thesis, Wesleyan University. Google ScholarDigital Library
    13. Dementhon, D., Kobla, V., and Doermann, D. 1998. Video summarization by curve simplifiation. In ACM Multimedia, 211–218. Google ScholarDigital Library
    14. Hauser, T. 2008. The Art of Wall-E. Chronicle Books LLC.Google Scholar
    15. Kang, H., Matsushita, Y., Tang, X., Chen, X., Hefei, P., and Beijing, P. 2006. Space-time video montage. In CVPR06, 1331–1338. Google ScholarDigital Library
    16. Kim, K., Essa, I., and Abowd, G. D. 2006. Interactive mosaic generation for video navigation. In ACM Multimedia, 655–658. Google ScholarDigital Library
    17. Kraaij, W., Smeaton, A., Over, P., and Arlandis, J. 2004. Trecvid 2004-an overview. In TRECVID video retrieval online proceedings.Google Scholar
    18. Kwatra, V., Schdl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graphics 22, 3, 277–286. Google ScholarDigital Library
    19. Ma, Y., and Zhang, H. 2002. A model of motion attention for video skimming. In Proc. Image Processing, Int’l Conf., vol. 1, I-129–I-132.Google Scholar
    20. Mei, T., Yang, B., Yang, S., and Hua, X. 2009. Video collage: presenting a video sequence using a single image. The Visual Computer 25, 1, 39–51. Google ScholarDigital Library
    21. Murch, W. 1995. In the Blink of an Eye: A Perspective on Film Editing. Silman-James Press, Los Angeles.Google Scholar
    22. Rother, C., Kumar, S., Kolmogorov, V., and Blake, A. 2005. Digital tapestry. In IEEE CVPR, I: 589–596. Google ScholarDigital Library
    23. Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. ACM Trans. Graphics 25, 3, 847–852. Google ScholarDigital Library
    24. Shipman, F., Girgensohn, A., and Wilcox, L. 2003. Generation of interactive multi-level video summaries. In ACM Multimedia, 392–401. Google ScholarDigital Library
    25. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR 2008.Google Scholar
    26. Sivic, J., Kaneva, B., Torralba, A., Avidan, S., and Freeman, W. 2008. Creating and exploring a large photorealistic virtual space. In IEEE CVPR Workshops, 2008., 1–8.Google Scholar
    27. Smith, M., and Kanade, T. 1995. Video skimming for quick browsing based on audio and image characterization. Technical Report CMU-CS-95-186, School of Computer Science, Carnegie Mellon University.Google Scholar
    28. Smith, M., and Kanade, T. 1997. Video skimming and characterization through the combination of image and language understanding techniques. In 1997 IEEE CVPR, 775–781. Google ScholarDigital Library
    29. Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: Extracting and packing panoramas for video browsing. In ACM Multimedia, 427–436. Google ScholarDigital Library
    30. Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1, 3. Google ScholarDigital Library
    31. Uchihashi, S., Foote, J., Girgensohn, A., and Boreczky, J. 1999. Video manga: generating semantically meaningful video summaries. In ACM Multimedia, ACM, 383–392. Google ScholarDigital Library
    32. Wang, T., Mei, T., Hua, X.-S., Liu, X., and Zhou, H.-Q. 2007. Video collage: A novel presentation of video sequence. In ICME, IEEE, 1479–1482.Google Scholar
    33. Yang, B., Mei, T., Sun, L.-F., Yang, S.-Q., and Hua, X.-S. 2008. Free-shaped video collage. Multi-Media Modeling (MMM), 175–185. Google ScholarDigital Library

ACM Digital Library Publication: