“Video tapestries with continuous temporal zoom” by Barnes, Goldman, Shechtman and Finkelstein

We present a novel approach for summarizing video in the form of a multiscale image that is continuous in both the spatial domain and across the scale dimension: There are no hard borders between discrete moments in time, and a user can zoom smoothly into the image to reveal additional temporal details. We call these artifacts tapestries because their continuous nature is akin to medieval tapestries and other narrative depictions predating the advent of motion pictures. We propose a set of criteria for such a summarization, and a series of optimizations motivated by these criteria. These can be performed as an entirely offline computation to produce high quality renderings, or by adjusting some optimization parameters the later stages can be solved in real time, enabling an interactive interface for video navigation. Our video tapestries combine the best aspects of two common visualizations, providing the visual clarity of DVD chapter menus with the information density and multiple scales of a video editing timeline representation. In addition, they provide continuous transitions between zoom levels. In a user study, participants preferred both the aesthetics and efficiency of tapestries over other interfaces for visual browsing.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graphics 23, 3, 294–302. Google ScholarDigital Library
2. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: pose selection and illustration. In ACM Intl. Conference on Computer Graphics and Interactive Techniques, 667–676. Google ScholarDigital Library
3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Trans. Graphics 28, 3. Google ScholarDigital Library
4. Berkhin, P. 2002. Grouping Multidimensional Data: A survey of clustering data mining techniques. Springer.Google Scholar
5. Bernstein, S. 1994. Film Production, Second Edition. Focal Press.Google Scholar
6. Boreczky, J., Girgensohn, A., Golovchinsky, G., and Uchihashi, S. 2000. An interactive comic book presentation for exploring video. In Proceedings of SIGCHI, ACM, 185–192. Google ScholarDigital Library
7. Bourdev, L., and Brandt, J. 2005. Robust object detection via soft cascade. In IEEE CVPR 2005, vol. 2. Google ScholarDigital Library
8. Chiu, P., Girgensohn, A., and Liu, Q. 2004. Stained-glass visualization for highly condensed video summaries. In IEEE ICME 2004.Google Scholar
9. Christel, M., Hauptmann, A., Wactlar, H., and Ng, T. 2002. Collages as dynamic summaries for news video. In ACM Multimedia, 561–569. Google ScholarDigital Library
10. Cockburn, A., Karlson, A., and Bederson, B. B. 2008. A review of overview+detail, zooming, and focus+context interfaces. ACM Comput. Surv. 41, 1, 1–31. Google ScholarDigital Library
11. Correa, C. D., and Ma, K.-L. 2010. Dynamic video narratives. ACM Trans. Graphics 29, 3. Google ScholarDigital Library
12. Davis, M. 1995. Media streams: representing video for retrieval and repurposing. PhD thesis, Wesleyan University. Google ScholarDigital Library
13. Dementhon, D., Kobla, V., and Doermann, D. 1998. Video summarization by curve simplifiation. In ACM Multimedia, 211–218. Google ScholarDigital Library
14. Hauser, T. 2008. The Art of Wall-E. Chronicle Books LLC.Google Scholar
15. Kang, H., Matsushita, Y., Tang, X., Chen, X., Hefei, P., and Beijing, P. 2006. Space-time video montage. In CVPR06, 1331–1338. Google ScholarDigital Library
16. Kim, K., Essa, I., and Abowd, G. D. 2006. Interactive mosaic generation for video navigation. In ACM Multimedia, 655–658. Google ScholarDigital Library
17. Kraaij, W., Smeaton, A., Over, P., and Arlandis, J. 2004. Trecvid 2004-an overview. In TRECVID video retrieval online proceedings.Google Scholar
18. Kwatra, V., Schdl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graphics 22, 3, 277–286. Google ScholarDigital Library
19. Ma, Y., and Zhang, H. 2002. A model of motion attention for video skimming. In Proc. Image Processing, Int’l Conf., vol. 1, I-129–I-132.Google Scholar
20. Mei, T., Yang, B., Yang, S., and Hua, X. 2009. Video collage: presenting a video sequence using a single image. The Visual Computer 25, 1, 39–51. Google ScholarDigital Library
21. Murch, W. 1995. In the Blink of an Eye: A Perspective on Film Editing. Silman-James Press, Los Angeles.Google Scholar
22. Rother, C., Kumar, S., Kolmogorov, V., and Blake, A. 2005. Digital tapestry. In IEEE CVPR, I: 589–596. Google ScholarDigital Library
23. Rother, C., Bordeaux, L., Hamadi, Y., and Blake, A. 2006. Autocollage. ACM Trans. Graphics 25, 3, 847–852. Google ScholarDigital Library
24. Shipman, F., Girgensohn, A., and Wilcox, L. 2003. Generation of interactive multi-level video summaries. In ACM Multimedia, 392–401. Google ScholarDigital Library
25. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR 2008.Google Scholar
26. Sivic, J., Kaneva, B., Torralba, A., Avidan, S., and Freeman, W. 2008. Creating and exploring a large photorealistic virtual space. In IEEE CVPR Workshops, 2008., 1–8.Google Scholar
27. Smith, M., and Kanade, T. 1995. Video skimming for quick browsing based on audio and image characterization. Technical Report CMU-CS-95-186, School of Computer Science, Carnegie Mellon University.Google Scholar
28. Smith, M., and Kanade, T. 1997. Video skimming and characterization through the combination of image and language understanding techniques. In 1997 IEEE CVPR, 775–781. Google ScholarDigital Library
29. Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: Extracting and packing panoramas for video browsing. In ACM Multimedia, 427–436. Google ScholarDigital Library
30. Truong, B. T., and Venkatesh, S. 2007. Video abstraction: A systematic review and classification. ACM Trans. Multimedia Comput. Commun. Appl. 3, 1, 3. Google ScholarDigital Library
31. Uchihashi, S., Foote, J., Girgensohn, A., and Boreczky, J. 1999. Video manga: generating semantically meaningful video summaries. In ACM Multimedia, ACM, 383–392. Google ScholarDigital Library
32. Wang, T., Mei, T., Hua, X.-S., Liu, X., and Zhou, H.-Q. 2007. Video collage: A novel presentation of video sequence. In ICME, IEEE, 1479–1482.Google Scholar
33. Yang, B., Mei, T., Sun, L.-F., Yang, S.-Q., and Hua, X.-S. 2008. Free-shaped video collage. Multi-Media Modeling (MMM), 175–185. Google ScholarDigital Library