“Schematic storyboarding for video visualization and editing” by Goldman, Curless, Salesin and Seitz

We present a method for visualizing short video clips in a single static image, using the visual language of storyboards. These schematic storyboards are composed from multiple input frames and annotated using outlines, arrows, and text describing the motion in the scene. The principal advantage of this storyboard representation over standard representations of video — generally either a static thumbnail image or a playback of the video clip in its entirety — is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. Our system renders a schematic storyboard layout based on a small amount of user interaction. We also demonstrate an interaction technique to scrub through time using the natural spatial dimensions of the storyboard. Potential applications include video editing, surveillance summarization, assembly instructions, composition of graphic novels, and illustration of camera technique for film studies.

1. Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 4, 294–301. Google ScholarDigital Library
2. Assa, J., Caspi, Y., and Cohen-Or, D. 2005. Action synopsis: Pose selection and illustration. ACM Transactions on Graphics (Proc. SIGGRAPH) 24, 3, 667–676. Google ScholarDigital Library
3. Cheong, L., and Huo, H. 2001. Shot change detection using scene-based constraint. Multimedia Tools and Applications 14, 2 (June), 175–186. Google ScholarDigital Library
4. Cutting, J. E. 2002. Representing motion in a static image: constraints and parallels in art, science, and popular culture. Perception 31, 1165–1193.Google ScholarCross Ref
5. Freeman, W. T., and Zhang, H. 2003. Shape-time photography. In Proc. Computer Vision and Pattern Recognition, 151–157.Google Scholar
6. Hart, J. 1999. The art of the storyboard: storyboarding for film, TV and animation. Focal Press.Google Scholar
7. Heng, W., and Ngan, K. 2001. An object-based shot boundary detection using edge tracing and tracking. Journal of Visual Communication and Image Representation 12, 3 (September), 217–239.Google ScholarDigital Library
8. Horn, B., Hilden, H., and Negahdaripour, S. 1988. Closed-form solution of absolute orientation using orthonormal matrices. Journal of the Optical Society of America A 5, 7, 1127–1135.Google ScholarCross Ref
9. Irani, M., and Anandan, P. 1998. Video indexing based on mosaic representations. IEEE Transactions on Pattern Analysis and Machine Intelligence 86, 5 (May), 905–921.Google Scholar
10. Jia, J., and Tang, C.-K. 2005. Eliminating structure and intensity misalignment in image stitching. In Proc. ICCV. Google ScholarDigital Library
11. Jojic, N., Basu, S., Petrovic, N., Frey, B., and Huang, T., 2003. Joint design of data analysis algorithms and user interface for video applications. presented at NIPS 2003 workshop on Machine Learning in User Interface, extended abstract at: http://research.microsoft.com/workshops/MLUI03/jojic.html.Google Scholar
12. Katz, S. D. 1991. Film directing shot by shot: visualizing from concept to screen. Michael Wiese Productions.Google Scholar
13. Kawagishi, Y., Hatsuyama, K., and Kondo, K. 2003. Cartoon blur: nonphotorealistic motion blur. In Proc. Comp. Graph. Intl., 276–281.Google Scholar
14. Kim, B., and Essa, I. 2005. Video-based nonphotorealistic and expressive illustration of motion. In Proc. Comp. Graph. Intl., 32–35. Google ScholarDigital Library
15. Kumar, M. P., Torr, P. H. S., and Zisserman, A. 2005. Learning layered motion segmentations of video. In Proc. ICCV, 33–40. Google ScholarDigital Library
16. Lee, M., Yang, Y., and Lee, S. 2001. Automatic video parsing using shot boundary detection and camera operation analysis. Pattern Recognition 34, 3 (March), 711–719.Google ScholarCross Ref
17. Li, Y., Zhang, T., and Tretter, D. 2001. An overview of video abstraction techniques. Tech. Rep. HPL-2001-191, HP Laboratories.Google Scholar
18. Massey, M., and Bender, W. 1996. Salient stills: process and practice. IBM Systems Journal 35, 3, 4, 557–574. Google ScholarDigital Library
19. Masuch, M., Schlechtweg, S., and Schultz, R. 1999. Speedlines: depicting motion in motionless pictures. In ACM SIGGRAPH 99 Conference abstracts and applications, 277. Google ScholarDigital Library
20. McCloud, S. 1993. Understanding Comics: The Invisible Art. Harper-Collins.Google Scholar
21. Nicolas, H., Manaury, A., Benois-Pineau, J., Dupuy, W., and Barba, D. 2004. Grouping video shots into scenes based on 1d mosaic descriptors. In Proc. Intl. Conf. on Image Proc., I: 637–640.Google Scholar
22. Nienhaus, M., and Döllner, J. 2003. Dynamic glyphs — depicting dynamics in images of 3D scenes. In Third International Symposium on Smart Graphics, 102–111. Google ScholarDigital Library
23. Rother, C., Kolmogorov, V., and Blake, A. 2004. “GrabCut” — interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 3, 309–314. Google ScholarDigital Library
24. Rubin, M. 2005. Droidmaker: George Lucas and the digital revolution. Triad Publishing. 327, 338.Google Scholar
25. Sangster, C., 2005. Personal Communication.Google Scholar
26. Taniguchi, Y., Akutsu, A., and Tonomura, Y. 1997. PanoramaExcerpts: extracting and packing panoramas for video browsing. In Proc. ACM Intl. Conf. on Multimedia, 427–436. Google ScholarDigital Library
27. Teodosio, L., and Bender, W. 1993. Salient video stills: Content and context preserved. In Proc. ACM Intl. Conf. on Multimedia, 39–46. Google ScholarDigital Library
28. Umeyama, S. 1991. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 4, 376–380. Google ScholarDigital Library
29. Ward, J. 1979. Perception and Pictorial Representation, vol. 1. Praeger, New York, ch. 13, “A piece of the action: Moving figures in still pictures”, 246–271.Google Scholar
30. Wexler, Y., and Simakov, D. 2005. Space-time scene manifolds. In Proc. ICCV, 858–863. Google ScholarDigital Library
31. Wood, D. N., Finkelstein, A., Hughes, J. F., Thayer, C. E., and Salesin, D. H. 1997. Multiperspective panoramas for cel animation. In Proc. SIGGRAPH, 243–250. Google ScholarDigital Library
32. Zelnik-Manor, L., Peters, G., and Perona, P. 2005. Squaring the circle in panoramas. In Proc. ICCV, 1292–1299. Google ScholarDigital Library