“A system for retargeting of streaming video”
Conference:
Type(s):
Title:
- A system for retargeting of streaming video
Session/Category Title: Resizing/montage
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a novel, integrated system for content-aware video retargeting. A simple and interactive framework combines key frame based constraint editing with numerous automatic algorithms for video analysis. This combination gives content producers high level control of the retargeting process. The central component of our framework is a non-uniform, pixel-accurate warp to the target resolution which considers automatic as well as interactively defined features. Automatic features comprise video saliency, edge preservation at the pixel resolution, and scene cut detection to enforce bilateral temporal coherence. Additional high level constraints can be added by the producer to guarantee a consistent scene composition across arbitrary output formats. For high quality video display we adopted a 2D version of EWA splatting eliminating aliasing artifacts known from previous work. Our method seamlessly integrates into postproduction and computes the reformatting in real-time. This allows us to retarget annotated video streams at a high quality to arbitary aspect ratios while retaining the intended cinematographic scene composition. For evaluation we conducted a user study which revealed a strong viewer preference for our method.
References:
1. Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Trans. Graph. 26, 3, 10. Google ScholarDigital Library
2. Botsch, M., Hornung, A., Zwicker, M., and Kobbelt, L. 2005. High-quality surface splatting on today’s GPUs. In Symposium on Point-Based Graphics, 17–24. Google ScholarCross Ref
3. Briggs, W. L., Henson, V. E., and McCormick, S. F. 2000. A multigrid tutorial: second edition. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. Google ScholarDigital Library
4. Buck, I. 2007. GPU computing with NVIDIA CUDA. In SIGGRAPH ’07 Course Notes. Google ScholarDigital Library
5. Chen, L.-Q., Xie, X., Fan, X., Ma, W.-Y., Zhang, H., and Zhou, H.-Q. 2003. A visual attention model for adapting images on small displays. Multimedia Syst. 9, 4, 353–364.Google ScholarDigital Library
6. David, H. A. 1963. The Method of Paired Comparisons. Charles Griffin&Company.Google Scholar
7. Deselaers, T., Dreuw, P., and Ney, H. 2008. Pan, zoom, scan — time-coherent, trained automatic video cropping. In CVPR.Google Scholar
8. Ell, T. A., and Sangwine, S. J. 2007. Hypercomplex fourier transforms of color images. IEEE Transactions on Image Processing 16, 1, 22–35. Google ScholarDigital Library
9. Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In Proceedings of Eurographics Symposium on Rendering, 297–303. Google ScholarDigital Library
10. Gonzalez, R. C., and Woods, R. E. 2002. Digital Image Processing. Prentice Hall. Google ScholarDigital Library
11. Greene, N., and Heckbert, P. S. 1986. Creating raster omnimax images from multiple perspective views using the elliptical weighted average filter. IEEE Comput. Graph. Appl. 6, 6, 21–27. Google ScholarDigital Library
12. Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. CVPR.Google Scholar
13. Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence 17, 1–3, 185–203.Google ScholarDigital Library
14. Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE PAMI 20, 11, 1254–1259. Google ScholarDigital Library
15. Knoche, H., Papaleo, M., Sasse, M. A., and Vanelli-Coralli, A. 2007. The kindest cut: Enhancing the user experience of mobile tv through adequate zooming. In ACM Multimedia, 87–96. Google ScholarDigital Library
16. Kraevoy, V., Sheffer, A., Shamir, A., and Cohen-Or, D. 2008. Non-homogeneous resizing of complex models. ACM Trans. Graph. 27, 5, 111. Google ScholarDigital Library
17. Liu, F., and Gleicher, M. 2006. Video retargeting: automating pan and scan. In ACM Multimedia, 241–250. Google ScholarDigital Library
18. Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3, 16. Google ScholarDigital Library
19. Rubinstein, M., Shamir, A., and Avidan, S. 2009. Multi-operator media retargeting. ACM Trans. Graph. 28, 3, 23. Google ScholarDigital Library
20. Schaefer, S., McPhail, T., and Warren, J. D. 2006. Image deformation using moving least squares. ACM Trans. Graph. 25, 3, 533–540. Google ScholarDigital Library
21. Segal, M., and Akeley, K., 2006. The OpenGL Graphics System: A Specification (Version 2.1). http://www.opengl.org.Google Scholar
22. Setlur, V., Takagi, S., Raskar, R., Gleicher, M., and Gooch, B. 2005. Automatic image retargeting. In MUM, 59–68. Google ScholarDigital Library
23. Viola, P. A., and Jones, M. J. 2004. Robust real-time face detection. IJCV 57, 2, 137–154. Google ScholarDigital Library
24. Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM Trans. Graph. 27, 5, 118. Google ScholarDigital Library
25. Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
26. Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Non-homogeneous content-driven video-retargeting. In ICCV, 1–6.Google Scholar
27. Zabih, R., Miller, J., and Mai, K. 1995. A feature-based algorithm for detecting and classifying scene breaks. In ACM Multimedia, 189–200. Google ScholarDigital Library
28. Zhang, Y.-F., Hu, S.-M., and Martin, R. R. 2008. Shrinkability maps for content-aware video resizing. In Pacific Graphics.Google Scholar
29. Zwicker, M., Pfister, H., van Baar, J., and Gross, M. H. 2002. Ewa splatting. IEEE Trans. Vis. Comput. Graph. 8, 3, 223–238. Google ScholarDigital Library
30. Zwicker, M., Räsänen, J., Botsch, M., Dachsbacher, C., and Pauly, M. 2004. Perspective accurate splatting. In Graphics Interface, 247–254. Google ScholarDigital Library
