“Practical temporal consistency for image-based graphics applications” by Lang, Wang, Aydin, Smolic and Gross
Conference:
Type:
Session Title:
- Image Processing
Title:
- Practical temporal consistency for image-based graphics applications
Presenter(s)/Author(s):
Abstract:
We present an efficient and simple method for introducing temporal consistency to a large class of optimization driven image-based computer graphics problems. Our method extends recent work in edge-aware filtering, approximating costly global regularization with a fast iterative joint filtering operation. Using this representation, we can achieve tremendous efficiency gains both in terms of memory requirements and running time. This enables us to process entire shots at once, taking advantage of supporting information that exists across far away frames, something that is difficult with existing approaches due to the computational burden of video data. Our method is able to filter along motion paths using an iterative approach that simultaneously uses and estimates per-pixel optical flow vectors. We demonstrate its utility by creating temporally consistent results for a number of applications including optical flow, disparity estimation, colorization, scribble propagation, sparse data up-sampling, and visual saliency computation.
References:
1. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., and Szeliski, R. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1, 1–31. Google ScholarDigital Library
2. Bhat, P., Zitnick, C. L., Cohen, M. F., and Curless, B. 2010. Gradientshop: A gradient-domain optimization framework for image and video filtering. ACM Trans. Graph. 29, 2. Google ScholarDigital Library
3. Chen, J., Paris, S., and Durand, F. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3, 103. Google ScholarDigital Library
4. Criminisi, A., Sharp, T., Rother, C., and Pérez, P. 2010. Geodesic image and video editing. ACM Trans. Graph. 29, 5, 134. Google ScholarDigital Library
5. Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In CVPR, 1141–1148.Google Scholar
6. Durand, F., and Dorsey, J. 2002. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21, 3, 257–266. Google ScholarDigital Library
7. Gastal, E. S. L., and Oliveira, M. M. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 4, 69. Google ScholarDigital Library
8. Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. In CVPR, IEEE Computer Society.Google Scholar
9. He, K., Sun, J., and Tang, X. 2010. Guided image filtering. In ECCV (1), Springer, vol. 6311 of Lecture Notes in Computer Science, 1–14. Google ScholarDigital Library
10. Höffken, M., Oberhoff, D., and Kolesnik, M. 2011. Temporal prediction and spatial regularization in differential optical flow. In ACIVS, Springer, vol. 6915 of Lecture Notes in Computer Science, 576–585. Google ScholarDigital Library
11. Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artif. Intell. 17, 1–3, 185–203.Google ScholarDigital Library
12. Hosni, A., Rhemann, C., Bleyer, M., and Gelautz, M. 2011. Temporally consistent disparity and optical flow via efficient spatio-temporal filtering. In PSIVT (1), Springer, vol. 7087 of Lecture Notes in Computer Science, 165–177. Google ScholarDigital Library
13. Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM Trans. Graph. 26, 3, 96. Google ScholarDigital Library
14. Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. H. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
15. Levin, A., Lischinski, D., and Weiss, Y. 2004. Colorization using optimization. ACM Trans. Graph. 23, 3, 689–694. Google ScholarDigital Library
16. Levin, A., Lischinski, D., and Weiss, Y. 2006. A closed form solution to natural image matting. In CVPR (1), IEEE Computer Society, 61–68. Google ScholarDigital Library
17. Nehab, D., Maximo, A., Lima, R. S., and Hoppe, H. 2011. Gpu-efficient recursive filtering and summed-area tables. ACM Trans. Graph. 30, 6, 176. Google ScholarDigital Library
18. Paris, S., Kornprobst, P., and Tumblin, J. 2009. Bilateral Filtering. Now Publishers Inc., Hanover, MA, USA. Google ScholarDigital Library
19. Perona, P., and Malik, J. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7, 629–639. Google ScholarDigital Library
20. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., and Gelautz, M. 2011. Fast cost-volume filtering for visual correspondence and beyond. In CVPR, IEEE, 3017–3024. Google ScholarDigital Library
21. Scharstein, D., and Szeliski, R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 1–3, 7–42. Google ScholarDigital Library
22. Sun, D., Roth, S., and Black, M. J. 2010. Secrets of optical flow estimation and their principles. In CVPR, 2432–2439.Google Scholar
23. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In ICCV, 839–846. Google ScholarDigital Library
24. Volz, S., Bruhn, A., Valgaerts, L., and Zimmer, H. 2011. Modeling temporal coherence for optical flow. In Proc. 13th International Conference on Computer Vision (ICCV), IEEE Computer Society Press, Barcelona. Google ScholarDigital Library
25. Wang, O., Lang, M., Frei, M., Hornung, A., Smolic, A., and Gross, M. H. 2011. Stereobrush: Interactive 2d to 3d conversion using discontinuous warps. In SBM, Eurographics Association, 47–54. Google ScholarDigital Library
26. Wildeboer, M. O., Yendo, T., Tehrani, M. P., and Tanimoto, M. 2010. A semi-automatic multi-view depth estimation method. Proceedings of SPIE, Visual Communications and Image Processing 7744.Google Scholar
27. Xiao, J., Cheng, H., Sawhney, H. S., Rao, C., and Isnardi, M. A. 2006. Bilateral filtering-based optical flow estimation with occlusion detection. In ECCV (1), Springer, vol. 3951 of Lecture Notes in Computer Science, 211–224. Google ScholarDigital Library
28. Yang, Q., Yang, R., Davis, J., and Nistér, D. 2007. Spatial-depth super resolution for range images. In CVPR, IEEE Computer Society.Google Scholar
29. Zimmer, H., Bruhn, A., and Weickert, J. 2011. Optic flow in harmony. International Journal of Computer Vision 93, 3, 368–388. Google ScholarDigital Library
