“Video deblurring for hand-held cameras using patch-based synthesis” by Cho, Wang and Lee

Videos captured by hand-held Cameras often contain significant camera shake, causing many frames to be blurry. Restoring shaky videos not only requires smoothing the camera motion and stabilizing the content, but also demands removing blur from video frames. However, video blur is hard to remove using existing single or multiple image deblurring techniques, as the blur kernel is both spatially and temporally varying. This paper presents a video deblurring method that can effectively restore sharp frames from blurry ones caused by camera shake. Our method is built upon the observation that due to the nature of camera shake, not all video frames are equally blurry. The same object may appear sharp on some frames while blurry on others. Our method detects sharp regions in the video, and uses them to restore blurry regions of the same content in nearby frames. Our method also ensures that the deblurred frames are both spatially and temporally coherent using patch-based synthesis. Experimental results show that our method can effectively remove complex video blur under the presence of moving objects and other outliers, which cannot be achieved using previous deconvolution-based approaches.

1. Agrawal, A., Xu, Y., and Raskar, R. 2009. Invertible motion blur in video. ACM Trans. Graphics 28, 3, 95:1–95:8. Google ScholarDigital Library
2. Baker, S., and Matthews, I. 2004. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision (IJCV) 56, 3, 221–255. Google ScholarDigital Library
3. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graphics 28, 3, 24:1–24:11. Google ScholarDigital Library
4. Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. 2010. The generalized PatchMatch correspondence algorithm. In Proc. ECCV 2010, 29–43. Google ScholarDigital Library
5. Buades, A., and Coll, B. 2005. A non-local algorithm for image denoising. In Proc. CVPR 2006, 60–65. Google ScholarDigital Library
6. Cai, J.-F., Ji, H., Liu, C., and Shen, Z. 2009. Blind motion deblurring using multiple images. J. Comput. Phys. 228, 5057–5071. Google ScholarDigital Library
7. Chen, J., Yuan, L., Tang, C.-K., and Quan, L. 2008. Robust dual motion deblurring. In Proc. CVPR 2008, 1–8.Google Scholar
8. Cho, S., and Lee, S. 2009. Fast motion deblurring. ACM Trans. Graphics 28, 5, 145:1–145:8. Google ScholarDigital Library
9. Cho, S., Matsushita, Y., and Lee, S. 2007. Removing non-uniform motion blur from images. In Proc. ICCV 2007, 1–8.Google Scholar
10. Cho, S., Wang, J., and Lee, S. 2011. Handling outliers in non-blind image deconvolution. In Proc. ICCV 2011, 495–502. Google ScholarDigital Library
11. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. Proc. ACM SIGGRAPH 2001, 341–346. Google ScholarDigital Library
12. Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., and Freeman, W. T. 2006. Removing camera shake from a single photograph. ACM Trans. Graphics 25, 3, 787–794. Google ScholarDigital Library
13. Freedman, G., and Fattal, R. 2011. Image and video up-scaling from local self-examples. ACM Trans. Graphics 30, 2, 12:1–12:11. Google ScholarDigital Library
14. Fried, D. L. 1978. Probability of getting a lucky short-exposure image through turbulence. J. Opt. Soc. Am. 68, 12, 1651–1657.Google ScholarCross Ref
15. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust L1 optimal camera paths. In Proc. CVPR 2011, 225–232. Google ScholarDigital Library
16. Gupta, A., Joshi, N., Zitnick, C. L., Cohen, M., and Curless, B. 2010. Single image deblurring using motion density functions. In Proc. ECCV 2010, 171–184. Google ScholarDigital Library
17. HaCohen, Y., Shechtman, E., Goldman, D. B., and Lischinski, D. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Graphics 30, 4, 70:1–70:10. Google ScholarDigital Library
18. Hirsch, M., Schuler, C. J., Harmeling, S., and Schölkopf, B. 2011. Fast removal of non-uniform camera shake. In Proc. ICCV 2011, 463–470. Google ScholarDigital Library
19. Joshi, N., and Cohen, M. 2010. Seeing mt. rainier: Lucky imaging for multi-image denoising, sharpening, and haze removal. In Proc. ICCP 2010, 1–8.Google Scholar
20. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. Graphics 24, 3, 795–802. Google ScholarDigital Library
21. Levin, A., Weiss, Y., Durand, F., and Freeman, W. T. 2011. Efficient marginal likelihood optimization in blind deconvolution. In Proc. CVPR 2011, 2657–2664. Google ScholarDigital Library
22. Li, Y., Kang, S. B., Joshi, N., Seitz, S. M., and Huttenlocher, D. P. 2010. Generating sharp panoramas from motion-blurred videos. In Proc. CVPR 2010, 2424–2431.Google Scholar
23. Liu, C., and Freeman, W. T. 2010. A high-quality video denoising algorithm based on reliable motion estimation. In Proc. ECCV 2010, 706–719. Google ScholarDigital Library
24. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graphics 30, 1, 4:1–4:10. Google ScholarDigital Library
25. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Analysis Machine Intelligence 28, 7, 1150–1163. Google ScholarDigital Library
26. Osher, S., and Rudin, L. I. 1990. Feature-oriented image enhancement using shock filters. SIAM Journal on Numerical Analysis 27, 4, 919–940. Google ScholarDigital Library
27. Shan, Q., Jia, J., and Agarwala, A. 2008. High-quality motion deblurring from a single image. ACM Trans. Graphics 27, 3, 73:1–73:10. Google ScholarDigital Library
28. Shi, J., and Tomasi, C. 1994. Good features to track. In Proc. CVPR 1994, 593–600.Google Scholar
29. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In Proc. CVPR 2008, 1–8.Google Scholar
30. Tai, Y.-W., Tan, P., and Brown, M. S. 2011. Richardson-lucy deblurring for scenes under a projective motion path. IEEE Trans. Pattern Analysis Machine Intelligence 33, 8, 1603–1618. Google ScholarDigital Library
31. Whyte, O., Sivic, J., Zisserman, A., and Ponce, J. 2010. Non-uniform deblurring for shaken images. In Proc. CVPR 2010, 491–498.Google Scholar