“Temporally coherent completion of dynamic video”
Conference:
Type(s):
Title:
- Temporally coherent completion of dynamic video
Session/Category Title: Video
Presenter(s)/Author(s):
Abstract:
We present an automatic video completion algorithm that synthesizes missing regions in videos in a temporally coherent fashion. Our algorithm can handle dynamic scenes captured using a moving camera. State-of-the-art approaches have difficulties handling such videos because viewpoint changes cause image-space motion vectors in the missing and known regions to be inconsistent. We address this problem by jointly estimating optical flow and color in the missing regions. Using pixel-wise forward/backward flow fields enables us to synthesize temporally coherent colors. We formulate the problem as a non-parametric patch-based optimization. We demonstrate our technique on numerous challenging videos.
References:
1. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. 2009. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. on Graphics 28, 3, 24.
2. Barnes, C., Shechtman, E., Goldman, D., and Finkelstein, A. 2010. The generalized patchmatch correspondence algorithm. In ECCV.
3. Barnes, C., Zhang, F.-L., Lou, L., Wu, X., and Hu, S.-M. 2015. PatchTable: Efficient patch queries for large datasets and applications. ACM Trans. on Graphics 34, 4, 97.
4. Bhat, K. S., Seitz, S. M., Hodgins, J. K., and Khosla, P. K. 2004. Flow-based video synthesis and editing. In ACM Trans. on Graphics, vol. 23, ACM, 360–363.
5. Criminisi, A., Pérez, P., and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE TIP 13, 9, 1200–1212.
6. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. on Graphics 31, 4, 82.
7. Drori, I., Cohen-Or, D., and Yeshurun, H. 2003. Fragment-based image completion. In ACM Trans. on Graphics, vol. 22, 303–312.
8. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In ICCV, vol. 2.
9. Granados, M., Kim, K. I., Tompkin, J., Kautz, J., and Theobalt, C. 2012. Background inpainting for videos with dynamic objects and a free-moving camera. In ECCV.
10. Granados, M., Tompkin, J., Kim, K., Grau, O., Kautz, J., and Theobalt, C. 2012. How not to be seenobject removal from videos of crowded scenes. In Computer Graphics Forum, vol. 31, 219–228.
11. Guillemot, C., and Le Meur, O. 2014. Image inpainting: Overview and recent advances. IEEE Signal Processing Magazine 31, 1, 127–144. Cross Ref
12. Huang, J.-B., Kang, S. B., Ahuja, N., and Kopf, J. 2014. Image completion using planar structure guidance. ACM Trans. on Graphics 33, 4, 129.
13. Ilan, S., and Shamir, A. 2015. A survey on data-driven video completion. Computer Graphics Forum 34, 6, 60–85.
14. Jamriška, O., Fišer, J., Asente, P., Lu, J., Shechtman, E., and Sykora, D. 2015. LazyFluids: Appearance transfer for fluid animations. ACM Trans. on Graphics 34, 4, 92.
15. Jia, J., Tai, Y.-W., Wu, T.-P., and Tang, C.-K. 2006. Video repairing under variable illumination using cyclic motions. IEEE TPAMI 28, 5, 832–839.
16. Kalantari, N. K., Shechtman, E., Barnes, C., Darabi, S., Goldman, D. B., and Sen, P. 2013. Patch-based high dynamic range video. ACM Trans. on Graphics 32, 6, 202–1.
17. Kwatra, V., Schödl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. In ACM Trans. on Graphics, vol. 22, 277–286.
18. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. In ACM Trans. on Graphics, vol. 24, 795–802.
19. Liu, C., and Freeman, W. T. 2010. A high-quality video de-noising algorithm based on reliable motion estimation. In ECCV.
20. Liu, C. 2009. Beyond pixels: Exploring new representations and applications for motion analysis. PhD thesis.
21. Mac Aodha, O., Humayun, A., Pollefeys, M., and Brostow, G. J. 2013. Learning a confidence measure for optical flow. IEEE TPAMI 35, 5, 1107–1120.
22. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE TPAMI 28, 7, 1150–1163.
23. Newson, A., Almansa, A., Fradet, M., Gousseau, Y., Pérez, P., et al. 2014. Video inpainting of complex scenes. SIAM Journal on Imaging Sciences.
24. Patwardhan, K. A., Sapiro, G., and Bertalmio, M. 2005. Video inpainting of occluding and occluded objects. In ICIP.
25. Patwardhan, K. A., Sapiro, G., and Bertalmío, M. 2007. Video inpainting under constrained camera motion. IEEE TIP 16, 2, 545–553.
26. Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L. V., Gross, M., and Sorkine-Hornung, A. 2016. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR.
27. Pritch, Y., Kav-Venaki, E., and Peleg, S. 2009. Shift-map image editing. In ICCV.
28. Roxas, M., Shiratori, T., and Ikeuchi, K. 2014. Video completion via spatio-temporally consistent motion inpainting. Information and Media Technologies 9, 4, 500–504.
29. Sevilla-Lara, L., Wulff, J., Sunkavalli, K., and Shechtman, E. 2015. Smooth loops from unconstrained video. Computer Graphics Forum 34, 4, 99–107.
30. Shechtman, E., Rav-Acha, A., Irani, M., and Seitz, S. 2010. Regenerative morphing. In CVPR.
31. Shiratori, T., Matsushita, Y., Tang, X., and Kang, S. B. 2006. Video completion by motion field transfer. In CVPR.
32. Strobel, M., Diebold, J., and Cremers, D. 2014. Flow and color inpainting for video completion. In German Conference on Pattern Recognition.
33. Wexler, Y., Shechtman, E., and Irani, M. 2007. Space-time completion of video. IEEE TPAMI 29, 3, 463–476.
34. Xue, T., Rubinstein, M., Liu, C., and Freeman, W. T. 2015. A computational approach for obstruction-free photography. ACM Trans. on Graphics 34, 4, 79.
