“Slippage-free background replacement for hand-held video” by Zhong, Yang, Qin, Lischinski, Cohen-Or, et al. …
Conference:
Type(s):
Title:
- Slippage-free background replacement for hand-held video
Session/Category Title: Moving Pictures
Presenter(s)/Author(s):
Abstract:
We introduce a method for replacing the background in a video of a moving foreground subject, when both the source video capturing the subject, and the target video capturing the new background scene, are natural videos, casually captured using a freely moving hand-held camera. We assume that the foreground subject has already been extracted, and focus on the challenging task of generating a video with a new background, such that the new background motion appears compatible with the original one. Failure to match the motion results in disturbing slippage or moonwalk artifacts, where the subject’s feet appear to slide or slip over the ground. While matching the motion across the entire frame is impossible for scenes with differing geometry, we aim to match the local motion of the ground in the vicinity of the subject. This is achieved by reordering and warping the available target background frames in a manner that optimizes a suitably designed objective function.
References:
1. Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. Graph. 23, 3 (Aug.), 584–591.
2. Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., and Szeliski, R. 2005. Panoramic video textures. ACM Trans. Graph. 24, 3 (July), 821–827.
3. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video SnapCut: Robust video object cutout using localized classifiers. ACM Trans. Graph. 28, 3 (July), 70:1–70:11.
4. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. Graph. 21, 3 (July), 243–248.
5. Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011) 30, 4, 89:1–89:9.
6. Flagg, M., Nakazawa, A., Zhang, Q., Kang, S. B., Ryu, Y. K., Essa, I., and Rehg, J. M. 2009. Human video textures. In Proceedings of the 2009 symposium on Interactive 3D graphics and games, ACM, 199–206.
7. Germann, M., Popa, T., Keiser, R., Ziegler, R., and Gross, M. 2012. Novel-view synthesis of outdoor sport events using an adaptive view-dependent geometry. Comp. Graph. Forum 31, 2pt1 (May), 325–333.
8. Gleicher, M. L., and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 5, 1, 2.
9. Goldstein, A., and Fattal, R. 2012. Video stabilization using epipolar geometry. ACM Trans. Graph. 31, 5, 126.
10. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust l1 optimal camera paths. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 225–232.
11. Hartley, R., and Zisserman, A. 2004. Multiple view geometry in computer vision, 2nd ed. Cambridge Univ Press.
12. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3 (July), 44:1–44:9.
13. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graph. 30, 1, 4.
14. Liu, S., Yuan, L., Tan, P., and Sun, J. 2013. Bundled camera paths for video stabilization. ACM Trans. Graph. 32, 4, 78.
15. Okabe, M., Anjyor, K., and Onai, R. 2011. Creating fluid animation from a single image using video database. Computer Graphics Forum 30, 7, 1973–1982.Cross Ref
16. Sand, P., and Teller, S. 2004. Video matching. ACM Trans. Graph. 23, 3, 592–599.
17. Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In Proc. 27th annual conference on Computer Graphics and interactive techniques, ACM Press/Addison-Wesley, 489–498.
18. Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proc. 23rd annual conference on Computer Graphics and interactive techniques, ACM, 259–268.
19. Steedly, D., Pal, C., and Szeliski, R. 2005. Efficiently registering video into panoramic mosaics. In Proc. ICCV ’05, vol. 2.
20. Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008. View and time interpolation in image space. Comp. Graph. Forum 27, 7, 1781–1787.Cross Ref
21. Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. ACM Trans. Graphics 29, 4, 125:1–125:10.
22. Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. ACM Trans. Graph. 24, 3 (July), 585–594.
23. Xu, F., Liu, Y., Stoll, C., Tompkin, J., Bharaj, G., Dai, Q., Seidel, H.-P., Kautz, J., and Theobalt, C. 2011. Video-based characters: Creating new human performances from a multi-view video database. ACM Trans. Graph. 30, 4 (July), 32:1–32:10.
24. Zhang, Y., Correa, C. D., and Ma, K.-L. 2011. Graph-based fire synthesis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation, 187–194.
