“Stereoscopic 3D copy & paste”
Conference:
Type(s):
Title:
- Stereoscopic 3D copy & paste
Session/Category Title: Image & video editing
Presenter(s)/Author(s):
Moderator(s):
Abstract:
With the increase in popularity of stereoscopic 3D imagery for film, TV, and interactive entertainment, an urgent need for editing tools to support stereo content creation has become apparent. In this paper we present an end-to-end system for object copy & paste in a stereoscopic setting to address this need. There is no straightforward extension of 2D copy & paste to support the addition of the third dimension as we show in this paper. For stereoscopic copy & paste we need to handle depth, and our core objective is to obtain a convincing 3D viewing experience. As one of the main contributions of our system, we introduce a stereo billboard method for stereoscopic rendering of the copied selection. Our approach preserves the stereo volume and is robust to the inevitable inaccuracies of the depth maps computed from a stereo pair of images. Our system also includes an interactive stereoscopic segmentation tool to achieve high quality object selection. Hence, we focus on intuitive and minimal user interaction, and our editing operations perform within interactive rates to provide immediate feedback.
References:
1. Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. on Graph. 23, 3 (Aug.), 584–591. Google ScholarDigital Library
2. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
3. Besl, P. J., and McKay, N. D. 1992. A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 239–256. Google ScholarDigital Library
4. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. on Pattern Anal. and Mach. Intell. 23, 11, 1222–1239. Google ScholarDigital Library
5. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. In Proc. of SIGGRAPH 2002, ACM Press / ACM SIGGRAPH, J. F. Hughes, Ed., ACM, 243–248. Google ScholarDigital Library
6. Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. on Pattern Anal. and Mach. Intell. 24, 603–619. Google ScholarDigital Library
7. Dong Seon, C., and Figueiredo, M. A. T. 2007. Cosegmentation for image sequences. Int. Conf. on Image Anal. and Proc., 635–640. Google ScholarDigital Library
8. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
9. Fuji, 2009. Finepix REAL 3D W1. http://www.fujifilm.com/products/3d/camera/finepix_real3dw1/.Google Scholar
10. Fukuda, K., Wilcox, L. M., Allison, R., and Howard, I. P. 2009. A reevaluation of the tolerance to vertical misalignment in stereopsis. Journal of Vision 9, 2 (February), 1–8.Google ScholarCross Ref
11. Georgiev, T. 2006. Covariant derivatives and vision. Proc. of European Conf. on Comp. Vision 4, 56–69. Google ScholarDigital Library
12. Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, second ed. Cambridge University Press, ISBN: 0521540518. Google ScholarDigital Library
13. Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth, Basic Mechanics & Depth Perception, vol. 1 & 2. I Porteous, Thornhill, Ontario.Google Scholar
14. Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Trans. on Graph. 25, 3 (July), 631–637. Google ScholarDigital Library
15. Koppal, S., Zitnick, C., Cohen, M., Kang, S., Ressler, B., and Colburn, A. 2010. A viewer-centric editor for stereoscopic cinema. IEEE Comp. Graph. and Appl. Preprint.Google Scholar
16. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. on Graph. 26, 3 (July). Google ScholarDigital Library
17. Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Tech. 53, 3, 030201.Google ScholarCross Ref
18. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., and Gross, M. 2010. Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. on Graph. 29, 4 (July). Google ScholarDigital Library
19. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. on Graph. 28, 3. Google ScholarDigital Library
20. Liu, J., Sun, J., and Shum, H.-Y. 2009. Paint selection. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
21. Loos, B. J., and Sloan, P.-P. 2010. Volumetric obscurance. In ACM Symp. on Interactive 3D Graph. and Games, ACM, New York, NY, USA, 151–156. Google ScholarDigital Library
22. Lu, F., Fu, Z., and Robles-Kelly, A. 2007. Efficient graph cuts for multiclass interactive image segmentation. Proc. of the Asian Conf. on Comp. vision, 134–144. Google ScholarDigital Library
23. Mammen, A. 1989. Transparency and antialiasing algorithms implemented with the virtual pixel maps technique. IEEE Comp. Graph. and Appl. 9, 4, 43–55. Google ScholarDigital Library
24. Ning, J., Zhang, L., Zhang, D., and Wu, C. 2010. Interactive image segmentation by maximal similarity based region merging. Pattern Recogn. 43, 2, 445–456. Google ScholarDigital Library
25. Patterson, R. 2007. Human factors of 3d displays. Journal of the Soc. for Information Disp., 15, 861–871.Google ScholarCross Ref
26. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. on Graph. 22, 3 (July), 313–318. Google ScholarDigital Library
27. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comp. Graph. and Appl. 21, 5, 34–41. Google ScholarDigital Library
28. Rhee, S.-M., Ziegler, R., Park, J., Naef, M., Gross, M., and Kim, M.-H. 2007. Low-cost telepresence for collaborative virtual environments. IEEE Trans on Vis. and Comp. Graph. 13, 1, 156–166. Google ScholarDigital Library
29. Rother, C., Kolmogorov, V., and Blake, A. 2004. “grab-cut”: interactive foreground extraction using iterated graph cuts. ACM Trans. on Graph. 23, 3 (Aug.), 309–314. Google ScholarDigital Library
30. Rother, C., Minka, T., Blake, A., and Kolmogorov, V. 2006. Cosegmentation of image pairs by histogram matching. IEEE Conf. on Comp. Vision and Pattern Recog., 993–1000. Google ScholarDigital Library
31. Scharstein, D., and Szeliski., R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Journal of Comp. Vision 47, 1/2/3, 7–42. Google ScholarDigital Library
32. Scharstein, D., and Szeliski, R., 2010. Middlebury Stereo Repository. http://vision.middlebury.edu/stereo/.Google Scholar
33. Shum, H. Y., Sun, J., Yamazaki, S., Li, Y., and Tang, C. K. 2004. Pop-up light field: An interactive image-based modeling and rendering system. ACM Trans. on Graph. 23, 2 (Aug.), 143–162. Google ScholarDigital Library
34. Smith, B., Zhang, L., and Jin, H. 2009. Stereo matching with nonparametric smoothness priors in feature space. IEEE Conf. on Comp. Vision and Pattern Recog., 485–492.Google Scholar
35. Taguchi, Y., Wilburn, B., and Zitnick, C. L. 2008. Stereo reconstruction with mixed pixels using adaptive over-segmentation. IEEE Conf. on Comp. Vision and Pattern Recog., 1–8.Google Scholar
36. The Foundry, 2010. Nuke – Ocula Plug-in. http://www.thefoundry.co.uk/.Google Scholar
37. Wang, J., and Cohen, M. F. 2008. Image and video matting: A survey. Foundations and Trends in Comp. Graph. and Vision 3, 2, 97–175. Google ScholarDigital Library
38. Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. Stereoscopic Disp. and Appl. 6803, 1, 68031E.Google ScholarCross Ref
39. Wang, L., Jin, H., Yang, R., and Gong, M. 2008. Stereoscopic inpainting: Joint color and depth completion from stereo images. In IEEE Conf. on Comp. Vision and Pattern Recog., 1–8.Google Scholar
40. Zitnick, C. L., and Kang, S. B. 2007. Stereo for image-based rendering using image over-segmentation. Int. Journal of Comp. Vision 75, 1, 49–65. Google ScholarDigital Library
41. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. on Graph. 23, 3 (Aug.), 600–608. Google ScholarDigital Library
42. Zitnick, C. L., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. IEEE Int. Conf. on Comp. Vision, 1308–1315. Google ScholarDigital Library
43. Zwicker, M., Pfister, H., van Baar, J., and Gross, M. 2002. Ewa splatting. IEEE Trans. on Vis. and Comp. Graph. 8, 3, 223–238. Google ScholarDigital Library
