“Nonlinear disparity mapping for stereoscopic 3D” by Lang, Sorkine-Hornung, Wang, Poulakos, Smolic, et al. …

This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is motivated by the observation that the displayed depth and the resulting 3D viewing experience are dictated by a complex combination of perceptual, technological, and artistic constraints. We first discuss the most important perceptual aspects of stereo vision and their implications for stereoscopic content creation. We then formalize these insights into a set of basic disparity mapping operators. These operators enable us to control and retarget the depth of a stereoscopic scene in a nonlinear and locally adaptive fashion. To implement our operators, we propose a new strategy based on stereoscopic warping of the input video streams. From a sparse set of stereo correspondences, our algorithm computes disparity and image-based saliency estimates, and uses them to compute a deformation of the input views so as to meet the target disparities. Our approach represents a practical solution for actual stereo production and display that does not require camera calibration, accurate dense depth maps, occlusion handling, or inpainting. We demonstrate the performance and versatility of our method using examples from live action post-production, 3D display size adaptation, and live broadcast. An additional user study and ground truth comparison further provide evidence for the quality and practical relevance of the presented work.

1. 3dtv.at, 2010. Stereoscopic player, Jan. http://www.3dtv.at/.Google Scholar
2. Agrawal, A., and Raskar, R. 2007. Gradient domain manipulation techniques in vision and graphics. In ICCV Courses.Google Scholar
3. Akeley, K., Watt, S. J., Girshick, A. R., and Banks, M. S. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. 23, 3, 804–813. Google ScholarDigital Library
4. Baker, S., and Matthews, I. 2004. Lucas-Kanade 20 years on: A unifying framework. IJCV 56, 3, 221–255. Google ScholarDigital Library
5. Banks, M. S., Gepshtein, S., and Landy, M. S. 2004. Why is spatial stereoresolution so low? Journal of Neuroscience 24, 2077–2089.Google ScholarCross Ref
6. Bleyer, M., Gelautz, M., Rother, C., and Rhemann, C. 2009. A stereo approach that handles the matting problem via image warping. In CVPR, 501–508.Google Scholar
7. Burt, P., and Juelsz, B. 1980. A disparity gradient limit for binocular fusion. Science 208, 4444 (5), 615–617.Google Scholar
8. Carroll, R., Agrawala, M., and Agarwala, A. 2009. Optimizing content-preserving projections for wide-angle images. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
9. Criminisi, A., Blake, A., Rother, C., Shotton, J., and Torr, P. H. 2007. Efficient dense stereo with occlusions for new view-synthesis by four-state dynamic programming. Int. J. Comput. Vision 71, 1, 89–110. Google ScholarDigital Library
10. Cutting, J. E., and Vishton, P. M. 1995. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In Handbook of perception and cognition, Perception of space and motion, W. Epstein and S. Rogers, Eds., vol. 5. Academic Press, San Diego, CA.Google Scholar
11. David, H. A. 1963. The Method of Paired Comparisons. Charles Griffin & Company.Google Scholar
12. Feldmann, I., Schreer, O., and Kauff, P. 2003. Nonlinear depth scaling for immersive video applications. WIAMIS.Google Scholar
13. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In SIGGRAPH, 43–54. Google ScholarDigital Library
14. Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. CVPR.Google Scholar
15. Guttmann, M., Wolf, L., and Cohen-Or, D. 2009. Semiautomatic stereo extraction from video footage. In ICCV.Google Scholar
16. Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (3), 1–30.Google ScholarCross Ref
17. Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth. Oxford University Press, New York, USA.Google Scholar
18. Kim, M.-B., Lee, S., Choi, C., Um, G.-M., Hur, N.-H., and Kim, J.-W. 2008. Depth scaling of multiview images for auto-multiscopic 3D monitors. In 3DTV08.Google Scholar
19. Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
20. Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Technology 53, 3, 030201.Google ScholarCross Ref
21. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH, 31–42. Google ScholarDigital Library
22. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
23. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91–110. Google ScholarDigital Library
24. Mahajan, D., Huang, F.-C., Matusik, W., Ramamoorthi, R., and Belhumeur, P. N. 2009. Moving gradients: a path-based method for plausible image interpolation. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
25. Matusik, W., and Pfister, H. 2004. 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3, 814–824. Google ScholarDigital Library
26. Mendiburu, B. 2009. 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Focal Press.Google Scholar
27. Mobile 3DTV, 2010. Stereo video data-sets, Jan. http://sp.cs.tut.fi/mobile3dtv/stereo-video/.Google Scholar
28. Neuman, R., 2009. Personal Communication with Robert Neuman, Chief Stereographer, Disney Animation Studios.Google Scholar
29. Paris, S., and Durand, F. 2006. A fast approximation of the bilateral filter using a signal processing approach. In ECCV (4), 568–580. Google ScholarDigital Library
30. Pritch, Y., Ben-Ezra, M., and Peleg, S. 2000. Automatic disparity control in stereo panoramas (omnistereo). In OMNIVIS. Google ScholarDigital Library
31. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann. Google ScholarDigital Library
32. Sattler, T., Leibe, B., and Kobbelt, L. 2009. SCRAMSAC: Improving RANSAC’s efficiency with a spatial consistency filter. In ICCV.Google Scholar
33. Seitz, S., and Dyer, C. 1996. View morphing. In SIGGRAPH 96, 21–30. Google ScholarDigital Library
34. Shade, J., Gortler, S. J., Li-wei, H., and Szeliski, R. 1998. Layered depth images. In SIGGRAPH, 231–242. Google ScholarDigital Library
35. Shamir, A., and Sorkine, O. 2009. Visual media retargeting. In SIGGRAPH ASIA Courses. Google ScholarDigital Library
36. Siegel, M., and Nagata, S. 2000. Just enough reality: Comfortable 3-D viewing via microstereopsis. IEEE Transactions on Circuits and Systems for Video Technology 10, 3 (4), 387–396. Google ScholarDigital Library
37. Smolic, A., Mller, K., Dix, K., Merkle, P., Kauff, P., and Wiegand, T. 2008. Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. In ICIP, IEEE, 2448–2451.Google Scholar
38. Stelmach, L. B., Tam, W. J., Meegan, D. V., and Vincent, A. 2000. Stereo image quality: effects of mixed spatio-temporal resolution. IEEE Transactions on Circuits and Systems for Video Technology 10, 2, 188–193. Google ScholarDigital Library
39. Sun, G., and Holliman, N. 2009. Evaluating methods for controlling depth perception in stereoscopic cinematography. Stereoscopic Displays and Virtual Reality Systems XX, Proceedings of SPIE 7237 (1).Google Scholar
40. the Foundry, 2010. Ocular, Nuke, Jan. http://www.thefoundry.co.uk/.Google Scholar
41. van den Hengel, A., Dick, A. R., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. Videotrace: rapid interactive scene modelling from video. ACM Trans. Graph. 26, 3, 86. Google ScholarDigital Library
42. Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. SPIE, vol. 6803.Google Scholar
43. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600–612. Google ScholarDigital Library
44. Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
45. Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., and Bischof, H. 2009. Anisotropic Huber-L1 optical flow. In British Machine Vision Conference (BMVC).Google Scholar
46. Weyrich, T., Deng, J., Barnes, C., Rusinkiewicz, S., and Finkelstein, A. 2007. Digital bas-relief from 3D scenes. ACM Trans. Graph. 26, 3, 32. Google ScholarDigital Library
47. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S. A. J., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3, 600–608. Google ScholarDigital Library