“Nonlinear disparity mapping for stereoscopic 3D” by Lang, Sorkine-Hornung, Wang, Poulakos, Smolic, et al. …

  • ©Manuel Lang, Alexander Sorkine-Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross




    Nonlinear disparity mapping for stereoscopic 3D



    This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is motivated by the observation that the displayed depth and the resulting 3D viewing experience are dictated by a complex combination of perceptual, technological, and artistic constraints. We first discuss the most important perceptual aspects of stereo vision and their implications for stereoscopic content creation. We then formalize these insights into a set of basic disparity mapping operators. These operators enable us to control and retarget the depth of a stereoscopic scene in a nonlinear and locally adaptive fashion. To implement our operators, we propose a new strategy based on stereoscopic warping of the input video streams. From a sparse set of stereo correspondences, our algorithm computes disparity and image-based saliency estimates, and uses them to compute a deformation of the input views so as to meet the target disparities. Our approach represents a practical solution for actual stereo production and display that does not require camera calibration, accurate dense depth maps, occlusion handling, or inpainting. We demonstrate the performance and versatility of our method using examples from live action post-production, 3D display size adaptation, and live broadcast. An additional user study and ground truth comparison further provide evidence for the quality and practical relevance of the presented work.


    1. 3dtv.at, 2010. Stereoscopic player, Jan. http://www.3dtv.at/.Google Scholar
    2. Agrawal, A., and Raskar, R. 2007. Gradient domain manipulation techniques in vision and graphics. In ICCV Courses.Google Scholar
    3. Akeley, K., Watt, S. J., Girshick, A. R., and Banks, M. S. 2004. A stereo display prototype with multiple focal distances. ACM Trans. Graph. 23, 3, 804–813. Google ScholarDigital Library
    4. Baker, S., and Matthews, I. 2004. Lucas-Kanade 20 years on: A unifying framework. IJCV 56, 3, 221–255. Google ScholarDigital Library
    5. Banks, M. S., Gepshtein, S., and Landy, M. S. 2004. Why is spatial stereoresolution so low? Journal of Neuroscience 24, 2077–2089.Google ScholarCross Ref
    6. Bleyer, M., Gelautz, M., Rother, C., and Rhemann, C. 2009. A stereo approach that handles the matting problem via image warping. In CVPR, 501–508.Google Scholar
    7. Burt, P., and Juelsz, B. 1980. A disparity gradient limit for binocular fusion. Science 208, 4444 (5), 615–617.Google Scholar
    8. Carroll, R., Agrawala, M., and Agarwala, A. 2009. Optimizing content-preserving projections for wide-angle images. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
    9. Criminisi, A., Blake, A., Rother, C., Shotton, J., and Torr, P. H. 2007. Efficient dense stereo with occlusions for new view-synthesis by four-state dynamic programming. Int. J. Comput. Vision 71, 1, 89–110. Google ScholarDigital Library
    10. Cutting, J. E., and Vishton, P. M. 1995. Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In Handbook of perception and cognition, Perception of space and motion, W. Epstein and S. Rogers, Eds., vol. 5. Academic Press, San Diego, CA.Google Scholar
    11. David, H. A. 1963. The Method of Paired Comparisons. Charles Griffin & Company.Google Scholar
    12. Feldmann, I., Schreer, O., and Kauff, P. 2003. Nonlinear depth scaling for immersive video applications. WIAMIS.Google Scholar
    13. Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The lumigraph. In SIGGRAPH, 43–54. Google ScholarDigital Library
    14. Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion Fourier transform. CVPR.Google Scholar
    15. Guttmann, M., Wolf, L., and Cohen-Or, D. 2009. Semiautomatic stereo extraction from video footage. In ICCV.Google Scholar
    16. Hoffman, D. M., Girshick, A. R., Akeley, K., and Banks, M. S. 2008. Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. Journal of Vision 8, 3 (3), 1–30.Google ScholarCross Ref
    17. Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth. Oxford University Press, New York, USA.Google Scholar
    18. Kim, M.-B., Lee, S., Choi, C., Um, G.-M., Hur, N.-H., and Kim, J.-W. 2008. Depth scaling of multiview images for auto-multiscopic 3D monitors. In 3DTV08.Google Scholar
    19. Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
    20. Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Technology 53, 3, 030201.Google ScholarCross Ref
    21. Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH, 31–42. Google ScholarDigital Library
    22. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
    23. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 2, 91–110. Google ScholarDigital Library
    24. Mahajan, D., Huang, F.-C., Matusik, W., Ramamoorthi, R., and Belhumeur, P. N. 2009. Moving gradients: a path-based method for plausible image interpolation. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
    25. Matusik, W., and Pfister, H. 2004. 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3, 814–824. Google ScholarDigital Library
    26. Mendiburu, B. 2009. 3D Movie Making: Stereoscopic Digital Cinema from Script to Screen. Focal Press.Google Scholar
    27. Mobile 3DTV, 2010. Stereo video data-sets, Jan. http://sp.cs.tut.fi/mobile3dtv/stereo-video/.Google Scholar
    28. Neuman, R., 2009. Personal Communication with Robert Neuman, Chief Stereographer, Disney Animation Studios.Google Scholar
    29. Paris, S., and Durand, F. 2006. A fast approximation of the bilateral filter using a signal processing approach. In ECCV (4), 568–580. Google ScholarDigital Library
    30. Pritch, Y., Ben-Ezra, M., and Peleg, S. 2000. Automatic disparity control in stereo panoramas (omnistereo). In OMNIVIS. Google ScholarDigital Library
    31. Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P. 2005. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting. Morgan Kaufmann. Google ScholarDigital Library
    32. Sattler, T., Leibe, B., and Kobbelt, L. 2009. SCRAMSAC: Improving RANSAC’s efficiency with a spatial consistency filter. In ICCV.Google Scholar
    33. Seitz, S., and Dyer, C. 1996. View morphing. In SIGGRAPH 96, 21–30. Google ScholarDigital Library
    34. Shade, J., Gortler, S. J., Li-wei, H., and Szeliski, R. 1998. Layered depth images. In SIGGRAPH, 231–242. Google ScholarDigital Library
    35. Shamir, A., and Sorkine, O. 2009. Visual media retargeting. In SIGGRAPH ASIA Courses. Google ScholarDigital Library
    36. Siegel, M., and Nagata, S. 2000. Just enough reality: Comfortable 3-D viewing via microstereopsis. IEEE Transactions on Circuits and Systems for Video Technology 10, 3 (4), 387–396. Google ScholarDigital Library
    37. Smolic, A., Mller, K., Dix, K., Merkle, P., Kauff, P., and Wiegand, T. 2008. Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. In ICIP, IEEE, 2448–2451.Google Scholar
    38. Stelmach, L. B., Tam, W. J., Meegan, D. V., and Vincent, A. 2000. Stereo image quality: effects of mixed spatio-temporal resolution. IEEE Transactions on Circuits and Systems for Video Technology 10, 2, 188–193. Google ScholarDigital Library
    39. Sun, G., and Holliman, N. 2009. Evaluating methods for controlling depth perception in stereoscopic cinematography. Stereoscopic Displays and Virtual Reality Systems XX, Proceedings of SPIE 7237 (1).Google Scholar
    40. the Foundry, 2010. Ocular, Nuke, Jan. http://www.thefoundry.co.uk/.Google Scholar
    41. van den Hengel, A., Dick, A. R., Thormählen, T., Ward, B., and Torr, P. H. S. 2007. Videotrace: rapid interactive scene modelling from video. ACM Trans. Graph. 26, 3, 86. Google ScholarDigital Library
    42. Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. SPIE, vol. 6803.Google Scholar
    43. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600–612. Google ScholarDigital Library
    44. Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
    45. Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., and Bischof, H. 2009. Anisotropic Huber-L1 optical flow. In British Machine Vision Conference (BMVC).Google Scholar
    46. Weyrich, T., Deng, J., Barnes, C., Rusinkiewicz, S., and Finkelstein, A. 2007. Digital bas-relief from 3D scenes. ACM Trans. Graph. 26, 3, 32. Google ScholarDigital Library
    47. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S. A. J., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3, 600–608. Google ScholarDigital Library

ACM Digital Library Publication: