“3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes” by Matusik and Pfister

  • ©Wojciech Matusik and Hanspeter Pfister




    3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes



    Three-dimensional TV is expected to be the next revolution in the history of television. We implemented a 3D TV prototype system with real-time acquisition, transmission, and 3D display of dynamic scenes. We developed a distributed, scalable architecture to manage the high computation and bandwidth demands. Our system consists of an array of cameras, clusters of network-connected PCs, and a multi-projector 3D display. Multiple video streams are individually encoded and sent over a broadband network to the display. The 3D display shows high-resolution (1024 × 768) stereoscopic color images for multiple viewpoints without special glasses. We implemented systems with rear-projection and front-projection lenticular screens. In this paper, we provide a detailed overview of our 3D TV system, including an examination of design choices and tradeoffs. We present the calibration and image alignment procedures that are necessary to achieve good image quality. We present qualitative results and some early user feedback. We believe this is the first real-time end-to-end 3D TV system with enough views and resolution to provide a truly immersive 3D experience.


    1. AKELEY, K., WATT, S., GIRSHICK, A., AND BANKS, M. 2004. A stereo display prototype with multiple focal distances. To appear in ACM Transaction on Graphics (Aug.). Google ScholarDigital Library
    2. BUEHLER, C., BOSSE, M., MCMILLAN, L., GORTLER, S., AND COHEN, M. 2001. Unstructured lumigraph rendering. In Computer Graphics, SIGGRAPH 2001 Proceedings, 425–432. Google ScholarDigital Library
    3. CARRANZA, J., THEOBALT, C., MAGNOR, M., AND SEIDEL, H. 2003. Free-viewpoint video of human actors. ACM Transactions on Graphics 22, 3, 569–577. Google ScholarDigital Library
    4. CHEN, S. E., AND WILLIAMS, L. 1993. View interpolation for image synthesis. In Computer Graphics, SIGGRAPH 93 Proceedings, 279–288. Google ScholarDigital Library
    5. FAVALORA, G., DORVAL, R., HALL, D., M., M. G., AND NAPOLI, J. 2001. Volumetric three-dimensional display system with rasterization hardware. In Stereoscopic Displays and Virtual Reality Systems VIII, vol. 4297 of SPIE Proceedings, 227–235.Google Scholar
    6. FEHN, C., KAUFF, P., DE BEECK, M. O., ERNST, F., IJSSELSTEIJN, W., POLLEFEYS, M., GOOL, L. V., OFEK, E., AND SEXTON, I. 2002. An evolutionary and optimised approach on 3D-TV. In Proceedings of International Broadcast Conference, 357–365.Google Scholar
    7. FLACK, J., HARMAN, P., AND FOX, S. 2003. Low bandwidth stereoscopic image encoding and transmission. In Stereoscopic Displays and Virtual Reality Systems X, vol. 5006 of Proceedings of SPIE, 206–215.Google Scholar
    8. GABOR, D. 1948. A new microscopic principle. Nature, 161 (May), 777–779.Google ScholarCross Ref
    9. GORTLER, S., GRZESZCZUK, R., SZELISKI, R., AND COHEN, M. 1996. The lumigraph. In Computer Graphics, SIGGRAPH 96 Proceedings, 43–54. Google ScholarDigital Library
    10. GROSS, M., WUERMLIN, S., NAEF, M., LAMBORAY, E., SPAGNO, C., KUNZ, A., KOLLER-MEIER, E., SVOBODA, T., GOOL, L. V., LANG, S., STREHLKE, K., MOERE, A. V., AND STAADT, O. 2003. blue-c: A spatially immersive display and 3D video portal for telepresence. ACM Transactions on Graphics 22, 3, 819–828. Google ScholarDigital Library
    11. HUEBSCHMAN, M., MUNJULURI, B., AND GARNER, H. R. 2003. Dynamic holographic 3-D image projection. Optics Express, 11, 437–445.Google ScholarCross Ref
    12. HUMPHREYS, G., HOUSTON, M., NG, Y., FRANK, R., AHERN, S., KIRCHNER, P., AND KLOSOWSKI, J. 2002. Chromium: A stream processing framework for interactive graphics on clusters. ACM Transactions on Graphics 21, 3, 693–703. Google ScholarDigital Library
    13. IVES, F. E., 1903. Parallax stereogram and process for making same. U.S. Patent Number 725,567, filed 1902, Apr.Google Scholar
    14. IVES, F. E. 1928. A camera for making parallax panoramagrams. Journal of the Optical Society of America, 17 (Dec.), 435–439.Google ScholarCross Ref
    15. IVES, H. E. 1929. Motion pictures in relief. Journal of the Optical Society of America, 18 (Feb.), 118–122.Google ScholarCross Ref
    16. IVES, H. E. 1931. The projection of parallax panoramagrams. Journal of the Optical Society of America, 21 (July), 397–409.Google ScholarCross Ref
    17. JAVIDI, B., AND OKANO, F., Eds. 2002. Three-Dimensional Television, Video, and Display Technologies. Springer-Verlag. Google ScholarDigital Library
    18. KAJIKI, Y., YOSHIKAWA, H., AND HONDA, T. 1996. Three-dimensional display with focused light array. In Practical Holography X, vol. 2652 of SPIE Proceedings, 106–116.Google Scholar
    19. KANADE, T., RANDER, P., AND NARAYANAN, P. 1997. Virtualized reality: Constructing virtual worlds from real scenes. IEEE Multimedia, Immersive Telepresence 4, 1 (Jan.), 34–47. Google ScholarDigital Library
    20. KANG, S. B., UYTTENDAELE, M., WINDER, S., AND SZELISKI, R. 2003. High dynamic range video. ACM Transactions on Graphics 22, 3, 319–325. Google ScholarDigital Library
    21. KANOLT, C. W., 1918. Photographic method and apparatus. U.S. Patent Number 1,260,682, filed 1915, Mar.Google Scholar
    22. LAMBORAY, E., WÜRMLIN, S., AND GROSS, M. 2004. Real-time streaming of point-based 3D video. In To appear in: Proceedings of IEEE Virtual Reality. Google ScholarDigital Library
    23. LEITH, E., AND UPATNIEKS, J. 1962. Reconstructed wavefronts and communication theory. Journal of the Optical Society of America 52, 10 (Oct.), 1123–1130.Google ScholarCross Ref
    24. LEVOY, M., AND HANRAHAN, P. 1996. Light field rendering. In Computer Graphics, SIGGRAPH 96 Proceedings, 31–42. Google ScholarDigital Library
    25. LI, K., CHEN, H., CHEN, Y., CLARK, D., COOK, P., DAMIANAKIS, S., ESSL, G., FINKELSTEIN, A., FUNKHOUSER, T., HOUSEL, T., KLEIN, A., LIU, Z., PRAUN, E., SAMANTA, R., SHEDD, B., SINGH, J. P., TZANETAKIS, G., AND ZHENG, J. 2002. Building and using a scalable display wall system. IEEE Computer Graphics and Applications 20, 4 (Dec.), 29–37. Google ScholarDigital Library
    26. LIAO, H., IWAHARA, M., HATA, N., SAKUMA, I., DOHI, T., KOIKE, T., MOMOI, Y., MINAKAWA, T., YAMASAKI, M., TAJIMA, F., AND TAKEDA, H. 2002. High-resolution integral videography autostereoscopic display using multi-projector. In Proceedings of the Ninth International Display Workshop, 1229–1232.Google Scholar
    27. LIPPMANN, G. 1908. Epreuves reversibles donnant la sensation du relief. Journal of Physics 7, 4 (Nov.), 821–825.Google Scholar
    28. MAENO, K., FUKAYA, N., NISHIKAWA, O., SATO, K., AND HONDA, T. 1996. Electroholographic display using 15-megapixel LCD. In Practical Holography X, vol. 2652 of SPIE Proceedings, 15–23.Google Scholar
    29. MAGNOR, M., RAMANATHAN, P., AND GIROD, B. 2003. Multiview coding for image-based rendering using 3-D scene geometry. IEEE Trans. Circuits and Systems for Video Technology 13, 11 (Nov.), 1092–1106. Google ScholarDigital Library
    30. MATUSIK, W., BUEHLER, C., RASKAR, R., GORTLER, S., AND MCMILLAN, L. 2000. Image-based visual hulls. In Computer Graphics, SIGGRAPH 2000 Proceedings, 369–374. Google ScholarDigital Library
    31. MCKAY, S., MAIR, G., MASON, S., AND REVIE, K. 2000. Membrane-mirrorbased autostereoscopic display for teleoperation and telepresence applications. In Stereoscopic Displays and Virtual Reality Systems VII, vol. 3957 of SPIE Proceedings, 198–207.Google Scholar
    32. MILLER, G. 1995. Volumetric hyper-reality, a computer graphics holy grail for the 21st century? In Proceedings of Graphics Interface ’95, Canadian Information Processing Society, 56–64.Google Scholar
    33. MOORE, J. R., DODGSON, N., TRAVIS, A., AND LANG, S. 1996. Time-multiplexed color autostereoscopic display. In Symposium on Steroscopic Displays and Applications VII, vol. 2653 of Proceedings of SPIE, 1–9.Google Scholar
    34. NAEMURA, T., TAGO, J., AND HARASHIMA, H. 2002. Realtime video-based modeling and rendering of 3D scenes. IEEE Computer Graphics and Applications (Mar.), 66–73. Google ScholarDigital Library
    35. NAKAJIMA, S., NAKAMURA, K., MASAMUNE, K., SAKUMA, I., AND DOHI, T. 2001. Three-dimensional medical imaging display with computer-generated integral photography. Computerized Medical Imaging and Graphics 25, 3, 235–241.Google ScholarCross Ref
    36. OKOSHI, T. 1976. Three-Dimensional Imaging Techniques. Academic Press.Google Scholar
    37. OOI, R., HAMAMOTO, T., NAEMURA, T., AND AIZAWA, K. 2001. Pixel independent random access image sensor for real time image-based rendering system. In IEEE International Conference on Image Processing, vol. II, 193–196.Google ScholarCross Ref
    38. PERLIN, K., PAXIA, S., AND KOLLIN, J. 2000. An autostereoscopic display. In SIGGRAPH 2000 Conference Proceedings, vol. 33, 319–326. Google ScholarDigital Library
    39. RAMANATHAN, P., KALMAN, M., AND GIROD, B. 2003. Rate-distortion optimized streaming of compressed light fields. In IEEE International Conference on Image Processing, 277–280.Google ScholarCross Ref
    40. RASKAR, R., WELCH, G., CUTTS, M., LAKE, A., STESIN, L., AND FUCHS, H. 1998. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of SIGGRAPH 98, 179–188. Google ScholarDigital Library
    41. RASKAR, R., BROWN, M., YANG, R., CHEN, W., WELCH, G., TOWLES, H., SEALES, B., AND FUCHS, H. 1999. Multi-projector displays using camera-based registration. In IEEE Visualization, 161–168. Google ScholarDigital Library
    42. SCHIRMACHER, H., MING, L., AND SEIDEL, H.-P. 2001. On-the-fly processing of generalized lumigraphs. In Proceedings of Eurographics 2001, vol. 20 of Computer Graphics Forum, Eurographics Association, 165–173.Google Scholar
    43. SEETZEN, H., HEIDRICH, W., STUERZLINGER, W., WARD, G., WHITEHEAD, L., TRENTACOSTE, M., GHOSH, A., AND VOROZCOVS, A. 2004. High dynamic range display systems. To appear in ACM Transaction on Graphics (Aug.). Google ScholarDigital Library
    44. SMOLIC, A., AND KIMATA, H., 2003. Report on 3DAV exploration. ISO/IEC JTCI/SC29/WG11 Document N5878, July.Google Scholar
    45. ST.-HILLAIRE, P., LUCENTE, M., SUTTER, J., PAPPU, R., C. J. SPARRELL, AND BENTON, S. 1995. Scaling up the MIT holographic video system. In Proceedings of the Fifth International Symposium on Display Holography, SPIE, 374–380.Google Scholar
    46. STANLEY, M., CONWAY, P., COOMBER, S., JONES, J., SCATTERGOOD, D., SLINGER, C., BANNISTER, B., BROWN, C., CROSSLAND, W., AND TRAVIS, A. 2000. A novel electro-optic modulator system for the production of dynamic images from giga-pixel computer generated holograms. In Practical Holography XIV and Holographic Materials VI, vol. 3956 of SPIE Proceedings, 13–22.Google Scholar
    47. STEWART, J., YU, J., GORTLER, S., AND MCMILLAN, L. 2003. A new reconstruction filter for undersampled light fields. In Eurographics Symposium on Rendering, ACM International Conference Proceeding Series, 150–156. Google ScholarDigital Library
    48. STONE, M. 2001. Color and brightness appearance issues in tiled displays. Computer Graphics and Applications 21, 6 (Sept.), 58–67. Google ScholarDigital Library
    49. TANIMOTO, M., AND FUJI, T., 2003. Ray-space coding using temporal and spatial predictions. ISO/IEC JTC1/SC29/WG11 Document M10410, Dec.Google Scholar
    50. WILBURN, B., SMULSKI, M., LEE, H. K., AND HOROWITZ, M. 2002. The light field video camera. In Media Processors 2002, vol. 4674 of SPIE, 29–36.Google Scholar
    51. YANG, J. C., EVERETT, M., BUEHLER, C., AND MCMILLAN, L. 2002. A real-time distributed light field camera. In Proceedings of the 13th Eurographics Workshop on Rendering, Eurographics Association, 77–86. Google ScholarDigital Library
    52. ZHANG, Z. 2000. A flexible new technique for camera calibration. IEEEE Transactions on Pattern Analysis and Machine Intelligence 22, 11, 1330–1334. Google ScholarDigital Library
    53. ZITNICK, L., KANG, S. B., UYTTENDAELE, M., WINDER, S., AND SZELISKI, R. 2004. High-quality video view interpolation using a layered representation. To appear in ACM Transaction on Graphics (Aug.). Google ScholarDigital Library

ACM Digital Library Publication: