“The office of the future: a unified approach to image-based modeling and spatially immersive displays” by Raskar, Welch, Cutts, Lake, Stesin, et al. …

We introduce ideas, proposed technologies, and initial results for an office of the future that is based on a unified application of computer vision and computer graphics in a system that combines and builds upon the notions of the CAVETM, tiled display systems, and image-based modeling. The basic idea is to use real-time computer vision techniques to dynamically extract per-pixel depth and reflectance information for the visible surfaces in the office including walls, furniture, objects, and people, and then to either project images on the surfaces, render images of the surfaces, or interpret changes in the surfaces. In the first case, one could designate every-day (potentially irregular) real surfaces in the office to be used as spatially immersive display surfaces, and then project high-resolution graphics and text onto those surfaces. In the second case, one could transmit the dynamic image-based models over a network for display at a remote site. Finally, one could interpret dynamic changes in the surfaces for the purposes of tracking, interaction, or augmented reality applications. To accomplish the simultaneous capture and display we envision an office of the future where the ceiling lights are replaced by computer controlled cameras and “smart” projectors that are used to capture dynamic image-based models with imperceptible structured light techniques, and to display high-resolution images on designated display surfaces. By doing both simultaneously on the designated display surfaces, one can dynamically adjust or autocalibrate for geometric, intensity, and resolution variations resulting from irregular or changing display surfaces, or overlapped projector images. Our current approach to dynamic image-based modeling is to use an optimized structured light scheme that can capture per-pixel depth and reflectance at interactive rates. Our system implementation is not yet imperceptible, but we can demonstrate the approach in the laboratory. Our approach to rendering on the designated (potentially irregular) display surfaces is to employ a two-pass projective texture scheme to generate images that when projected onto the surfaces appear correct to a moving head-tracked observer. We present here an initial implementation of the overall vision, in an office-like setting, and preliminary

1. Bajaj, C.L., F. Bernardini, and G. Xu. “Automatic reconstruction of surfaces and scalar fields from 3D scans,” SIGGRAPH 95 Conference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley, pp. 109-118, August 1995.
2. Bennett, David T. Chairman and Co-Founder of Alternate Realities Corporation. Internet: http://www.virtual-reality.com. 215 Southport Drive Suite 1300 Morrisville, NC 27560.
3. Bowen, Loftin, R. “Hands Across the Atlantic,” IEEE Computer Graphics and Applications, Vol. 17, No. 2, pp. 78-79, March-April 1997.
4. Bryson, Steve, David Zeltzer, Mark T. Bolas, Bertrand de La Chapelle, and David Bennett. “The Future of Virtual Reality: Head Mounted Displays Versus Spatially Immersive Displays,” SIGGRAPH 97 Conference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley, pp. 485-486, August 1997.
5. Buxton, W., Sellen, A. & Sheasby, M. “Interfaces for multiparty videoconferencing,” In K. Finn, A. Sellen & S. Wilber (Eds.). Video Mediated Communication. Hillsdale, N.J.: Erlbaum, pp. 385-400, 1997.
6. Capin, Tolga K., Hansrudi Noser, Daniel Thalmann, Igor Sunday Pandzic and Nadia Magnenat Thalman. “Virtual Human Representation and Communication in VLNet,” IEEE Computer Graphics and Applications, Vol. 17, No. 2, pp. 42-53, March-April 1997.
7. Chi, Vern, Matt Cutts, Henry Fuchs, Kurtis Keller, Greg Welch, Mark Bloomenthal, Elaine Cohen, Sam Drake, Russ Fish, Rich Riesenfeld. 1998. “A Wide Field-of-View Camera Cluster”, University of North Carolina at Chapel Hill, Dept of Computer Science, Technical Report TR98-018.
8. Chien, C.H., Y.B. Sire, and J.K. Aggarwal. “Generation of volume/surface octree from range data,” In The Computer Graphics Society Conference on Computer Vision and Pattern Recognition, pp. 254-260, June 1988.
9. Clodfelter, Robert M. “Predicting Display System Performance,” Presented at the 1996 IMAGE Conference, Scottsdale, AZ, pp. 1-5, June 23-28, 1996.
10. Conner, D.B, Cutts, M., Fish, R., Fuchs, H., Holden, L., Jacobs, M., Loss, B., Markosian, L., Riesenfeld, R., and Turk, G. “An Immersive Tool for Wide-Area Collaborative Design,” TeamCAD, the First Graphics Visualization, and Usability (GVU) Workshop on Collaborative Design. Atlanta, Georgia, May 12-13, 1997.
11. Connolly, C.I. “Cumulative generation of octree models from range data,” Proceedings, Int’l. Conference Robotics, pp. 25-32, March 1984.
12. Cruz-Neira, Carolina, Daniel J. Sandin, and Thomas A. DeFanti. “Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE,” Computer Graphics, SIGGRAPH Annual Conference Proceedings, 1993.
13. Curless, Brian, and Marc Levoy. “A Volumetric Method for Building Complex Models from Range Images,” SIGGRAPH 96 onference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley. pp. 303-312, 1996.
14. DePiero, F.W., and Trivedi, M.M., “3-D Computer Vision using Structured Light: Design, Calibration, and Implementation Issues,” Advances in Computers(43), 1996, Academic Press, pp.243-278
15. Dias, Jos6 Miguel Salles, Ricardo Galli, Ant6nio Carlos Almeida, Carlos A. C. Belo, and Jos6 Manuel Rebord~. “mWorld: A Multiuser 3D Virtual Environment,” IEEE Computer Graphics and Applications, Vol. 17, No. 2., pp. 55-64, March-April 1997.
16. Dorsey, Julie O’B., Fransco X. Sillion, Donald Greenberg. “Design and Simulation of Opera Lighting and Projection Effects,” SIGGRAPH 91 Conference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley, pp. 41-50, 1991.
17. Edelsbrunner, H. and E.E Mucke. “Three-dimensional alpha shapes,” In Workshop on Volume Visualization, pp. 75-105, October 1992.
18. Fuchs, Henry, Gary Bishop, Kevin Arthur, Leonard McMillan, Ruzena Bajcsy, Sang Lee, Hany Farid, and Takeo Kanade. “Virtual Space Teleconferencing Using a Sea of Cameras,” Proceedings of the First International Symposium on Medical Robotics and Computer Assisted Surgery, (Pittsburgh, PA.) Sept 22-24, 1994.
19. Gajewska, Hania, Jay Kistler, Mark S. Manasse, and David D. Redell. “Argo: A System for Distributed Collaboration,” (DEC, Multimedia ’94)
20. Gibbs, Simon, Constantin Arapis and Christian J. Breiteneder. “TELEPORT-Towards Immersive Copresence,” accepted for publication in ACM Multimedia Systems Journal, 1998.
21. Goncalves, Luis, Enrico Di Bernardo, Enrico Ursella, Pietro Perona. “Monocular Tracking of the Human Arm in 3D,” Proc. of the 5th Inter. Conf. on Computer Vision, ICCV 95.
22. Hilton, A., A.J. Toddart, J. Illingworth, and T. Windeatt. “Reliable surface reconstruction from multiple range images,” In Fouth European Conference on Computer Vision, Volume 1, pp. 117-126. April 1996.
23. Holmes, Richard E. “Common Projector and Display Modules for Aircraft Simulator Visual Systems,” Presented at the IMAGE V Conference, Phoenix, AZ, June 19-22, pp. 81-88, 1990.
24. Hoppe, Hugues, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle. “Surface Reconstruction from Unorganized Points,” SIGGRAPH 92 Conference Proceedings, Annual Conference Series, ACM SIGGRAPH, Addison-Wesley, pp. 71-76, 1992.
25. Hornbeck, Larry J., “Deformable-Mirror Spatial Light Modulators,”Proceedings SPIE, Vol. 1150, Aug 1989.
26. Hornbeck, Larry J., “Digital Light Processing for High- Brightness High-Resolution Applications,” {cited 21 April 1998}. Available from http://www.ti.com/dlp/docs/business/resources/ white/hornbeck.pdf, 1995.
27. Ichikawa, Y., Okada, K., Jeong, G., Tanaka, S. and Matsushita, Y.: “MAJIC Videoconferencing System: Experiments, Evaluation and Improvement’,” In Proceedings of ECSCW’95, pp. 279-292, Sept. 1995.
28. Ishii, Hiroshi, Minoru Kobayashi, Kazuho Arita. “Iterative Design of Seamless Collaboration Media,” CACM, Volume 37, Number 8, pp. 83-97, August 1994.
29. Jarvis, Kevin. “Real-time 60Hz Distortion Correction on a Silicon Graphics IG,” Real-time Graphics, Vol. 5, No. 7, pp. 6-7. February 1997.
30. Kanade, Takeo and Haruhiko Asada. “Noncontact Visual Three-Dimensional Ranging Devices,” Proceedings of SPIE: 3D Machine Perception. Volume 283, Pages 48-54. April 23-24, 1981.
31. Kanade, Takeo, Hiroshi Kano, Shigeru Kimura, Atsushi Yoshida, Kazuo Oda. “Development of a Video-Rate Stereo Machine,” Proceedings of International Robotics and Systems Conference (IROS ’95). pp. 95-100, Pittsburgh, PA., August 5-9, 1995.
32. Lacroix, Michel. “A HDTV Projector For Wide Field of View Flight Simulators,” pp. 493-500. Presented at the IMAGE VI Conference, Scottsdale, AZ, July 14-17, 1992.
33. Lamotte, Wire, Eddy Flerackers, Frank Van Reeth, Rae Earnshaw, Joao Mena De Matos. Visinet: Collaborative 3D Visualization and VR over ATM Networks. IEEE Computer Graphics and Applications, Vol. 17, No. 2, pp. 66-75, March-April 1997.
34. Lehner, Valerie D., and Thomas A. DeFanti. “Distributed Virtual Reality: Supporting Remote Collaboration in Vehicle Design,” IEEE Computer Graphics and Applications, Vol. 17, No. 2, pp. 13-17, March-April 1997.
35. Lyon, Paul. “Edge-blending Multiple Projection Displays On A Dome Surface To Form Continuous Wide Angle Fields-of-View,” pp. 203-209. Proceedings of 7th I/ITEC, 1985.
36. Macedonia, Michale R. and Stefan Noll. “A Transatlantic Research and Development Environment,” IEEE Computer Graphics and Applications, Vol. 17, No. 2, pp. 76-82, March-April 1997.
37. Mandeville, J., T. Furness, M. Kawahata, D. Campbell, E Danset, A. Dahl, J. Dauner, J. Davidson, K. Kandie, and E Schwartz. “GreenSpace: Creating a Distributed Virtual Environment for Global Applications,” Proceedings of IEEE Networked Virtual Reality Workshop, 1995.
38. Nayar, Shree, Masahiro Watanabe, Minori Noguchi. “Realtime Focus Range Sensor,” Columbia University, CUCS-028-94.
39. Neumann, Ulrich and Henry Fuchs, “A Vision of Telepresence for Medical Consultation and Other Applications,” Proceedings of the Sixth International Symposium on Robotics Research, Hidden Valley, PA, Oct. 1-5, 1993, pp. 565-571.
40. Ohya, Jun, Kitamura, Yasuichi, Takemura, Haruo, et al. “Realtime Reproduction of 3D Human Images in Virtual Space Teleconferencing,” IEEE Virtual Reality International Symposium. Sep 1993.
41. Panorama Project, WWW description. {cited 13 January 1998}. Available from http://www.tnt.uni-hannover.de/project/eu/ panorama/overview.html
42. Raskar, Ramesh, Matt Cutts, Greg Welch, Wolfgang Sttierzlinger. “Efficient Image Generation for Multiprojector and Multisurface Displays,” University of North Carolina at Chapel Hill, Dept of Computer Science, Technical Report TR98-016, 1998.
43. Raskar, Ramesh, Henry Fuchs, Greg Welch, Adam Lake, Matt Cutts. “3D Talking Heads : Image Based Modeling at Interactive rate using Structured Light Projection,”University of North Carolina at Chapel Hill, Dept of Computer Science, Technical Report TR98-017, 1998
44. Segal, Mark, Carl Korobkin, Rolf van Widenfelt, Jim Foran and Paul Haeberli, “Fast Shadows and Lighting Effects using Texture Mapping,” Computer Graphics (SIGGRAPH 92 Proceedings), pp. 249-252, July, 1992.
45. Tsai, Roger Y. “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision,” Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 364-374, 1986.
46. Underkoffler, J. “A View From the Luminous Room,” Personal Technologies, Vol. 1, No. 2, pp. 49-59, June 1997.
47. Underkoffler, J., and Hiroshi Ishii. “Illuminating Light: An Optical Design Tool with a Luminous-Tangible Interface,” Proceedings of CHI ’98, ACM, April 1998.