“Achieving eye contact in a one-to-many 3D video teleconferencing system” by Jones, Lang, Fyffe, Yu, Busch, et al. …

We present a set of algorithms and an associated display system capable of producing correctly rendered eye contact between a three-dimensionally transmitted remote participant and a group of observers in a 3D teleconferencing system. The participant’s face is scanned in 3D at 30Hz and transmitted in real time to an autostereoscopic horizontal-parallax 3D display, displaying him or her over more than a 180° field of view observable to multiple observers. To render the geometry with correct perspective, we create a fast vertex shader based on a 6D lookup table for projecting 3D scene vertices to a range of subject angles, heights, and distances. We generalize the projection mathematics to arbitrarily shaped display surfaces, which allows us to employ a curved concave display surface to focus the high speed imagery to individual observers. To achieve two-way eye contact, we capture 2D video from a cross-polarized camera reflected to the position of the virtual participant’s eyes, and display this 2D video feed on a large screen in front of the real participant, replicating the viewpoint of their virtual self. To achieve correct vertical perspective, we further leverage this image to track the position of each audience member’s eyes, allowing the 3D display to render correct vertical perspective for each of the viewers around the device. The result is a one-to-many 3D teleconferencing system able to reproduce the effects of gaze, attention, and eye contact generally missing in traditional teleconferencing systems.

1. Argyle, M., and Cook, M. 1976. Gaze and Mutual Gaze. Cambridge University Press, London.Google Scholar
2. Chen, M. 2002. Leveraging the asymmetric sensitivity of eye contact for videoconference. In CHI ’02: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 49–56. Google ScholarDigital Library
3. Gemmell, J., Toyama, K., Zitnick, C., Kang, T., and Seitz, S. 2000. Gaze awareness for video-conferencing: a software approach. Multimedia, IEEE 7, 4 (Oct-Dec), 26–35. Google ScholarDigital Library
4. Grayson, D. M., and Monk, A. F. 2003. Are you looking at me? Eye contact and desktop video conferencing. ACM Trans. Comput.-Hum. Interact. 10, 3, 221–243. Google ScholarDigital Library
5. Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L. V., Lang, S., Strehlke, K., De Moere, A. V., and Staadt, O. 2003. blue-c: A spatially immersive display and 3d video portal for telepresence. ACM Transactions on Graphics 22, 3 (July), 819–827. Google ScholarDigital Library
6. Ishii, H., Kobayashi, M., and Grudin, J. 1993. Integration of interpersonal space and shared workspace: Clearboard design and experiments. ACM Trans. Inf. Syst. 11, 4, 349–375. Google ScholarDigital Library
7. Jerald, J., and Daily, M. 2002. Eye gaze correction for video-conferencing. In ETRA ’02: Proceedings of the 2002 symposium on Eye tracking research & applications, ACM, New York, NY, USA, 77–81. Google ScholarDigital Library
8. Jones, A., McDowall, I., Yamada, H., Bolas, M., and Debevec, P. 2007. Rendering for an interactive 360ř light field display. ACM Transactions on Graphics 26, 3 (July), 40:1–40:10. Google ScholarDigital Library
9. Kajiya, J. T., and Kay, T. L. 1989. Rendering fur with three dimensional textures. In Computer Graphics (Proceedings of SIGGRAPH 89), 271–280. Google ScholarDigital Library
10. Liu, J., Beldie, I. P., and Wöpking, M. 1995. A computational approach to establish eye-contact in videocommunication. In in Videocommunication, Int. Workshop on Stereoscopic and Three Dimentional Imaging, 229–234.Google Scholar
11. Matusik, W., and Pfister, H. 2004. 3D tv: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Transactions on Graphics 23, 3 (Aug.), 814–824. Google ScholarDigital Library
12. Morris, E. 2004. The Fog of War: 13 questions and answers on the filmmaking of Errol Morris. FLM Magazine. http://www.errolmorris.com/content/eyecontact/interrotron.html.Google Scholar
13. Muhlbach, L., Bocker, M., and Prussog, A. 1995. Telepresence in videocommunications: a study on stereoscopy and individual eye contact. Human Factors 37, 2, 290–305.Google ScholarCross Ref
14. Mukawa, N., Oka, T., Arai, K., and Yuasa, M. 2005. What is connected by mutual gaze?: user’s behavior in video-mediated communication. In CHI ’05: CHI ’05 extended abstracts on Human factors in computing systems, ACM, New York, NY, USA, 1677–1680. Google ScholarDigital Library
15. Nayar, S. K., Krishnan, G., Grossberg, M. D., and Raskar, R. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Transactions on Graphics 25, 3 (July), 935–944. Google ScholarDigital Library
16. Nguyen, D., and Canny, J. 2005. Multiview: spatially faithful group video conferencing. In CHI ’05: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 799–808. Google ScholarDigital Library
17. Nguyen, D. T., and Canny, J. 2007. Multiview: improving trust in group video conferencing through spatial faithfulness. In CHI ’07: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 1465–1474. Google ScholarDigital Library
18. Ostromoukhov, V. 2001. A simple and efficient error-diffusion algorithm. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 567–572. Google ScholarDigital Library
19. Ott, M., Lewis, J. P., and Cox, I. 1993. Teleconferencing eye contract using a virtual camera. In CHI ’93: INTERACT ’93 and CHI ’93 conference companion on Human factors in computing systems, ACM, New York, NY, USA, 109–110. Google ScholarDigital Library
20. Perlin, K., Paxia, S., and Kollin, J. S. 2000. An autostereoscopic display. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 319–326. Google ScholarDigital Library
21. Poulin, P., and Fournier, A. 1990. A model for anisotropic reflection. In Computer Graphics (Proceedings of SIGGRAPH 90), 273–282. Google ScholarDigital Library
22. Prussog, A., Muhlbach, L., and Bocker, M. 1994. Telepresence in videocommunications. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting, Human Factors and Ergonomics Society, Santa Monica, CA, USA, vol. 1, 180–184.Google Scholar
23. Quante, B., and Muhlbach, L. 1999. Eye-contact in multipoint videoconferencing. In Proceedings of the 17th International Symposium on Human Factors in Telecommunication.Google Scholar
24. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. 1998. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 179–188. Google ScholarDigital Library
25. Rees, J. 2008. Critics pan CNN’s fake election holograms. New Zealand Herald (Nov 7).Google Scholar
26. Rose, D. A. D., and Clarke, P. M. 1995. A review of eye-to-eye videoconferencing techniques. BT technology journal 13, 4, 127–131.Google Scholar
27. Sandin, D. J., Margolis, T., Ge, J., Girado, J., Peterka, T., and Defanti, T. A. 2005. The varrier autostereoscopic virtual reality display. In SIGGRAPH ’05: ACM SIGGRAPH 2005 Papers, ACM, New York, NY, USA, 894–903. Google ScholarDigital Library
28. Sellen, A. J. 1995. Remote conversations: The effects of mediating talk with technology. Human Computer Interaction 10, 401–444. Google ScholarDigital Library
29. Steinmeyer, J. 1999. The Science behind the Ghost: A Brief History of Pepper’s Ghost. Hahne.Google Scholar
30. Taguchi, Y., Koike, T., Takahashi, K., and Naemura, T. 2009. Transcaip: A live 3d tv system using a camera array and an integral photography display with interactive control of viewing parameters. Accepted to IEEE Transactions on Visualization and Computer Graphics. Google ScholarDigital Library
31. Vertegaal, R. 1999. The gaze groupware system: mediating joint attention in multiparty communication and collaboration. In CHI ’99: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 294–301. Google ScholarDigital Library
32. Viola, P., and Jones, M. J. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2, 137–154. Google ScholarDigital Library
33. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Transactions on Graphics 24, 3 (Aug), 765–776. Google ScholarDigital Library
34. Woods, E., Mason, P., and Billinghurst, M. 2003. Magicmouse: an inexpensive 6-degree-of-freedom mouse. In GRAPHITE 2003, 285–286. Google ScholarDigital Library
35. Yang, R., and Zhang, Z. 2004. Eye gaze correction with stereovision for video-teleconferencing. Pattern Analysis and Machine Intelligence, IEEE Transactions on 26, 7 (July), 956–960. Google ScholarDigital Library
36. Yang, J. C., Everett, M., Buehler, C., and Mcmillan, L. 2002. A real-time distributed light field camera. In Rendering Techniques 2002: 13th Eurographics Workshop on Rendering, 77–86. Google ScholarDigital Library
37. Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.Google Scholar
38. Zhang, Z. 2000. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 11, 1330–1334. Google ScholarDigital Library
39. Zwicker, M., Matusik, W., Durand, F., and Pfister, H. 2006. Antialiasing for automultiscopic 3D displays. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 73–82. Google ScholarDigital Library