“Interacting With Virtual Actors” by Perlin

Conference:

• SIGGRAPH 1995
• More from SIGGRAPH 1995:  

Type(s):

• E-Tech

Entry Number: 05

Title:

Interacting With Virtual Actors

Program Title:

Interactive Entertainment

Presenter(s):

• Ken Perlin

Collaborator(s):

• Athomas Goldberg
• Maria Augusta
• Leo Cadaval
• Troy Downing
• Mehmet Karaul
• Kouchen Lin
• Dan Moss
• Ruggero Ruschioni
• Eduardo Toledo
• Daniel Wey
• Marcelo Knorich-Zuffo

Project Affiliation:

NYU Media Research Lab

Description:

The New York University Media Research Laboratory (NYU- MRL) aims to create a sense of social space and interaction, to create virtual actors that respond believably in social ways in real time, without the intervention of human pup­peteers, and to effectively use a distributed network for com­puting the behavior of the vir­tual actors. These are enabling technologies for interactive television and embodied meta­ verses.

History

In 1985, NYU-MRL used pseudo-random functions to create natural surface textures of surprisingly realistic appear­ance without modeling the underlying physics. This work led, for example, to a popular texture “noise function,” the skin on the “Jurassic Park” dinosaurs, and the atmosphere effects in “The Lion King.”

Last year, the lab applied the same approach to the problem of building real-time graphic puppets that appear to be emotionally responsive. Instead of trying to model the correct physics of human movement, they applied noise functions, together with rhyth­mic motions. The results, which were surprisingly lifelike, can be used to convey very subtle expressions of personality and body language. They were demonstrated in the short film “Danse Interactif” in the SIGGRAPH 94 Electronic Theatre.

NYU-MRL shares an interest in real-time parallel computation with the Laboratorio de Sistemas Integraveis at the University of Sao Paolo. Together, the two labs have now successfully extended these techniques to real-time social interaction between groups of puppets.

Basic Functionalities

In this project, participants enter a darkened corridor and see a virtual room through a “picture window.” A few virtu­al people in the virtual room are interacting socially with each other. The virtual room is rendered on a Silicon Graphics workstation and rear projected via a high-resolution projector. A ceiling-mounted video tracker tracks each par­ticipant’s position. There is no trained operator intervening in the activities. Naive partici­pants interact on their own, in complex ways, with the virtual actors.

The virtual people in the virtual room interact with each other. They move about, perhaps give each other things, look out the window, or shake hands. It’s a typical social gathering. To the participant, their “talking” is heard as indistinct murmuring. It is clear that they are talking (even their tone of voice can be dis­cerned), but it is not clear what language they are speaking.

The participants are interesting to these virtual people. One or two of them might glance over to see what a participant is up to. Another might walk over to the “window” and peer at the participant. As the participant moves about, virtual people follow the movements with
their eyes. One might point, or wave. If the participant knocks on the window, one of them might knock back, and the sound is audible. If the partici­pant gets too close, the virtual person near the “window” steps back, to maintain a safe distance. Two or more people might argue about the partici­pant’s presence, pointing and waving or raising their voices. Participants try to get the atten­tion of the virtual people in the virtual room. They try to be noticed, and to provoke a reaction.

To the participant, the virtual people seem to be from anoth­er country or even another planet. They do not speak the same language, but there is an attempt to communicate, mostly through body lan­guage.

Technical Information

Each actor is computed on a different computer in a local area network (LAN). They com­municate with each other over the LAN in real time at interac­ tive frame rates. The actors actually pass entire behavior procedures to each other over the network at each animation frame, giving each other com­ plex instructions and hints on how to respond to each other and allowing them to coordi­nate their responses to the par­ ticipant.

The behavior description lan­guage that is passed around the network is the same lan­guage that was originally developed for NYU-MRL’s inter­ active texture work in 1 984. It has special features not usually found in programming lan­guages (including very light­ weight procedure definitions and a powerful method for fast dynamic scoping) that are nec­essary to realize this sort of performance.

Research Agenda

This research focuses on situa­tions in which people commu­nicate richly through body lan­guage, such as parties, bar scenes, and meetings. The goals of this project include:

• Creating a sense of social space and interaction.
• Making embodied charac­ters believable and respon­sive to users in real time.
• Effective use of a distributive network for computing the behavior of actors.
• Generating “suspended dis­ belief” among the partici­pants, so they feel the expe­rience is really happening.

In particular, NYU-MRL is exploring how to convey peripheral awareness, approach/avoidance,
“paying attention,” “listening,” etc. The lab also studies immersive scenarios involving two or more walls, to deter­ mine how simulated body lan­guage will help to convey the impression of various compet­ing social or attention-getting activities.

Ultimately, NYU-MRL’s goal is to determine to what extent the mere rhythm of interpersonal interaction can be encoded, in order to convey the impression of social complexity. For exam­ ple, could one structure entire narratives in this manner?

Assessment of Potential Future Impact

Consider this possible sce­nario: you send your “agent” to the Metaverse library to get some information. Along the way, the agent encounters your neighbor strolling the Metaverse. You’ve already told your agent that if she encoun­ters this person, she must remind him of your lunch date tomorrow.

Alternatively, for networked video games, this NYU-MRL work demonstrates that it is feasible to generate fully 3D, interactive characters. This allows networked video games to break away from the current restrictive reliance on canned CD-ROM-based video footage and animation clips.

This project also shows that it is feasible, using current tech­nology, to apply these tech­niques non-invasively. People can interact in their own homes with real-time personal agents who have convincing and interesting personalities and live behind wall-size screens. Such agents can be used for entertainment, infor­mation access, or simply for company.

Additional Images:

PDF:

• Interacting With Virtual Actors

Overview Page:

SIGGRAPH 1995: Interactive Communities