“Enhanced Reality: A New Frontier for Computer Entertainment” by Marks

Enhanced reality is a new form of computer entertainment that combines live video and computer graphics to produce real-time, movie-like special effects. Because the user is directly involved, enhanced reality can be more personalized and more engaging than traditional computer entertainment (video games). In the enhanced reality demonstrations of this exhibit, participants interact with a virtual character, play with virtual butterflies, interact with virtual crawling spiders, and engage in magic duels.

Enhanced reality is targeted specifically at home computer entertainment, for use in a typical living room or family room environment. When necessary, participants use simple props to enhance the interaction process; this enables a successful user experience despite the unstructured background and widely varying lighting conditions.

The techniques used to achieve enhanced reality fall into two categories:

1. Interpretation, which consists of processing video input to extract information about the participant and the environment, such as the 3D position of special props or a model of the lighting of the scene.
2. Enhancement, which consists of modifying the video image to produce a desired effect, such as rendering synthetic objects that look real.

This work is implemented on PlayStation 2 and displayed on a standard TV set. An inexpensive (<$100) IEEE 1394 Webcam is used for video input. The interaction props are simple plastic and/or foam toys.