“Electronic Perception Technology – Everyday Devices that “See”” by Bamji, Gokturk, Liu, Rafii and Spare
Conference:
- SIGGRAPH 2003
-
More from SIGGRAPH 2003:
Entry Number: 07
Title:
- Electronic Perception Technology - Everyday Devices that “See”
Presenter(s):
Project Affiliation:
- Canesta, Inc.
Description:
Abstract
Machines and digital devices today are essentially “blind,” unable to perceive the world around them so they can react and interact with it, except for the most specialized or expensive applications. Canesta has invented a breakthrough electronic perception technology that enables everyday devices (particularly small and inexpensive ones) with the ability to perceive the world around them as humans do, giving rise to new levels of functionality and convenience previously unimaginable. This project examines this technology and its first application – a projection keyboard for mobile and wireless devices. Future applications of the technology such as gesture processing for video games and user interface, occupant sensing for automotive airbag deployment, and intruder detection for security systems are also explored.
Introduction
The traditional approach to computer vision, perceiving specific objects or other identifiable features in a scene given only a 2-dimensional image, is a profoundly difficult problem. Many approaches have been developed through research over the last 60 years. Many depend upon either a complex physical setup, such as multiple cameras and controlled light sources, and virtually all require substantial amounts of computing power.
Canesta has made a significant breakthrough in this regard by its invention of a low cost electronic perception technology. The technology includes new types of CMOS chip-based image sensors, similar in size, power, and cost to commodity-priced video camera chips, that are uniquely able to resolve the three-dimensional features of a scene relevant to specific applications.
How electronic perception technology works
Electronic perception technology has two principal components: electronic perception sensor chips and proprietary image processing software embedded in such chips. The following discussion relates to one class of these chips as described in several U.S. patents recently granted to Canesta.
First, a diffuse infrared light source illuminates objects in a scene. The electronic perception sensor chip then measures the time it takes for light to travel from the scene to each pixel within the chip, providing a range value at each pixel (in addition to the brightness). The sensor chip has an array of these pixels which each generate the distance information simultaneously, resulting in a 3-dimensional depth map of the entire scene that updates dynamically – at over 30 frames per second. Embedded imaging software then uses the depth map to perceive and identify objects, and passes this information off to the host device at a higher level of abstraction.
Applications: everyday devices that “see”
The first application of electronic perception technology is the Canesta Keyboard Perception Chipset™. This chipset enables a projection keyboard capable of being fully integrated into cell phones, PDAs, or other mobile devices. When so equipped, the mobile device projects an image of a full-sized keyboard onto any flat surface, and the user “types” on this image. Canesta’s electronic perception technology resolves the user’s finger movements in real time into keystrokes provided to the device.
The future applications are limited only by one’s imagination. If you “sight-enable” an electronic device, it suddenly can react to and interact with the world in an entirely new way. For example, an automobile intelligent airbag system could sense the size and position of the occupant so that the airbag could deploy appropriately, reducing injury. Or a security system could easily detect the difference between a an intruder and normal activity, like a pet moving or child going to the bathroom in the night, dramatically reducing false alarms and increasing use. And video or virtual reality games could be freed from any mechanical input whatsoever. The game sees the player, and reacts to his actions.
When such capability can be added to virtually any product in a cost-efficient manner, designers will let their imaginations run free.
Other Information:
References
BAMJI, C. 2001. CMOS-compatible three-dimensional image sensor IC. US Patent # 6,323,94.
BAMJI, C., CHARBON, E., 2003. Methods for CMOS-compatible three-dimensional image sensing using quantum efficiency modulation. US Patent # 6,515,740.
CANESTA, INC. 2002. Whitepaper, Enabling Everyday Devices to “See”(www.canesta.com).
TOMASI, C., RAFII, A., AND TORUNOGLU, I. 2003. Canesta’s Projection Keyboard. In Communications of the ACM, ACM
