“Integrated learning for interactive synthetic characters”

The ability to learn is a potentially compelling and important quality for interactive synthetic characters. To that end, we describe a practical approach to real-time learning for synthetic characters. Our implementation is grounded in the techniques of reinforcement learning and informed by insights from animal training. It simplifies the learning task for characters by (a) enabling them to take advantage of predictable regularities in their world, (b) allowing them to make maximal use of any supervisory signals, and (c) making them easy to train by humans.We built an autonomous animated dog that can be trained with a technique used to train real dogs called “clicker training”. Capabilities demonstrated include being trained to recognize and use acoustic patterns as cues for actions, as well as to synthesize new actions from novel paths through its motion space.A key contribution of this paper is to demonstrate that by addressing the three problems of state, action, and state-action space discovery at the same time, the solution for each becomes easier. Finally, we articulate heuristics and design principles that make learning practical for synthetic characters.

1. BALLARD, D. 1997. An Introduction to Natural Computation. MIT Press, Cambridge, MA. Google Scholar
2. BLUMBERG, B., AND GAYLEAN, T. 1995. Multi-level direction of autonomous creatures for real-time virtual environments. In Proceedings of SIGGRAPH 1995, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
3. BURKE, R., ISLA, D., DOWNIE, M., IVANOV, Y., AND BLUMBERG, B. 2001. Creature smarts: The art and architecture of a virtual brain. In Proceedings of the Computer Game Developers Conference.Google Scholar
4. BURKE, R. 2001. Its about Time:Temporal Representation for Synthetic Characters. Master’s thesis, The Media Lab, MIT.Google Scholar
5. COPPINGER, R., AND COPPINGER, L. 2001. Dogs: A Startling New Understanding of Canine Origin, Behavior, and Evolution. Scribner, New York, NY.Google Scholar
6. DOWNIE, M. 2000. behavior, animation, music: the music and movement of synthetic characters. Master’s thesis, The Media Lab, MIT.Google Scholar
7. DRESCHER, G. 1991. Made-Up Minds:A Constructivist Approach to Artificial Intelligence. MIT Press, Cambridge MA. Google Scholar
8. EVANS, R. 2002. Varieties of learning. In AI Game Programming Wisdom, E. Rabin, Ed. Charles River Media, Hingham MA. Google Scholar
9. FALOUTSOS, P., VAN DE PANNE, M., AND TERZOPOLOUS, D. 2001. Composible controllers for physics-based character animation. In Proceedings of SIGGRAPH 2001, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
10. FUNGE, J., TU, X., AND TERZOPOLOUS, D. 1999. Cognitive modeling: Knowledge, reasoning and planning for intelligent characters. In Proceedings of SIGGRAPH 1999, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
11. GALLISTEL, C. R., AND GIBBON, J. 2000. Time, rate and conditioning. Psychological Review 107.Google Scholar
12. GLEICHER, M. 1998. Retargetting motion to new characters. In Proceedings of SIGGRAPH 1998, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
13. GOULD, J., AND GOULD, C. 1999. The Animal Mind. W.H. Freeman, New York, NY.Google Scholar
14. GRAND, S., CLIFF, D., AND MALHOTRA, A. 1996. Creatures: Artificial life autonomous agents for home entertainment. In Proceedings of the Autonomous Agents ’97 Conference. Google Scholar
15. GRZESZCZUK, R., AND TERZOPOULOS, D. 1995. Automated learning of muscle-actuated locomotion through control abstraction. In Proceedings of SIGGRAPH 1995, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
16. GRZESZCZUK, R., TERZOPOULOS, D., AND HINTON, G. 1998. Neuroanimator: Fast neural network emulation and control of physics-based models. In Proceedings of SIGGRAPH 1998, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
17. HODGINS, J., AND POLLARD, N. 1997. Adapting simulated behaviors for new characters. In Proceedings of SIGGRAPH 1997, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
18. ISLA, D., BURKE, R., DOWNIE, M., AND BLUMBERG, B. 2001. A layered brain architecture for synthetic creatures. In Proceedings of The International Joint Conference on Artificial Intelligence. Google Scholar
19. ISLA, D. 2001. The Virtual Hippocampus: Spatial Common Sense for Synthetic Creatures. Master’s thesis, The Media Lab, MIT.Google Scholar
20. IVANOV, Y., BLUMBERG, B., AND PENTLAND, A. 2001. Expectation maximization for weakly labeled data. In Proceedings of the 18th International Conference on Machine Learning. Google Scholar
21. IVANOV, Y. 2001. State Discovery for Autonomous Creatures. PhD thesis, The Media Lab, MIT.Google Scholar
22. KAELBLING, L. 1990. Learning in embedded systems. PhD thesis, Stanford University. Google Scholar
23. KAPLAN, F., OUDEYER, P.-Y., KUBINYI, E., AND MIKLOSI, A. 2001. Taming robots with clicker training : a solution for teaching complex behaviors. In Proceedings of the 9th European workshop on learning robots, LNAI, Springer, M. Quoy, P. Gaussier, and J. L. Wyatt, Eds.Google Scholar
24. LINDSAY, S. 2000. Applied Dog Behavior and Training. Iowa State University Press, Ames, IA.Google Scholar
25. LORENZ, K., AND LEYAHUSEN, P. 1973. Motivation of Human and Animal Behavior: An Ethological View. Van Nostrand Reinhold Co., New York, NY.Google Scholar
26. LORENZ, K. 1981. The Foundations of Ethology. Springer-Verlag, New York, NY.Google Scholar
27. MITCHELL, K. 1997. Machine Learning. McGraw Hill, New York, NY. Google Scholar
28. PERLIN, K., AND GOLDBERG, A. 1996. Improv: A system for scripting interactive actors in virtual worlds. In Proceedings of SIGGRAPH 1996, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
29. PRYOR, K. 1999. Clicker Training for Dogs. Sunshine Books, Inc., Waltham, MA.Google Scholar
30. RABINER, L., AND JUANG, B.-H. 1993. Fundamentals of Speech Recognition. Prentice Hall, Englewood Cliffs, NJ. Google Scholar
31. RAMIREZ, K. 1999. Animal Training:Successful Animal Management Through Positive Reinforcement. Shedd Aquarium, Chicago, IL.Google Scholar
32. RESNER, B., STERN, A., AND FRANK, A. 1997. The truth about catz and dogz. In The Computer Games Developer Conference.Google Scholar
33. REYNOLDS, C. 1987. Flocks, herds and schools: A distributed behavioral model. In Proceedings of SIGGRAPH 1987, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
34. ROSE, C., COHEN, M., AND BODENHEIMER, B. 1999. Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics And Applications 18, 5. Google Scholar
35. SHETTLEWORTH, S. J. 1998. Cognition, Evolution and Behavior. Oxford University Press, New York, NY.Google Scholar
36. SUTTON, R., AND BARTO, A. 1998. Reinforcement Learning: An Introduction. MIT Press, Cambridge MA. Google Scholar
37. SUTTON, R. 1991. Reinforcement learning architectures for animats. In The First International Conference on Simulation of Adaptive Behavior, MIT Press, Paris, Fr. Google Scholar
38. THERRIEN, C. 1989. Decision Estimation and Classification: An Introduction to Pattern Recognition and Related Topics. John Wiley and Sons, New York, NY. Google Scholar
39. TOMLINSON, B., AND BLUMBERG, B. 2002. Alphawolf: Social learning, emotion and development in autonomous virtual agents. In First GSFC/JPL Workshop on Radical Agent Concepts.Google Scholar
40. TU, X., AND TERZOPOULOS, D. 1994. Artificial fishes: Physics, locomotion, perception, behavior. In Proceedings of SIGGRAPH 1994, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
41. VAN DE PANNE, M., AND FIUME., E. 1993. Sensor-actuator networks. In Proceedings of SIGGRAPH 1993, ACM Press / ACM SIGGRAPH, Computer Graphics Proceedings, Annual Conference Series, ACM. Google Scholar
42. VAN DE PANNE, M., KIM, R., AND FIUME., E. 1994. Synthesizing parameterized motions. In 5th Eurographics Workshop on Simulation and Animation.Google Scholar
43. WATKINS, C. J., AND DAYAN, P. 1992. Q-learning. Machine Learning 8. Google Scholar
44. WILKES, G. 1995. Click and Treat Training Kit. Click and Treat Inc., Mesa, AZ.Google Scholar
45. YOON, S., BLUMBERG, B., AND SCHNEIDER, G. 2000. Motivation-driven learning for interactive synthetic characters. In Proceedings of the Fourth International Conference on Autonomous Agents. Google Scholar
46. YOON, S., BURKE, R., AND BLUMBERG, B. 2000. Interactive training for synthetic characters. In Proceedings of AAAI 2000. Google Scholar