“Performance animation from low-dimensional control signals” by Chai and Hodgins

This paper introduces an approach to performance animation that employs video cameras and a small set of retro-reflective markers to create a low-cost, easy-to-use system that might someday be practical for home use. The low-dimensional control signals from the user’s performance are supplemented by a database of pre-recorded human motion. At run time, the system automatically learns a series of local models from a set of motion capture examples that are a close match to the marker locations captured by the cameras. These local models are then used to reconstruct the motion of the user as a full-body animation. We demonstrate the power of this approach with real-time control of six different behaviors using two video cameras and a small set of retro-reflective markers. We compare the resulting animation to animation from commercial motion capture equipment with a full set of markers.

1. Aha, D. 1997. Editorial, special issue on lazy learning. In Artificial Intelligence Review. 11(1–5):1–6. Google ScholarDigital Library
2. Arikan, O., and Forsyth, D. A. 2002. Interactive motion generation from examples. In ACM Transactions on Graphics. 21(3):483–490. Google ScholarDigital Library
3. Arikan, O., Forsyth, D. A., and O’Brien, J. F. 2003. Motion synthesis from annotations. In ACM Transactions on Graphics. 22(3):402–408. Google ScholarDigital Library
4. Atkeson, C. G., Moore, A. W., and Schaal, S. 1997a. Locally weighted learning. In Artificial Intelligence Review. 11(1–5):11–73. Google ScholarDigital Library
5. Atkeson, C. G., Moore, A. W., and Schaal, S. 1997b. Locally weighted learning for control. In Artificial Intelligence Review. 11(1–5):75–113. Google ScholarDigital Library
6. Badler, N. I., Hollick, M., and Granieri, J. 1993. Real-time control of a virtual human using minimal sensors. In Presence. 2(1):82–86.Google ScholarDigital Library
7. Bazaraa, M. S., Sherali, H. D., and Shetty, C. 1993. Nonlinear Programming: Theory and Algorithms. John Wiley and Sons Ltd. 2nd Edition.Google Scholar
8. Bishop, C. 1996. Neural Network for Pattern Recognition. Cambridge University Press. Google ScholarDigital Library
9. Brand, M., and Hertzmann, A. 2000. Style machines. In Proceedings of ACM SIGGRAPH 2000. 183–192. Google ScholarDigital Library
10. Brand, M. 1999. Shadow puppetry. In Proceedings of IEEE International Conference on Computer Vision. 1237–1244. Google ScholarDigital Library
11. Bregler, C., and Omohundro, S. 1995. Nonlinear image interpolation using manifold learning. In Advances in Neural Information Processing Systems 7. 973–980.Google Scholar
12. Cheung, G., Baker, S., and Kanade, T. 2003. Shape-from-silhouette of articulated object and its use for human body kinematics estimation and motion capture. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 77–84. Google ScholarDigital Library
13. Daelemans, W., and Van De Bosch. A. 2001. Treetalk: Memory-based word phonemisation. In Data-Driven Techniques in Speech Synthesis, Kluwer. 149–172.Google Scholar
14. Dontcheva, M., Yngve, G., and Popovic, Z. 2003. Layered acting for character animation. In ACM Transactions on Graphics. 22(3):409–416. Google ScholarDigital Library
15. Freeman, W. T., Anderson, D., Beardsley, P., Dodge, C., Kage, H., Kyuma, K., Miyake, Y., Roth, M., Tanaka, K., Weissman, C., and Yerazunis, W. 1998. Computer vision for interactive computer graphics. In IEEE Computer Graphics and Applications. 18(3):42–53. Google ScholarDigital Library
16. Fukunaga, K., and Olsen. D. 1971. An algorithm for finding intrinsic dimensionality of data. In IEEE Transactions on Computers. C-20:176–183.Google Scholar
17. Grochow. K., Martin, S. L., Hertzmann, A., and Popovic, Z. 2004. Style-based inverse kinematics. In ACM Transactions on Graphics. 23(3):522–531. Google ScholarDigital Library
18. Guo. S., and Roberge, J. 1996. A high level control mechanism for human locomotion based on parametric frame space interpolation. In Eurographics Workshop on Computer Animation and Simulation ’96. 95–107. Google ScholarDigital Library
19. Hinton. G., Revow, M., and Dayan, P. 1995. Recognizing handwritten digits using mixtures of linear models. In Advances in Neural Information Processing Systems 7. 1015–1022.Google Scholar
20. Howe. N., Leventon, M., and Freeman, W. 1999. Bayesian reconstruction of 3d human motion from single-camera video. In Advances in Neural Information Processing Systems 12. 820–826.Google Scholar
21. Konami Boxing and Police 911 GAME, 2001. http://www.konami.com.Google Scholar
22. Kovar. L., and Gleicher, M. 2004. Automated extraction and parameterization of motions in large data sets. In ACM Transactions on Graphics. 23(3):559–568. Google ScholarDigital Library
23. Kovar. L., Gleicher, M., and Pighin, F. 2002. Motion graphs. In ACM Transactions on Graphics. 21(3):473–482. Google ScholarDigital Library
24. Lawrence, N. D. 2004. Gaussian process latent variable models for visualization of high dimensional data. In Advances in Neural Information Processing Systems 16. 329–336.Google Scholar
25. Lee, J., Chai, J., Reitsma, P., Hodgins, J., and Pollard, N. 2002. Interactive control of avatars animated with human motion data. In ACM Transactions on Graphics. 21(3):491–500. Google ScholarDigital Library
26. Li, Y., Wang, T., and Shum, H.-Y. 2002. Motion texture: A two-level statistical model for character synthesis. In ACM Transactions on Graphics. 21(3):465–472. Google ScholarDigital Library
27. Mardia, K., Kent, J., and Bibby, M. 1979. Multivariate Analysis. Academy Press.Google Scholar
28. MicroStrain 3DM-G, 2004. http://www.microstrain.com.Google Scholar
29. Oore, S., Terzopoulos, D., and Hinton, G. 2002. A desktop input device and interface for interactive 3d character. In Proceedings of Graphics Interface 2002. 133–140.Google Scholar
30. Pullen, K., and Bregler, C. 2002. Motion capture assisted animation: Texturing and synthesis. In ACM Transactions on Graphics. 21(3):501–508. Google ScholarDigital Library
31. Ren, L., Shakhnarovich, G., Hodgins, J. K., Pfister, H., and Viola, P. A. 2004. Learning silhouette features for control of human motion. In Computer Science Technical Reports 2004, Carnegie Mellon University. CMU-CS-04-165.Google Scholar
32. Rose, C., Cohen, M. F., and Bodenheimer, B. 1998. Verbs and adverbs: Multidimensional motion interpolation. In IEEE Computer Graphics and Applications. 18(5):32–40. Google ScholarDigital Library
33. Roweis, S., and Saul, L. 2000. Nonlinear dimensionality reduction by locally linear embedding. In Science. 290(5500):2323–2326.Google ScholarCross Ref
34. Safonova, A., Hodgins, J., and Pollard, N. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. In ACM Transactions on Graphics. 23(3):514–521. Google ScholarDigital Library
35. Scholkopf, B., Smola, A., and Muller, K.-R. 1999. Kernel principal component analysis. In Advances in Kernel Methods-SV Learning, MIT Press. 327–352. Google ScholarDigital Library
36. Semwal, S., Hightower, R., and Stansfield, S. 1998. Mapping algorithms for real-time control of an avatar using eight sensors. In Presence. 7(1):1–21. Google ScholarDigital Library
37. Shin, H. J., Lee, J., Gleicher, M., and Shin, S. Y. 2001. Computer puppetry: An importance-based approach. In ACM Transactions on Graphics. 20(2):67–94. Google ScholarDigital Library
38. Sidenbladh, H., Black, M. J., and Sigal, L. 2002. Implicit probabilistic models of human motion for synthesis and tracking. In European Conference on Computer Vision. 784–800. Google ScholarDigital Library
39. Sony Eye Toy Systems, 2003. http://www.eyetoy.com.Google Scholar
40. Stone, M. 1974. Cross-validatory choice and assessment of statistical predictions. In Journal of the Royal Statistical Society. 36:111–147.Google Scholar
41. Tenenbaum, J., Silva, V., and Langford, J. 2000. A global geometric framework for nonlinear dimensionality reduction. In Science. 290(5500):2319–2323.Google ScholarCross Ref
42. Vicon Systems, 2004. http://www.vicon.com.Google Scholar
43. Wiley, D. J., and Hahn, J. K. 1997. Interpolation synthesis of articulated figure motion. In IEEE Computer Graphics and Applications. 17(6):39–45. Google ScholarDigital Library
44. Xsens Mt-9, 2004. http://www.xsens.com.Google Scholar
45. Xu, G., and Zhang, Z. 1996. Epipolar Geometry in Stereo, Motion, and Object Recognition: A Unified Approach. Kluwer. Google ScholarDigital Library
46. Yamane, K., and Nakamura, Y. 2003. Natural motion animation through constraining and deconstraining at will. In IEEE Transactions on Visualization and Computer Graphics. 9(3):352–360. Google ScholarDigital Library
47. Yamane, K., Kuffner, J. J., and Hodgins, J. K. 2004. Synthesizing animations of human manipulation tasks. In ACM Transactions on Graphics. 23(3):532–539. Google ScholarDigital Library
48. Yin, K., and Pai, D. K. 2003. Footsee: An interactive animation system. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 329–338. Google ScholarDigital Library
49. Zhang, Z. 1999. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the International Conference on Computer Vision. 666–673.Google ScholarCross Ref