“Eyecatch: simulating visuomotor coordination for object interception” by Yeo, Lesmana, Neog and Pai

  • ©Sang Hoon Yeo, Martin Lesmana, Debanga R. Neog, and Dinesh K. Pai




    Eyecatch: simulating visuomotor coordination for object interception



    We present a novel framework for animating human characters performing fast visually guided tasks, such as catching a ball. The main idea is to consider the coordinated dynamics of sensing and movement. Based on experimental evidence about such behaviors, we propose a generative model that constructs interception behavior online, using discrete submovements directed by uncertain visual estimates of target movement. An important aspect of this framework is that eye movements are included as well, and play a central role in coordinating movements of the head, hand, and body. We show that this framework efficiently generates plausible movements and generalizes well to novel scenarios.


    1. Abe, Y., and Popović, J. 2006. Interactive animation of dynamic manipulation. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on Computer animation, Eurographics Association, 195–204. Google ScholarDigital Library
    2. Bauml, B., Schmidt, F., Wimbock, T., Birbach, O., Dietrich, A., Fuchs, M., Friedl, W., Frese, U., Borst, C., Grebenstein, M., et al. 2011. Catching flying balls and preparing coffee: Humanoid rollin’justin performs dynamic and sensitive tasks. In Robotics and Automation (ICRA), 2011 IEEE International Conference on, IEEE, 3443–3444.Google ScholarCross Ref
    3. Beek, P., and Lewbel, A. 1995. The science of juggling. Scientific American 273, 5, 92–97.Google ScholarCross Ref
    4. Berthoz, A. 2000. The brain’s sense of movement. Harvard Univ Pr.Google Scholar
    5. Birbach, O., Frese, U., and Bauml, B. 2011. Realtime perception for catching a flying ball with a mobile humanoid. In Robotics and Automation (ICRA), 2011 IEEE International Conference on, IEEE, 5955–5962.Google Scholar
    6. Carpenter, R., and Williams, M. 1995. Neural computation of log likelihood in control of saccadic eye movements. Nature 377, 6544, 59–62.Google Scholar
    7. Cooper, S., Hertzmann, A., and Popović, Z. 2007. Active learning for real-time motion controllers. In ACM Transactions on Graphics (TOG), vol. 26, ACM, 5. Google ScholarDigital Library
    8. Crawford, J., Martinez-Trujillo, J., and Klier, E. 2003. Neural control of three-dimensional eye and head movements. Current opinion in neurobiology 13, 6, 655–662.Google Scholar
    9. Dessing, J., Peper, C., Bullock, D., and Beek, P. 2005. How position, velocity, and temporal information combine in the prospective control of catching: Data and model. Journal of cognitive neuroscience 17, 4, 668–686. Google ScholarDigital Library
    10. Dipietro, L., Krebs, H., Fasoli, S., Volpe, B., and Hogan, N. 2009. Submovement changes characterize generalization of motor recovery after stroke. Cortex 45, 3, 318–324.Google ScholarCross Ref
    11. Flash, T., and Hogan, N. 1985. The coordination of arm movements: an experimentally confirmed mathematical model. The journal of Neuroscience 5, 7, 1688–1703.Google ScholarCross Ref
    12. Francik, J., and Szarowicz, A. 2005. Character animation with decoupled behaviour and smart objects. In 6th International Conference on Computer Games CGAIMS, Louisville, Kentucky, USA.Google Scholar
    13. Freedman, E., Stanford, T., and Sparks, D. 1996. Combined eye-head gaze shifts produced by electrical stimulation of the superior colliculus in rhesus monkeys. Journal of neurophysiology 76, 2, 927–952.Google ScholarCross Ref
    14. Freedman, E. 2001. Interactions between eye and head control signals can account for movement kinematics. Biological cybernetics 84, 6, 453–462.Google Scholar
    15. Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., and Sasse, M. 2003. The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. in Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, 529–536. Google ScholarDigital Library
    16. Gillies, M., and Dodgson, N. 1999. Ball catching: An example of psychologically-based behavioural animation. Eurographics UK.Google Scholar
    17. Grillon, H., and Thalmann, D. 2009. Simulating gaze attention behaviors for crowds. Computer Animation and Virtual Worlds 20, 2-3, 111–119. Google ScholarDigital Library
    18. Grochow, K., Martin, S. L., Hertzmann, A., and Popović, Z. 2004. Style-based inverse kinematics. In ACM SIGGRAPH 2004 Papers, ACM, New York, NY, USA, SIGGRAPH ’04, 522–531. Google ScholarDigital Library
    19. Gu, E., and Badler, N. 2006. Visual attention and eye gaze during multiparty conversations with distractions. In Intelligent Virtual Agents, Springer, 193–204. Google ScholarDigital Library
    20. Hayhoe, M., and Ballard, D. 2005. Eye movements in natural behavior. Trends in cognitive sciences 9, 4, 188–194.Google Scholar
    21. Hebb, D. 1949. The organization of behavior: A neuropsychological theory. Lawrence Erlbaum.Google Scholar
    22. Hove, B., and Slotine, J. 1991. Experiments in robotic catching. In American Control Conference, 1991, IEEE, 380–386.Google Scholar
    23. Itti, L. 2003. Realistic avatar eye and head animation using a neurobiological model of visual attention. Tech. rep., DTIC Document.Google Scholar
    24. Itti, L. 2006. Quantitative modelling of perceptual salience at human eye position. Visual cognition 14, 4–8, 959–984.Google Scholar
    25. Johansson, R., Westling, G., Bäckström, A., and Flanagan, J. 2001. Eye-hand coordination in object manipulation. the Journal of Neuroscience 21, 17, 6917–6932.Google Scholar
    26. Lanman, J., Bizzi, E., and Allum, J. 1978. The coordination of eye and head movement during smooth pursuit. Brain Research 153, 1, 39–53.Google ScholarCross Ref
    27. Lee, S., and Terzopoulos, D. 2006. Heads up!: biomechanical modeling and neuromuscular control of the neck. In ACM Transactions on Graphics (TOG), vol. 25, ACM, 1188–1198. Google ScholarDigital Library
    28. Lee, S., Badler, J., and Badler, N. 2002. Eyes alive. In ACM Transactions on Graphics (TOG), vol. 21, ACM, 637–644. Google ScholarDigital Library
    29. Lee, S., Sifakis, E., and Terzopoulos, D. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics (TOG) 28, 4, 99. Google ScholarDigital Library
    30. Leigh, R., and Zee, D. 1999. The neurology of eye movements. No. 55. Oxford Univ Pr.Google Scholar
    31. Liu, C. 2009. Dextrous manipulation from a grasping pose. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 59. Google ScholarDigital Library
    32. McIntyre, J., Zago, M., Berthoz, A., Lacquaniti, F., et al. 2001. Does the brain model newton’s laws? Nature Neuroscience 4, 7, 693–694.Google ScholarCross Ref
    33. McKee, S. 1981. A local mechanism for differential velocity detection. Vision Research 21, 4, 491–500.Google ScholarCross Ref
    34. Meyer, C., Lasker, A., and Robinson, D. 1985. The upper limit of human smooth pursuit velocity. Vision Research 25, 4, 561–563.Google ScholarCross Ref
    35. Novak, K., Miller, L., and Houk, J. 2002. The use of overlapping submovements in the control of rapid hand movements. Experimental Brain Research 144, 3, 351–364.Google ScholarCross Ref
    36. Orban de Xivry, J., and Lefèvre, P. 2007. Saccades and pursuit: two outcomes of a single sensorimotor process. The Journal of Physiology 584, 1, 11–23.Google ScholarCross Ref
    37. Pelachaud, C., and Bilvi, M. 2003. Modelling gaze behavior for conversational agents. In Intelligent Virtual Agents, Springer, 93–100.Google ScholarCross Ref
    38. Peters, C., and Qureshi, A. 2010. A head movement propensity model for animating gaze shifts and blinks of virtual characters. Computers & Graphics 34, 6, 677–687. Google ScholarDigital Library
    39. Plamondon, R. 1995. A kinematic theory of rapid human movements. Biological Cybernetics 72, 4, 295–307.Google ScholarDigital Library
    40. Pollard, N., and Zordan, V. 2005. Physically based grasping control from example. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, ACM, 311–318. Google ScholarDigital Library
    41. Riley, M., and Atkeson, C. 2002. Robot catching: Towards engaging human-humanoid interaction. Autonomous Robots 12, 1, 119–128. Google ScholarDigital Library
    42. Robinson, D., Gordon, J., and Gordon, S. 1986. A model of the smooth pursuit eye movement system. Biological Cybernetics 55, 1, 43–57. Google ScholarDigital Library
    43. Robinson, D. 1965. The mechanics of human smooth pursuit eye movement. The Journal of Physiology 180, 3, 569.Google ScholarCross Ref
    44. Rohrer, B., and Hogan, N. 2003. Avoiding spurious submovement decompositions: a globally optimal algorithm. Biological cybernetics 89, 3, 190–199.Google Scholar
    45. Shao, W., and Terzopoulos, D. 2005. Autonomous pedestrians. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, ACM, 19–28. Google ScholarDigital Library
    46. Soechting, J., and Terzuolo, C. 1987. Organization of arm movements. motion is segmented. Neuroscience 23, 1, 39–51.Google ScholarCross Ref
    47. Starkes, J., Helsen, W., and Elliott, D. 2002. A menage a trois: the eye, the hand and on-line processing. Journal of sports sciences 20, 3, 217–224.Google ScholarCross Ref
    48. Sternad, D., and Schaal, S. 1999. Segmentation of end-point trajectories does not imply segmented control. Experimental Brain Research 124, 1, 118–136.Google ScholarCross Ref
    49. Sueda, S., Kaufman, A., and Pai, D. K. 2008. Musculotendon simulation for hand animation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3, 83:1–83:8. Google ScholarDigital Library
    50. Terzopoulos, D., and Rabie, T. 1997. Animat vision: Active vision in artificial animals. Videre. Journal of Computer Vision Research 1, 1, 2–19.Google Scholar
    51. Tsang, W., Singh, K., and Fiume, E. 2005. Helping hand: an anatomically accurate inverse dynamics solution for unconstrained hand motion. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation, ACM, 319–328. Google ScholarDigital Library
    52. Tu, X., and Terzopoulos, D. 1994. Artificial fishes: Physics, locomotion, perception, behavior. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, ACM, 43–50. Google ScholarDigital Library
    53. Urtasun, R., Fleet, D. J., Geiger, A., Popović, J., Darrell, T. J., and Lawrence, N. D. 2008. Topologically-constrained latent variable models. In Proceedings of the 25th international conference on Machine learning, ACM, New York, NY, USA, ICML ’08, 1080–1087. Google ScholarDigital Library
    54. Vallbo, A., and Wessberg, J. 1993. Organization of motor output in slow finger movements in man. The Journal of physiology 469, 1, 673.Google ScholarCross Ref
    55. Wang, J. M., Fleet, D. J., and Hertzmann, A. 2007. Multifactor gaussian process models for style-content separation. In Proceedings of the 24th international conference on Machine learning, ACM, New York, NY, USA, ICML ’07, 975–982. Google ScholarDigital Library
    56. Wolpert, D., Ghahramani, Z., and Jordan, M. 1995. An internal model for sensorimotor integration. Science 269, 5232, 1880.Google ScholarCross Ref
    57. Woodworth, R. 1899. Accuracy of voluntary movement. The Psychological Review: Monograph Supplements 3, 3, i.Google ScholarCross Ref
    58. Yamane, K., Kuffner, J., and Hodgins, J. 2004. Synthesizing animations of human manipulation tasks. In ACM Transactions on Graphics (TOG), vol. 23, ACM, 532–539. Google ScholarDigital Library
    59. Young, L., and Stark, L. 1963. Variable feedback experiments testing a sampled data model for eye tracking movements. Human Factors in Electronics, IEEE Transactions on, 1, 38–51.Google Scholar
    60. Zago, M., McIntyre, J., Senot, P., and Lacquaniti, F. 2009. Visuo-motor coordination and internal models for object interception. Experimental Brain Research 192, 4, 571–604.Google ScholarCross Ref

ACM Digital Library Publication: