“Deep learning of biomimetic sensorimotor control for biomechanical human animation” by Nakada, Zhou, Chen, Weiss and Terzopoulos

  • ©Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos



Entry Number: 56


    Deep learning of biomimetic sensorimotor control for biomechanical human animation


Session Title: Virtually Human



    We introduce a biomimetic framework for human sensorimotor control, which features a biomechanically simulated human musculoskeletal model actuated by numerous muscles, with eyes whose retinas have nonuniformly distributed photoreceptors. The virtual human’s sensorimotor control system comprises 20 trained deep neural networks (DNNs), half constituting the neuromuscular motor subsystem, while the other half compose the visual sensory subsystem. Directly from the photoreceptor responses, 2 vision DNNs drive eye and head movements, while 8 vision DNNs extract visual information required to direct arm and leg actions. Ten DNNs achieve neuromuscular control—2 DNNs control the 216 neck muscles that actuate the cervicocephalic musculoskeletal complex to produce natural head movements, and 2 DNNs control each limb; i.e., the 29 muscles of each arm and 39 muscles of each leg. By synthesizing its own training data, our virtual human automatically learns efficient, online, active visuomotor control of its eyes, head, and limbs in order to perform nontrivial tasks involving the foveation and visual pursuit of target objects coupled with visually-guided limb-reaching actions to intercept the moving targets, as well as to carry out drawing and writing tasks.


    1. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio. 2010. Theano: A CPU and GPU math compiler in Python. In Proc. 9th Python in Science Conference. Austin, TX, 1–7.Google Scholar
    2. A.L. Cruz Ruiz, C. Pontonnier, N. Pronost, and G. Dumont. 2017. Muscle-based control for character animation. Computer Graphics Forum 36, 6 (2017), 122–147. Google ScholarDigital Library
    3. M. F. Deering. 2005. A photon accurate model of the human eye. ACM Transactions on Graphics 24, 3 (2005), 649–658. Google ScholarDigital Library
    4. P. Faloutsos, M. van de Panne, and D. Terzopoulos. 2001. Composable controllers for physics-based character animation. In Proc. 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). Los Angeles, CA, 251–260. Google ScholarDigital Library
    5. Y. Fan, J. Litven, and D.K. Pai. 2014. Active volumetric musculoskeletal systems. ACM Transactions on Graphics 33, 4 (2014), 152. Google ScholarDigital Library
    6. R. Featherstone. 2014. Rigid Body Dynamics Algorithms. Springer, New York, NY. Google ScholarDigital Library
    7. T. Geijtenbeek, M. Van De Panne, and A.F. Van Der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics 32, 6 (2013), 206. Google ScholarDigital Library
    8. I. Goodfellow, Y. Bengio, and A. Courville. 2016. Deep Learning. MIT Press, Cambridge, MA. Google ScholarDigital Library
    9. R. Grzeszczuk, D. Terzopoulos, and G. Hinton. 1998. NeuroAnimator: Fast neural network emulation and control of physics-based models. In Computer Graphics Proceedings, Annual Conference Series. Orlando, FL, 9–20. Proc. ACM SIGGRAPH 98. Google ScholarDigital Library
    10. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE International Conference on Computer Vision. Santiago, Chile, 1026–1034. Google ScholarDigital Library
    11. J.K. Hodgins, W.L. Wooten, D.C. Brogan, and J.F. O’Brien. 1995. Animating human athletics. In Proc. ACM SIGGRAPH ’95 Conference. Los Angeles, CA, 71–78. Google ScholarDigital Library
    12. D. Holden, T. Komura, and J. Saito. 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics 36, 4 (2017), 42. Google ScholarDigital Library
    13. W. Huang, M. Kapadia, and D. Terzopoulos. 2010. Full-body hybrid motor control for reaching. In Motion in Games (Lecture Notes in Computer Science, Vol. 6459). Springer-Verlag, Berlin, 36–47. Google ScholarDigital Library
    14. A.-E. Ichim, P. Kadleček, L. Kavan, and M. Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics 36, 4 (2017), 153. Google ScholarDigital Library
    15. P. Kadleček, A.-E. Ichim, T. Liu, J. Křivánek, and L. Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics 35, 6 (2016), 213. Google ScholarDigital Library
    16. K. Kähler, J. Haber, H. Yamauchi, and H.-P. Seidel. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. San Antonio, TX, 55–63. Google ScholarDigital Library
    17. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. Technical Report. arXiv preprint arXiv:1412.6980.Google Scholar
    18. S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics 28, 4 (2009), 99:1–17. Google ScholarDigital Library
    19. S.-H. Lee and D. Terzopoulos. 2006. Heads Up! Biomechanical modeling and neuromuscular control of the neck. ACM Transactions on Graphics 23, 212 (2006), 1188–1198. Proc. ACM SIGGRAPH 2006. Google ScholarDigital Library
    20. Y. Lee, M.S. Park, T. Kwon, and J. Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics 33, 6 (2014), 218. Google ScholarDigital Library
    21. Y. Lee, D. Terzopoulos, and K. Waters. 1995. Realistic modeling for facial animation. In Computer Graphics Proceedings, Annual Conference Series (Proc. ACM SIGGRAPH 95). Los Angleles, CA, 55–62. Google ScholarDigital Library
    22. M. Lesmana, A. Landgren, P.-E. Forssén, and D.K. Pai. 2014. Active gaze stabilization. In Proc. Indian Conference on Computer Vision, Graphics, and Image Processing. Bangalore, India, Article 81, 8 pages. Google ScholarDigital Library
    23. M. Lesmana and D.K. Pai. 2011. A biologically inspired controller for fast eye movements. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, Shanghai, China, 3670–3675.Google Scholar
    24. L. Liu and J. Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep Q-learning. ACM Transactions on Graphics 36, 3 (2017), 29. Google ScholarDigital Library
    25. M. Nakada, H. Chen, and D. Terzopoulos. 2018. Deep learning of biomimetic visual perception for virtual humans. In Proc. ACM Symposium on Applied Perception (SAP ’18). Vancouver, BC, 1–8. Google ScholarDigital Library
    26. M. Nakada and D. Terzopoulos. 2015. Deep learning of neuromuscular control for biomechanical human animation. In Advances in Visual Computing (Lecture Notes in Computer Science, Vol. 9474). Springer, Berlin, 339–348. Proc. International Symposium on Visual Computing, Las Vegas, NV, December 2015.Google Scholar
    27. X.B. Peng, G. Berseth, K. Yin, and M. Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics 36, 4 (2017), 41. Google ScholarDigital Library
    28. T.F. Rabie and D. Terzopoulos. 2000. Active perception in virtual humans. In Proc. Vision Interface 2000. Montreal, Canada, 16–22.Google Scholar
    29. P. Sachdeva, S. Sueda, S. Bradley, M. Fain, and D.K. Pai. 2015. Biomechanical simulation and control of hands and tendinous systems. ACM Transactions on Graphics 34, 4 (2015), 42. Google ScholarDigital Library
    30. E.L. Schwartz. 1977. Spatial mapping in the primate sensory projection: Analytic structure and relevance to perception. Biological Cybernetics 25, 4 (1977), 181–194. Google ScholarDigital Library
    31. W. Si, S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2014. Realistic biomechanical simulation and control of human swimming. ACM Transactions on Graphics 34, 1, Article 10 (Nov. 2014), 15 pages. Google ScholarDigital Library
    32. E. Sifakis, I. Neverov, and R. Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 1, 212 (2005), 417–425. Google ScholarDigital Library
    33. S. Sueda, A. Kaufman, and D.K. Pai. 2008. Musculotendon simulation for hand animation. ACM Transactions on Graphics 27, 3 (Aug. 2008), 83. Google ScholarDigital Library
    34. D. Terzopoulos and T.F. Rabie. 1995. Animat vision: Active vision with artificial animals. In Proc. Fifth International Conference on Computer Vision (ICCV ’95). Cambridge, MA, 840–845. Google ScholarDigital Library
    35. D. Terzopoulos and K. Waters. 1990. Physically-based facial modelling, analysis, and animation. Computer Animation and Virtual Worlds 1, 2 (1990), 73–80.Google Scholar
    36. J.M. Wang, S.R. Hamner, S.L. Delp, and V. Koltun. 2012. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics 31, 4, Article 25 (2012), 11 pages. Google ScholarDigital Library
    37. Q. Wei, S. Sueda, and D.K. Pai. 2010. Biomechanical simulation of human eye movement. In Biomedical Simulation (Lecture Notes in Computer Science), Vol. 5958. Springer-Verlag, Berlin, 108–118. Google ScholarDigital Library
    38. S.H. Yeo, M. Lesmana, D.R. Neog, and D.K. Pai. 2012. Eyecatch: Simulating visuomotor coordination for object interception. ACM Transactions on Graphics 31, 4 (2012), 1–10. Google ScholarDigital Library

ACM Digital Library Publication: