“Deep learning of biomimetic sensorimotor control for biomechanical human animation” by Nakada, Zhou, Chen, Weiss and Terzopoulos
Conference:
Type:
Entry Number: 56
Title:
- Deep learning of biomimetic sensorimotor control for biomechanical human animation
Session/Category Title: Virtually Human
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We introduce a biomimetic framework for human sensorimotor control, which features a biomechanically simulated human musculoskeletal model actuated by numerous muscles, with eyes whose retinas have nonuniformly distributed photoreceptors. The virtual human’s sensorimotor control system comprises 20 trained deep neural networks (DNNs), half constituting the neuromuscular motor subsystem, while the other half compose the visual sensory subsystem. Directly from the photoreceptor responses, 2 vision DNNs drive eye and head movements, while 8 vision DNNs extract visual information required to direct arm and leg actions. Ten DNNs achieve neuromuscular control—2 DNNs control the 216 neck muscles that actuate the cervicocephalic musculoskeletal complex to produce natural head movements, and 2 DNNs control each limb; i.e., the 29 muscles of each arm and 39 muscles of each leg. By synthesizing its own training data, our virtual human automatically learns efficient, online, active visuomotor control of its eyes, head, and limbs in order to perform nontrivial tasks involving the foveation and visual pursuit of target objects coupled with visually-guided limb-reaching actions to intercept the moving targets, as well as to carry out drawing and writing tasks.
References:
1. J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio. 2010. Theano: A CPU and GPU math compiler in Python. In Proc. 9th Python in Science Conference. Austin, TX, 1–7.Google Scholar
2. A.L. Cruz Ruiz, C. Pontonnier, N. Pronost, and G. Dumont. 2017. Muscle-based control for character animation. Computer Graphics Forum 36, 6 (2017), 122–147. Google ScholarDigital Library
3. M. F. Deering. 2005. A photon accurate model of the human eye. ACM Transactions on Graphics 24, 3 (2005), 649–658. Google ScholarDigital Library
4. P. Faloutsos, M. van de Panne, and D. Terzopoulos. 2001. Composable controllers for physics-based character animation. In Proc. 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). Los Angeles, CA, 251–260. Google ScholarDigital Library
5. Y. Fan, J. Litven, and D.K. Pai. 2014. Active volumetric musculoskeletal systems. ACM Transactions on Graphics 33, 4 (2014), 152. Google ScholarDigital Library
6. R. Featherstone. 2014. Rigid Body Dynamics Algorithms. Springer, New York, NY. Google ScholarDigital Library
7. T. Geijtenbeek, M. Van De Panne, and A.F. Van Der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics 32, 6 (2013), 206. Google ScholarDigital Library
8. I. Goodfellow, Y. Bengio, and A. Courville. 2016. Deep Learning. MIT Press, Cambridge, MA. Google ScholarDigital Library
9. R. Grzeszczuk, D. Terzopoulos, and G. Hinton. 1998. NeuroAnimator: Fast neural network emulation and control of physics-based models. In Computer Graphics Proceedings, Annual Conference Series. Orlando, FL, 9–20. Proc. ACM SIGGRAPH 98. Google ScholarDigital Library
10. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. IEEE International Conference on Computer Vision. Santiago, Chile, 1026–1034. Google ScholarDigital Library
11. J.K. Hodgins, W.L. Wooten, D.C. Brogan, and J.F. O’Brien. 1995. Animating human athletics. In Proc. ACM SIGGRAPH ’95 Conference. Los Angeles, CA, 71–78. Google ScholarDigital Library
12. D. Holden, T. Komura, and J. Saito. 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics 36, 4 (2017), 42. Google ScholarDigital Library
13. W. Huang, M. Kapadia, and D. Terzopoulos. 2010. Full-body hybrid motor control for reaching. In Motion in Games (Lecture Notes in Computer Science, Vol. 6459). Springer-Verlag, Berlin, 36–47. Google ScholarDigital Library
14. A.-E. Ichim, P. Kadleček, L. Kavan, and M. Pauly. 2017. Phace: Physics-based face modeling and animation. ACM Transactions on Graphics 36, 4 (2017), 153. Google ScholarDigital Library
15. P. Kadleček, A.-E. Ichim, T. Liu, J. Křivánek, and L. Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics 35, 6 (2016), 213. Google ScholarDigital Library
16. K. Kähler, J. Haber, H. Yamauchi, and H.-P. Seidel. 2002. Head shop: Generating animated head models with anatomical structure. In Proc. 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. San Antonio, TX, 55–63. Google ScholarDigital Library
17. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. Technical Report. arXiv preprint arXiv:1412.6980.Google Scholar
18. S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics 28, 4 (2009), 99:1–17. Google ScholarDigital Library
19. S.-H. Lee and D. Terzopoulos. 2006. Heads Up! Biomechanical modeling and neuromuscular control of the neck. ACM Transactions on Graphics 23, 212 (2006), 1188–1198. Proc. ACM SIGGRAPH 2006. Google ScholarDigital Library
20. Y. Lee, M.S. Park, T. Kwon, and J. Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics 33, 6 (2014), 218. Google ScholarDigital Library
21. Y. Lee, D. Terzopoulos, and K. Waters. 1995. Realistic modeling for facial animation. In Computer Graphics Proceedings, Annual Conference Series (Proc. ACM SIGGRAPH 95). Los Angleles, CA, 55–62. Google ScholarDigital Library
22. M. Lesmana, A. Landgren, P.-E. Forssén, and D.K. Pai. 2014. Active gaze stabilization. In Proc. Indian Conference on Computer Vision, Graphics, and Image Processing. Bangalore, India, Article 81, 8 pages. Google ScholarDigital Library
23. M. Lesmana and D.K. Pai. 2011. A biologically inspired controller for fast eye movements. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, Shanghai, China, 3670–3675.Google Scholar
24. L. Liu and J. Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep Q-learning. ACM Transactions on Graphics 36, 3 (2017), 29. Google ScholarDigital Library
25. M. Nakada, H. Chen, and D. Terzopoulos. 2018. Deep learning of biomimetic visual perception for virtual humans. In Proc. ACM Symposium on Applied Perception (SAP ’18). Vancouver, BC, 1–8. Google ScholarDigital Library
26. M. Nakada and D. Terzopoulos. 2015. Deep learning of neuromuscular control for biomechanical human animation. In Advances in Visual Computing (Lecture Notes in Computer Science, Vol. 9474). Springer, Berlin, 339–348. Proc. International Symposium on Visual Computing, Las Vegas, NV, December 2015.Google Scholar
27. X.B. Peng, G. Berseth, K. Yin, and M. Van De Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics 36, 4 (2017), 41. Google ScholarDigital Library
28. T.F. Rabie and D. Terzopoulos. 2000. Active perception in virtual humans. In Proc. Vision Interface 2000. Montreal, Canada, 16–22.Google Scholar
29. P. Sachdeva, S. Sueda, S. Bradley, M. Fain, and D.K. Pai. 2015. Biomechanical simulation and control of hands and tendinous systems. ACM Transactions on Graphics 34, 4 (2015), 42. Google ScholarDigital Library
30. E.L. Schwartz. 1977. Spatial mapping in the primate sensory projection: Analytic structure and relevance to perception. Biological Cybernetics 25, 4 (1977), 181–194. Google ScholarDigital Library
31. W. Si, S.-H. Lee, E. Sifakis, and D. Terzopoulos. 2014. Realistic biomechanical simulation and control of human swimming. ACM Transactions on Graphics 34, 1, Article 10 (Nov. 2014), 15 pages. Google ScholarDigital Library
32. E. Sifakis, I. Neverov, and R. Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 1, 212 (2005), 417–425. Google ScholarDigital Library
33. S. Sueda, A. Kaufman, and D.K. Pai. 2008. Musculotendon simulation for hand animation. ACM Transactions on Graphics 27, 3 (Aug. 2008), 83. Google ScholarDigital Library
34. D. Terzopoulos and T.F. Rabie. 1995. Animat vision: Active vision with artificial animals. In Proc. Fifth International Conference on Computer Vision (ICCV ’95). Cambridge, MA, 840–845. Google ScholarDigital Library
35. D. Terzopoulos and K. Waters. 1990. Physically-based facial modelling, analysis, and animation. Computer Animation and Virtual Worlds 1, 2 (1990), 73–80.Google Scholar
36. J.M. Wang, S.R. Hamner, S.L. Delp, and V. Koltun. 2012. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics 31, 4, Article 25 (2012), 11 pages. Google ScholarDigital Library
37. Q. Wei, S. Sueda, and D.K. Pai. 2010. Biomechanical simulation of human eye movement. In Biomedical Simulation (Lecture Notes in Computer Science), Vol. 5958. Springer-Verlag, Berlin, 108–118. Google ScholarDigital Library
38. S.H. Yeo, M. Lesmana, D.R. Neog, and D.K. Pai. 2012. Eyecatch: Simulating visuomotor coordination for object interception. ACM Transactions on Graphics 31, 4 (2012), 1–10. Google ScholarDigital Library
