“Scalable muscle-actuated human simulation and control” by Lee, Park, Lee and Lee

  • ©Seunghwan Lee, Moon Seok Park, Kyoungmin Lee, and Jehee Lee




    Scalable muscle-actuated human simulation and control

Session/Category Title: Learning to Move



    Many anatomical factors, such as bone geometry and muscle condition, interact to affect human movements. This work aims to build a comprehensive musculoskeletal model and its control system that reproduces realistic human movements driven by muscle contraction dynamics. The variations in the anatomic model generate a spectrum of human movements ranging from typical to highly stylistic movements. To do so, we discuss scalable and reliable simulation of anatomical features, robust control of under-actuated dynamical systems based on deep reinforcement learning, and modeling of pose-dependent joint limits. The key technical contribution is a scalable, two-level imitation learning algorithm that can deal with a comprehensive full-body musculoskeletal model with 346 muscles. We demonstrate the predictive simulation of dynamic motor skills under anatomical conditions including bone deformity, muscle weakness, contracture, and the use of a prosthesis. We also simulate various pathological gaits and predictively visualize how orthopedic surgeries improve post-operative gaits.


    1. Ijaz Akhter and Michael J. Black. 2015. Pose-conditioned joint angle limits for 3D human pose reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1446–1455.Google Scholar
    2. Edith M Arnold, Samuel R Ward, Richard L Lieber, and Scott L Delp. 2010. A model of the lower limb for analysis of human movement. Annals of biomedical engineering 38, 2 (2010), 269–279.Google ScholarCross Ref
    3. Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence. Google ScholarDigital Library
    4. Glen Berseth, Cheng Xie, Paul Cernek, and Michiel Van de Panne. 2018. Progressive reinforcement learning with distillation for multi-skilled motion control. arXiv preprint arXiv.1802.04765 (2018).Google Scholar
    5. Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. 2018. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions on Graphics 37, 6 (2018). Google ScholarDigital Library
    6. Michael Damsgaard, John Rasmussen, Søren T. Christensen, Egidijus Surma, and Mark D. Zee. 2006. Analysis of musculoskeletal systems in the AnyBody Modeling System. Simulation Modelling Practice and Theory 14, 8 (2006), 1100 — 1111.Google ScholarCross Ref
    7. Scott L. Delp, Frank C. Anderson, Anderson S. Arnold, Peter Loan, Ayman Habib, T. John, Eran Guendelman, and Darryl G. Thelen. 2007. OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement. IEEE Transactions on Biomedical Engineering 54, 11 (2007), 1940–1950.Google ScholarCross Ref
    8. Danny Driess, Heiko Zimmermann, Simon Wolfen, Dan Suissa, Daniel Haeufle, Daniel Hennes, Marc Toussaint, and Syn Schmitt. 2018. Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 6461–6468.Google ScholarCross Ref
    9. Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible Muscle-Based Locomotion for Bipedal Creatures. ACM Transactions on Graphics 32, 6 (2013). Google ScholarDigital Library
    10. Daseong Han, Junyong Noh, Xiaogang Jin, Joseph S. Shin, and Sung Y. Shin. 2014. On-line real-time physics-based predictive motion control with balance recovery. In Computer Graphics Forum, Vol. 33. 245–254. Google ScholarDigital Library
    11. Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).Google Scholar
    12. Yifeng Jiang and C Karen Liu. 2018. Data-driven approach to simulating realistic human joint constraints. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1098–1103.Google ScholarCross Ref
    13. Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing personalized anatomical models for physics-based body animation. ACM Transactions on Graphics 35, 6, Article 213 (2016). Google ScholarDigital Library
    14. Łukasz Kidziński, Sharada Prasanna Mohanty, Carmichael Ong, Zhewei Huang, et al. 2018. Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. arXiv preprint arXiv:1804.00361 (2018).Google Scholar
    15. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Seungbae Bang, Jinwook Kim, Michael J. Black, and Sung-Hee Lee. 2017. Data-driven Physics for Human Soft Tissue Animation. ACM Transactions on Graphics 36, 4, Article 54 (2017), 12 pages. Google ScholarDigital Library
    16. Taku Komura, Yoshihisa Shinagawa, and Tosiyasu L Kunii. 2000. Creating and retargetting motion by the musculoskeletal human body model. The Visual Computer 16, 5 (2000), 254–270.Google ScholarCross Ref
    17. Jehee Lee. 2008. Representing Rotations and Orientations in Geometric Computing. IEEE Computer Graphics and Applications 28, 2 (2008), 75–83. Google ScholarDigital Library
    18. Jeongseok Lee, Michael X. Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S. Srinivasa, Mike Stilman, and C. Karen Liu. 2018a. DART: Dynamic Animation and Robotics Toolkit. The Journal of Open Source Software 3, 22 (2018), 500.Google ScholarCross Ref
    19. Sunghee Lee, Eftychios Sifakis, and Demetri Terzopoulos. 2009. Comprehensive biomechanical modeling and simulation of the upper body. ACM Transactions on Graphics 28, 4, Article 99 (2009). Google ScholarDigital Library
    20. Seunghwan Lee, Ri Yu, Jungnam Park, Mridul Aanjaneya, Eftychios Sifakis, and Jehee Lee. 2018b. Dexterous Manipulation and Control with Volumetric Muscles. ACM Transactions on Graphics 37, 4, Article 57 (2018), 13 pages. Google ScholarDigital Library
    21. Sung Hee Lee and Demetri Terzopoulos. 2006. Heads Up!: Biomechanical Modeling and Neuromuscular Control of the Neck. ACM Transactions on Graphics 25, 3 (2006), 1188–1198. Google ScholarDigital Library
    22. Yoonsang Lee, Sungeun Kim, and Jehee Lee. 2010. Data-driven biped control. ACM Transactions on Graphics 29, 4 (2010), 129. Google ScholarDigital Library
    23. Yoonsang Lee, Kyungho Lee, Soon-Sun Kwon, Jiwon Jeong, Carol O’Sullivan, Moon Seok Park, and Jehee Lee. 2015. Push-Recovery Stability of Biped Locomotion. ACM Transactions on Graphics 34, 6 (2015). Google ScholarDigital Library
    24. Yoonsang Lee, Moon Seok Park, Taesoo Kwon, and Jehee Lee. 2014. Locomotion control for many-muscle humanoids. ACM Transactions on Graphics 33, 6, Article 218 (2014). Google ScholarDigital Library
    25. Andrew Levy, Robert Platt, and Kate Saenko. 2019. Hierarchical reinforcement learning with hindsight. In Proceedings of the International Conference on Learning Representations (ICLR).Google Scholar
    26. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).Google Scholar
    27. Libin Liu and Jessica Hodgins. 2017. Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning. ACM Transactions on Graphics 36, 3 (2017). Google ScholarDigital Library
    28. Libin Liu and Jessica Hodgins. 2018. Learning Basketball Dribbling Skills Using Trajectory Optimization and Deep Reinforcement Learning. ACM Transactions on Graphics 37, 4 (2018). Google ScholarDigital Library
    29. John E. Lloyd, Ian Stavness, and Sidney Fels. 2012. ARTISYNTH: a fast interactive biomechanical modeling toolkit combining multibody and finite element simulation.Google Scholar
    30. Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017).Google Scholar
    31. Brian Mirtich and John Canny. 1995. Impulse-based simulation of rigid bodies. In Proceedings of the 1995 symposium on Interactive 3D graphics. 181-ff. Google ScholarDigital Library
    32. Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos. 2018. Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation. ACM Transactions on Graphics 37, 4, Article 56 (2018), 15 pages. Google ScholarDigital Library
    33. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.Google Scholar
    34. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018a. Deep-Mimic: Example-guided Deep Reinforcement Learning of Physics-based Character Skills. ACM Transactions on Graphics 37, 4, Article 143 (2018). Google ScholarDigital Library
    35. Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. 2017. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Transaction on Graphics 36, 4, Article 41 (2017). Google ScholarDigital Library
    36. Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018b. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Transactions on Graphics 37, 6, Article 178 (2018).Google ScholarDigital Library
    37. Xue Bin Peng and Michiel van de Panne. 2017. Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?. In Proceedings of the ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA ’17). Article 12, 13 pages. Google ScholarDigital Library
    38. Prashant Sachdeva, Shinjiro Sueda, Susanne Bradley, Mikhail Fain, and Dinesh K Pai. 2015. Biomechanical simulation and control of hands and tendinous systems. ACM Transactions on Graphics 34, 4, Article 42 (2015). Google ScholarDigital Library
    39. Shunsuke Saito, Zi-Ye Zhou, and Ladislav Kavan. 2015. Computational bodybuilding: Anatomically-based modeling of human bodies. ACM Transactions on Graphics 34, 4, Article 41 (2015). Google ScholarDigital Library
    40. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning. 1889–1897. Google ScholarDigital Library
    41. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google Scholar
    42. Eftychios Sifakis, Igor Neverov, and Ronald Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (2005). Google ScholarDigital Library
    43. Kwang Won Sok, Manmyung Kim, and Jehee Lee. 2007. Simulating biped behaviors from human motion data. ACM Transactions on Graphics 26, 3 (2007). Google ScholarDigital Library
    44. Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. 2008. Musculotendon Simulation for Hand Animation. ACM Transactions on Graphics 27, 3, Article 83 (2008), 8 pages. Google ScholarDigital Library
    45. Darryl G Thelen et al. 2003. Adjustment of muscle mechanics model parameters to simulate dynamic contractions in older adults. Transactions-American Society Of Mechanical Engineers Journal Of Biomechanical Engineering 125, 1 (2003), 70–77.Google ScholarCross Ref
    46. Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. 3540–3549. Google ScholarDigital Library
    47. Andreas Wachter and Lorenz T. Biegler. 2006. On the Implementation of an Interior-point Filter Line-search Algorithm for Large-scale Nonlinear Programming. Math. Program. 106, 1 (2006), 25–57. Google ScholarDigital Library
    48. Jack M. Wang, Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. 2012. Optimizing Locomotion Controllers Using Biologically-based Actuators and Objectives. ACM Transaction on Graphics 31, 4, Article 25 (2012). Google ScholarDigital Library
    49. Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. 2017. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems. 5320–5329. Google ScholarDigital Library
    50. Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to Train Your Dragon: Example-guided Control of Flapping Flight. ACM Transactions on Graphics 36, 6, Article 198 (2017), 13 pages. Google ScholarDigital Library
    51. Jungdam Won, Jungnam Park, and Jehee Lee. 2018. Aerobatics Control of Flying Creatures via Self-regulated Learning. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia ’18). Article 181. Google ScholarDigital Library
    52. KangKang Yin, Kevin Loken, and Michiel van de Panne. 2007. SIMBICON: Simple Biped Locomotion Control. ACM Transaction on Graphics 26, 3, Article 105 (2007). Google ScholarDigital Library
    53. Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-energy Locomotion. ACM Transactions on Graphics 37, 4, Article 144 (2018), 12 pages. Google ScholarDigital Library
    54. Felix E Zajac. 1989. Muscle and tendon Properties models scaling and application to biomechanics and motor. Critical reviews in biomedical engineering 17, 4 (1989), 359–411.Google Scholar
    55. Felix E Zajac, Richard R Neptune, and Steven A Kautz. 2003. Biomechanics and muscle coordination of human walking: part II: lessons from dynamical simulations and clinical implications. Gait & posture 17, 1 (2003), 1–17.Google Scholar

ACM Digital Library Publication:

Overview Page: