“Learning a family of motor skills from a single motion clip” by Lee, Lee, Lee and Lee

We present a new algorithm that learns a parameterized family of motor skills from a single motion clip. The motor skills are represented by a deep policy network, which produces a stream of motions in physics simulation in response to user input and environment interaction by navigating continuous action space. Three novel technical components play an important role in the success of our algorithm. First, it explicitly constructs motion parameterization that maps action parameters to their corresponding motions. Simultaneous learning of motion parameterization and motor skills significantly improves the performance and visual quality of learned motor skills. Second, continuous-time reinforcement learning is adopted to explore temporal variations as well as spatial variations in motion parameterization. Lastly, we present a new automatic curriculum generation method that explores continuous action space more efficiently. We demonstrate the flexibility and versatility of our algorithm with highly dynamic motor skills that can be parameterized by task goals, body proportions, physical measurements, and environmental conditions.

1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265–283.Google ScholarDigital Library
2. Yeuhi Abe, C Karen Liu, and Zoran Popović. 2004. Momentum-based parameterization of dynamic character motion. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation. 173–182.Google ScholarDigital Library
3. Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020a. Skeleton-Aware Networks for Deep Motion Retargeting. ACM Transactions on Graphics 39, 4 (2020), 62.Google ScholarDigital Library
4. Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020b. Unpaired Motion Style Transfer from Video to Animation. ACM Transactions on Graphics 39, 4 (2020), 64.Google ScholarDigital Library
5. Adobe Systems Incs. 2018. Mixamo. https://www.mixamo.comGoogle Scholar
6. Shailen Agrawal, Shuo Shen, and Michiel van de Panne. 2014. Diverse motions and character shapes for simulated skills. IEEE transactions on visualization and computer graphics 20, 10 (2014).Google ScholarCross Ref
7. Shailen Agrawal and Michiel van de Panne. 2016. Task-based Locomotion. ACM Transactions on Graphics 35, 4 (2016).Google ScholarDigital Library
8. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight Experience Replay. In Advances in Neural Information Processing Systems. 5048–5058.Google ScholarDigital Library
9. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. 41–48.Google ScholarDigital Library
10. Kevin Bergamin, Simon Claver, Daniel Holden, and James Richard Forbes. 2019. DReCon: Data-Driven Responsive Control of Physics-Based Characters. ACM Transactions on Graphics 38, 6, Article 1 (2019).Google ScholarDigital Library
11. Gaurav Bharaj, Stelian Coros, Bernhard Thomaszewski, James Tompkin, Bernd Bickel, and Hanspeter Pfister. 2015. Computational Design of Walking Automata. In Proceedings of the 14th ACM SIGGRAPH / Eurographics Symposium on Computer Animation. 93–100.Google ScholarDigital Library
12. Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. 2018. Learning to Dress: Synthesizing Human Dressing Motion via Deep Reinforcement Learning. ACM Transactions on Graphics 37, 6, Article 179 (2018).Google ScholarDigital Library
13. CMU. 2002. CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.eduGoogle Scholar
14. Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 33–42.Google ScholarDigital Library
15. Keith Grochow, Steven L. Martin, Aaron Hertzmann, and Zoran Popović. 2004. Style-Based Inverse Kinematics. ACM Transactions on Graphics 23, 3 (2004), 522–531.Google ScholarDigital Library
16. Perttu Hämäläinen, Perttu, Joose Rajamäki, and C. Karen Liu. 2015. Online Control of Simulated Humanoids Using Particle Belief Propagation. ACM Transactions on Graphics 34, 4, Article 81 (2015).Google ScholarDigital Library
17. Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. 2008. Real-time motion retargeting to highly varied user-created morphologies. ACM Transactions on Graphics 27, 3 (2008), 1–11.Google ScholarDigital Library
18. Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).Google Scholar
19. Edmond S. L. Ho, Taku Komura, and Chiew-Lan Tai. 2010. Spatial Relationship Preserving Character Motion Adaptation. ACM Transactions on Graphics 29, 4, Article 33 (2010).Google ScholarDigital Library
20. Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020. Learned Motion Matching. ACM Transactions on Graphics 39, 4, Article 53 (2020).Google ScholarDigital Library
21. Seokpyo Hong, Daseong Han, Kyungmin Cho, Joseph S. Shin, and Junyong Noh. 2019. Physics-Based Full-Body Soccer Motion Control for Dribbling and Shooting. ACM Transactions on Graphics 38, 4, Article 74 (2019).Google ScholarDigital Library
22. Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C. Karen Liu. 2019. Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation. ACM Transactions on Graphics 38, 4, Article 72 (2019).Google ScholarDigital Library
23. Manmyung Kim, Kyunglyul Hyun, Jongmin Kim, and Jehee Lee. 2009. Synchronized multi-character motion editing. ACM transactions on graphics 28, 3, Article 79 (2009).Google Scholar
24. Jeongseok Lee, Michael X Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S Srinivasa, Mike Stilman, and C Karen Liu. 2018a. DART: Dynamic animation and robotics toolkit. The Journal of Open Source Software 3, 22 (2018).Google ScholarCross Ref
25. Jehee Lee and Sung Yong Shin. 1999. A hierarchical approach to interactive motion editing for human-like figures. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 39–48.Google ScholarDigital Library
26. Kyungho Lee, Seyoung Lee, and Jehee Lee. 2018b. Interactive character animation by learning multi-objective control. ACM Transactions on Graphics 37, 6, Article 180 (2018).Google ScholarDigital Library
27. Kang Hoon Lee, Myung Geol Choi, and Jehee Lee. 2006. Motion Patches: Building blocks for virtual environments annotated with motion data. ACM Transactions on Graphics 25, 3 (2006), 898–906.Google ScholarDigital Library
28. Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019. Scalable Muscle-actuated Human Simulation and Control. ACM Transactions on Graphics 38, 4, Article 73 (2019).Google ScholarDigital Library
29. Sergey Levine and Vladlen Koltun. 2013. Guided policy search. In International Conference on Machine Learning. 1–9.Google ScholarDigital Library
30. Sergey Levine, Jack M. Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012. Continuous Character Control with Low-Dimensional Embeddings. ACM Transactions on Graphics 31, 4, Article 28 (2012).Google ScholarDigital Library
31. C. Karen Liu, Aaron Hertzmann, and Zoran Popović. 2005. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization. ACM Transactions on Graphics 24, 3 (2005), 1071–1081.Google ScholarDigital Library
32. Libin Liu, KangKang Yin, and Baining Guo. 2015. Improving Sampling-Based Motion Control. Computer Graphics Forum 34, 2 (2015), 415–423.Google ScholarDigital Library
33. Libin Liu, KangKang Yin, Michiel van de Panne, and Baining Guo. 2012. Terrain Runner: Control, Parameterization, Composition, and Planning for Highly Dynamic Motions. ACM Transactions on Graphics 31, 6, Article 154 (2012).Google ScholarDigital Library
34. Libin Liu, KangKang Yin, Michiel van de Panne, Tianjia Shao, and Weiwei Xu. 2010. Sampling-Based Contact-Rich Motion Control. ACM Transactions on Graphics 29, 4, Article 128 (2010).Google ScholarDigital Library
35. Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. 2020. CARL: Controllable Agent with Reinforcement Learning for Quadruped Locomotion. ACM Transactions on Graphics 39, 4, Article 38 (2020).Google ScholarDigital Library
36. Anna Majkowska and Petros Faloutsos. 2007. Flipping with physics: motion editing for acrobatics. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation. 35–44.Google Scholar
37. Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. 2019. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems (2019).Google ScholarCross Ref
38. James McCann, Nancy S Pollard, and Siddartha S Srinavisa. 2006. Physics-Based Motion Retiming. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 205–214.Google ScholarDigital Library
39. Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2020. Catch amp; Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks. ACM Transactions on Graphics 39, 4, Article 39 (2020), 14 pages.Google ScholarDigital Library
40. Sehee Min, Jungdam Won, Seunghwan Lee, Jungnam Park, and Jehee Lee. 2019. SoftCon: Simulation and Control of Soft-Bodied Animals with Biomimetic Actuators. ACM Transactions on Graphics 38, 6, Article 208 (2019).Google ScholarDigital Library
41. Igor Mordatch and Emo Todorov. 2014. Combining the benefits of function approximation and trajectory optimization.. In Robotics: Science and Systems.Google Scholar
42. Igor Mordatch, Emanuel Todorov, and Zoran Popović. 2012. Discovery of Complex Behaviors through Contact-Invariant Optimization. ACM Transactions on Graphics 31, 4, Article 43 (2012).Google ScholarDigital Library
43. Tomohiko Mukai and Shigeru Kuriyama. 2005. Geostatistical Motion Interpolation. ACM Transactions on Graphics 24, 3 (2005), 1062–1070.Google ScholarDigital Library
44. Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and J. Lee. 2019. Learning predict-and-simulate policies from unorganized human motion data. ACM Transactions on Graphics 38, 6, Article 205 (2019).Google ScholarDigital Library
45. Sang Il Park, Hyun Joon Shin, and Sung Yong Shin. 2002. On-Line Locomotion Generation Based on Motion Blending. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 105–111.Google ScholarDigital Library
46. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics 37, 4, Article 143 (2018).Google ScholarDigital Library
47. Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. 2017. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Transactions on Graphics 36, 4, Article 41 (2017).Google ScholarDigital Library
48. Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. In Advances in Neural Information Processing Systems. 3686–3697.Google Scholar
49. Charles Rose, Michael F Cohen, and Bobby Bodenheimer. 1998. Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Applications 18, 5 (1998), 32–40.Google ScholarDigital Library
50. Kwang Won Sok, Katsu Yamane, Jehee Lee, and Jessica Hodgins. 2010. Editing dynamic human motions via momentum and force. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer animation. 11–20.Google ScholarDigital Library
51. Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. 2020. Local Motion Phases for Learning Multi-Contact Character Movements. ACM Transactions on Graphics 39, 4, Article 54 (2020).Google ScholarDigital Library
52. Jack M Wang, David J Fleet, and Aaron Hertzmann. 2007. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence 30, 2 (2007), 283–298.Google Scholar
53. Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2020. A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters. ACM Transactions on Graphics 39, 4, Article 33 (2020).Google ScholarDigital Library
54. Jungdam Won and Jehee Lee. 2019. Learning body shape variation in physics-based characters. ACM Transactions on Graphics 38, 6, Article 207 (2019).Google ScholarDigital Library
55. Jungdam Won, Kyungho Lee, Carol O’Sullivan, Jessica K Hodgins, and Jehee Lee. 2014. Generating and ranking diverse multi-character interactions. ACM Transactions on Graphics 33, 6, Article 219 (2014).Google ScholarDigital Library
56. Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to Train Your Dragon: Example-Guided Control of Flapping Flight. ACM Transactions on Graphics 36, 6, Article 198 (2017).Google ScholarDigital Library
57. Jungdam Won, Jungnam Park, and Jehee Lee. 2018. Aerobatics control of flying creatures via self-regulated learning. ACM Transactions on Graphics 37, 6, Article 181 (2018).Google ScholarDigital Library
58. Zhaoming Xie, Hung Yu Ling, Nam Hee Kim, and Michiel van de Panne. 2020. ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation.Google Scholar
59. Yuting Ye and C. Karen Liu. 2012. Synthesis of Detailed Hand Manipulations Using Contact Sampling. ACM Transactions on Graphics 31, 4, Article 41 (2012).Google ScholarDigital Library
60. Zhiqi Yin and KangKang Yin. 2020. Linear Time Stable PD Controllers for Physics-Based Character Animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 191–200.Google ScholarDigital Library
61. Ri Yu, Hwangpil Park, and Jehee Lee. 2019. Figure Skating Simulation from Video. Computer Graphics Forum 38, 7 (2019), 225–234.Google ScholarCross Ref
62. Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Transactions on Graphics 37, 4, Article 144 (2018).Google ScholarDigital Library
63. M Ersin Yumer and Niloy J Mitra. 2016. Spectral Style Transfer for Human Motion between Independent Actions. ACM Transactions on Graphics 35, 4 (2016).Google Scholar