“Mode-adaptive neural networks for quadruped motion control” by Zhang, Starke, Komura and Saito

  • ©He Zhang, Sebastian Starke, Taku Komura, and Jun Saito



Entry Number: 145


    Mode-adaptive neural networks for quadruped motion control


Session Title: Animation Control



    Quadruped motion includes a wide variation of gaits such as walk, pace, trot and canter, and actions such as jumping, sitting, turning and idling. Applying existing data-driven character control frameworks to such data requires a significant amount of data preprocessing such as motion labeling and alignment. In this paper, we propose a novel neural network architecture called Mode-Adaptive Neural Networks for controlling quadruped characters. The system is composed of the motion prediction network and the gating network. At each frame, the motion prediction network computes the character state in the current frame given the state in the previous frame and the user-provided control signals. The gating network dynamically updates the weights of the motion prediction network by selecting and blending what we call the expert weights, each of which specializes in a particular movement. Due to the increased flexibility, the system can learn consistent expert weights across a wide range of non-periodic/periodic actions, from unstructured motion capture data, in an end-to-end fashion. In addition, the users are released from performing complex labeling of phases in different gaits. We show that this architecture is suitable for encoding the multi-modality of quadruped locomotion and synthesizing responsive motion in real-time.


    1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI’16). USENIX Association, Berkeley, CA, USA, 265–283. http://dl.acm.org/citation.cfm?id=3026877.3026899 Google ScholarDigital Library
    2. Mazen Al Borno, Martin De Lasa, and Aaron Hertzmann. 2013. Trajectory optimization for full-body movements with complex contacts. IEEE Trans on Vis and Comp Graph 19, 8 (2013), 1405–1414. Google ScholarDigital Library
    3. Okan Arikan and David A Forsyth. 2002. Interactive motion generation from examples. ACM Trans on Graph 21, 3 (2002), 483–490. Google ScholarDigital Library
    4. Emad Barsoum, John Kender, and Zicheng Liu. 2017. HP-GAN: Probabilistic 3D human motion prediction via GAN. CoRR abs/1711.09561 (2017). arXiv:1711.09561 http://arxiv.org/abs/1711.09561Google Scholar
    5. Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. 2015. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297 (2015). https://arxiv.org/abs/1511.06297Google Scholar
    6. Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. 2016. Learning feed-forward one-shot learners. In Proc. NIPS. 523–531. Google ScholarDigital Library
    7. Jinxiang Chai and Jessica K Hodgins. 2005. Performance Animation from Low-dimensional Control Signals. ACM Trans on Graph 24, 3 (2005), 686–696. Google ScholarDigital Library
    8. Xiaobin Chang, Timothy M Hospedales, and Tao Xiang. 2018. Multi-level factorisation net for person re-identification. arXiv preprint arXiv:1803.09132 (2018). https://arxiv.org/abs/1803.09132v2Google Scholar
    9. Simon Clavet. 2016. Motion Matching and The Road to Next-Gen Animation. In Proc. of GDC 2016.Google Scholar
    10. Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). CoRR abs/1511.07289 (2015). http://arxiv.org/abs/1511.07289Google Scholar
    11. Stelian Coros, Andrej Karpathy, Ben Jones, Lionel Reveret, and Michiel Van De Panne. 2011. Locomotion skills for simulated quadrupeds. ACM Trans on Graph 30, 4 (2011), 59. Google ScholarDigital Library
    12. David Eigen, Marc’ Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314 (2013).Google Scholar
    13. Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. 2015. Recurrent network models for human dynamics. In Proc. ICCV. 4346–4354. Google ScholarDigital Library
    14. Keith Grochow, Steven L Martin, Aaron Hertzmann, and Zoran Popović. 2004. Style-based inverse kinematics. ACM Trans on Graph 23, 3 (2004), 522–531. Google ScholarDigital Library
    15. Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. 2008. Real-time motion retargeting to highly varied user-created morphologies. ACM Trans on Graph 27, 3 (2008), 27. Google ScholarDigital Library
    16. Daniel Holden, Taku Komura, and Jun Saito. 2017. Phase-functioned neural networks for character control. ACM Trans on Graph 36, 4 (2017), 42. Google ScholarDigital Library
    17. Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Trans on Graph 35, 4 (2016). Google ScholarDigital Library
    18. Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning Motion Manifolds with Convolutional Autoencoders. In SIGGRAPH Asia 2015 Technical Briefs. Article 18, 4 pages. Google ScholarDigital Library
    19. Eugene Hsu, Kari Pulli, and Jovan Popovic. 2005. Style Translation for Human Motion. ACM Trans on Graph 24, 3 (2005), 1082–1089. Google ScholarDigital Library
    20. Ting-Chieh Huang, Yi-Jheng Huang, and Wen-Chieh Lin. 2013. Real-time horse gait synthesis. Computer Animation and Virtual Worlds 24, 2 (2013), 87–95.Google ScholarCross Ref
    21. Leslie Ikemoto, Okan Arikan, and David Forsyth. 2009. Generalizing motion edits with gaussian processes. ACM Trans on Graph 28, 1 (2009), 1. Google ScholarDigital Library
    22. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural Computation 3, 1 (1991), 79–87.Google ScholarCross Ref
    23. Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the EM algorithm. Neural Computation 6, 2 (1994), 181–214. Google ScholarDigital Library
    24. Ahmad Abdul Karim, Thibaut Gaudin, Alexandre Meyer, Axel Buendia, and Saida Bouakaz. 2013. Procedural locomotion of multilegged characters in dynamic environments. Computer Animation and Virtual Worlds 24, 1 (2013), 3–15.Google ScholarCross Ref
    25. Lucas Kovar and Michael Gleicher. 2004. Automated Extraction and Parameterization of Motions in Large Data Sets. ACM Trans on Graph 23, 3 (2004), 559–568. Google ScholarDigital Library
    26. Lucas Kovar, Michael Gleicher, and Frédéric Pighin. 2002. Motion graphs. ACM Trans on Graph 21, 3 (2002), 473–482. Google ScholarDigital Library
    27. Paul G Kry, Lionel Revéret, François Faure, and M-P Cani. 2009. Modal locomotion: Animating virtual characters with natural vibrations. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 289–298.Google Scholar
    28. Manfred Lau and James J Kuffher. 2005. Behavior planning for character animation. In Proc. SCA. 271–280. Google ScholarDigital Library
    29. Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard. 2002. Interactive control of avatars animated with human motion data. ACM Trans on Graph 21, 3 (2002), 491–500. Google ScholarDigital Library
    30. Yongjoon Lee, Kevin Wampler, Gilbert Bernstein, Jovan Popović, and Zoran Popović. 2010. Motion fields for interactive character locomotion. ACM Trans on Graph 29, 6 (2010), 138. Google ScholarDigital Library
    31. Sergey Levine and Jovan Popović. 2012. Physically Plausible Simulation for Character Animation. In Proc. SCA. 221–230. http://dl.acm.org/citation.cfm?id=2422356.2422388 Google ScholarDigital Library
    32. Sergey Levine, Jack M Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012. Continuous character control with low-dimensional embeddings. ACM Trans on Graph 31, 4 (2012), 28. Google ScholarDigital Library
    33. Zimo Li, Yi Zhou, Shuangjiu Xiao, Chong He, and Hao Li. 2017. Auto-Conditioned LSTM Network for Extended Complex Human Motion Synthesis. arXiv preprint arXiv:1707.05363 (2017). https://arxiv.org/abs/1707.05363Google Scholar
    34. C. Karen Liu, Aaron Hertzmann, and Zoran Popović;. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Trans on Graph 24, 3 (2005), 1071–1081. Google ScholarDigital Library
    35. C. Karen Liu and Zoran Popović. 2002. Synthesis of complex dynamic character motion from simple animations. ACM Trans on Graph 21, 3 (2002), 408–416. Google ScholarDigital Library
    36. Libin Liu and Jessica Hodgins. 2017. Learning to schedule control fragments for physics-based characters using deep q-learning. ACM Trans on Graph 36, 3 (2017), 29. Google ScholarDigital Library
    37. Ilya Loshchilov and Frank Hutter. 2017. Fixing Weight Decay Regularization in Adam. CoRR abs/1711.05101 (2017). arXiv:1711.05101 http://arxiv.org/abs/1711.05101Google Scholar
    38. Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017a. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017). https://arxiv.org/abs/1707.02201Google Scholar
    39. Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017b. Learning human behaviors from motion capture by adversarial imitation. CoRR abs/1707.02201 (2017). arXiv:1707.02201 http://arxiv.org/abs/1707.02201Google Scholar
    40. Jianyuan Min and Jinxiang Chai. 2012. Motion graphs++: a compact generative model for semantic motion analysis and synthesis. ACM Trans on Graph 31, 6 (2012), 153. Google ScholarDigital Library
    41. Tomohiko Mukai and Shigeru Kuriyama. 2005. Geostatistical motion interpolation. ACM Trans on Graph 24, 3 (2005), 1062–1070. Google ScholarDigital Library
    42. Xue Bin Peng, Glen Berseth, and Michiel van de Panne. 2015. Dynamic Terrain Traversal Skills Using Reinforcement Learning. ACM Trans on Graph 34, 4, Article 80 (2015), 80:1–80:11 pages. Google ScholarDigital Library
    43. Xue Bin Peng, Glen Berseth, and Michiel van de Panne. 2016. Terrain-Adaptive Locomotion Skills Using Deep Reinforcement Learning. ACM Trans on Graph 35, 4 (2016). Google ScholarDigital Library
    44. Marc H. Raibert and Jessica K. Hodgins. 1991. Animation of dynamic legged locomotion. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1991, Providence, RI, USA, April 27–30, 1991. 349–358. Google ScholarDigital Library
    45. Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. arXiv preprint arXiv:1705.08045 (2017). https://arxiv.org/abs/1705.08045Google Scholar
    46. Charles Rose, Michael F. Cohen, and Bobby Bodenheimer. 1998. Verbs and Adverbs: Multidimensional Motion Interpolation. IEEE Computer Graphics and Applications 18, 5 (1998), 32–40. Google ScholarDigital Library
    47. Alla Safonova and Jessica K Hodgins. 2007. Construction and optimal search of interpolated motion graphs. ACM Trans on Graph 26, 3 (2007), 106. Google ScholarDigital Library
    48. Alla Safonova, Jessica K Hodgins, and Nancy S Pollard. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. ACM Trans on Graph 23, 3 (2004), 514–521. Google ScholarDigital Library
    49. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. CoRR abs/1701.06538 (2017). arXiv:1701.06538 http://arxiv.org/abs/1701.06538Google Scholar
    50. Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helfen, Meinard Müller, Hans-Peter Seidel, and Bernd Eberhardt. 2011. Motion reconstruction using sparse accelerometer data. ACM Trans on Graph 30, 3 (2011), 18. Google ScholarDigital Library
    51. Graham W Taylor and Geoffrey E Hinton. 2009. Factored conditional restricted Boltzmann machines for modeling motion style. In Proc. ICML. ACM, 1025–1032. Google ScholarDigital Library
    52. Graham W Taylor, Geoffrey E Hinton, and Sam T Roweis. 2011. Two distributed-state models for generating high-dimensional time series. The Journal of Machine Learning Research 12 (2011), 1025–1068. Google ScholarDigital Library
    53. Michiel van de Panne. 1996. Parameterized gait synthesis. IEEE Computer Graphics and Applications 16, 2 (1996), 40–49. Google ScholarDigital Library
    54. Kevin Wampler and Zoran Popović. 2009. Optimal gait and form for animal locomotion. ACM Trans on Graph 28, 3 (2009), 60. Google ScholarDigital Library
    55. Kevin Wampler, Zoran Popović, and Jovan Popović. 2014. Generalizing locomotion style to new animals with inverse optimal regression. ACM Trans on Graph 33, 4 (2014), 49. Google ScholarDigital Library
    56. J.M. Wang, D.J. Fleet, and A. Hertzmann. 2008. Gaussian Process Dynamical Models for Human Motion. Pattern Analysis and Machine Intelligence, IEEE Trans. on 30, 2 (Feb 2008), 283–298. D0I Google ScholarDigital Library
    57. Shihong Xia, Congyi Wang, Jinxiang Chai, and Jessica Hodgins. 2015. Realtime Style Transfer for Unlabeled Heterogeneous Human Motion. ACM Trans on Graph 34, 4 (2015), 119:1–119:10. Google ScholarDigital Library
    58. Yuting Ye and C Karen Liu. 2012. Synthesis of detailed hand manipulations using contact sampling. ACM Trans on Graph 31, 4 (2012), 41. Google ScholarDigital Library
    59. KangKang Yin, Kevin Loken, and Michiel Van de Panne. 2007. Simbicon: Simple biped locomotion control. ACM Trans on Graph 26, 3 (2007), 105. Google ScholarDigital Library
    60. Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. 2012. Twenty years of mixture of experts. IEEE Trans on Neural Networks and Learning Systems 23, 8 (2012), 1177–1193.Google ScholarCross Ref

ACM Digital Library Publication: