“DOC: Differentiable Optimal Control for Retargeting Motions onto Legged Robots” by Grandia, Knoop, Schumacher, Hutter and Bächer
Conference:
Type:
Title:
- DOC: Differentiable Optimal Control for Retargeting Motions onto Legged Robots
Session/Category Title: Character Animation
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Legged robots are designed to perform highly dynamic motions. However, it remains challenging for users to retarget expressive motions onto these complex systems. In this paper, we present a Differentiable Optimal Control (DOC) framework that facilitates the transfer of rich motions from either animals or animations onto these robots. Interfacing with either motion capture or animation data, we formulate retargeting objectives whose parameters make them agnostic to differences in proportions and numbers of degrees of freedom between input and robot. Optimizing these parameters over the manifold spanned by optimal state and control trajectories, we minimize the retargeting error. We demonstrate the utility and efficacy of our modeling by applying DOC to a Model-Predictive Control (MPC) formulation, showing retargeting results for a family of robots of varying proportions and mass distribution. With a hardware deployment, we further show that the retargeted motions are physically feasible, while MPC ensures that the robots retain their capability to react to unexpected disturbances.
References:
1. Evan Ackerman. 2021. How Boston Dynamics Taught Its Robots to Dance. IEEE Spectrum (Jan. 7 2021). https://spectrum.ieee.org/how-boston-dynamics-taught-its-robots-to-dance
2. Brandon Amos, Ivan Jimenez, Jacob Sacks, Byron Boots, and J Zico Kolter. 2018. Differentiable MPC for end-to-end planning and control. Advances in neural information processing systems 31 (2018).
3. Yuichiro Aoyama, George Boutselis, Akash Patel, and Evangelos A. Theodorou. 2021. Constrained Differential Dynamic Programming Revisited. In 2021 IEEE International Conference on Robotics and Automation (ICRA).
4. Taylor Apgar, Patrick Clary, Kevin Green, Alan Fern, and Jonathan Hurst. 2018. Fast Online Trajectory Optimization for the Bipedal Robot Cassie. In Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation.
5. Ko Ayusawa and Eiichi Yoshida. 2017. Motion retargeting for humanoid robots based on simultaneous morphing parameter identification and motion optimization. IEEE Transactions on Robotics 33, 6 (2017), 1343–1357.
6. Dimitri Bertsekas. 2012. Dynamic programming and optimal control: Volume I. Vol. 1. Athena scientific.
7. Thomas Bi, Péter Fankhauser, Dario Bellicoso, and Marco Hutter. 2018. Real-Time Dance Generation to Music for a Legged Robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
8. Gerardo Bledt, Matthew J. Powell, Benjamin Katz, Jared Di Carlo, Patrick M. Wensing, and Sangbae Kim. 2018. MIT Cheetah 3: Design and Control of a Robust, Dynamic Quadruped Robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
9. Steven Bohez, Saran Tunyasuvunakool, Philemon Brakel, Fereshteh Sadeghi, Leonard Hasenclever, Yuval Tassa, Emilio Parisotto, Jan Humplik, Tuomas Haarnoja, Roland Hafner, et al. 2022. Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors. arXiv preprint arXiv:2203.17138 (2022).
10. SR McReynolds AE Bryson Jr. 1965. A successive sweep method for solving optimal programming problems. In Proc 1965 Joint Automatic Control Conference.
11. Sungjoon Choi, Matt Pan, and Joohyung Kim. 2020. Nonparametric motion retargeting for humanoid robots on shared latent space. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals.
12. Sungjoon Choi, Min Jae Song, Hyemin Ahn, and Joohyung Kim. 2021. Self-Supervised Motion Retargeting with Safety Guarantee. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 8097–8103.
13. Stelian Coros, Andrej Karpathy, Ben Jones, Lionel Reveret, and Michiel Van De Panne. 2011. Locomotion skills for simulated quadrupeds. ACM Trans. Graph. (TOG) 30, 4 (2011).
14. Marco Da Silva, Yeuhi Abe, and Jovan Popović. 2008. Simulation of human motion data using short-horizon model-predictive control. In Computer Graphics Forum, Vol. 27. Wiley Online Library.
15. Behzad Dariush, Michael Gienger, Arjun Arumbakkam, Christian Goerick, Youding Zhu, and Kikuo Fujimura. 2008a. Online and markerless motion retargeting with kinematic constraints. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.
16. Behzad Dariush, Michael Gienger, Bing Jian, Christian Goerick, and Kikuo Fujimura. 2008b. Whole body humanoid control from human motion descriptors. In 2008 IEEE International Conference on Robotics and Automation.
17. Kourosh Darvish, Yeshasvi Tirupachuri, Giulio Romualdi, Lorenzo Rapetti, Diego Ferigo, Francisco Javier Andrade Chavez, and Daniele Pucci. 2019. Whole-Body Geometric Retargeting for Humanoid Robots. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids).
18. J.F.O. De O. Pantoja and D.Q. Mayne. 1989. A sequential quadratic programming algorithm for discrete optimal control problems with control inequality constraints. In Proceedings of the 28th IEEE Conference on Decision and Control,.
19. Farbod Farshidian, Edo Jelavic, Asutosh Satapathy, Markus Giftthaler, and Jonas Buchli. 2017a. Real-time motion planning of legged robots: A model predictive control approach. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).
20. Farbod Farshidian, Michael Neunert, Alexander W Winkler, Gonzalo Rey, and Jonas Buchli. 2017b. An efficient optimal planning and control framework for quadrupedal locomotion. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE.
21. Christian Gehring, Stelian Coros, Marco Hutter, Carmine Dario Bellicoso, Huub Heijnen, Remo Diethelm, Michael Bloesch, Peter Fankhauser, Jemin Hwangbo, Mark Hoepflinger, and Roland Siegwart. 2016. Practice Makes Perfect: An Optimization-Based Approach to Controlling Agile Motions for a Quadruped Robot. IEEE Robotics & Automation Magazine 23, 1 (2016).
22. Michael J. Gielniak, C. Karen Liu, and Andrea L. Thomaz. 2010. Secondary action in robot motion. In 19th International Symposium in Robot and Human Interactive Communication. IEEE, Viareggio, Italy.
23. Michael J. Gielniak and Andrea L. Thomaz. 2011. Spatiotemporal correspondence as a metric for human-like robot motion. In 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).
24. Markus Giftthaler and Jonas Buchli. 2017. A projection approach to equality constrained iterative linear quadratic optimal control. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).
25. Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques – SIGGRAPH ’98. ACM Press.
26. Ruben Grandia, Farbod Farshidian, René Ranftl, and Marco Hutter. 2019. Feedback MPC for torque-controlled legged robots. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.
27. Ruben Grandia, Fabian Jenelten, Shaohui Yang, Farbod Farshidian, and Marco Hutter. 2022. Perceptive Locomotion through Nonlinear Model Predictive Control. https://arxiv.org/pdf/2208.08373.pdf (2022).
28. C.R. Hargraves and S.W. Paris. 1987. Direct trajectory optimization using nonlinear programming and collocation. Journal of Guidance, Control, and Dynamics 10, 4 (1987).
29. Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. 2018. Learning an Embedding Space for Transferable Robot Skills. https://openreview.net/forum?id=rk07ZXZRb
30. Chris Hecker, Bernd Raabe, Ryan W. Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. 2008. Real-time motion retargeting to highly varied user-created morphologies. ACM Trans. Graph. (TOG) 27, 3 (2008).
31. Shayan Hoshyari, Hongyi Xu, Espen Knoop, Stelian Coros, and Moritz Bächer. 2019. Vibration-minimizing motion retargeting for robotic characters. ACM Trans. Graph. (TOG) 38, 4 (2019).
32. Taylor A. Howell, Brian E. Jackson, and Zachary Manchester. 2019. ALTRO: A Fast Solver for Constrained Trajectory Optimization. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
33. David H. Jacobson and David Q. Mayne. 1970. Differential Dynamic Programming. American Elsevier Publishing Company.
34. Dongho Kang, Flavio De Vincenti, Naomi C. Adami, and Stelian Coros. 2022. Animal Motions on Legged Robots Using Nonlinear Model Predictive Control. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.
35. Dongho Kang, Simon Zimmermann, and Stelian Coros. 2021. Animal Gaits on Quadrupedal Robots Using Motion Matching and Model-Based Control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE.
36. Benjamin Katz, Jared Di Carlo, and Sangbae Kim. 2019. Mini cheetah: A platform for pushing the limits of dynamic quadruped control. In 2019 International Conference on Robotics and Automation (ICRA). IEEE.
37. Sunwoo Kim, Maks Sorokin, Jehee Lee, and Sehoon Ha. 2022. HumanConQuad: Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning. In SIGGRAPH Asia 2022 Emerging Technologies. ACM.
38. Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. 2021. Learning a family of motor skills from a single motion clip. ACM Transactions on Graphics 40, 4 (2021).
39. He Li and Patrick M. Wensing. 2020. Hybrid Systems Differential Dynamic Programming for Whole-Body Motion Planning of Legged Robots. IEEE Robotics and Automation Letters 5, 4 (2020).
40. Tianyu Li, Jungdam Won, Sehoon Ha, and Akshara Rai. 2021. Model-based Motion Imitation for Agile, Diverse and Generalizable Quadupedal Locomotion. arXiv preprint arXiv:2109.13362 (2021).
41. Weiwei Li and Emanuel Todorov. 2004. Iterative linear quadratic regulator design for nonlinear biological movement systems.. In ICINCO (1).
42. T. C. Lin and J. S. Arora. 1991. Differential dynamic programming technique for constrained optimal control. Computational Mechanics 9, 1 (1991).
43. David Mayne. 1966. A Second-order Gradient Method for Determining Optimal Trajectories of Non-linear Discrete-time Systems. Internat. J. Control 3, 1 (1966).
44. S. K. Mitter. 1966. Successive approximation methods for the solution of optimal control problems. Automatica 3, 3 (1966).
45. Daniel M. Murray and Sidney J. Yakowitz. 1979. Constrained differential dynamic programming and its application to multireservoir control. Water Resources Research 15, 5 (1979).
46. Shin’ichiro Nakaoka, Atsushi Nakazawa, Fumio Kanehiro, Kenji Kaneko, Mitsuharu Morisawa, Hirohisa Hirukawa, and Katsushi Ikeuchi. 2007. Learning from Observation Paradigm: Leg Task Models for Enabling a Biped Humanoid Robot to Imitate Human Dances. The International Journal of Robotics Research 26, 8 (2007).
47. Michael Neunert, Cédric de Crousaz, Fadri Furrer, Mina Kamel, Farbod Farshidian, Roland Siegwart, and Jonas Buchli. 2016. Fast nonlinear Model Predictive Control for unified trajectory optimization and tracking. In 2016 IEEE International Conference on Robotics and Automation (ICRA).
48. Michael Neunert, Farbod Farshidian, Alexander W. Winkler, and Jonas Buchli. 2017. Trajectory Optimization Through Contacts and Automatic Gait Discovery for Quadrupeds. IEEE Robotics and Automation Letters 2, 3 (2017).
49. Koichi Nishiwaki, Satoshi Kagami, Yasuo Kuniyoshi, Masayuki Inaba, and Hirochika Inoue. 2002. Online generation of humanoid walking motion based on a fast generation method of motion pattern that follows desired zmp. In IEEE/RSJ international conference on intelligent robots and systems, Vol. 3. IEEE, 2684–2689.
50. Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization. Springer, New York, NY, USA.
51. Alex Oshin, Matthew D. Houghton, Michael J. Acheson, Irene M. Gregory, and Evangelos A. Theodorou. 2022. Parameterized Differential Dynamic Programming.
52. Christian Ott, Dongheui Lee, and Yoshihiko Nakamura. 2008. Motion capture based human motion recognition and imitation by direct marker control. In Humanoids 2008 – 8th IEEE-RAS International Conference on Humanoid Robots.
53. Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. 2020. Learning agile robotic locomotion skills by imitating animals. Robotics: Science and Systems (2020).
54. Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. 2022. ASE: large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. on Graph. (TOG) 41, 4 (2022).
55. Brian Plancher, Zachary Manchester, and Scott Kuindersma. 2017. Constrained un-scented dynamic programming. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
56. N.S. Pollard, J.K. Hodgins, M.J. Riley, and C.G. Atkeson. 2002. Adapting human motion for the control of a humanoid robot. In Proceedings 2002 IEEE International Conference on Robotics and Automation, Vol. 2. IEEE, Washington, DC, USA.
57. Michael Posa, Scott Kuindersma, and Russ Tedrake. 2016. Optimization and stabilization of trajectories for constrained dynamical systems. In 2016 IEEE International Conference on Robotics and Automation (ICRA).
58. Marc H Raibert and Jessica K Hodgins. 1991. Animation of dynamic legged locomotion. In Proceedings of the 18th annual conference on Computer graphics and interactive techniques. 349–358.
59. Lionel Reveret, Laurent Favreau, Christine Depraz, and Marie-Paule Cani. 2005. Morphable model of quadrupeds skeletons for animating 3d animals. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation. 135–142.
60. Quentin Rouxel, Kai Yuan, Ruoshi Wen, and Zhibin Li. 2022. Multicontact Motion Retargeting Using Whole-Body Optimization of Full Kinematics and Sequential Force Equilibrium. IEEE/ASME Transactions on Mechatronics (2022).
61. Hoseok Ryu, Minseok Kim, Seungwhan Lee, Moon Seok Park, Kyoungmin Lee, and Jehee Lee. 2021. Functionality-Driven Musculature Retargeting. Computer Graphics Forum 40, 1 (2021).
62. Christian Schumacher, Espen Knoop, and Moritz Bächer. 2021. A Versatile Inverse Kinematics Formulation for Retargeting Motions onto Robots with Kinematic Loops. (2021).
63. Jian Shi, Peter B Luh, Shi-Chung Chang, and Tsu-Shuan Chang. 1990. A Method for Constrained Dynamic Optimization Problems. In 1990 American Control Conference.
64. Ljiljana Skrba, Lionel Reveret, Franck Hétroy, Marie-Paule Cani, and Carol O’Sullivan. 2009. Animating quadrupeds: methods and applications. In Computer Graphics Forum, Vol. 28. Wiley Online Library.
65. Jean-Pierre Sleiman, Farbod Farshidian, and Marco Hutter. 2021. Constraint Handling in Continuous-Time DDP-Based Model Predictive Control. In 2021 IEEE International Conference on Robotics and Automation (ICRA).
66. Laura Smith, J Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. 2021. Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World. arXiv preprint arXiv:2110.05457 (2021).
67. Kwang Won Sok, Manmyung Kim, and Jehee Lee. 2007. Simulating biped behaviors from human motion data. ACM Trans. Graph. 26, 3 (2007).
68. Wei Sun, Evangelos A. Theodorou, and Panagiotis Tsiotras. 2014. Continuous-time differential dynamic programming with terminal constraints. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). IEEE.
69. Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. 2018. Sim-to-real: Learning agile locomotion for quadruped robots. Robotics: Science and Systems (2018).
70. Yuval Tassa, Tom Erez, and Emanuel Todorov. 2012. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE.
71. Yuval Tassa, Nicolas Mansard, and Emo Todorov. 2014. Control-limited differential dynamic programming. In 2014 IEEE International Conference on Robotics and Automation (ICRA).
72. Michael Taylor, Sergey Bashkirov, Javier Fernandez Rico, Ike Toriyama, Naoyuki Miyada, Hideki Yanagisawa, and Kensaku Ishizuka. 2021. Learning Bipedal Robot Locomotion from Human Movement. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2797–2803.
73. E. Todorov and Weiwei Li. 2005. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005.
74. Tarik Tosun, Ross Mead, and Robert Stengel. 2014. A general method for kinematic retargeting: Adapting poses between humans and robots. In ASME International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers.
75. Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional VAEs. ACM Transactions on Graphics 41, 4 (2022).
76. Jungdam Won and Jehee Lee. 2019. Learning body shape variation in physics-based characters. ACM Trans. Graph. (TOG) 38, 6 (2019), 1–12.
77. Weitao Xi and C. David Remy. 2014. Optimal gaits and motions for legged robots. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.
78. Zhaoming Xie, C. Karen Liu, and Kris Hauser. 2017. Differential dynamic programming with nonlinear constraints. In 2017 IEEE International Conference on Robotics and Automation (ICRA).
79. Katsu Yamane, Stuart O Anderson, and Jessica K Hodgins. 2010. Controlling humanoid robots with human motion data: Experimental validation. In 2010 10th IEEE-RAS International Conference on Humanoid Robots. IEEE.
80. Chuanyu Yang, Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. 2020. Multi-expert learning of adaptive legged locomotion. Science Robotics 5, 49 (2020), eabb2174.
81. Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu. 2022a. ControlVAE: ModelBased Learning of Generative Controllers for Physics-Based Characters. ACM Transactions on Graphics 41, 6 (2022).
82. Qingfeng Yao, Jilong Wang, Shuyu Yang, Cong Wang, Hongyin Zhang, Qifeng Zhang, and Donglin Wang. 2022b. Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning. arXiv preprint arXiv:2203.05973 (2022).
83. Yuting Ye and C. Karen Liu. 2010. Optimal feedback control for character animation using an abstract model. ACM Transactions on Graphics 29, 4 (2010).
84. Fulong Yin, Annan Tang, Liangwei Xu, Yue Cao, Yu Zheng, Zhengyou Zhang, and Xiangyu Chen. 2021. Run Like a Dog: Learning Based Whole-Body Control Framework for Quadruped Gait Style Transfer. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
85. He Zhang, Sebastian Starke, Taku Komura, and Jun Saito. 2018. Mode-adaptive neural networks for quadruped motion control. ACM Trans. Graph. (TOG) 37, 4 (2018), 1–11.
86. Allan Zhao, Jie Xu, Mina Konaković-Luković, Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. 2020. Robogrammar: graph grammar for terrain-optimized robot design. ACM Trans. on Graph. (TOG) 39, 6 (2020).
87. Ziyi Zhou, Bruce Wingo, Nathan Boyd, Seth Hutchinson, and Ye Zhao. 2022. Momentum-Aware Trajectory Optimization and Control for Agile Quadrupedal Locomotion. arXiv preprint arXiv:2203.01548 (2022).
88. Victor Brian Zordan and Jessica K. Hodgins. 2002. Motion capture-driven simulations that hit and react. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation (SCA ’02). Association for Computing Machinery, New York, NY, USA.
