“PuppetMaster: robotic animation of marionettes” by Zimmermann, Poranne, Bern and Coros
Conference:
Type:
Title:
- PuppetMaster: robotic animation of marionettes
Session/Category Title: Motion is in Control
Presenter(s)/Author(s):
Abstract:
We present a computational framework for robotic animation of real-world string puppets. Also known as marionettes, these articulated figures are typically brought to life by human puppeteers. The puppeteer manipulates rigid handles that are attached to the puppet from above via strings. The motions of the marionette are therefore governed largely by gravity, the pull forces exerted by the strings, and the internal forces arising from mechanical articulation constraints. This seemingly simple setup conceals a very challenging and nuanced control problem, as marionettes are, in fact, complex coupled pendulum systems. Despite this, in the hands of a master puppeteer, marionette animation can be nothing short of mesmerizing. Our goal is to enable autonomous robots to animate marionettes with a level of skill that approaches that of human puppeteers. To this end, we devise a predictive control model that accounts for the dynamics of the marionette and kinematics of the robot puppeteer. The input to our system consists of a string puppet design and a target motion, and our trajectory planning algorithm computes robot control actions that lead to the marionette moving as desired. We validate our methodology through a series of experiments conducted on an array of marionette designs and target motions. These experiments are performed both in simulation and using a physical robot, the human-sized, dual arm ABB YuMi® IRB 14000.
References:
1. Thomas Auzinger, Wolfgang Heidrich, and Bernd Bickel. 2018. Computational design of nanostructural color for additive manufacturing. ACM Trans. Graph. 37, 4 (2018), 159:1–159:16. Google ScholarDigital Library
2. Yunfei Bai, Wenhao Yu, and C. Karen Liu. 2016. Dexterous Manipulation of Cloth. Comput. Graph. Forum 35, 2 (2016), 523–532.Google ScholarCross Ref
3. Jernej Barbič, Marco da Silva, and Jovan Popović. 2009. Deformable Object Animation Using Reduced Optimal Control. ACM Trans. Graph. 28, 3, Article 53 (July 2009), 9 pages. Google ScholarDigital Library
4. James M. Bern, Kai-Hung Chang, and Stelian Coros. 2017. Interactive design of animated plushies. ACM Trans. Graph. 36, 4 (2017), 80:1–80:11. Google ScholarDigital Library
5. M. Bernard and K. Kondak. 2009. Generic slung load transportation system using small size helicopters. In 2009 IEEE International Conference on Robotics and Automation. 3258–3264. Google ScholarDigital Library
6. Morten Bisgaard, Jan Bendtsen, and Anders La Cour-Harbo. 2009. Modelling of Generic Slung Load System. In AIAA Modeling and Simulation Technologies Conference and Exhibit. American Institute of Aeronautics and Astronautics.Google Scholar
7. Yang Cao, Shengtai Li, Linda Petzold, and Radu Serban. 2003. Adjoint sensitivity analysis for differential-algebraic equations: The adjoint DAE system and its numerical solution. SIAM Journal on Scientific Computing 24, 3 (2003), 1076–1089. Google ScholarDigital Library
8. I.-Ming Chen, Raymond Tay, Shusong Xing, and Song Huat Yeo. 2004. Marionette: From Traditional Manipulation to Robotic Manipulation. In International Symposium on History of Machines and Mechanisms. Springer, Dordrecht, 119–133.Google ScholarCross Ref
9. Alexander Clegg, Jie Tan, Greg Turk, and C. Karen Liu. 2015. Animating human dressing. ACM Trans. Graph. 34, 4 (2015), 116:1–116:9. Google ScholarDigital Library
10. Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. 2018. Learning to dress: synthesizing human dressing motion via deep reinforcement learning. ACM Trans. Graph. 37, 6 (2018), 179:1–179:10. Google ScholarDigital Library
11. Michael F. Cohen. 1992. Interactive Spacetime Control for Animation. SIGGRAPH Comput. Graph. 26, 2 (July 1992), 293–302. Google ScholarDigital Library
12. Stelian Coros, Bernhard Thomaszewski, Gioacchino Noris, Shinjiro Sueda, Moira Forberg, Robert W. Sumner, Wojciech Matusik, and Bernd Bickel. 2013. Computational design of mechanical characters. ACM Trans. Graph. 32, 4 (2013), 83:1–83:12. Google ScholarDigital Library
13. Cédric De Crousaz, Farbod Farshidian, and Jonas Buchli. 2014. Aggressive optimal control for agile flight with a slung load. In in IROS 2014 Workshop on Machine Learning in Planning and Control of Robot Motion.Google Scholar
14. Tao Du, Adriana Schulz, Bo Zhu, Bernd Bickel, and Wojciech Matusik. 2016. Computational multicopter design. ACM Trans. Graph. 35, 6 (2016), 227:1–227:10. http://dl.acm.org/citation.cfm?id=2982427 Google ScholarDigital Library
15. Simon Duenser, James M. Bern, Roi Poranne, and Stelian Coros. 2018. Interactive Robotic Manipulation of Elastic Objects. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1–5, 2018. 3476–3481.Google Scholar
16. A. Faust, I. Palunko, P. Cruz, R. Fierro, and L. Tapia. 2013. Learning swing-free trajectories for UAVs with a suspended load. In 2013 IEEE International Conference on Robotics and Automation. 4902–4909.Google Scholar
17. Damien Gauge, Stelian Coros, Sandro Mani, and Bernhard Thomaszewski. 2014. Interactive Design of Modular Tensegrity Characters. In The Eurographics / ACM SIGGRAPH Symposium on Computer Animation, SCA 2014, Copenhagen, Denmark, 2014. 131–138. Google ScholarDigital Library
18. Moritz Geilinger, Roi Poranne, Ruta Desai, Bernhard Thomaszewski, and Stelian Coros. 2018. Skaterbots: Optimization-based Design and Motion Synthesis for Robotic Creatures with Legs and Wheels. In Proceedings of ACM SIGGRAPH, ACM Transactions on Graphics (TOG) (Ed.), Vol. 37. ACM. Google ScholarDigital Library
19. Gaël Guennebaud, Benoît Jacob, et al. 2010. Eigen v3. http://eigen.tuxfamily.org.Google Scholar
20. John Hauser. 2002. A Projection Operator Approach To The Optimization Of Trajectory Functionals. IFAC Proceedings Volumes 35, 1 (2002), 377 — 382. 15th IFAC World Congress.Google ScholarCross Ref
21. Richard H. Jackson and Garth P. Mccormick. 1988. Second-order Sensitivity Analysis in Factorable Programming: Theory and Applications. Math. Program. 41, 1–3 (May 1988), 1–27. Google ScholarDigital Library
22. E. Johnson and T. D. Murphey. 2007. Dynamic Modeling and Motion Planning for Marionettes: Rigid Bodies Articulated by Massless Strings. In Proceedings 2007 IEEE International Conference on Robotics and Automation. 330–335.Google Scholar
23. Mickaël Ly, Romain Casati, Florence Bertails-Descoubes, Mélina Skouras, and Laurence Boissieux. 2018. Inverse Elastic Shell Design with Contact and Friction. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia ’18). ACM, New York, NY, USA, Article 201, 16 pages. Google ScholarDigital Library
24. Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus H. Gross. 2011. Example-based elastic materials. ACM Trans. Graph. 30, 4 (2011), 72:1–72:8. Google ScholarDigital Library
25. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid Control Using the Adjoint Method. In ACM SIGGRAPH 2004 Papers (SIGGRAPH ’04). ACM, New York, NY, USA, 449–456. Google ScholarDigital Library
26. Vittorio Megaro, Bernhard Thomaszewski, Maurizio Nitti, Otmar Hilliges, Markus H. Gross, and Stelian Coros. 2015. Interactive design of 3D-printable robotic creatures. ACM Trans. Graph. 34, 6 (2015), 216:1–216:9. Google ScholarDigital Library
27. Vittorio Megaro, Jonas Zehnder, Moritz Bächer, Stelian Coros, Markus H. Gross, and Bernhard Thomaszewski. 2017. A computational design tool for compliant mechanisms. ACM Trans. Graph. 36, 4 (2017), 82:1–82:12. Google ScholarDigital Library
28. E. Mizutani. 2015. On Pantoja’s problem allegedly showing a distinction between differential dynamic programming and stagewise Newton methods. Internat. J. Control 88, 9 (2015), 1702–1711.Google ScholarCross Ref
29. Todd D Murphey and E. Johnson. 2011. Control aesthetics in software architecture for robotic marionettes. In American Control Conference (ACC), 2011. IEEE, IEEE, 3825–3830.Google Scholar
30. D. M. Murray and S. J. Yakowitz. 1984. Differential dynamic programming and Newton’s method for discrete optimal control problems. Journal of Optimization Theory and Applications 43, 3 (01 Jul 1984), 395–414.Google ScholarCross Ref
31. I. Palunko, A. Faust, P. Cruz, L. Tapia, and R. Fierro. 2013. A reinforcement learning approach towards autonomous suspended load manipulation using aerial robots. In 2013 IEEE International Conference on Robotics and Automation. 4896–4901.Google Scholar
32. Zherong Pan and Dinesh Manocha. 2018. Active Animations of Reduced Deformable Models with Environment Interactions. ACM Trans. Graph. 37, 3, Article 36 (Aug. 2018), 17 pages. Google ScholarDigital Library
33. J. F. A. De O. Pantoja. 1988. Differential dynamic programming and Newton’s method. Internat. J. Control 47, 5 (1988), 1539–1553.Google ScholarCross Ref
34. Jesús Pérez, Miguel A. Otaduy, and Bernhard Thomaszewski. 2017. Computational design and automated fabrication of kirchhoff-plateau surfaces. ACM Trans. Graph. 36, 4 (2017), 62:1–62:12. Google ScholarDigital Library
35. J. Schultz and T. Murphey. 2012. Trajectory generation for underactuated control of a suspended mass. In 2012 IEEE International Conference on Robotics and Automation. 123–129.Google Scholar
36. Adriana Schulz, Cynthia R. Sung, Andrew Spielberg, Wei Zhao, Robin Cheng, Eitan Grinspun, Daniela Rus, and Wojciech Matusik. 2017. Interactive robogami: An end-to-end system for design of robots with ground locomotion. I. J. Robotics Res. 36, 10 (2017), 1131–1147. Google ScholarDigital Library
37. Christian Schulz, Christoph von Tycowicz, Hans-Peter Seidel, and Klaus Hildebrandt. 2014. Animating Deformable Objects Using Sparse Spacetime Constraints. ACM Trans. Graph. 33, 4, Article 109 (July 2014), 10 pages. Google ScholarDigital Library
38. Mélina Skouras, Bernhard Thomaszewski, Stelian Coros, Bernd Bickel, and Markus H. Gross. 2013. Computational design of actuated deformable characters. ACM Trans. Graph. 32, 4 (2013), 82:1–82:10. Google ScholarDigital Library
39. Peng Song, Xiaofei Wang, Xiao Tang, Chi-Wing Fu, Hongfei Xu, Ligang Liu, and Niloy J. Mitra. 2017. Computational design of wind-up toys. ACM Trans. Graph. 36, 6 (2017), 238:1–238:13. Google ScholarDigital Library
40. Koushil Sreenath and Vijay Kumar. 2013. Dynamics, Control and Planning for Cooperative Manipulation of Payloads Suspended by Cables from Multiple Quadrotor Robots.Google Scholar
41. K. Sreenath, N. Michael, and V. Kumar. 2013. Trajectory generation and control of a quadrotor with a cable-suspended load – A differentially-flat hybrid system. In 2013 IEEE International Conference on Robotics and Automation. 4888–4895.Google Scholar
42. Bernhard Thomaszewski, Stelian Coros, Damien Gauge, Vittorio Megaro, Eitan Grinspun, and Markus H. Gross. 2014. Computational design of linkage-based characters. ACM Trans. Graph. 33, 4 (2014), 64:1–64:9. Google ScholarDigital Library
43. Emanuel Todorov. 2014. Convex and analytically-invertible dynamics with contacts and constraints: Theory and implementation in MuJoCo. In 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, May 31 – June 7, 2014. 6054–6061.Google ScholarCross Ref
44. Andrew Witkin and Michael Kass. 1988. Spacetime Constraints. SIGGRAPH Comput. Graph. 22, 4 (June 1988), 159–168. Google ScholarDigital Library
45. Hongyi Xu, Espen Knoop, Stelian Coros, and Moritz Bächer. 2018. Bend-it: design and fabrication of kinetic wire characters. ACM Trans. Graph. 37, 6 (2018), 239:1–239:15. Google ScholarDigital Library
46. Katsu Yamane, Jessica K. Hodgins, and H. Benjamin Brown. 2004. Controlling a motorized marionette with human motion capture data. International Journal of Humanoid Robotics 01, 04 (Dec. 2004), 651–669.Google ScholarCross Ref
47. D. Zameroski, G. Starr, J. Wood, and R. Lumia. 2006. Swing-free trajectory generation for dual cooperative manipulators using dynamic programming. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. 1997–2003.Google Scholar
48. Ran Zhang, Thomas Auzinger, Duygu Ceylan, Wilmot Li, and Bernd Bickel. 2017. Functionality-aware retargeting of mechanisms to 3D shapes. ACM Trans. Graph. 36, 4 (2017), 81:1–81:13. Google ScholarDigital Library
