“Learning bicycle stunts” by Tan, Gu, Liu and Turk
Conference:
Type(s):
Title:
- Learning bicycle stunts
Session/Category Title: Controlling Character
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a general approach for simulating and controlling a human character that is riding a bicycle. The two main components of our system are offline learning and online simulation. We simulate the bicycle and the rider as an articulated rigid body system. The rider is controlled by a policy that is optimized through offline learning. We apply policy search to learn the optimal policies, which are parameterized with splines or neural networks for different bicycle maneuvers. We use Neuroevolution of Augmenting Topology (NEAT) to optimize both the parametrization and the parameters of our policies. The learned controllers are robust enough to withstand large perturbations and allow interactive user control. The rider not only learns to steer and to balance in normal riding situations, but also learns to perform a wide variety of stunts, including wheelie, endo, bunny hop, front wheel pivot and back hop.
References:
1. Allen, B., and Faloutsos, P. 2009. Evolved controllers for simulated locomotion. In Motion in Games, Lecture Notes in Computer Science, 219–230. Google ScholarDigital Library
2. Andrews, S., and Kry, P. 2013. Goal directed multi-finger manipulation: Control policies and analysis. Computers & Graphics 37, 7, 830–839. Google ScholarDigital Library
3. Auslander, J., Fukunaga, A., Partovi, H., Christensen, J., Hsu, L., Reiss, P., Shuman, A., Marks, J., and Ngo, J. T. 1995. Further experience with controller-based automatic motion synthesis for articulated figures. ACM Trans. Graph. 14, 4 (Oct.), 311–336. Google ScholarDigital Library
4. BBC. 2005. Bicycle chosen as best invention. BBC News.Google Scholar
5. Boyan, J. A., and Moore, A. W. 1995. Generalization in reinforcement learning: Safely approximating the value function. In Advances in Neural Information Processing Systems 7, MIT Press, 369–376.Google Scholar
6. Carvallo, M. E. 1900. Théorie du mouvement du monocycle et de la bicyclette. Journal de L’Ecole Polytechnique 5.Google Scholar
7. Chambaron, S., Berberian, B., Delbecque, L., Ginhac, D., and Cleeremans, A. 2009. Implicit motor learning in discrete and continuous tasks: Toward a possible account of discrepant results. Handbook of Motor Skills: Development, Impairment, and Therapy, 139–155. Google ScholarDigital Library
8. Collins, R. N. 1963. A mathematical analysis of the stability of two-wheeled vehicles. PhD thesis, University of Wisconsin. Google ScholarDigital Library
9. Coros, S., Beaudoin, P., and van de Panne, M. 2009. Robust task-based control policies for physics-based characters. ACM Trans. Graph. 28, 5 (Dec.), 170:1–170:9. Google ScholarDigital Library
10. Coros, S., Beaudoin, P., and van de Panne, M. 2010. Generalized biped walking control. ACM Transctions on Graphics 29, 4, Article 130. Google ScholarDigital Library
11. Coros, S., Karpathy, A., Jones, B., Reveret, L., and van de Panne, M. 2011. Locomotion skills for simulated quadrupeds. ACM Transactions on Graphics 30, 4. Google ScholarDigital Library
12. da Silva, M., Abe, Y., and Popović, J. 2008. Interactive simulation of stylized human locomotion. In ACM SIGGRAPH 2008 Papers, ACM, New York, NY, USA, SIGGRAPH ’08, 82:1–82:10. Google ScholarDigital Library
13. de Lasa, M., and Hertzmann, A. 2009. Prioritized optimization for task-space control. In International Conference on Intelligent Robots and Systems (IROS). Google ScholarDigital Library
14. Geijtenbeek, T., and Pronost, N. 2012. Interactive Character Animation Using Simulated Physics: A State-of-the-Art Review. Computer Graphics Forum 31, 8, 2492–2515. Google ScholarDigital Library
15. Geijtenbeek, T., van de Panne, M., and van der Stappen, A. F. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics 32, 6. Google ScholarDigital Library
16. Grzeszczuk, R., and Terzopoulos, D. 1995. Automated learning of muscle-actuated locomotion through control abstraction. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 63–70. Google ScholarDigital Library
17. Hansen, N. 2009. The CMA Evolution Strategy: A Tutorial.Google Scholar
18. Heidrich-Meisner, V., and Igel, C. 2008. Evolution strategies for direct policy search. In Proceedings of the 10th International Conference on Parallel Problem Solving from Nature: PPSN X, Springer-Verlag, Berlin, Heidelberg, 428–437. Google ScholarDigital Library
19. Hinton, G. E. 2007. Learning multiple layers of representation. Trends in Cognitive Sciences 11, 428–434.Google ScholarCross Ref
20. Hodgins, J. K., Sweeney, P. K., and Lawrence, D. G. 1992. Generating natural-looking motion for computer animation. In Proceedings of the Conference on Graphics Interface ’92, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 265–272. Google ScholarDigital Library
21. Hodgins, J. K., Wooten, W. L., Brogan, D. C., and O’Brien, J. F. 1995. Animating human athletics. In SIGGRAPH, 71–78. Google ScholarDigital Library
22. Jones, D. E. H. 1970. The Stability of the bicycle. Physics Today 23, 34–40.Google ScholarCross Ref
23. Klein, F., and Sommerfeld, A. 1910. Stabilität des fahrrads. Über die Theorie des Kreisels, Ch. IX, Section 8, 863–884.Google Scholar
24. Kooijman, J. D. G., Meijaard, J. P., Papadopoulos, J. M., Ruina, A., and Schwab, A. L. 2011. A Bicycle Can Be Self-Stable Without Gyroscopic or Caster Effects. Science 332, 6027 (Apr.), 339–342.Google ScholarCross Ref
25. Kwon, T., and Hodgins, J. 2010. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Eurographics Association, Aire-la-Ville, Switzerland, SCA ’10, 129–138. Google ScholarDigital Library
26. Laszlo, J., van de Panne, M., and Fiume, E. 1996. Limit cycle control and its application to the animation of balancing and walking. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’96, 155–162. Google ScholarDigital Library
27. Levine, S., and Koltun, V. 2013. Guided policy search. In ICML ’13: Proceedings of the 30th International Conference on Machine Learning.Google Scholar
28. Levine, S., Wang, J. M., Haraux, A., Popović, Z., and Koltun, V. 2012. Continuous character control with low-dimensional embeddings. ACM Trans. Graph. 31, 4 (July), 28:1–28:10. Google ScholarDigital Library
29. Meijaard, J. P., Papadopoulos, J. M., Ruina, A., and Schwab, A. L. 2007. Linearized dynamics euqations for the balance and steer of a bicycle: a benchmark and review. Proceedings of the Royal Society A.Google Scholar
30. Mordatch, I., de Lasa, M., and Hertzmann, A. 2010. Robust Physics-Based Locomotion Using Low-Dimensional Planning. ACM Transactions on Graphics 29, 3. Google ScholarDigital Library
31. Muico, U., Lee, Y., Popović, J., and Popović, Z. 2009. Contact-aware nonlinear control of dynamic characters. In ACM SIGGRAPH 2009 Papers, ACM, New York, NY, USA, SIGGRAPH ’09, 81:1–81:9. Google ScholarDigital Library
32. Ng, A. Y., and Jordan, M. 2000. Pegasus: A policy search method for large MDPs and POMDPs. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, UAI’00, 406–415. Google ScholarDigital Library
33. Ng, A. Y., and Russell, S. J. 2000. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, ICML ’00, 663–670. Google ScholarDigital Library
34. Ngo, J. T., and Marks, J. 1993. Spacetime constraints revisited. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’93, 343–350. Google ScholarDigital Library
35. Peters, J., and Schaal, S. 2008. Reinforcement learning of motor skills with policy gradients. Neural Networks 21, 4 (May), 682–697. Google ScholarDigital Library
36. Pratt, J. E., Chew, C.-M., Torres, A., Dilworth, P., and Pratt, G. A. 2001. Virtual model control: An intuitive approach for bipedal locomotion. Int’l J. Robotic Research. 20, 2, 129–143.Google ScholarCross Ref
37. Randløv, J., and Alstrøm, P. 1998. Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Morgan Kauffman, San Francisco, CA, USA, J. W. Shavlik, Ed., 463–471. Google ScholarDigital Library
38. Rankine, W. J. M. 1870. On the dynamical principles of the motion of velocipedes. The Engineer.Google Scholar
39. Sims, K. 1994. Evolving virtual creatures. In Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’94, 15–22. Google ScholarDigital Library
40. Singh, D. V. 1964. Advanced concepts of the stability of two-wheeled vehicle-application of mathematical analysis to actual vehicles. PhD thesis, University of Wisconsin.Google Scholar
41. Smith, R., 2008. Open dynamics engine. http://www.ode.org/.Google Scholar
42. Stanley, K. O., and Miikkulainen, R. 2002. Evolving neural networks through augmenting topologies. Evol. Comput. 10, 2 (June), 99–127. Google ScholarDigital Library
43. Sutton, R. S., and Barto, A. G. 1998. Introduction to Reinforcement Learning, 1st ed. MIT Press, Cambridge, MA, USA. Google ScholarDigital Library
44. Tan, J., Gu, Y., Turk, G., and Liu, C. K. 2011. Articulated swimming creatures. In ACM SIGGRAPH 2011 papers, ACM, SIGGRAPH ’11, 58:1–58:12. Google ScholarDigital Library
45. Tan, J., Turk, G., and Liu, C. K. 2012. Soft body locomotion. ACM Trans. Graph. 31, 4 (July), 26:1–26:11. Google ScholarDigital Library
46. Thrun, S., and Schwartz, A. 1993. Issues in using function approximation for reinforcement learning. In In Proceedings of the Fourth Connectionist Models Summer School, Erlbaum.Google Scholar
47. Treuille, A., Lee, Y., and Popović, Z. 2007. Near-optimal character animation with continuous control. ACM Trans. Graph. 26, 3 (July). Google ScholarDigital Library
48. Tsai, Y.-Y., Lin, W.-C., Cheng, K. B., Lee, J., and Lee, T.-Y. 2010. Real-time physics-based 3D biped character animation using an inverted pendulum model. IEEE Transactions on Visualization and Computer Graphics 16, 2 (Mar.), 325–337. Google ScholarDigital Library
49. van de Panne, M., and Fiume, E. 1993. Sensor-actuator networks. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH ’93, 335–342. Google ScholarDigital Library
50. van de Panne, M., and Lee, C. 2003. Ski stunt simulator: Experiments with interactive dynamics. In Proceedings of the 14th Western Computer Graphics Symposium.Google Scholar
51. Van Zytveld, P. 1975. A Method for the Automatic Stabilization of an Unmanned Bicycle. Department of Aeronautics and Astronautics, Stanford University.Google Scholar
52. Wang, J. M., Fleet, D. J., and Hertzmann, A. 2009. Optimizing walking controllers. ACM Trans. Graph. 28, 5 (Dec.), 168:1–168:8. Google ScholarDigital Library
53. Wang, J. M., Fleet, D. J., and Hertzmann, A. 2010. Optimizing walking controllers for uncertain inputs and environments. ACM Trans. Graph. 29, 4 (July), 73:1–73:8. Google ScholarDigital Library
54. Wang, J. M., Hamner, S. R., Delp, S. L., and Koltun, V. 2012. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Trans. Graph. 31, 4 (July), 25:1–25:11. Google ScholarDigital Library
55. Whipple, F. J. W. 1899. The stability of the motion of a bicycle. Quarterly Journal of Pure and Applied Mathematics 30, 312–348.Google Scholar
56. Wu, J.-c., and Popović, Z. 2003. Realistic modeling of bird flight animations. In ACM SIGGRAPH 2003 Papers, ACM, New York, NY, USA, SIGGRAPH ’03, 888–895. Google ScholarDigital Library
57. Ye, Y., and Liu, C. K. 2010. Optimal feedback control for character animation using an abstract model. In SIGGRAPH ’10: ACM SIGGRAPH 2010 papers, ACM, New York, NY, USA, 1–9. Google ScholarDigital Library
58. Yin, K., Loken, K., and van de Panne, M. 2007. SIMBICON: simple biped locomotion control. In ACM SIGGRAPH 2007 papers, SIGGRAPH ’07. Google ScholarDigital Library
59. Yin, K., Coros, S., Beaudoin, P., and van de Panne, M. 2008. Continuation methods for adapting simulated skills. ACM Trans. Graph. 27, 3. Google ScholarDigital Library
60. Zhao, P., and van de Panne, M. 2005. User interfaces for interactive control of physics-based 3D characters. In Proceedings of the 2005 Symposium on Interactive 3D Graphics and Games, ACM, New York, NY, USA, I3D ’05, 87–94. Google ScholarDigital Library
