“Fluid directed rigid body control using deep reinforcement learning” by Ma, Tian, Pan, Ren and Manocha
Conference:
Type(s):
Entry Number: 96
Title:
- Fluid directed rigid body control using deep reinforcement learning
Session/Category Title: Fluids 2: Vortex Boogaloo
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We present a learning-based method to control a coupled 2D system involving both fluid and rigid bodies. Our approach is used to modify the fluid/rigid simulator’s behavior by applying control forces only at the simulation domain boundaries. The rest of the domain, corresponding to the interior, is governed by the Navier-Stokes equation for fluids and Newton-Euler’s equation for the rigid bodies. We represent our controller using a general neural-net, which is trained using deep reinforcement learning. Our formulation decomposes a control task into two stages: a precomputation training stage and an online generation stage. We utilize various fluid properties, e.g., the liquid’s velocity field or the smoke’s density field, to enhance the controller’s performance. We set up our evaluation benchmark by letting controller drive fluid jets move on the domain boundary and allowing them to shoot fluids towards a rigid body to accomplish a set of challenging 2D tasks such as keeping a rigid body balanced, playing a two-player ping-pong game, and driving a rigid body to sequentially hit specified points on the wall. In practice, our approach can generate physically plausible animations.
References:
1. Nadir Akinci, Markus Ihmsen, Gizem Akinci, Barbara Solenthaler, and Matthias Teschner. 2012. Versatile Rigid-fluid Coupling for Incompressible SPH. ACM Trans. Graph. 31, 4, Article 62 (July 2012), 8 pages. Google ScholarDigital Library
2. Christopher Batty, Florence Bertails, and Robert Bridson. 2007. A Fast Variational Framework for Accurate Solid-fluid Coupling. ACM Trans. Graph. 26, 3, Article 100 (July 2007). Google ScholarDigital Library
3. Markus Becker and Matthias Teschner. 2007. Weakly Compressible SPH for Free Surface Flows. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA ’07). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 209–217. Google ScholarDigital Library
4. Markus Becker, Hendrik Tessendorf, and Matthias Teschner. 2009. Direct Forcing for Lagrangian Rigid-Fluid Coupling. IEEE Transactions on Visualization and Computer Graphics 15, 3 (May 2009), 493–503. Google ScholarDigital Library
5. Boris Bonev, Lukas Prantl, and Nils Thuerey. 2017. Pre-computed Liquid Spaces with Generative Neural Networks. arXiv to appear (Apr 2017), 14.Google Scholar
6. Nuttapong Chentanez and Matthias Müller. 2011. Real-time Eulerian Water Simulation Using a Restricted Tall Cell Grid. In ACM SIGGRAPH 2011 Papers (SIGGRAPH ’11). ACM, New York, NY, USA, Article 82, 10 pages. Google ScholarDigital Library
7. Marc Claesen and Bart De Moor. 2015. Hyperparameter Search in Machine Learning. arXiv preprint arXiv:1502.02127 (2015).Google Scholar
8. Douglas Enright, Stephen Marschner, and Ronald Fedkiw. 2002. Animation and Rendering of Complex Water Surfaces. ACM Trans. Graph. 21, 3 (July 2002), 736–744. Google ScholarDigital Library
9. Isabelle Fantoni and Rogelio Lozano. 2001. Non-Linear Control for Underactuated Mechanical Systems. Springer-Verlag New York, Inc., Secaucus, NJ, USA. Google ScholarDigital Library
10. Raanan Fattal and Dani Lischinski. 2004. Target-driven Smoke Animation. ACM Trans. Graph. 23, 3 (Aug. 2004), 441–448. Google ScholarDigital Library
11. Paul Glasserman and David D. Yao. 1992. Some Guidelines and Guarantees for Common Random Numbers. Manage. Sci. 38, 6 (June 1992), 884–908.Google Scholar
12. Nikolaus Hansen and Andreas Ostermeier. 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evol. Comput. 9, 2 (June 2001), 159–195. Google ScholarDigital Library
13. Sham Kakade. 2001. A Natural Policy Gradient. In Advances in Neural Information Processing Systems 14 (NIPS 2001), Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani (Eds.). MIT Press, 1531–1538. Google ScholarDigital Library
14. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).Google Scholar
15. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems – Volume 1 (NIPS’12). Curran Associates Inc., USA, 1097–1105. Google ScholarDigital Library
16. Alexander Kurganov and Guergana Petrova. 2007. A Second-Order Well-Balanced Positivity Preserving Central-Upwind Scheme for the Saint-Venant System. Commun. Math. Sci. 5, 1 (03 2007), 133–160. https://projecteuclid.org:443/euclid.cms/1175797625Google Scholar
17. Anita T. Layton and Michiel van de Panne. 2002. A numerically efficient and stable algorithm for animating water waves. The Visual Computer 18, 1 (01 Feb 2002), 41–53.Google Scholar
18. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. 2016. End-to-end Training of Deep Visuomotor Policies. J. Mach. Learn. Res. 17, 1 (Jan. 2016), 1334–1373. Google ScholarDigital Library
19. Libin Liu and Jessica Hodgins. 2017. Learning to Schedule Control Fragments for Physics-Based Characters Using Deep Q-Learning. ACM Trans. Graph. 36, 3, Article 29 (June 2017), 14 pages. Google ScholarDigital Library
20. Frank Losasso, Jerry Talton, Nipun Kwatra, and Ronald Fedkiw. 2008. Two-Way Coupled SPH and Particle Level Set Fluid Simulation. IEEE Transactions on Visualization and Computer Graphics 14, 4 (July 2008), 797–804. Google ScholarDigital Library
21. Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. In Advances in Neural Information Processing Systems 30 (NIPS 2017). 6382–6393.Google Scholar
22. Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction. Springer Berlin Heidelberg, Berlin, Heidelberg, 52–59. Google ScholarDigital Library
23. Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid Control Using the Adjoint Method. ACM Trans. Graph. 23, 3 (Aug. 2004), 449–456. Google ScholarDigital Library
24. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research), Maria Fiorina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. PMLR, New York, New York, USA, 1928–1937. Google ScholarDigital Library
25. Patrick Mullen, Keenan Crane, Dmitry Pavlov, Yiying Tong, and Mathieu Desbrun. 2009. Energy-preserving Integrators for Fluid Animation. ACM Trans. Graph. 28, 3, Article 38 (July 2009), 8 pages. Google ScholarDigital Library
26. J. Nocedal and S. J. Wright. 2006. Numerical Optimization (2nd ed.). Springer, New York.Google Scholar
27. Zherong Pan, Jin Huang, Yiying Tong, Changxi Zheng, and Hujun Bao. 2013. Interactive Localized Liquid Motion Editing. ACM Trans. Graph. 32, 6, Article 184 (Nov. 2013), 10 pages. Google ScholarDigital Library
28. Zherong Pan and Dinesh Manocha. 2017. Efficient Solver for Spacetime Control of Smoke. ACM Trans. Graph. 36, 5, Article 162 (July 2017), 13 pages. Google ScholarDigital Library
29. Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. 2017. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Trans. Graph. 36, 4, Article 41 (July 2017), 13 pages. Google ScholarDigital Library
30. Jovan Popović, Steven M. Seitz, and Michael Erdmann. 2003. Motion Sketching for Control of Rigid-body Simulations. ACM Trans. Graph. 22, 4 (Oct. 2003), 1034–1054. Google ScholarDigital Library
31. Jovan Popović, Steven M. Seitz, Michael Erdmann, Zoran Popović, and Andrew Witkin. 2000. Interactive Manipulation of Rigid Body Simulations. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’00). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 209–217. Google ScholarDigital Library
32. Karthik Raveendran, Chris Wojtan, Nils Thuerey, and Greg Turk. 2014. Blending Liquids. ACM Trans. Graph. 33, 4, Article 137 (July 2014), 10 pages. Google ScholarDigital Library
33. Avi Robinson-Mosher, Tamar Shinar, Jon Gretarsson, Jonathan Su, and Ronald Fedkiw. 2008. Two-way Coupling of Fluids to Rigid and Deformable Solids and Shells. In ACM SIGGRAPH 2008 Papers (SIGGRAPH ’08). ACM, New York, NY, USA, Article 46, 9 pages. Google ScholarDigital Library
34. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015a. Trust Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research), Francis Bach and David Blei (Eds.), Vol. 37. PMLR, Lille, France, 1889–1897. Google ScholarDigital Library
35. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015b. High-Dimensional Continuous Control Using Generalized Advantage Estimation. (2015). arXiv:arXiv:1506.02438Google Scholar
36. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. (2017). arXiv:arXiv:1707.06347Google Scholar
37. Lin Shi and Yizhou Yu. 2005. Taming Liquids for Rapidly Changing Targets. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA ’05). ACM, New York, NY, USA, 229–236. Google ScholarDigital Library
38. Nils Thuerey. 2016. Interpolations of Smoke and Liquid Simulations. ACM Trans. Graph. 36, 1, Article 3 (Sept. 2016), 16 pages. Google ScholarDigital Library
39. J. Tompson, K. Schlachter, P. Sprechmann, and K. Perlin. 2016. Accelerating Eulerian Fluid Simulation With Convolutional Networks. ArXiv e-prints (July 2016). arXiv:cs.CV/1607.03597Google Scholar
40. Adrien Treuille, Antoine McNamara, Zoran Popović, and Jos Stam. 2003. Keyframe Control of Smoke Simulations. ACM Trans. Graph. 22, 3 (July 2003), 716–723. Google ScholarDigital Library
41. Christopher D. Twigg and Doug L. James. 2007. Many-worlds Browsing for Control of Multibody Dynamics. ACM Trans. Graph. 26, 3, Article 14 (July 2007). Google ScholarDigital Library
42. Christopher D. Twigg and Doug L. James. 2008. Backward Steps in Rigid Body Simulation. ACM Trans. Graph. 27, 3, Article 25 (Aug. 2008), 10 pages. Google ScholarDigital Library
43. Jack M. Wang, Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. 2012. Optimizing Locomotion Controllers Using Biologically-based Actuators and Objectives. ACM Trans. Graph. 31, 4, Article 25 (July 2012), 11 pages. Google ScholarDigital Library
44. Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to Train Your Dragon: Example-guided Control of Flapping Flight. ACM Trans. Graph. 36, 6, Article 198 (Nov. 2017), 13 pages. Google ScholarDigital Library
45. KangKang Yin, Kevin Loken, and Michiel van de Panne. 2007. SIMBICON: Simple Biped Locomotion Control. In ACM SIGGRAPH 2007 Papers (SIGGRAPH ’07). ACM, New York, NY, USA, Article 105. Google ScholarDigital Library
46. Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, and Rob Fergus. 2010. Deconvolutional networks. 2528–2535.Google Scholar
47. Bo Zhu, Wenlong Lu, Matthew Cong, Byungmoon Kim, and Ronald Fedkiw. 2013. A New Grid Structure for Domain Extension. ACM Trans. Graph. 32, 4, Article 63 (July 2013), 12 pages. Google ScholarDigital Library
48. Yongning Zhu and Robert Bridson. 2005. Animating Sand As a Fluid. ACM Trans. Graph. 24, 3 (July 2005), 965–972. Google ScholarDigital Library
