“Closed-loop control of direct ink writing via reinforcement learning” by Piovarči, Foshey, Xu, Erps, Babaei, et al. …

Enabling additive manufacturing to employ a wide range of novel, functional materials can be a major boost to this technology. However, making such materials printable requires painstaking trial-and-error by an expert operator, as they typically tend to exhibit peculiar rheological or hysteresis properties. Even in the case of successfully finding the process parameters, there is no guarantee of print-to-print consistency due to material differences between batches. These challenges make closed-loop feedback an attractive option where the process parameters are adjusted on-the-fly. There are several challenges for designing an efficient controller: the deposition parameters are complex and highly coupled, artifacts occur after long time horizons, simulating the deposition is computationally costly, and learning on hardware is intractable. In this work, we demonstrate the feasibility of learning a closed-loop control policy for additive manufacturing using reinforcement learning. We show that approximate, but efficient, numerical simulation is sufficient as long as it allows learning the behavioral patterns of deposition that translate to real-world experiences. In combination with reinforcement learning, our model can be used to discover control policies that outperform baseline controllers. Furthermore, the recovered policies have a minimal sim-to-real gap. We showcase this by applying our control policy in-vivo on a single-layer printer using low and high viscosity materials.

1. Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. 2019. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113 (2019).Google Scholar
2. Ivanna Baturynska, Oleksandr Semeniuta, and Kristian Martinsen. 2018. Optimization of process parameters for powder bed fusion additive manufacturing by combination of machine learning and finite element method: A conceptual framework. Procedia Cirp 67 (2018), 227–232.Google ScholarCross Ref
3. Jan Bender, Matthias Müller, Miguel A Otaduy, Matthias Teschner, and Miles Macklin. 2014. A survey on position-based simulation methods in computer graphics. In Computer graphics forum, Vol. 33. Wiley Online Library, 228–251.Google Scholar
4. John Parker Burg. 1975. Maximum Entropy Spectral Analysis. Stanford University.Google Scholar
5. Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. 2018. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Trans. Graph. 37, 6 (2018).Google ScholarDigital Library
6. Erwin Coumans and Yunfei Bai. 2016. Pybullet, a python module for physics simulation for games, robotics and machine learning. (2016).Google Scholar
7. Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. 2018. End-to-end differentiable physics for learning and control. Advances in neural information processing systems 31 (2018), 7178–7189.Google Scholar
8. Sarah Elliott and Maya Cakmak. 2018. Robotic cleaning through dirt rearrangement planning with learned transition models. In ICRA 2018. IEEE.Google ScholarDigital Library
9. Timothy Erps, Michael Foshey, Mina Konaković Luković, Wan Shou, Hanns Hagen Goetzke, Herve Dietsch, Klaus Stoll, Bernhard von Vacano, and Wojciech Matusik. 2021. Accelerated Discovery of 3D Printing Materials Using Data-Driven Multi-Objective Optimization. arXiv:2106.15697Google Scholar
10. Wei Gao, Yunbo Zhang, Devarajan Ramanujan, Karthik Ramani, Yong Chen, Christopher B Williams, Charlie CL Wang, Yung C Shin, Song Zhang, and Pablo D Zavattieri. 2015. The status, challenges, and future of additive manufacturing in engineering. Computer-Aided Design 69 (2015), 65–89.Google ScholarDigital Library
11. Angus Johnson. 2015. Clipper – an open source freeware library for clipping and offsetting lines and polygons. http://www.angusj.com/delphi/clipper.php.Google Scholar
12. Branden Kappes, Senthamilaruvi Moorthy, Dana Drake, Henry Geerlings, and Aaron Stebner. 2018. Machine learning to optimize additive manufacturing parameters for laser powder bed fusion of Inconel 718. In Proceedings of the 9th International Symposium on Superalloy 718 & Derivatives: Energy, Aerospace, and Industrial Applications. Springer, 595–610.Google ScholarCross Ref
13. Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. 2019. ABC: A Big CAD Model Dataset For Geometric Deep Learning. In CVPR.Google Scholar
14. Jeongseok Lee, Michael X Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S Srinivasa, Mike Stilman, and C Karen Liu. 2018. Dart: Dynamic animation and robotics toolkit. Journal of Open Source Software 3, 22 (2018), 500.Google ScholarCross Ref
15. Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019. Scalable muscle-actuated human simulation and control. ACM Trans. Graph. 38, 4 (2019).Google ScholarDigital Library
16. Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. 2019a. Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids. In International Conference on Learning Representations.Google Scholar
17. Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. 2019b. Propagation networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation. IEEE.Google ScholarDigital Library
18. Chenang Liu, David Roberson, and Zhenyu Kong. 2017. Textural analysis-based online closed-loop quality control for additive manufacturing processes. In IIE Annual Conference. Proceedings. Institute of Industrial and Systems Engineers (IISE).Google Scholar
19. Libin Liu and Jessica Hodgins. 2018. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Trans. Graph. 37, 4 (2018).Google ScholarDigital Library
20. Pingchuan Ma, Yunsheng Tian, Zherong Pan, Bo Ren, and Dinesh Manocha. 2018. Fluid directed rigid body control using deep reinforcement learning. ACM Trans. Graph. 37, 4 (2018).Google ScholarDigital Library
21. Miles Macklin and Matthias Müller. 2013. Position based fluids. ACM Trans. Graph. 32, 4 (2013).Google ScholarDigital Library
22. Larry Marple. 1980. A new autoregressive spectrum analysis algorithm. IEEE Transactions on Acoustics, Speech, and Signal Processing 28, 4 (1980), 441–454.Google ScholarCross Ref
23. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533.Google Scholar
24. Mojtaba Mozaffar, Arindam Paul, Reda Al-Bahrani, Sarah Wolff, Alok Choudhary, Ankit Agrawal, Kornel Ehmann, and Jian Cao. 2018. Data-driven prediction of the high-dimensional thermal history in directed energy deposition processes via recurrent neural networks. Manufacturing letters 18 (2018), 35–39.Google Scholar
25. Matthias Müller, David Charypar, and Markus H Gross. 2003. Particle-based fluid simulation for interactive applications.. In Symposium on Computer animation.Google Scholar
26. Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. 2007. Position based dynamics. Journal of Visual Communication and Image Representation 18, 2 (2007).Google ScholarDigital Library
27. Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. 2018. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In ICRA 2018. IEEE.Google ScholarDigital Library
28. Francis Ogoke and Amir Barati Farimani. 2021. Thermal control of laser powder bed fusion using deep reinforcement learning. Additive Manufacturing 46 (2021).Google Scholar
29. Junhyuk Oh, Satinder Singh, and Honglak Lee. 2017. Value Prediction Network. In NIPS.Google Scholar
30. Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. 37, 4 (2018).Google ScholarDigital Library
31. Connor Schenck and Dieter Fox. 2018. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning. PMLR, 317–335.Google Scholar
32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).Google Scholar
33. David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. 2017. The predictron: End-to-end learning and planning. In International Conference on Machine Learning. PMLR, 3191–3199.Google Scholar
34. Pitchaya Sitthi-Amorn, Javier E Ramos, Yuwang Wangy, Joyce Kwan, Justin Lan, Wenshou Wang, and Wojciech Matusik. 2015. MultiFab: a machine vision assisted platform for multi-material 3D printing. ACM Trans. Graph. 34, 4 (2015), 1–11.Google ScholarDigital Library
35. Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. 2018. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning. PMLR, 4732–4741.Google Scholar
36. Chao Tang, Jie Lun Tan, and Chee How Wong. 2018. A numerical investigation on the physical mechanisms of single track defects in selective laser melting. International Journal of Heat and Mass Transfer 126 (2018), 957–968.Google ScholarCross Ref
37. Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 5026–5033.Google ScholarCross Ref
38. Marc A Toussaint, Kelsey Rebecca Allen, Kevin A Smith, and Joshua B Tenenbaum. 2018. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics: Science and Systems Foundation (2018).Google Scholar
39. Chengcheng Wang, Xipeng Tan, Erjia Liu, and Shu Beng Tor. 2018. Process parameter optimization and mechanical properties for additively manufactured stainless steel 316L parts by selective electron beam melting. Materials & Design 147 (2018).Google Scholar
40. Chengcheng Wang, XP Tan, SB Tor, and CS Lim. 2020. Machine learning in additive manufacturing: State-of-the-art and perspectives. Additive Manufacturing (2020).Google Scholar
41. Jun Wu, Niels Aage, Rüdiger Westermann, and Ole Sigmund. 2018. Infill Optimization for Additive Manufacturing—Approaching Bone-Like Porous Structures. IEEE Transactions on Visualization and Computer Graphics 24, 2 (2018), 1127–1140.Google ScholarCross Ref
42. Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. 2019. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv:1910.13439 (2019).Google Scholar
43. Jie Xu, Tao Du, Michael Foshey, Beichen Li, Bo Zhu, Adriana Schulz, and Wojciech Matusik. 2019. Learning to fly: computational controller design for hybrid UAVs with reinforcement learning. ACM Trans. Graph. 38, 4 (2019).Google ScholarDigital Library
44. Wentao Yan, Ya Qian, Wenjun Ge, Stephen Lin, Wing Kam Liu, Feng Lin, and Gregory J Wagner. 2018. Meso-scale modeling of multiple-layer fabrication process in selective electron beam melting: inter-layer/track voids formation. Materials & Design 141 (2018), 210–219.Google ScholarCross Ref
45. Bing Yao, Farhad Imani, and Hui Yang. 2018. Markov decision process for image-guided additive manufacturing. IEEE Robotics and Automation Letters 3, 4 (2018), 2792–2798.Google ScholarCross Ref
46. Ri Yu, Hwangpil Park, and Jehee Lee. 2019. Figure skating simulation from video. In Computer graphics forum, Vol. 38. Wiley Online Library, 225–234.Google Scholar
47. Yunbo Zhang, Wenhao Yu, C Karen Liu, Charlie Kemp, and Greg Turk. 2020. Learning to manipulate amorphous materials. ACM Trans. Graph. 39, 6 (2020).Google ScholarDigital Library
48. Haisen Zhao, Fanglin Gu, Qi-Xing Huang, Jorge Garcia, Yong Chen, Changhe Tu, Bedrich Benes, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. 2016. Connected Fermat Spirals for Layered Fabrication. ACM Trans. Graph. 35, 4 (jul 2016).Google ScholarDigital Library
49. Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A Dataset of 10,000 3D-Printing Models. arXiv preprint arXiv:1605.04797 (2016).Google Scholar