“A gradient-based framework for 3D print appearance optimization” by Nindel, Iser, Rittig, Wilkie and Křivánek

In full-color inkjet 3D printing, a key problem is determining the material configuration for the millions of voxels that a printed object is made of. The goal is a configuration that minimises the difference between desired target appearance and the result of the printing process. So far, the techniques used to find such a configuration have relied on domain-specific methods or heuristic optimization, which allowed only a limited level of control over the resulting appearance.We propose to use differentiable volume rendering in a continuous material-mixture space, which leads to a framework that can be used as a general tool for optimising inkjet 3D printouts. We demonstrate the technical feasibility of this approach, and use it to attain fine control over the fabricated appearance, and high levels of faithfulness to the specified target.

1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. http://tensorflow.org/ Software available from tensorflow.org.Google Scholar
2. Navid Ansari, Omid Alizadeh-Mousavi, Hans-Peter Seidel, and Vahid Babaei. 2020. Mixed integer ink selection for spectral reproduction. ACM Transactions on Graphics (Proc. SIGGRAPH Asia) 39, 6 (Nov. 2020), 255:1–255:16. Google ScholarDigital Library
3. Thomas Auzinger, Wolfgang Heidrich, and Bernd Bickel. 2018. Computational design of nanostructural color for additive manufacturing. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (July 2018), 159:1–159:16. Google ScholarDigital Library
4. Vahid Babaei, Kiril Vidimče, Michael Foshey, Alexandre Kaspar, Piotr Didyk, and Wojciech Matusik. 2017. Color Contoning for 3D Printing. ACM Transactions on Graphics (Proc. SIGGRAPH) 36, 4 (July 2017), 124:1–124:15. Google ScholarDigital Library
5. Atılım Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2017. Automatic Differentiation in Machine Learning: a Survey. Journal of Machine Learning Research 18, 1 (2017), 5595–5637.Google ScholarDigital Library
6. Alan Brunton, Can Ates Arikan, Tejas Madan Tanksale, and Philipp Urban. 2018. 3D Printing Spatially Varying Color and Translucency. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (July 2018), 157:1–157:13. Google ScholarDigital Library
7. Alan Brunton, Can Ates Arikan, and Philipp Urban. 2015. Pushing the Limits of 3D Color Printing: Error Diffusion with Translucent Materials. ACM Transactions on Graphics 35, 1 (Dec. 2015), 4:1–4:13. Google ScholarDigital Library
8. Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2020. Towards Learning-based Inverse Subsurface Scattering. In 2020 IEEE International Conference on Computational Photography, ICCP 2020, Saint Louis, MO, USA, April 24-26, 2020. IEEE, New York, NY, USA, 1–12. Google ScholarCross Ref
9. Guan-Hao Chen, Chun-Ling Yang, Lai-Man Po, and Sheng-Li Xie. 2006. Edge-Based Structural Similarity for Image Quality Assessment. In IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, Vol. 2. IEEE, New York, NY, USA, 933–936.Google Scholar
10. Yue Dong, Jiaping Wang, Fabio Pellacini, Xin Tong, and Baining Guo. 2010. Fabricating spatially-varying subsurface scattering. ACM Transactions on Graphics (Proc. SIGGRAPH) 29, 4 (July 2010), 62:1–62:10. Google ScholarDigital Library
11. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research 12, Jul (2011), 2121–2159.Google ScholarDigital Library
12. John Duchi, Shai Shalev-Shwartz, Yoram Singer, and Tushar Chandra. 2008. Efficient Projections onto the L1-Ball for Learning in High Dimensions. In 25th International Conference on Machine Learning (Helsinki, Finland) (ICML ’08). Association for Computing Machinery, New York, NY, USA, 272–279. Google ScholarDigital Library
13. Oskar Elek, Denis Sumin, Ran Zhang, Tim Weyrich, Karol Myszkowski, Bernd Bickel, Alexander Wilkie, and Jaroslav Křivánek. 2017. Scattering-aware Texture Reproduction for 3D Printing. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 36, 6 (Nov. 2017), 241:1–241:15. Google ScholarDigital Library
14. Roland W Fleming and Heinrich H Bülthoff. 2005. Low-level image cues in the perception of translucent materials. ACM Transactions on Applied Perception (TAP) 2, 3 (2005), 346–382.Google ScholarDigital Library
15. J. R. Frisvad, S. A. Jensen, J. S. Madsen, A. Correia, L. Yang, S. K. S. Gregersen, Y. Meuret, and P.-E. Hansen. 2020. Survey of Models for Acquiring the Optical Properties of Translucent Materials. Computer Graphics Forum 39, 2 (2020), 729–755. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14023 Google ScholarCross Ref
16. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. In European Conference on Computer Vision. Springer, Berlin, Germany, 685–701. Google ScholarCross Ref
17. Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse volume rendering with material dictionaries. ACM Transactions on Graphics (TOG) 32, 6 (2013), 162:1–162:13. Google ScholarDigital Library
18. Miloš Hašan, Martin Fuchs, Wojciech Matusik, Hanspeter Pfister, and Szymon Rusinkiewicz. 2010. Physical Reproduction of Materials with Specified Subsurface Scattering. ACM Trans. Graph. 29, 4, Article 61 (July 2010), 10 pages. Google ScholarDigital Library
19. Roman Hochuli, Samuel Powell, Simon Arridge, and Ben Cox. 2016. Quantitative photoacoustic tomography using forward and adjoint Monte Carlo models of radiance. Journal of biomedical optics 21, 12 (2016), 126004.Google ScholarCross Ref
20. Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Trans. Graph. 35, 1 (2015), 1–1.Google ScholarDigital Library
21. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG) 37, 6 (July 2018), 125:1–125:12. Google ScholarDigital Library
22. M Ronnier Luo, Guihua Cui, and Bryan Rigg. 2001. The development of the CIE 2000 colour-difference formula: CIEDE2000. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 26, 5 (2001), 340–350.Google Scholar
23. A. Luongo, V. Falster, M. B. Doest, M. M. Ribo, E. R. Eiriksson, D. B. Pedersen, and J. R. Frisvad. 2020. Microstructure Control in 3D Printing with Digital Light Processing. Computer Graphics Forum 39, 1 (2020), 347–359. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13807 Google ScholarCross Ref
24. M. Magnor, G. Kindlmann, N. Duric, and C. Hansen. 2004. Constrained inverse volume rendering for planetary nebulae. In IEEE Visualization 2004. IEEE, New York, NY, USA, 83–90. Google ScholarDigital Library
25. K McLaren. 1976. XIII – The development of the CIE 1976 (L* a* b*) uniform colour space and colour-difference formula. Journal of the Society of Dyers and Colourists 92, 9 (1976), 338–341.Google ScholarCross Ref
26. Peter Morovič, Ján Morovič, Ingeborg Tastl, Melanie Gottwals, and Gary Dispoto. 2019. Co-optimization of color and mechanical properties by volumetric voxel control. Struct Multidisc Optim 60, 3 (Sept. 2019), 895–908. Google ScholarDigital Library
27. Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative Backpropagation: An Adjoint Method for Lightning-Fast Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 39, 4 (July 2020), 146:1–146:15. Google ScholarDigital Library
28. Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (Dec. 2019), 203:1–203:17. Google ScholarDigital Library
29. Isabel Molina Orihuela and Mehran Ebrahimi. 2019. An Efficient Algorithm for Computing the Derivative of Mean Structural Similarity Index Measure. In Image Analysis and Recognition, Fakhri Karray, Aurélio Campilho, and Alfred Yu (Eds.). Springer International Publishing, Cham, 55–66.Google Scholar
30. Marios Papas, Christian Regg, Wojciech Jarosz, Bernd Bickel, Philip Jackson, Wojciech Matusik, Steve Marschner, and Markus Gross. 2013. Fabricating translucent materials using continuous pigment mixtures. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–12.Google ScholarDigital Library
31. Michal Piovarči, Michael Foshey, Vahid Babaei, Szymon Rusinkiewicz, Wojciech Matusik, and Piotr Didyk. 2020. Towards spatially varying gloss reproduction for 3D printing. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–13.Google ScholarDigital Library
32. Jens Preiss, Felipe Fernandes, and Philipp Urban. 2014. Color-image quality assessment: From prediction to optimization. IEEE Transactions on Image Processing 23, 3 (2014), 1366–1378.Google ScholarDigital Library
33. Olivier Rouiller, Bernd Bickel, Jan Kautz, Wojciech Matusik, and Marc Alexa. 2013. 3D-printing spatially varying BRDFs. IEEE computer graphics and applications 33, 6 (2013), 48–57.Google ScholarDigital Library
34. Gaurav Sharma, Wencheng Wu, and Edul N Dalal. 2005. The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 30, 1 (2005), 21–30.Google Scholar
35. Kfir Shem-Tov, Sai Praveen Bangaru, Anat Levin, and Ioannis Gkioulekas. 2020. Towards Reflectometry from Interreflections. In 2020 IEEE International Conference on Computational Photography (ICCP). IEEE, New York, NY, USA, 1–12. Google ScholarCross Ref
36. Liang Shi, Vahid Babaei, Changil Kim, Michael Foshey, Yuanming Hu, Pitchaya Sitthi-Amorn, Szymon Rusinkiewicz, and Wojciech Matusik. 2018. Deep multispectral painting reproduction via multi-layer, custom-ink printing. ACM Trans. Graph. 37, 6 (Dec. 2018), 1–15. Google ScholarDigital Library
37. Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-Aware Scattering Compensation for 3D Printing. ACM Transactions on Graphics (Proc. SIGGRAPH) 38, 4 (July 2019), 111:1–111:14. Google ScholarDigital Library
38. Philipp Urban, Tejas Madan Tanksale, Alan Brunton, Bui Minh Vu, and Shigeki Nakauchi. 2019. Redefining A in RGBA: Towards a Standard for Graphical 3D Printing. ACM Trans. Graph. 38, 3 (June 2019), 21:1–21:14. Google ScholarDigital Library
39. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
40. Zhou Wang, Eero P Simoncelli, and Alan C Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. IEEE, New York, NY, USA, 1398–1402.Google ScholarCross Ref
41. Douglas R Wyman, Michael S Patterson, and Brian C Wilson. 1989a. Similarity relations for anisotropic scattering in Monte Carlo simulations of deeply penetrating neutral particles. J. Comput. Phys. 81, 1 (1989), 137–150.Google ScholarDigital Library
42. Douglas R Wyman, Michael S Patterson, and Brian C Wilson. 1989b. Similarity relations for the interaction parameters in radiation transport. Applied optics 28, 24 (1989), 5243–5249.Google Scholar
43. Bei Xiao, Bruce Walter, Ioannis Gkioulekas, Todd Zickler, Edward Adelson, and Kavita Bala. 2014. Looking against the light: How perception of translucency depends on lighting direction. Journal of vision 14, 3 (2014), 17–17.Google ScholarCross Ref
44. Bei Xiao, Shuang Zhao, Ioannis Gkioulekas, Wenyan Bi, and Kavita Bala. 2020. Effect of geometric sharpness on translucent material perception. Journal of vision 20, 7 (2020), 10–10.Google ScholarCross Ref
45. Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik. 2013. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Transactions on Image Processing 23, 2 (2013), 684–695.Google ScholarDigital Library
46. Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Transactions on Graphics (TOG) 38, 6 (Nov. 2019), 227:1–227:16. Google ScholarDigital Library
47. Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. 2011. FSIM: A feature similarity index for image quality assessment. IEEE transactions on Image Processing 20, 8 (2011), 2378–2386.Google Scholar
48. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Conf. Comp. Vision & Pat. Rec. (CVPR). IEEE, New York, NY, USA, 586–595.Google ScholarCross Ref
49. Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz. 2016a. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging 3, 1 (2016), 47–57.Google ScholarCross Ref
50. Shuang Zhao, Ravi Ramamoorthi, and Kavita Bala. 2014. High-order similarity relations in radiative transfer. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1–12.Google ScholarDigital Library
51. Shuang Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016b. Downsampling scattering parameters for rendering anisotropic media. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1–11.Google ScholarDigital Library
52. Quan Zheng, Vahid Babaei, Gordon Wetzstein, Hans-Peter Seidel, Matthias Zwicker, and Gurprit Singh. 2020. Neural Light Field 3D Printing. ACM Trans. Graph. 39, 6, Article 207 (Nov. 2020), 12 pages. Google ScholarDigital Library