“End-to-end Procedural Material Capture with Proxy-Free Mixed-Integer Optimization” by Li, Matusik and Shi
Conference:
Type(s):
Title:
- End-to-end Procedural Material Capture with Proxy-Free Mixed-Integer Optimization
Session/Category Title: Material Acquisition
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Node-graph-based procedural materials are vital to 3D content creation within the computer graphics industry. Leveraging the expressive representation of procedural materials, artists can effortlessly generate diverse appearances by altering the graph structure or node parameters. However, manually reproducing a specific appearance is a challenging task that demands extensive domain knowledge and labor. Previous research has sought to automate this process by converting artist-created material graphs into differentiable programs and optimizing node parameters against a photographed material appearance using gradient descent. These methods involve implementing differentiable filter nodes [Shi et al. 2020] and training differentiable neural proxies for generator nodes to optimize continuous and discrete node parameters [Hu et al. 2022a] jointly. Nevertheless, Neural Proxies exhibits critical limitations, such as long training times, inaccuracies, fixed resolutions, and confined parameter ranges, which hinder their scalability towards the broad spectrum of production-grade material graphs. These constraints fundamentally stem from the absence of faithful and efficient implementations of generic noise and pattern generator nodes, both differentiable and non-differentiable. Such deficiency prevents the direct optimization of continuous and discrete generator node parameters without relying on surrogate models.We present Diffmat v2, an improved differentiable procedural material library, along with a fully-automated, end-to-end procedural material capture framework that combines gradient-based optimization and gradient-free parameter search to match existing production-grade procedural materials against user-taken flash photos. Diffmat v2 expands the range of differentiable material graph nodes in Diffmat [Shi et al. 2020] by adding generic noise/pattern generator nodes and user-customizable per-pixel filter nodes. This allows for the complete translation and optimization of procedural materials across various categories without the need for external proprietary tools or pre-cached noise patterns. Consequently, our method can capture a considerably broader array of materials, encompassing those with highly regular or stochastic geometries. We demonstrate that our end-to-end approach yields a closer match to the target than MATch [Shi et al. 2020] and Neural Proxies [Hu et al. 2022a] when starting from initially unmatched continuous and discrete parameters.
References:
1. Substance 3D Designer Adobe. 2022a. Adobe. https://substance3d.adobe.com/documentation/sddoc/substance-3d-designer-102400008.html.
2. Substance 3D Designer Function Nodes Overview Adobe. 2022b. Adobe. https://substance3d.adobe.com/documentation/sddoc/function-nodes-overview-102400052.html.
3. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2013. Practical SVBRDF capture in the frequency domain. ACM Trans. Graph. 32, 4 (July 2013), 1–12.
4. Miika Aittala, Tim Weyrich, and Jaakko Lehtinen. 2015. Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. 34, 4 (July 2015), 1–13.
5. Ken-ichi Anjyo. 1988. A Simple Spectral Approach to Stochastic Modelling for Natural Objects. In EG 1988-Technical Papers. Eurographics Association.
6. Louis-Philippe Asselin, Denis Laurendeau, and Jean-Francois Lalonde. 2020. Deep SVBRDF Estimation on Real Materials.
7. Sai Praveen Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39, 6 (Nov. 2020), 1–18.
8. Pietro Belotti, Christian Kirches, Sven Leyffer, Jeff Linderoth, James Luedtke, and Ashutosh Mahajan. 2013. Mixed-integer nonlinear optimization. Acta Numerica 22 (2013), 1–131.
9. Robert L Cook and Tony DeRose. 2005. Wavelet noise. ACM Trans. Graph. 24, 3 (July 2005), 803–811.
10. Kristin J Dana, Bram van Ginneken, Shree K Nayar, and Jan J Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Trans. Graph. 18, 1 (Jan. 1999), 1–34.
11. Sebastien Deguy, Albert Benassi, and Université Blaise Pascal. 2001. A Flexible Noise Model For Designing Maps. http://citeseerx.ist.psu.edu › viewdoc › summary- http://citeseerx.ist.psu.edu › viewdoc › summary (2001).
12. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image SVBRDF capture with a rendering-aware deep network. ACM Trans. Graph. 37, 4 (Aug. 2018), 1–15.
13. Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2019. Flexible SVBRDF capture with a multi-image deep network. Comput. Graph. Forum 38, 4 (July 2019), 1–13.
14. J-M Dischler, K Maritaud, B Lévy, and D Ghazanfarpour. 2002. Texture particles. Comput. Graph. Forum 21, 3 (Sept. 2002), 401–410.
15. Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. 2014. Appearance-from-motion: recovering spatially varying surface reflectance under unknown lighting. ACM Trans. Graph. 33, 6 (Nov. 2014), 1–12.
16. Yue Dong, Jiaping Wang, Xin Tong, John Snyder, Yanxiang Lan, Moshe Ben-Ezra, and Baining Guo. 2010. Manifold bootstrapping for SVBRDF capture. In ACM SIGGRAPH 2010 papers (Los Angeles, California) (SIGGRAPH ’10, Article 98). Association for Computing Machinery, New York, NY, USA, 1–10.
17. David S. Ebert, F. Kenton Musgrave, Darwyn Peachey, Ken Perlin, and Steven Worley. 2002. Texturing and Modeling: A Procedural Approach (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
18. Christian Eisenacher, Chuck Tappan, Brent Burley, Daniel Teece, and Arthur Shek. 2010. Example-based texture synthesis on Disney’s Tangled. In ACM SIGGRAPH 2010 Talks (Los Angeles, California) (SIGGRAPH ’10, Article 32). Association for Computing Machinery, New York, NY, USA, 1.
19. Alain Fournier, Don Fussell, and Loren Carpenter. 1982. Computer rendering of stochastic models. Commun. ACM 25, 6 (June 1982), 371–384.
20. Bruno Galerne, Ares Lagae, Sylvain Lefebvre, and George Drettakis. 2012. Gabor noise by example. ACM Trans. Graph. 31, 4 (July 2012), 1–9.
21. B Galerne, A Leclaire, and L Moisan. 2017. Texton noise. Comput. Graph. Forum 36, 8 (Dec. 2017), 205–218.
22. Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. 2019. Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Trans. Graph. 38, 4 (July 2019), 1–15.
23. Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. 2003. Linear light source reflectometry. ACM Trans. Graph. 22, 3 (July 2003), 749–758.
24. Geoffrey Y Gardner. 1984. Simulation of natural scenes using textured quadric surfaces. SIGGRAPH Comput. Graph. 18, 3 (Jan. 1984), 11–20.
25. Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture Synthesis Using Convolutional Neural Networks. In Advances in Neural Information Processing Systems, C Cortes, N Lawrence, D Lee, M Sugiyama, and R Garnett (Eds.), Vol. 28. Curran Associates, Inc.
26. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A Wilson, and Paul Debevec. 2009. Estimating specular roughness and anisotropy from second order spherical gradient illumination. Comput. Graph. Forum 28, 4 (June 2009), 1161–1170.
27. Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A Wilson, and Paul Debevec. 2010. Circularly polarized spherical illumination reflectometry. In ACM SIGGRAPH Asia 2010 papers (Seoul, South Korea) (SIGGRAPH ASIA ’10, Article 162). Association for Computing Machinery, New York, NY, USA, 1–12.
28. Abhijeet Ghosh, Tim Hawkins, Pieter Peers, Sune Frederiksen, and Paul Debevec. 2008. Practical modeling and acquisition of layered facial reflectance.
29. Guillaume Gilet, Basile Sauvage, Kenneth Vanhoey, Jean-Michel Dischler, and Djamchid Ghazanfarpour. 2014. Local random-phase noise for procedural texturing. ACM Trans. Graph. 33, 6 (Nov. 2014), 1–11.
30. Alexander Goldberg, Matthias Zwicker, and Frédo Durand. 2008. Anisotropic noise. ACM Trans. Graph. 27, 3 (Aug. 2008), 1–8.
31. Dan B Goldman, Brian Curless, Aaron Hertzmann, and Steven M Seitz. 2010. Shape and spatially-varying BRDFs from photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32, 6 (June 2010), 1060–1071.
32. Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Měch, Tamy Boubekeur, and Niloy J Mitra. 2022. MatFormer: a generative model for procedural materials. ACM Trans. Graph. 41, 4 (July 2022), 1–12.
33. Jie Guo, Shuichang Lai, Chengzhi Tao, Yuelong Cai, Lei Wang, Yanwen Guo, and Ling-Qi Yan. 2021. Highlight-aware two-stream network for single-image SVBRDF acquisition. ACM Trans. Graph. 40, 4 (July 2021), 1–14.
34. Yu Guo, Milos Hasan, Lingqi Yan, and Shuang Zhao. 2019. A Bayesian Inference Framework for Procedural Material Parameter Estimation. (Dec. 2019). arXiv:1912.01067 [cs.GR]
35. Yu Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. 2020. MaterialGAN: reflectance capture using a generative SVBRDF model. ACM Trans. Graph. 39, 6 (Nov. 2020), 1–13.
36. Eric Heitz and Fabrice Neyret. 2018. High-Performance By-Example Noise using a Histogram-Preserving Blending Operator. Proc. ACM Comput. Graph. Interact. Tech. 1, 2 (Aug. 2018), 1–25.
37. Philipp Henzler, Valentin Deschaintre, Niloy J Mitra, and Tobias Ritschel. 2021. Generative modelling of BRDF textures from flash images. ACM Trans. Graph. 40, 6 (Dec. 2021), 1–13.
38. Michael Holroyd, Jason Lawrence, and Todd Zickler. 2010. A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance. ACM Trans. Graph. 29, 4 (July 2010), 1–12.
39. Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019a. A novel framework for inverse procedural texture modeling. ACM Trans. Graph. 38, 6 (Nov. 2019), 1–14.
40. Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2022a. Node Graph Optimization Using Differentiable Proxies. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22, Article 5). Association for Computing Machinery, New York, NY, USA, 1–9.
41. Yiwei Hu, Chengan He, Valentin Deschaintre, Julie Dorsey, and Holly Rushmeier. 2022b. An Inverse Procedural Modeling Pipeline for SVBRDF Maps. ACM Trans. Graph. 41, 2 (Jan. 2022), 1–17.
42. Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. 2019b. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG) 38, 6 (2019), 201.
43. Zhuo Hui, Kalyan Sunkavalli, Joon-Young Lee, Sunil Hadap, Jian Wang, and Aswin C Sankaranarayanan. 2017. Reflectance Capture Using Univariate Sampling of BRDFs.
44. Kaizhang Kang, Cihui Xie, Chengan He, Mingqi Yi, Minyi Gu, Zimin Chen, Kun Zhou, and Hongzhi Wu. 2019. Learning efficient illumination multiplexing for joint capture of reflectance and shape. ACM Trans. Graph. 38, 6 (Dec. 2019), 1–12.
45. Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. 2020. Differentiable Rendering: A Survey. (June 2020). arXiv:2006.12057 [cs.CV]
46. A Lagae, S Lefebvre, R Cook, T DeRose, G Drettakis, D S Ebert, J P Lewis, K Perlin, and M Zwicker. 2010a. A survey of procedural noise functions. Comput. Graph. Forum 29, 8 (Dec. 2010), 2579–2600.
47. Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. 2009. Procedural noise using sparse Gabor convolution. In ACM SIGGRAPH 2009 papers (New Orleans, Louisiana) (SIGGRAPH ’09, Article 54). Association for Computing Machinery, New York, NY, USA, 1–10.
48. Ares Lagae, Peter Vangorp, Toon Lenaerts, and Philip Dutré. 2010b. Procedural isotropic stochastic textures by example. Comput. Graph. 34, 4 (Aug. 2010), 312–321.
49. Jason Lawrence, Aner Ben-Artzi, Christopher DeCoro, Wojciech Matusik, Hanspeter Pfister, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2006. Inverse shade trees for non-parametric material representation and editing. ACM Trans. Graph. 25, 3 (July 2006), 735–745.
50. Jon Lee and Sven Leyffer. 2011. Mixed integer nonlinear programming. Vol. 154. Springer Science & Business Media.
51. L Lefebvre and Pierre Poulin. 2000. Analysis and Synthesis of Structural Textures. Graphics Interface (2000).
52. Sylvain Lefebvre, Samuel Hornus, and Anass Lasram. 2010. By-example synthesis of architectural textures. In ACM SIGGRAPH 2010 papers (Los Angeles, California) (SIGGRAPH ’10, Article 84). Association for Computing Machinery, New York, NY, USA, 1–8.
53. John-Peter Lewis. 1984. Texture synthesis for digital painting. SIGGRAPH Comput. Graph. 18, 3 (Jan. 1984), 245–252.
54. J P Lewis. 1986. Methods for stochastic spectral synthesis. In Proceedings on Graphics Interface ’86/Vision Interface ’86 (Vancouver, British Columbia, Canada). Canadian Information Processing Society, CAN, 173–179.
55. J P Lewis. 1987. Generalized stochastic subdivision. ACM Trans. Graph. 6, 3 (July 1987), 167–190.
56. J P Lewis. 1989. Algorithms for solid noise synthesis. SIGGRAPH Comput. Graph. 23, 3 (July 1989), 263–270.
57. T M Li, M Aittala, F Durand, and J Lehtinen. 2018a. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (2018).
58. Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2017. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. 36, 4 (July 2017), 1–11.
59. Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. 2018b. Materials for masses: SVBRDF acquisition with a single mobile phone image. In Proceedings of the European Conference on Computer Vision (ECCV). 72–87.
60. Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2018c. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. 37, 6 (Dec. 2018), 1–11.
61. Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (Nov. 2019), 1–14.
62. Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. 2018. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. 37, 6 (Dec. 2018), 1–12.
63. Giljoo Nam, Joo Ho Lee, Hongzhi Wu, Diego Gutierrez, and Min H Kim. 2016. Simultaneous acquisition of microscale reflectance and normals. ACM Trans. Graph. 35, 6 (Nov. 2016), 1–11.
64. M Nimier-David, D Vicini, T Zeltner, and others. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on (2019).
65. M Olano, Hartj. C, W Heidrich, B Mark, and K Perlin. 2002. Real-time shading languages. ACM SIGGRAPH 2002 Course (2002).
66. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32 (2019).
67. K Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics (1985).
68. K Perlin. 2002. Improving noise. Proceedings of the 29th annual conference on (2002).
69. Emil Praun, Adam Finkelstein, and Hugues Hoppe. 2000. Lapped textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’00). ACM Press/Addison-Wesley Publishing Co., USA, 465–470.
70. Peiran Ren, Jiaping Wang, John Snyder, Xin Tong, and Baining Guo. 2011. Pocket reflectometry. ACM Trans. Graph. 30, 4 (July 2011), 1–10.
71. J Riviere, P Peers, and A Ghosh. 2016. Mobile surface reflectometry. Comput. Graph. Forum 35, 1 (Feb. 2016), 191–202.
72. Dietmar Saupe. 1988. Algorithms for random fractals. In The Science of Fractal Images, Michael F Barnsley, Robert L Devaney, Benoit B Mandelbrot, Heinz-Otto Peitgen, Dietmar Saupe, Richard F Voss, Heinz-Otto Peitgen, and Dietmar Saupe (Eds.). Springer New York, New York, NY, 71–136.
73. Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020. Match: differentiable material graphs for procedural material capture. ACM Trans. Graph. 39, 6 (Nov. 2020), 1–15.
74. Borom Tunwattanapong, Graham Fyffe, Paul Graham, Jay Busch, Xueming Yu, Abhijeet Ghosh, and Paul Debevec. 2013. Acquiring reflectance and shape from continuous spherical harmonic illumination. ACM Trans. Graph. 32, 4 (July 2013), 1–12.
75. Jarke J van Wijk. 1991. Spot noise texture synthesis for data visualization. SIGGRAPH Comput. Graph. 25, 4 (July 1991), 309–318.
76. Richard F Voss. 1988. Fractals in nature: From characterization to simulation. In The Science of Fractal Images, Michael F Barnsley, Robert L Devaney, Benoit B Mandelbrot, Heinz-Otto Peitgen, Dietmar Saupe, Richard F Voss, Heinz-Otto Peitgen, and Dietmar Saupe (Eds.). Springer New York, New York, NY, 21–70.
77. Jiaping Wang, Shuang Zhao, Xin Tong, John Snyder, and Baining Guo. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Trans. Graph. 27, 3 (Aug. 2008), 1–9.
78. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera. IEEE Trans. Vis. Comput. Graph. 22, 8 (Aug. 2016), 2012–2023.
79. Geoff Wyvill and Kevin Novins. 1999. Filtered noise and the fourth dimension. In ACM SIGGRAPH 99 Conference abstracts and applications (Los Angeles, California, USA) (SIGGRAPH ’99). Association for Computing Machinery, New York, NY, USA, 242.
80. Zexiang Xu, Jannik Boll Nielsen, Jiyang Yu, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Minimal BRDF sampling for two-shot near-field reflectance acquisition. ACM Trans. Graph. 35, 6 (Nov. 2016), 1–12.
81. Wenjie Ye, Xiao Li, Yue Dong, Pieter Peers, and Xin Tong. 2018. Single image surface appearance modeling with self-augmented CNNs and inexact supervision. Comput. Graph. Forum 37, 7 (Oct. 2018), 201–211.
82. Yu-Ying Yeh, Zhengqin Li, Yannick Hold-Geoffroy, Rui Zhu, Zexiang Xu, Milos Hasan, Kalyan Sunkavalli, and Manmohan Chandraker. 2022. PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes. (June 2022).
83. Jiyang Yu, Zexiang Xu, Matteo Mannino, Henrik Wann Jensen, and Ravi Ramamoorthi. 2016. Sparse sampling for image-based SVBRDF acquisition.
84. C Zhang, B Miller, K Yan, I Gkioulekas, and others. 2020. Path-space differentiable rendering. ACM transactions on (2020).
85. S Zhao, W Jakob, and T M Li. 2020. Physics-based differentiable rendering: from theory to implementation. ACM SIGGRAPH 2020 Courses (2020).
86. Xilong Zhou and Nima Khademi Kalantari. 2021. Adversarial single-image SVBRDF estimation with hybrid training. Comput. Graph. Forum 40, 2 (May 2021), 315–325.
87. Zhiming Zhou, Guojun Chen, Yue Dong, David Wipf, Yong Yu, John Snyder, and Xin Tong. 2016. Sparse-as-possible SVBRDF acquisition. ACM Trans. Graph. 35, 6 (Nov. 2016), 1–12.
