“One Noise to Rule Them All: Learning a Unified Model of Spatially-Varying Noise Patterns”
Conference:
Type(s):
Title:
- One Noise to Rule Them All: Learning a Unified Model of Spatially-Varying Noise Patterns
Presenter(s)/Author(s):
Abstract:
We present a generative model that can synthesize a diverse array of spatially-varying noise patterns, even without spatially-varying training data. Our model offers a versatile and controllable way to generate noise textures for computer graphics applications. We also demonstrate its utility in inverse procedural material design.
References:
[1]
Adobe. 2023a. Adobe Substance 3D Designer. https://www.adobe.com/products/substance3d-designer.html.
[2]
Adobe. 2023b. Adobe Substance 3D Documentation. https://helpx.adobe.com/substance-3d-designer/substance-compositing-graphs/nodes-reference-for-substance-compositing-graphs/node-library.html.
[3]
Martin Arjovsky, Soumith Chintala, and L?on Bottou. 2017. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 214–223. https://proceedings.mlr.press/v70/arjovsky17a.html
[4]
Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. 2017. Learning Texture Manifolds with the Periodic Spatial GAN. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 469–477. https://proceedings.mlr.press/v70/bergmann17a.html
[5]
Blender. 2023. Blender. https://www.blender.org.
[6]
Robert L Cook and Tony DeRose. 2005. Wavelet noise. ACM Transactions on Graphics (TOG) 24, 3 (2005), 803–811.
[7]
Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. 2012. Image Melding: Combining Inconsistent Images using Patch-based Synthesis. ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH 2012) 31, 4, Article 82 (2012), 82:1–82:10 pages.
[8]
Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 (2021), 8780–8794.
[9]
Julie Dorsey and Pat Hanrahany. 2006. Modeling and rendering of metallic patinas. In ACM SIGGRAPH 2006 Courses. 2–es.
[10]
David Ebert, Kenton Musgrave, Darwyn Peachey, Ken Perlin, and Steve Worley. 2002. Texturing And Modeling. A Procedural Approach (3 ed.). Morgan Kaufmann.
[11]
Alain Fournier, Don Fussell, and Loren Carpenter. 1982. Computer rendering of stochastic models. Commun. ACM 25, 6 (1982), 371–384.
[12]
Eric Galin, Eric Gu?rin, Adrien Peytavie, Guillaume Cordonnier, Marie-Paule Cani, Bedrich Benes, and James Gain. 2019. A review of digital terrain modeling. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 553–577.
[13]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015. Texture Synthesis Using Convolutional Neural Networks. In NeurIPS.
[14]
Jean-David G?nevaux, ?ric Galin, Eric Gu?rin, Adrien Peytavie, and Bedrich Benes. 2013. Terrain generation using procedural models based on hydrology. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1–13.
[15]
Paul Guerrero, Milo? Ha?an, Kalyan Sunkavalli, Radom?r M?ch, Tamy Boubekeur, and Niloy J Mitra. 2022. MatFormer: A generative model for procedural materials. arXiv preprint arXiv:2207.01044 (2022).
[16]
Yu Guo, Milo? Ha?an, Lingqi Yan, and Shuang Zhao. 2020. A bayesian inference framework for procedural material parameter estimation. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 255–266.
[17]
Nicholas Guttenberg. 2023. Diffusion with Offset Noise. (2023).
[18]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).
[19]
Damien Hinsinger, Fabrice Neyret, and Marie-Paule Cani. 2002. Interactive animation of ocean waves. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 161–166.
[20]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic
[21]
models. Advances in neural information processing systems 33 (2020), 6840–6851.
[22]
Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019. A novel framework for inverse procedural texture modeling. ACM Transactions on Graphics (ToG) 38, 6 (2019), 1–14.
[23]
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2022. Node graph optimization using differentiable proxies. In ACM SIGGRAPH 2022 conference proceedings. 1–9.
[24]
Zhizhong Huang, Junping Zhang, Yi Zhang, and Hongming Shan. 2021. DU-GAN: Generative adversarial networks with dual-domain U-Net-based discriminators for low-dose CT denoising. IEEE Transactions on Instrumentation and Measurement 71 (2021), 1–12.
[25]
Michael J Jones and Tomaso Poggio. 1998. Multidimensional morphable models. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271). IEEE, 683–688.
[26]
Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. 2020. UCSG-NET- Unsupervised Discovering of Constructive Solid Geometry Tree. In Advances in Neural Information Processing Systems.
[27]
Ares Lagae, Sylvain Lefebvre, Rob Cook, Tony DeRose, George Drettakis, David S Ebert, John P Lewis, Ken Perlin, and Matthias Zwicker. 2010. A survey of procedural noise functions. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 2579–2600.
[28]
Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutr?. 2009. Procedural noise using sparse Gabor convolution. ACM Transactions on Graphics (TOG) 28, 3 (2009), 1–10.
[29]
Chuan Li and Michael Wand. 2016. Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 702–716.
[30]
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. 2024. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 5404–5411.
[31]
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. 2023. InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation. arXiv:2309.06380 [cs.LG]
[32]
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
[33]
Wojciech Matusik, Matthias Zwicker, and Fr?do Durand. 2005. Texture design using a simplicial complex of morphable textures. ACM Transactions on Graphics (TOG) 24, 3 (2005), 787–794.
[34]
Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning. PMLR, 8162–8171.
[35]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis with Spatially-Adaptive Normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
[36]
Ken Perlin. 1985. An Image Synthesizer. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’85). Association for Computing Machinery, New York, NY, USA, 287–296.
[37]
Ken Perlin. 2002. Improving noise. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 681–682.
[38]
Lara Raad, Axel Davy, Agn?s Desolneux, and Jean-Michel Morel. 2018. A survey of exemplar-based texture synthesis. Annals of Mathematical Sciences and Applications 3, 1 (2018), 89–148.
[39]
Daxuan Ren, Jianmin Zheng, Jianfei Cai, Jiatong Li, Haiyong Jiang, Zhongang Cai, Junzhe Zhang, Liang Pan, Mingyuan Zhang, Haiyu Zhao, et al. 2021. CSG-Stump: A Learning Friendly CSG-Like Representation for Interpretable Shape Parsing. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12478–12487.
[40]
Carlos Rodriguez-Pardo and Elena Garces. 2023. SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps. IEEE Transactions on Visualization and Computer Graphics 29, 6 (jun 2023), 2914–2925.
[41]
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. 2020. A u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8207–8216.
[42]
Omry Sendik and Daniel Cohen-Or. 2017. Deep Correlations for Texture Synthesis. ACM Trans. Graph. 36, 5, Article 161 (jul 2017), 15 pages.
[43]
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. 2021. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. 3531–3539.
[44]
Liang Shi, Beichen Li, Milo? Ha?an, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020. MATch: Differentiable Material Graphs for Procedural Material Capture. ACM Trans. Graph. 39, 6 (Dec. 2020), 1–15.
[45]
Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising Diffusion Implicit Models. arXiv:2010.02502 (October 2020). https://arxiv.org/abs/2010.02502
[46]
Yang Song. 2021. Generative Modeling by Estimating Gradients of the Data Distribution. https://yang-song.net/blog/2021/score/.
[47]
Thibault Tricard, Semyon Efremov, C?dric Zanni, Fabrice Neyret, Jon?s Mart?nez, and Sylvain Lefebvre. 2019. Procedural phasor noise. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–13.
[48]
Dmitry Ulyanov, Vadim Lebedev, Vedaldi Andrea, and Victor Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1349–1357.
[49]
Soobin Um, Suhyeon Lee, and Jong Chul Ye. 2024. Don’t Play Favorites: Minority Guidance for Diffusion Models. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=3NmO9lY4Jn
[50]
Roland Van Der Linden, Ricardo Lopes, and Rafael Bidarra. 2013. Procedural generation of dungeons. IEEE Transactions on Computational Intelligence and AI in Games 6, 1 (2013), 78–89.
[51]
Steven Worley. 1996. A cellular texture basis function. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96). Association for Computing Machinery, New York, NY, USA, 291–294.
[52]
Yuxin Wu and Kaiming He. 2018. Group Normalization. In Computer Vision – ECCV 2018 – 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XIII (Lecture Notes in Computer Science, Vol. 11217), Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, 3–19.
[53]
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. 2022. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In ICLR.
[54]
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In International Conference on Computer Vision (ICCV).
[55]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[56]
Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Nima Khademi Kalantari. 2023b. PhotoMat: A Material Generator Learned from Single Flash Photos. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
[57]
Yang Zhou, Kaijian Chen, Rongjun Xiao, and Hui Huang. 2023a. Neural Texture Synthesis with Guided Correspondence. In Conference on Computer Vision and Pattern Recognition (CVPR). 18095–18104.
[58]
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018. Non-stationary Texture Synthesis by Adversarial Expansion. ACM Transactions on Graphics (Proc. SIGGRAPH) 37, 4 (2018), 49:1–49:13.
