“Mesh Neural Cellular Automata” by Pajouheshgar, Xu, Mordvintsev, Niklasson, Zhang, et al. …
Conference:
Type(s):
Title:
- Mesh Neural Cellular Automata
Presenter(s)/Author(s):
Abstract:
MeshNCA directly creates dynamic textures on 3D meshes without UV maps. MeshNCA’s training targets include PBR textures, text prompts, and motion-vector fields. Trained only on an Icosphere mesh, we create textures on unseen meshes and interactively edit the synthesized textures, both in real time. Our online demo showcases the capabilities.
References:
[1]
Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth, Michael J Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. International Journal of Computer Vision 92, 1 (2011), 1–31.
[2]
Alexey Bokhovkin, Shubham Tulsiani, and Angela Dai. 2023. Mesh2Tex: Generating Mesh Textures from Image Queries. arXiv preprint arXiv:2304.05868 (2023).
[3]
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR]. Stanford University — Princeton University — Toyota Technological Institute at Chicago.
[4]
Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019. Learning to predict 3d objects with an interpolation-based differentiable renderer. Advances in neural information processing systems 32 (2019).
[5]
Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, and Adrien Bousseau. 2018. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (ToG) 37, 4 (2018), 1–15.
[6]
K. C. Dharma, Clayton T. Morrison, and Bradley Walls. 2023. Texture Generation Using a Graph Generative Adversarial Network and Differentiable Rendering. In Image and Vision Computing, Wei Qi Yan, Minh Nguyen, and Martin Stommel (Eds.). Springer Nature Switzerland, Cham, 388–401.
[7]
Alexei A Efros and Thomas K Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 2. IEEE, 1033–1038.
[8]
Kurt W Fleischer, David H Laidlaw, Bena L Currin, and Alan H Barr. 1995. Cellular texture generation. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 239–248.
[9]
WT Freeman and Ce Liu. 2011. Markov random fields for super-resolution and texture synthesis. Advances in Markov Random Fields for Vision and Image Processing 1, 155–165 (2011), 3.
[10]
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. StyleGAN-NADA: CLIP-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–13.
[11]
Leon Gatys, Alexander S Ecker, and Matthias Bethge. 2015. Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems 28 (2015).
[12]
St?phane Gobron and Norishige Chiba. 1999. 3D surface cellular automata and their applications. The Journal of Visualization and Computer Animation 10, 3 (1999), 143–158.
[13]
Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2021. Learning graph cellular automata. Advances in Neural Information Processing Systems 34 (2021), 20983–20994.
[14]
Robin Green. 2003. Spherical harmonic lighting: The gritty details. In Archives of the game developers conference, Vol. 56. 4.
[15]
Paul Guerrero, Milo? Ha?an, Kalyan Sunkavalli, Radom?r M?ch, Tamy Boubekeur, and Niloy J Mitra. 2022. MatFormer: A Generative Model for Procedural Materials. ACM Transactions on Graphics 41, 4 (2022).
[16]
Jorge Gutierrez, Julien Rabin, Bruno Galerne, and Thomas Hurtut. 2020. On demand solid texture synthesis using deep 3d networks. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 511–530.
[17]
Jianwei Han, Kun Zhou, Li-Yi Wei, Minmin Gong, Hujun Bao, Xinming Zhang, and Baining Guo. 2006. Fast example-based surface texture synthesis via discrete optimization. The Visual Computer 22 (2006), 918–925.
[18]
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. 2020. Learning a neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8356–8364.
[19]
Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019. A novel framework for inverse procedural texture modeling. ACM Transactions on Graphics (ToG) 38, 6 (2019), 1–14.
[20]
Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2023. Generating Procedural Materials from Text or Image Prompts. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
[21]
Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022. ReLU Fields: The Little Non-Linearity That Could. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 27, 9 pages.
[22]
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10051–10060.
[23]
Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien-Tsin Wong. 2007. Solid Texture Synthesis from 2D Exemplars. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007) 26, 3 (2007), 2:1–2:9.
[24]
Sylvain Lefebvre and Hugues Hoppe. 2006. Appearance-Space Texture Synthesis. ACM Trans. Graph. 25, 3 (jul 2006), 541–548.
[25]
Jiabao Lei, Yabin Zhang, Kui Jia, et al. 2022. Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. Advances in Neural Information Processing Systems 35 (2022), 30923–30936.
[26]
Yiwei Ma, Xiaoqing Zhang, Xiaoshuai Sun, Jiayi Ji, Haowei Wang, Guannan Jiang, Weilin Zhuang, and Rongrong Ji. 2023. X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2749–2760.
[27]
St?phane M?rillou and Djamchid Ghazanfarpour. 2008. A survey of aging and weathering phenomena in computer graphics. Computers & Graphics 32, 2 (2008), 159–174.
[28]
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13492–13502.
[29]
Alexander Mordvintsev and Eyvind Niklasson. 2021. ?NCA: Texture Generation with Ultra-Compact Neural Cellular Automata. arXiv preprint arXiv:2111.13545 (2021).
[30]
Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. 2018. Differentiable image parameterizations. Distill 3, 7 (2018), e12.
[31]
Alexander Mordvintsev, Ettore Randazzo, and Eyvind Niklasson. 2021. Differentiable Programming of Reaction-Diffusion Patterns. In ALIFE 2022: The 2022 Conference on Artificial Life. MIT Press.
[32]
Alexander Mordvintsev, Ettore Randazzo, Eyvind Niklasson, and Michael Levin. 2020. Growing Neural Cellular Automata. Distill (2020). https://distill.pub/2020/growing-ca.
[33]
John von Neumann. 1966. Theory of self-reproducing automata. Math. Comp. 21 (1966), 745.
[34]
Eyvind Niklasson, Alexander Mordvintsev, Ettore Randazzo, and Michael Levin. 2021. Self-organising textures. Distill 6, 2 (2021), e00027–003.
[35]
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. 2019. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4531–4540.
[36]
Ehsan Pajouheshgar, Yitao Xu, Tong Zhang, and Sabine S?sstrunk. 2023. DyNCA: Realtime Dynamic Texture Synthesis Using Neural Cellular Automata. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20742–20751.
[37]
Joao Paulo. 2019. Free seamless PBR textures with Diffuse, Normal, Displacement, Occlusion and Roughness Maps. https://3dtextures.me/
[38]
Darwyn R Peachey. 1985. Solid texturing of complex surfaces. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques. 279–286.
[39]
Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985), 287–296.
[40]
Tiziano Portenier, Siavash Arjomand Bigdeli, and Orcun Goksel. 2020. Gramgan: Deep 3d texture synthesis from 2d exemplars. Advances in Neural Information Processing Systems 33 (2020), 6994–7004.
[41]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
[42]
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. 2023. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 (2023).
[43]
Liang Shi, Beichen Li, Milo? Ha?an, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020. Match: Differentiable material graphs for procedural material capture. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–15.
[44]
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nie?ner, and Angela Dai. 2022. Texturify: Generating textures on 3d shape surfaces. In European Conference on Computer Vision. Springer, 72–88.
[45]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.1556
[46]
Yiren Song. 2022. CLIPTexture: Text-Driven Texture Synthesis. In Proceedings of the 30th ACM International Conference on Multimedia. 5468–5476.
[47]
Lyne P Tchapmi, Trishiet Ray, Micael Tchapmi, Bokui Shen, Roberto Martin-Martin, and Silvio Savarese. 2022. Generating procedural 3D materials from images using neural networks. In Proceedings of the 2022 4th International Conference on Image, Video and Signal Processing. 32–40.
[48]
Matthew Tesfaldet, Marcus A Brubaker, and Konstantinos G Derpanis. 2018. Two-stream convolutional networks for dynamic texture synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6703–6712.
[49]
AM Turing. 1952. The Chemical Basis of Morphogenesis. Philosophical Transactions of the Royal Society of London Series B 237, 641 (1952), 37–72.
[50]
Greg Turk. 1991. Generating textures on arbitrary surfaces using reaction-diffusion. Acm Siggraph Computer Graphics 25, 4 (1991), 289–298.
[51]
Greg Turk. 2001. Texture synthesis on surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 347–354.
[52]
Bruno Vallet and Bruno L?vy. 2008. Spectral geometry processing with manifold harmonics. In Computer Graphics Forum, Vol. 27. Wiley Online Library, 251–260.
[53]
Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. 355–360.
[54]
Lexing Ying, Aaron Hertzmann, Henning Biermann, and Denis Zorin. 2001. Texture and shape synthesis on surfaces. In Rendering Techniques 2001: Proceedings of the Eurographics Workshop in London, United Kingdom, June 25–27, 2001 12. Springer, 301–312.
[55]
Jiaxuan You, Zhitao Ying, and Jure Leskovec. 2020. Design space for graph neural networks. Advances in Neural Information Processing Systems 33 (2020), 17009–17021.
[56]
Cem Yuksel, Sylvain Lefebvre, and Marco Tarini. 2019. Rethinking texture mapping. In Computer graphics forum, Vol. 38. Wiley Online Library, 535–551.
[57]
Jingdan Zhang, Kun Zhou, Luiz Velho, Baining Guo, and Heung-Yeung Shum. 2003. Synthesis of progressively-variant textures on arbitrary surfaces. In ACM SIGGRAPH 2003 Papers. 295–302.
