“Controllable Neural Style Transfer for Dynamic Meshes”
Conference:
Type(s):
Title:
- Controllable Neural Style Transfer for Dynamic Meshes
Presenter(s)/Author(s):
Abstract:
A novel mesh neural style transfer technique is presented. We improve upon the previous stylization works through implicit re-parametrizations of meshes, controllable style orientations, better temporal coherency treatment, and volume conservation. These improvements enable high-quality mesh stylizations, creating unique looks for both simulations and 3D assets.
References:
[1]
Joshua Aurand, Raphael Ortiz, Silvia Nauer, and Vinicius C. Azevedo. 2022. Efficient Neural Style Transfer for Volumetric Simulations. ACM Transactions on Graphics 41, 6 (Nov. 2022). https://doi.org/10.1145/3550454.3555517 Publisher: Association for Computing Machinery.
[2]
Robert Bridson, Jim Houriham, and Marcus Nordenstam. 2007. Curl-noise for procedural fluid flow. ACM Transactions on Graphics 26, 3 (July 2007), 46?es. https://doi.org/10.1145/1276377.1276435
[3]
Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017. StyleBank: An Explicit Representation for Neural Image Style Transfer. https://doi.org/10.48550/arXiv.1703.09210 arXiv:1703.09210 [cs].
[4]
Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, and Kui Jia. 2022. TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition. Technical Report. https://cyw-3d.github.io/tango.
[5]
Yingshu Chen, Guocheng Shao, Ka Chun Shum, Binh-Son Hua, and Sai-Kit Yeung. 2023. Advances in 3D Neural Stylization: A Survey. https://doi.org/10.48550/arXiv.2311.18328 arXiv:2311.18328 [cs].
[6]
Fang Da, David Hahn, Christopher Batty, Chris Wojtan, and Eitan Grinspun. 2016. Surface-only liquids. ACM Transactions on Graphics 35, 4 (July 2016), 1?12. https://doi.org/10.1145/2897824.2925899
[7]
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. 2017. A Learned Representation For Artistic Style. https://doi.org/10.48550/arXiv.1610.07629 arXiv:1610.07629 [cs].
[8]
Anna Fr?hst?ck, Ibraheem Alhashim, and Peter Wonka. 2019. TileGAN: synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics 38, 4 (Aug. 2019), 1?11. https://doi.org/10.1145/3306346.3322993
[9]
William Gao, Noam Aigerman, Thibault Groueix, Vova Kim, and Rana Hanocka. 2023. TextDeformer: Geometry Manipulation using Text Guidance. In ACM SIGGRAPH 2023 Conference Proceedings(SIGGRAPH ?23). Association for Computing Machinery, New York, NY, USA, 1?11. https://doi.org/10.1145/3588432.3591552
[10]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2414?2423. https://doi.org/10.1109/CVPR.2016.265
[11]
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. 2017. Exploring the structure of a real-time, arbitrary neural artistic stylization network. In Procedings of the British Machine Vision Conference 2017. British Machine Vision Association, London, UK, 114. https://doi.org/10.5244/C.31.114
[12]
Pav Grochola, Filippo Maccari, Young Joon Lee, and Edmond Boulet-Gilly. 2023. Linework in Spider-Man Across the Spider-Verse: An artistic driven approach to linework generation. In ACM SIGGRAPH 2023 Talks. ACM, Los Angeles CA USA, 1?2. https://doi.org/10.1145/3587421.3595456
[13]
Jie Guo, Mengtian Li, Zijing Zong, Yuntao Liu, Jingwu He, Yanwen Guo, and Ling-Qi Yan. 2021. Volumetric appearance stylization with stylizing kernel prediction network. ACM Transactions on Graphics 40, 4 (July 2021), 162:1?162:15. https://doi.org/10.1145/3450626.3459799
[14]
Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. 2020. Deep geometric texture synthesis. ACM Transactions on Graphics 39, 4 (July 2020). https://doi.org/10.1145/3386569.3392471 arXiv:2007.00074 Publisher: Association for Computing Machinery.
[15]
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. 2024. Style Aligned Image Generation via Shared Attention. https://doi.org/10.48550/arXiv.2312.02133 arXiv:2312.02133 [cs].
[16]
Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 327?340. https://doi.org/10.1145/383259.383295
[17]
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 2020-Decem, NeurIPS 2020 (2020), 1?25. arXiv:2006.11239.
[18]
Hsin-Ping Huang, Hung-Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. 2021. Learning to Stylize Novel Views. https://doi.org/10.48550/arXiv.2105.13509 arXiv:2105.13509 [cs].
[19]
Xun Huang and Serge Belongie. 2017. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, Venice, 1510?1519. https://doi.org/10.1109/ICCV.2017.167
[20]
Lukas H?llein, Justin Johnson, and Matthias Nie?ner. 2022. StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions. Technical Report. https://lukashoel.github.io/stylemesh/
[21]
Yongcheng Jing, Yang Liu, Yezhou Yang, Zunlei Feng, Yizhou Yu, Dacheng Tao, and Mingli Song. 2018. Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields. In Computer Vision ? ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII. Springer-Verlag, Berlin, Heidelberg, 244?260. https://doi.org/10.1007/978-3-030-01261-8_15
[22]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Computer Vision ? ECCV 2016. Vol. 9906. Springer International Publishing, Cham, 694?711. https://doi.org/10.1007/978-3-319-46475-6_43 Series Title: Lecture Notes in Computer Science.
[23]
Hongyuan Kang, Xiao Dong, Juan Cao, and Zhonggui Chen. 2023. Neural style transfer for 3D meshes. Graphical Models 129 (Oct. 2023), 101198. https://doi.org/10.1016/j.gmod.2023.101198
[24]
Paul Kanyuk, Vinicius Azevedo, Raphael Ortiz, and Jingwei Tang. 2023. Singed Silhouettes and Feed Forward Flames: Volumetric Neural Style Transfer for Expressive Fire Simulation. In ACM SIGGRAPH 2023 Talks. ACM, Los Angeles CA USA, 1?2. https://doi.org/10.1145/3587421.3595435
[25]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, 3907?3916. https://doi.org/10.1109/CVPR.2018.00411
[26]
Byungsoo Kim, Vinicius C. Azevedo, Markus Gross, and Barbara Solenthaler. 2019. Transport-based neural style transfer for smoke simulations. ACM Transactions on Graphics 38, 6 (Nov. 2019). https://doi.org/10.1145/3355089.3356560 arXiv:1905.07442 Publisher: Association for Computing Machinery.
[27]
Byungsoo Kim, Vinicius C. Azevedo, Markus Gross, and Barbara Solenthaler. 2020. Lagrangian neural style transfer for fluids. ACM Transactions on Graphics 39, 4 (July 2020). https://doi.org/10.1145/3386569.3392473 arXiv:2005.00803 Publisher: Association for Computing Machinery.
[28]
M. Kohlbrenner, U. Finnendahl, T. Djuren, and M. Alexa. 2021. Gauss Stylization: Interactive Artistic Mesh Modeling based on Preferred Surface Normals. Computer Graphics Forum 40, 5 (2021), 33?43. https://doi.org/10.1111/cgf.14355 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14355.
[29]
Nicholas Kolkin, Michal Kucera, Sylvain Paris, Daniel Sykora, Eli Shechtman, and Greg Shakhnarovich. 2022. Neural Neighbor Style Transfer. (March 2022). http://arxiv.org/abs/2203.13215 arXiv:2203.13215.
[30]
Chuan Li and Michael Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA, 2479?2486. https://doi.org/10.1109/CVPR.2016.272
[31]
Shaohua Li, Xinxing Xu, Liqiang Nie, and Tat-Seng Chua. 2017d. Laplacian-Steered Neural Style Transfer. In Proceedings of the 25th ACM international conference on Multimedia. ACM, Mountain View California USA, 1716?1724. https://doi.org/10.1145/3123266.3123425
[32]
Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. 2019. Learning Linear Transformations for Fast Image and Video Style Transfer. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Long Beach, CA, USA, 3804?3812. https://doi.org/10.1109/CVPR.2019.00393
[33]
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. 2017a. Diversified Texture Synthesis with Feed-Forward Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, HI, 266?274. https://doi.org/10.1109/CVPR.2017.36
[34]
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. 2017b. Universal Style Transfer via Feature Transforms. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper_files/paper/2017/hash/49182f81e6a13cf5eaa496d51fea6406-Abstract.html
[35]
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. 2017c. Demystifying Neural Style Transfer. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, Melbourne, Australia, 2230?2236. https://doi.org/10.24963/ijcai.2017/310
[36]
Hsueh Ti Derek Liu and Alec Jacobson. 2019. Cubic stylization. ACM Transactions on Graphics 38, 6 (Nov. 2019). https://doi.org/10.1145/3355089.3356495 arXiv:1910.02926 Publisher: Association for Computing Machinery.
[37]
Hsueh-Ti Derek Liu and Alec Jacobson. 2021. Normal-Driven Spherical Shape Analogies. Computer Graphics Forum 40, 5 (2021), 45?55. https://doi.org/10.1111/cgf.14356 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14356.
[38]
Hsueh-Ti Derek Liu, Vladimir G. Kim, Siddhartha Chaudhuri, Noam Aigerman, and Alec Jacobson. 2020. Neural subdivision. ACM Transactions on Graphics 39, 4 (Aug. 2020). https://doi.org/10.1145/3386569.3392418
[39]
Hsueh-ti Derek Liu, Michael Tao, Alec Jacobson, and Hsueh-Ti Derek Liu. 2018. Paparazzi: Surface Editing by way of Multi-View Image Processing. 1, 1 (2018), 11. https://doi.org/10.1145/8888888.7777777
[40]
Kunhao Liu, Fangneng Zhan, Yiwen Chen, Jiahui Zhang, Yingchen Yu, Abdulmotaleb El Saddik, Shijian Lu, and Eric Xing. 2023c. StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields. (March 2023). http://arxiv.org/abs/2303.10598 arXiv:2303.10598.
[41]
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. 2023b. One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. http://arxiv.org/abs/2306.16928 arXiv:2306.16928 [cs].
[42]
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 2023a. Zero-1-to-3: Zero-shot One Image to 3D Object. https://doi.org/10.48550/arXiv.2303.11328 arXiv:2303.11328 [cs].
[43]
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. https://doi.org/10.48550/arXiv.1711.05101 arXiv:1711.05101 [cs, math].
[44]
Chongyang Ma, Haibin Huang, Alla Sheffer, Evangelos Kalogerakis, and Rui Wang. 2014. Analogy-Driven 3D Style Transfer. Technical Report. Volume: 33 Issue: 2.
[45]
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2021. Text2Mesh: Text-Driven Neural Stylization for Meshes. (Dec. 2021). arXiv:2112.03221.
[46]
Shailesh Mishra and Jonathan Granskog. 2022. CLIP-based Neural Neighbor Style Transfer for 3D Assets. https://doi.org/10.48550/arXiv.2208.04370 arXiv:2208.04370 [cs].
[47]
Matthias M?ller. 2009. Fast and robust tracking of fluid surfaces. In Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New Orleans Louisiana, 237?245. https://doi.org/10.1145/1599470.1599501
[48]
Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. 2022. SNeRF: Stylized Neural Implicit Representations for 3D Scenes. ACM Transactions on Graphics 41, 4 (July 2022). https://doi.org/10.1145/3528223.3530107 arXiv:2207.02363 Publisher: Association for Computing Machinery.
[49]
Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large steps in inverse rendering of geometry. ACM Transactions on Graphics 40, 6 (Dec. 2021). https://doi.org/10.1145/3478513.3480501 Publisher: Association for Computing Machinery.
[50]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K?pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. http://arxiv.org/abs/1912.01703 arXiv:1912.01703 [cs, stat].
[51]
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2022. DreamFusion: Text-to-3D using 2D Diffusion. (Sept. 2022). http://arxiv.org/abs/2209.14988 _eprint: 2209.14988.
[52]
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. 2023. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. http://arxiv.org/abs/2306.17843 arXiv:2306.17843 [cs].
[53]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. https://doi.org/10.48550/arXiv.2103.00020 arXiv:2103.00020 [cs].
[54]
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. 2020. Accelerating 3D Deep Learning with PyTorch3D. https://doi.org/10.48550/arXiv.2007.08501 arXiv:2007.08501 [cs].
[55]
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. 2023. TEXTure: Text-Guided Texturing of 3D Shapes. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings. ACM, Los Angeles CA USA, 1?11. https://doi.org/10.1145/3588432.3591503
[56]
Eric Risser, Pierre Wilmot, and Connelly Barnes. 2017. Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses. http://arxiv.org/abs/1701.08893 arXiv:1701.08893 [cs].
[57]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2022-June (Dec. 2022), 10674?10685. https://doi.org/10.1109/CVPR52688.2022.01042 ISBN: 9781665469463 _eprint: 2112.10752.
[58]
Omry Sendik and Daniel Cohen-Or. 2017. Deep Correlations for Texture Synthesis. ACM Transactions on Graphics 36, 5 (Oct. 2017), 1?15. https://doi.org/10.1145/3015461
[59]
Falong Shen, Shuicheng Yan, and Gang Zeng. 2018. Neural Style Transfer via Meta Networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, 8061?8069. https://doi.org/10.1109/CVPR.2018.00841
[60]
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. 2023. MVDream: Multi-view Diffusion for 3D Generation. 2 (2023), 1?18. http://arxiv.org/abs/2308.16512 arXiv:2308.16512.
[61]
Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. (2014). https://doi.org/10.48550/ARXIV.1409.1556 Publisher: arXiv Version Number: 6.
[62]
Arunachalam Somasundaram, Levi Biasco, and Damon Riesberg. 2023. Dynamic Mesh Sharpening. In ACM SIGGRAPH 2023 Talks. ACM, Los Angeles CA USA, 1?2. https://doi.org/10.1145/3587421.3595428
[63]
Olga Sorkine and Marc Alexa. 2007. As-rigid-as-possible surface modeling. In Proceedings of the fifth Eurographics symposium on Geometry processing(SGP ?07). Eurographics Association, Goslar, DEU, 109?116.
[64]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going Deeper with Convolutions. https://doi.org/10.48550/arXiv.1409.4842 arXiv:1409.4842 [cs].
[65]
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. 2016. Texture networks: feed-forward synthesis of textures and stylized images. In Proceedings of the 33rd International Conference on International Conference on Machine Learning – Volume 48(ICML?16). JMLR.org, New York, NY, USA, 1349?1357.
[66]
Xin Wang, Geoffrey Oxholm, Da Zhang, and Yuan-Fang Wang. 2017. Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Honolulu, HI, 7178?7186. https://doi.org/10.1109/CVPR.2017.759
[67]
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. 2023. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. (2023), 1?34. http://arxiv.org/abs/2305.16213 arXiv:2305.16213.
[68]
Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 2021. 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Montreal, QC, Canada, 12436?12445. https://doi.org/10.1109/ICCV48922.2021.01223
[69]
Hang Zhang and Kristin Dana. 2019. Multi-style Generative Network for Real-Time Transfer. In Computer Vision ? ECCV 2018 Workshops. Vol. 11132. Springer International Publishing, Cham, 349?365. https://doi.org/10.1007/978-3-030-11018-5_32 Series Title: Lecture Notes in Computer Science.
[70]
Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. 2022. ARF: Artistic Radiance Fields. https://doi.org/10.48550/arXiv.2206.06360 arXiv:2206.06360 [cs].
[71]
Yizhong Zhang, Huamin Wang, Shuai Wang, Yiying Tong, and Kun Zhou. 2012. A Deformable Surface Model for Real-Time Water Drop Animation. IEEE Transactions on Visualization and Computer Graphics 18, 8 (Aug. 2012), 1281?1289. https://doi.org/10.1109/TVCG.2011.141
[72]
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics 37, 4 (Aug. 2018), 1?13. https://doi.org/10.1145/3197517.3201285
