“Toonify3D: StyleGAN-based 3D Stylized Face Generator”
Conference:
Type(s):
Title:
- Toonify3D: StyleGAN-based 3D Stylized Face Generator
Presenter(s)/Author(s):
Abstract:
We present a StyleGAN-based 3D stylized face generation framework, Toonify3D, which automatically creates full-head 3D stylized avatar mesh models and supports GAN-based 3D facial expression editing.
References:
[1]
3DScanStore. 2023. 3DScanStore. https://www.3dscanstore.com
[2]
Rameen Abdal, Hsin-Ying Lee, Peihao Zhu, Menglei Chai, Aliaksandr Siarohin, Peter Wonka, and Sergey Tulyakov. 2023. 3DAvatarGAN: Bridging Domains for Personalized Editable Avatars. In Proc. CVPR.
[3]
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. 2021. StyleFlow: Attribute-Conditioned Exploration of StyleGAN-Generated Images Using Conditional Continuous Normalizing Flows. ACM Trans. Graph. (2021).
[4]
Victoria Fernandez Abrevaya, Adnane Boukhayma, Philip H.S. Torr, and Edmond Boyer. 2020. Cross-Modal Deep Face Normals With Deactivable Skip Connections. In Proc. CVPR.
[5]
Brian Amberg, Sami Romdhani, and Thomas Vetter. 2007. Optimal step nonrigid ICP algorithms for surface registration. In Proc. CVPR.
[6]
David Beniaguev. 2022. Synthetic Faces High Quality (SFHQ) dataset. https://github.com/SelfishGene/SFHQ-dataset
[7]
Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proc. SIGGRAPH.
[8]
Blender. 2023. Blender – a 3D modelling and rendering package. http://www.blender.org
[9]
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. 2022. Efficient geometry-aware 3D generative adversarial networks. In Proc. CVPR.
[10]
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. 2021. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR.
[11]
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019. Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set. In Proc. CVPR Workshops.
[12]
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Trans. Graph. (2022).
[13]
Pablo Garrido, Michael Zollh?fer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick P?rez, and Christian Theobalt. 2016. Reconstruction of personalized 3D face rigs from monocular video. ACM Trans. Graph. (2016).
[14]
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. 2021. StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. In Proc. ICLR.
[15]
Xiaoguang Han, Chang Gao, and Yizhou Yu. 2017. DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling. ACM Trans. Graph. (2017).
[16]
Aaron Hertzmann and Steven M Seitz. 2005. Example-based photometric stereo: Shape reconstruction with general, varying brdfs. IEEE TPAMI (2005).
[17]
Wonjong Jang, Gwangjin Ju, Yucheol Jung, Jiaolong Yang, Xin Tong, and Seungyong Lee. 2021. StyleCariGAN: Caricature generation via StyleGAN feature map modulation. ACM Trans. Graph. (2021).
[18]
Haibo Jin, Shengcai Liao, and Ling Shao. 2021. Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild. IJCV (2021).
[19]
Wonjoon Jin, Nuri Ryu, Geonung Kim, Seung-Hwan Baek, and Sunghyun Cho. 2022. Dr.3D: Adapting 3D GANs to Artistic Drawings. In Proc. ACM SIGGRAPH Asia.
[20]
Yucheol Jung, Wonjong Jang, Soongjin Kim, Jiaolong Yang, Xin Tong, and Seungyong Lee. 2022. Deep deformable 3d caricatures with learned shape control. In Proc. ACM SIGGRAPH.
[21]
Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proc. CVPR.
[22]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR.
[23]
Seung Wook Kim, Karsten Kreis, Daiqing Li, Antonio Torralba, and Sanja Fidler. 2022. Polymorphic-GAN: Generating aligned samples across multiple domains with learned morph maps. In Proc. CVPR.
[24]
Yushi Lan, Xuyi Meng, Shuai Yang, Chen Change Loy, and Bo Dai. 2023. E3DGE: Self-Supervised Geometry-Aware Encoder for Style-based 3D GAN Inversion. In Proc. CVPR.
[25]
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017. Learning a model of facial shape and expression from 4D scans.ACM Trans. Graph. (2017).
[26]
Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or. 2022. Large: Latent-based regression through gan semantics. In Proc. CVPR.
[27]
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. 2021. Few-shot image generation via cross-domain correspondence. In Proc. CVPR.
[28]
Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E Green, and Nassir Navab. 2021. Segmentation in style: Unsupervised semantic image segmentation with stylegan and clip. arXiv preprint arXiv:2107.12518 (2021).
[29]
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. 2020. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:2011.00844 (2020).
[30]
Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, and Bo Dai. 2021. A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis. NeurIPS (2021).
[31]
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D face model for pose and illumination invariant face recognition. In Proc. AVSS.
[32]
Justin NM Pinkney and Doron Adler. 2020. Resolution dependent GAN interpolation for controllable image synthesis between domains. In Proc. NeurIPS workshop on Machine Learning for Creativity and Design.
[33]
Yuda Qiu, Xiaojie Xu, Lingteng Qiu, Yan Pan, Yushuang Wu, Weikai Chen, and Xiaoguang Han. 2021. 3DCaricShop: A dataset and a baseline method for single-view 3d caricature face reconstruction. In Proc. CVPR.
[34]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In Proc. ICML.
[35]
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2021. Encoding in style: a stylegan encoder for image-to-image translation. In Proc. CVPR.
[36]
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2022. Pivotal Tuning for Latent-based Editing of Real Images. ACM Trans. Graph. (2022).
[37]
Matan Sela, Elad Richardson, and Ron Kimmel. 2017. Unrestricted facial geometry reconstruction using image-to-image translation. In Proc. ICCV.
[38]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. SfSNet: Learning shape, reflectance and illuminance of faces in the wild. In Proc. CVPR.
[39]
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. 2020. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE TPAMI (2020).
[40]
Yichun Shi, Divyansh Aggarwal, and Anil K Jain. 2021. Lifting 2d stylegan for 3d-aware face generation. In Proc. CVPR.
[41]
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proc. ICLR.
[42]
Ayush Tewari, Michael Zollhofer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt. 2017. Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proc. ICCV Workshops.
[43]
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021. Designing an encoder for stylegan image manipulation. ACM Trans. Graph. (2021).
[44]
Hao Wang, Guosheng Lin, Steven CH Hoi, and Chunyan Miao. 2022. 3D cartoon face generation with controllable expressions from a single GAN image. arXiv preprint arXiv:2207.14425 (2022).
[45]
Zongze Wu, Yotam Nitzan, Eli Shechtman, and Dani Lischinski. 2022. StyleAlign: Analysis and Applications of Aligned StyleGAN Models. In Proc. ICLR.
[46]
Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. 2021. Generative hierarchical features from synthesizing images. In Proc. CVPR.
[47]
Shuai Yang, Liming Jiang, Ziwei Liu, and Chen Change Loy. 2022. Pastiche Master: Exemplar-based high-resolution portrait style transfer. In Proc. CVPR.
[48]
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. 2021. DatasetGAN: Efficient labeled data factory with minimal human effort. In Proc. CVPR.
[49]
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020. In-domain gan inversion for real image editing. In Proc. ECCV.
