“DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling”

  • ©Xiaoguang Han, Chang Gao, and Yizhou Yu




    DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling


Session Title: Sketching & Curves



    Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.


    1. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 151–160. Google ScholarDigital Library
    2. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 187–194. Google ScholarDigital Library
    3. Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online modeling for realtime facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 40.Google ScholarDigital Library
    4. Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 2013. 3D shape regression for real-time facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 41.Google ScholarDigital Library
    5. Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014. Faceware-house: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (2014), 413–425. Google ScholarDigital Library
    6. Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016. Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics (TOG) 35, 4 (2016), 126.Google ScholarDigital Library
    7. Lyndsey Clarke, Min Chen, and Benjamin Mora. 2011. Automatic generation of 3D caricatures based on artistic deformation styles. IEEE transactions on visualization and computer graphics 17, 6 (2011), 808–821. Google ScholarDigital Library
    8. Kevin Dale, Kalyan Sunkavalli, Micah K Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video face replacement. ACM Transactions on Graphics (TOG) 30, 6 (2011), 130.Google ScholarDigital Library
    9. Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive contours for conveying shape. ACM Transactions on Graphics (TOG) 22, 3 (2003), 848–855. Google ScholarDigital Library
    10. Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4 (2012), 31–1. Google ScholarDigital Library
    11. Lubin Fan, Ruimin Wang, Linlin Xu, Jiansong Deng, and Ligang Liu. 2013. Modeling by drawing with shadow guidance. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 157–166. Google ScholarCross Ref
    12. Wei-Wen Feng, Byung-Uck Kim, and Yizhou Yu. 2008. Real-time data driven deformation using kernel canonical correlation analysis. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 91. Google ScholarDigital Library
    13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.Google Scholar
    14. Orn Gunnarsson and Steve Maddock. 2007. A statistically-assisted sketch-based interface for creating arbitrary 3-dimensional faces. In Proc. Theory and Practice of Computer Graphics.Google Scholar
    15. Haibin Huang, Evangelos Kalogerakis, ME Yumer, and Radomir Mech. 2016. Shape synthesis from sketches via procedural models and convolutional networks. IEEE Transactions on Visualization and Computer Graphics (2016).Google Scholar
    16. Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: A Sketching Interface for 3D Freeform Design. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 409–416. Google ScholarDigital Library
    17. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).Google Scholar
    18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.Google Scholar
    19. Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.Google ScholarDigital Library
    20. Thomas Lewiner, Thales Vieira, Dimas Martínez, Adelailson Peixoto, Vinícius Mello, and Luiz Velho. 2011. Interactive 3D caricature from harmonic exaggeration. Computers & Graphics 35, 3 (2011), 586–595. Google ScholarDigital Library
    21. Bo Li, Yijuan Lu, Fuqing Duan, Shuilong Dong, Yachun Fan, Lu Qian, Hamid Laga, Haisheng Li, Yuxiang Li, Peng Liu, and others. 2016. 3D Sketch-Based 3D Shape Retrieval. (2016).Google Scholar
    22. Junfa Liu, Yiqiang Chen, Chunyan Miao, Jinjing Xie, Charles X Ling, Xingyu Gao, and Wen Gao. 2009. Semi-Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 2104–2116.Google ScholarDigital Library
    23. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007a. FiberMesh: designing freeform surfaces with 3D curves. ACM transactions on graphics (TOG) 26, 3 (2007), 41.Google Scholar
    24. Andrew Nealen, Olga Sorkine, Marc Alexa, and Daniel Cohen-Or. 2007b. A sketch-based interface for detail-preserving mesh editing. In ACM SIGGRAPH 2007 courses. ACM, 42. Google ScholarDigital Library
    25. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Trans. Graph. 35, 4, Article 130 (July 2016), 11 pages.Google ScholarDigital Library
    26. Luke Olsen, Faramarz F Samavati, Mario Costa Sousa, and Joaquim A Jorge. 2009. Sketch-based modeling: A survey. Computers & Graphics 33, 1 (2009), 85–103. Google ScholarDigital Library
    27. Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow aligned surfacing of curve networks. ACM Transactions on Graphics (TOG) 34, 4 (2015), 127.Google ScholarDigital Library
    28. Alec Rivers, Frédo Durand, and Takeo Igarashi. 2010. 3D modeling with silhouettes. Vol. 29. ACM. Google ScholarDigital Library
    29. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 234–241. Google ScholarCross Ref
    30. Shunsuke Saito, Tianye Li, and Hao Li. 2016. Real-Time Facial Segmentation and Performance Capture from RGB Input. arXiv preprint arXiv:1604.02647 (2016).Google Scholar
    31. Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35, 4 (2016), 119.Google ScholarDigital Library
    32. Matan Sela, Yonathan Aflalo, and Ron Kimmel. 2015. Computational caricaturization of surfaces. Computer Vision and Image Understanding 141 (2015), 1–17. Google ScholarDigital Library
    33. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: shading concept sketches using cross-section curves. ACM Transactions on Graphics 31, 4 (2012). Google ScholarDigital Library
    34. Karan Singh and Eugene Fiume. 1998. Wires: a geometric deformation technique. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 405–414. Google ScholarDigital Library
    35. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. ACM, 175–184. Google ScholarDigital Library
    36. Robert W Sumner and Jovan Popović. 2004. Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23, 3 (2004), 399–405. Google ScholarDigital Library
    37. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-based Linear 3D Face Models. ACM Trans. Graph. 30, 4 (July 2011), 76:1–76:10.Google ScholarDigital Library
    38. Roberto C Cavalcante Vieira, Creto A Vidal, and Joaquim Bento Cavalcante-Neto. 2013. Three-Dimensional Face Caricaturing by Anthropometric Distortions. In 2013 XXVI Conference on Graphics, Patterns and Images. IEEE, 163–170.Google ScholarDigital Library
    39. Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 426–433. Google ScholarDigital Library
    40. Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1875–1883.Google ScholarCross Ref
    41. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 77. Google ScholarDigital Library
    42. Jinjing Xie, Yiqiang Chen, Junfa Liu, Chunyan Miao, and Xingyu Gao. 2009. Interactive 3D caricature generation based on double sampling. In Proceedings of the 17th ACM international conference on Multimedia. ACM, 745–748. Google ScholarDigital Library
    43. Xiaohua Xie, Kai Xu, Niloy J Mitra, Daniel Cohen-Or, Wenyong Gong, Qi Su, and Baoquan Chen. 2013. Sketch-to-Design: Context-Based Part Assembly. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 233–245.Google Scholar
    44. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics 33, 4 (2014). Google ScholarDigital Library
    45. Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu. 2013. Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics (TOG) 32, 4 (2013), 123.Google ScholarDigital Library
    46. Fei Yang, Jue Wang, Eli Shechtman, Lubomir Bourdev, and Dimitri Metaxas. 2011. Expression flow for 3D-aware face component transfer. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 60.Google ScholarDigital Library
    47. Qian Yu, Feng Liu, Yi-Zhe SonG, Tao Xiang, Timothy Hospedales, and Chen Change Loy. 2016. Sketch Me That Shoe. In Computer Vision and Pattern Recognition. Google ScholarCross Ref
    48. Yizhou Yu, Kun Zhou, Dong Xu, Xiaohan Shi, Hujun Bao, Baining Guo, and Heung-Yeung Shum. 2004. Mesh editing with poisson-based gradient field manipulation. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 644–651. Google ScholarDigital Library

ACM Digital Library Publication: