“DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling”
Conference:
Type(s):
Title:
- DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling
Session/Category Title: Sketching & Curves
Presenter(s)/Author(s):
Moderator(s):
Abstract:
Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.
References:
1. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3d curve models. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 151–160. Google ScholarDigital Library
2. Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 187–194. Google ScholarDigital Library
3. Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online modeling for realtime facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 40.Google ScholarDigital Library
4. Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 2013. 3D shape regression for real-time facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 41.Google ScholarDigital Library
5. Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014. Faceware-house: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3 (2014), 413–425. Google ScholarDigital Library
6. Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016. Real-time facial animation with image-based dynamic avatars. ACM Transactions on Graphics (TOG) 35, 4 (2016), 126.Google ScholarDigital Library
7. Lyndsey Clarke, Min Chen, and Benjamin Mora. 2011. Automatic generation of 3D caricatures based on artistic deformation styles. IEEE transactions on visualization and computer graphics 17, 6 (2011), 808–821. Google ScholarDigital Library
8. Kevin Dale, Kalyan Sunkavalli, Micah K Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video face replacement. ACM Transactions on Graphics (TOG) 30, 6 (2011), 130.Google ScholarDigital Library
9. Doug DeCarlo, Adam Finkelstein, Szymon Rusinkiewicz, and Anthony Santella. 2003. Suggestive contours for conveying shape. ACM Transactions on Graphics (TOG) 22, 3 (2003), 848–855. Google ScholarDigital Library
10. Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4 (2012), 31–1. Google ScholarDigital Library
11. Lubin Fan, Ruimin Wang, Linlin Xu, Jiansong Deng, and Ligang Liu. 2013. Modeling by drawing with shadow guidance. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 157–166. Google ScholarCross Ref
12. Wei-Wen Feng, Byung-Uck Kim, and Yizhou Yu. 2008. Real-time data driven deformation using kernel canonical correlation analysis. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 91. Google ScholarDigital Library
13. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.Google Scholar
14. Orn Gunnarsson and Steve Maddock. 2007. A statistically-assisted sketch-based interface for creating arbitrary 3-dimensional faces. In Proc. Theory and Practice of Computer Graphics.Google Scholar
15. Haibin Huang, Evangelos Kalogerakis, ME Yumer, and Radomir Mech. 2016. Shape synthesis from sketches via procedural models and convolutional networks. IEEE Transactions on Visualization and Computer Graphics (2016).Google Scholar
16. Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: A Sketching Interface for 3D Freeform Design. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 409–416. Google ScholarDigital Library
17. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).Google Scholar
18. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.Google Scholar
19. Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.Google ScholarDigital Library
20. Thomas Lewiner, Thales Vieira, Dimas Martínez, Adelailson Peixoto, Vinícius Mello, and Luiz Velho. 2011. Interactive 3D caricature from harmonic exaggeration. Computers & Graphics 35, 3 (2011), 586–595. Google ScholarDigital Library
21. Bo Li, Yijuan Lu, Fuqing Duan, Shuilong Dong, Yachun Fan, Lu Qian, Hamid Laga, Haisheng Li, Yuxiang Li, Peng Liu, and others. 2016. 3D Sketch-Based 3D Shape Retrieval. (2016).Google Scholar
22. Junfa Liu, Yiqiang Chen, Chunyan Miao, Jinjing Xie, Charles X Ling, Xingyu Gao, and Wen Gao. 2009. Semi-Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 2104–2116.Google ScholarDigital Library
23. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007a. FiberMesh: designing freeform surfaces with 3D curves. ACM transactions on graphics (TOG) 26, 3 (2007), 41.Google Scholar
24. Andrew Nealen, Olga Sorkine, Marc Alexa, and Daniel Cohen-Or. 2007b. A sketch-based interface for detail-preserving mesh editing. In ACM SIGGRAPH 2007 courses. ACM, 42. Google ScholarDigital Library
25. Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Trans. Graph. 35, 4, Article 130 (July 2016), 11 pages.Google ScholarDigital Library
26. Luke Olsen, Faramarz F Samavati, Mario Costa Sousa, and Joaquim A Jorge. 2009. Sketch-based modeling: A survey. Computers & Graphics 33, 1 (2009), 85–103. Google ScholarDigital Library
27. Hao Pan, Yang Liu, Alla Sheffer, Nicholas Vining, Chang-Jian Li, and Wenping Wang. 2015. Flow aligned surfacing of curve networks. ACM Transactions on Graphics (TOG) 34, 4 (2015), 127.Google ScholarDigital Library
28. Alec Rivers, Frédo Durand, and Takeo Igarashi. 2010. 3D modeling with silhouettes. Vol. 29. ACM. Google ScholarDigital Library
29. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 234–241. Google ScholarCross Ref
30. Shunsuke Saito, Tianye Li, and Hao Li. 2016. Real-Time Facial Segmentation and Performance Capture from RGB Input. arXiv preprint arXiv:1604.02647 (2016).Google Scholar
31. Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG) 35, 4 (2016), 119.Google ScholarDigital Library
32. Matan Sela, Yonathan Aflalo, and Ron Kimmel. 2015. Computational caricaturization of surfaces. Computer Vision and Image Understanding 141 (2015), 1–17. Google ScholarDigital Library
33. Cloud Shao, Adrien Bousseau, Alla Sheffer, and Karan Singh. 2012. CrossShade: shading concept sketches using cross-section curves. ACM Transactions on Graphics 31, 4 (2012). Google ScholarDigital Library
34. Karan Singh and Eugene Fiume. 1998. Wires: a geometric deformation technique. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. ACM, 405–414. Google ScholarDigital Library
35. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. ACM, 175–184. Google ScholarDigital Library
36. Robert W Sumner and Jovan Popović. 2004. Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23, 3 (2004), 399–405. Google ScholarDigital Library
37. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-based Linear 3D Face Models. ACM Trans. Graph. 30, 4 (July 2011), 76:1–76:10.Google ScholarDigital Library
38. Roberto C Cavalcante Vieira, Creto A Vidal, and Joaquim Bento Cavalcante-Neto. 2013. Three-Dimensional Face Caricaturing by Anthropometric Distortions. In 2013 XXVI Conference on Graphics, Patterns and Images. IEEE, 163–170.Google ScholarDigital Library
39. Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. 2005. Face transfer with multilinear models. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 426–433. Google ScholarDigital Library
40. Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3d shape retrieval using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1875–1883.Google ScholarCross Ref
41. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 77. Google ScholarDigital Library
42. Jinjing Xie, Yiqiang Chen, Junfa Liu, Chunyan Miao, and Xingyu Gao. 2009. Interactive 3D caricature generation based on double sampling. In Proceedings of the 17th ACM international conference on Multimedia. ACM, 745–748. Google ScholarDigital Library
43. Xiaohua Xie, Kai Xu, Niloy J Mitra, Daniel Cohen-Or, Wenyong Gong, Qi Su, and Baoquan Chen. 2013. Sketch-to-Design: Context-Based Part Assembly. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 233–245.Google Scholar
44. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D curve networks from 2D sketches via selective regularization. ACM Transactions on Graphics 33, 4 (2014). Google ScholarDigital Library
45. Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu. 2013. Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics (TOG) 32, 4 (2013), 123.Google ScholarDigital Library
46. Fei Yang, Jue Wang, Eli Shechtman, Lubomir Bourdev, and Dimitri Metaxas. 2011. Expression flow for 3D-aware face component transfer. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 60.Google ScholarDigital Library
47. Qian Yu, Feng Liu, Yi-Zhe SonG, Tao Xiang, Timothy Hospedales, and Chen Change Loy. 2016. Sketch Me That Shoe. In Computer Vision and Pattern Recognition. Google ScholarCross Ref
48. Yizhou Yu, Kun Zhou, Dong Xu, Xiaohan Shi, Hujun Bao, Baining Guo, and Heung-Yeung Shum. 2004. Mesh editing with poisson-based gradient field manipulation. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 644–651. Google ScholarDigital Library
