“The face of art: landmark detection and geometric style in portraits” by Yaniv, Newman and Shamir
Conference:
Type:
Session Title:
- Photo Science
Title:
- The face of art: landmark detection and geometric style in portraits
Presenter(s)/Author(s):
Abstract:
Facial Landmark detection in natural images is a very active research domain. Impressive progress has been made in recent years, with the rise of neural-network based methods and large-scale datasets. However, it is still a challenging and largely unexplored problem in the artistic portraits domain. Compared to natural face images, artistic portraits are much more diverse. They contain a much wider style variation in both geometry and texture and are more complex to analyze. Moreover, datasets that are necessary to train neural networks are unavailable.We propose a method for artistic augmentation of natural face images that enables training deep neural networks for landmark detection in artistic portraits. We utilize conventional facial landmarks datasets, and transform their content from natural images into “artistic face” images. In addition, we use a feature-based landmark correction step, to reduce the dependency between the different facial features, which is necessary due to position and shape variations of facial landmarks in artworks. To evaluate our landmark detection framework, we created an “Artistic-Faces” dataset, containing 160 artworks of various art genres, artists and styles, with a large variation in both geometry and texture. Using our method, we can detect facial features in artistic portraits and analyze their geometric style. This allows the definition of signatures for artistic styles of artworks and artists, that encode both the geometry and the texture style. It also allows us to present a geometric-aware style transfer method for portraits.
References:
1. J. Alabort-i-Medina, E. Antonakos, J. Booth, P. Snape, and S. Zafeiriou. 2014. Menpo: A Comprehensive Platform for Parametric Image Alignment and Visual Deformable Models. In Proceedings of the ACM International Conference on Multimedia (MM ’14). ACM, New York, NY, USA, 679–682. Google ScholarDigital Library
2. P.-N. Belhumeur, D.-W. Jacobs, D.-J. Kriegman, and N. Kumar. 2013. Localizing parts of faces using a consensus of exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 12 (2013), 2930–2940. Google ScholarDigital Library
3. Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. 2013. Style and Abstraction in Portrait Sketching. ACM Transactions on Graphics 32(4) (SIGGRAPH Conference Proceedings) 32, 4, Article 55 (2013), 12 pages. Google ScholarDigital Library
4. F.-L. Bookstein. 1989. Principal Warps: Thin-Plate Splines and the Decomposition of Deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence 11, 6 (1989), 567–585. Google ScholarDigital Library
5. A. Bulat and G. Tzimiropoulos. 2016. Convolutional aggregation of local evidence for large pose face alignment. In Proceedings of the British Machine Vision Conference (BMVC).Google Scholar
6. A. Bulat and G. Tzimiropoulos. 2017. How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1021–1030.Google Scholar
7. Kaidi Cao, Jing Liao, and Lu Yuan. 2018. CariGANs: Unpaired Photo-to-caricature Translation. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia ’18). Article 244, 14 pages. Google ScholarDigital Library
8. X. Cao, Y. Wei, F. Wen, and J. Sun. 2014. Face Alignment by Explicit Shape Regression. Proceedings of the International Journal of Computer Vision (IJCV) 107, 2 (2014), 177–190. Google ScholarDigital Library
9. W.-T. Chu and W.-W. Li. 2017. Manga FaceNet: Face Detection in Manga based on Deep Neural Network. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR). 412–415. Google ScholarDigital Library
10. T.-F. Cootes and C.-J. Taylor. 1992. Active Shape Models – `Smart Snakes’. In Proceedings of the British Machine Vision Conference (BMVC). 266–275.Google ScholarCross Ref
11. P. Dollar, P. Welinder, and P. Perona. 2010. Cascaded Pose Regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1078–1085.Google Scholar
12. X. Dong, Y. Yan, W. Ouyang, and Y. Yang. 2018. Style Aggregated Network for Facial Landmark Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 379–388.Google Scholar
13. L.-G. Farkas, T.-A. Hreczko, J.-C. Kolar, and I.-R. Munro. 1985. Vertical and Horizontal Proportions of the Face in Young Adult North American Caucasians: Revision of Neoclassical Canons. Plastic and Reconstructive Surgery (PRS) 75, 3 (1985), 328–337.Google ScholarCross Ref
14. L.-A. Gatys, A.-S. Ecker, and M. Bethge. 2015. A Neural Algorithm of Artistic Style. (2015). arXiv:arXiv:1508.06576Google Scholar
15. G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. 2017. Exploring the structure of a Real-time, Arbitrary Neural Artistic Stylization Network. In Proceedings of the British Machine Vision Conference (BMVC).Google Scholar
16. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Nets. In Proceedings of the Neural Information Processing Systems (NIPS). 2672–2680. Google ScholarDigital Library
17. Jing Huo, Wenbin Li, Yinghuan Shi, Yang Gao, and Hujun Yin. 2018. WebCaricature: a benchmark for caricature recognition. In British Machine Vision Conference.Google Scholar
18. P.-T. Jackson, A. Atapour-Abarghouei, S. Bonner, T. Breckon, and B. Obara. 2018. Style Augmentation: Data Augmentation via Style Randomization. (2018). arXiv:arXiv:1809.05375Google Scholar
19. M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. 2015. Spatial Transformer Networks. In Advances in Neural Information Processing Systems (NIPS). 2017–2025. Google ScholarDigital Library
20. S. Jha, N. Agarwal, and S. Agarwal. 2018. Bringing Cartoons to Life: Towards Improved Cartoon Face Detection and Recognition Systems. (2018). arXiv:arXiv:1804.01753Google Scholar
21. Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song. 2017. Neural Style Transfer: A Review. (2017). arXiv:arXiv:1705.04058Google Scholar
22. J. Johnson, A. Alahi, and L. Fei-Fei. 2016. Perceptual Losses for Realtime Style Transfer and Super-Resolution. In Proceedings of the European Conference on Computer Vision (ECCV). 694–711.Google Scholar
23. Parneet Kaur, Hang Zhang, and Kristin J. Dana. 2017. Photo-Realistic Facial Texture Transfer. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (2017), 2097–2105.Google Scholar
24. V. Kazemi and J. Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1867–1874. Google ScholarDigital Library
25. Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758. Google ScholarDigital Library
26. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2015).Google Scholar
27. M. Kowalski, J. Naruniec, and T. Trzcinski. 2017. Deep Alignment Network: A Convolutional Neural Network for Robust Face Alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2034–2043.Google Scholar
28. V. Le, J. Brandt, Z. Lin, L. Bourdev, and T.-S. Huang. 2012. Interactive facial feature localization. In Proceedings of the European Conference on Computer Vision (ECCV). 679–692. Google ScholarDigital Library
29. C. Li and M. Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2479–2486.Google Scholar
30. S. Li, X. Xu, L. Nie, and T.-S. Chua. 2017. Laplacian-steered Neural Style Transfer. In Proceedings of ACM on Multimedia Conference (ACM). 1716–1724. Google ScholarDigital Library
31. J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou. 2017. A Deep Regression Architecture with Two-Stage Re-initialization for High Performance Facial Landmark Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3691–3700.Google Scholar
32. M. Minear and D. Park. 2004. A Lifespan Database of Adult Facial Stimuli. Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc 36 (12 2004), 630–3.Google Scholar
33. N.-V. Nguyen, C. Rigaud, and J.-C. Burie. 2017. Comic characters Detection using Deep Learning. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), Vol. 3. 42–46.Google ScholarCross Ref
34. Toru Ogawa, Atsushi Otsubo, Rei Narita, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018. Object Detection for Comics using Manga109 Annotations. CoRR abs/1803.08670 (2018).Google Scholar
35. S. Ren, X. Cao, Y. Wei, and J. Sun. 2016. Face Alignment via Regressing Local Binary Features. IEEE Transactions on Image Processing 25, 3 (March 2016), 1233–1245.Google ScholarDigital Library
36. C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 2013. 300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 397–403. Google ScholarDigital Library
37. J.-M. Saragih, S. Lucey, and J.-F. Cohn. 2011. Deformable Model Fitting by Regularized Landmark Mean-Shift. Proceedings of the International Journal of Computer Vision (IJCV) 91, 2 (2011), 200–215. Google ScholarDigital Library
38. K. Schmid, D. Marx, and A. Samal. 2008. Computation of face attractiveness index based on neoclassic canons, symmetry and golden ratio. Pattern Recognition 41, 8 (2008), 2710–2717. Google ScholarDigital Library
39. Ahmed A. S. Seleim, Mohamed A. Elgharib, and Linda Doyle. 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph. 35 (2016), 129:1–129:18. Google ScholarDigital Library
40. Y. Shi, D. Deb, and A.-K. Jain. 2018. WarpGAN: Automatic Caricature Generation. (2018). arXiv:arXiv:1811.10100Google Scholar
41. M. Stricker, O. Augereau, K. Kise, and M. Iwata. 2018. Facial Landmark Detection for Manga Images. (2018). arXiv:arXiv:1811.03214Google Scholar
42. W. Sun and K. Kise. 2010. Similar Partial Copy Detection of Line Drawings Using a Cascade Classifier and Feature Matching. In Proceedings of the International Workshop on Computational Forensics (IWCF). 121–132. Google ScholarDigital Library
43. Y. Sun, X. Wang, and X. Tang. 2013. Deep Convolutional Network Cascade for Facial Point Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3476–3483. Google ScholarDigital Library
44. G. Trigeorgis, P. Snape, M. Nicolaou, E. Antonakos, and S. Zafeiriou. 2016. Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4177–4187.Google Scholar
45. D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In Proceedings of the International Conference on Machine Learning (ICML). 1349–1357. Google ScholarDigital Library
46. X. Xiong and F. De la Torre. 2013. Supervised Descent Method and its Applications to Face Alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 532–539. Google ScholarDigital Library
47. X. Glorot Y. and Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), Vol. 9. 249–256.Google Scholar
48. H. Yanagisawa, D. Ishii, and H. Watanabe. 2014. Face Detection for Comic Images with Deformable Part Model. In Proceedings of the Image Electronics and Visual Computing Workshop (IIEEJ).Google Scholar
49. J. Yang, Q. Liu, and K. Zhang. 2017. Stacked Hourglass Network for Robust Facial Landmark Localisation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2025–2033.Google Scholar
50. X. Yu, F. Zhou, and M. Chandraker. 2016. Deep Deformation Network for Object Landmark Localization. In Proceedings of the European Conference on Computer Vision (ECCV). 52–70.Google Scholar
51. H. Zhang, Q. Li, Z. Sun, and Y. Liu. 2018. Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection. IEEE Transactions on Information Forensics and Security 13, 10 (2018), 2409–2422.Google ScholarCross Ref
52. K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. 2016. Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks. IEEE Signal Processing Letters 23, 10 (2016), 1499–1503.Google ScholarCross Ref
53. J.-Y. Zhu, T. Park, P. Isola, and A.-A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2242–2251.Google ScholarCross Ref
54. S. Zhu, C. Li, C. Change Loy, and X. Tang. 2015. Face Alignment by Coarse-to-Fine Shape Searching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4998–5006.Google Scholar
55. S. Zhu, C. Li, C.-C. Loy, and X. Tang. 2016. Unconstrained Face Alignmentvia Cascaded Compositional Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3409–3417.Google Scholar
56. X. Zhu and D. Ramana. 2012. Face detection, pose estimation, and landmark localization in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2879–2886. Google ScholarDigital Library
