“The face of art: landmark detection and geometric style in portraits” by Yaniv, Newman and Shamir

  • ©Jordan Yaniv, Yael Newman, and Ariel Shamir

Conference:


Type:


Title:

    The face of art: landmark detection and geometric style in portraits

Session/Category Title:   Photo Science


Presenter(s)/Author(s):



Abstract:


    Facial Landmark detection in natural images is a very active research domain. Impressive progress has been made in recent years, with the rise of neural-network based methods and large-scale datasets. However, it is still a challenging and largely unexplored problem in the artistic portraits domain. Compared to natural face images, artistic portraits are much more diverse. They contain a much wider style variation in both geometry and texture and are more complex to analyze. Moreover, datasets that are necessary to train neural networks are unavailable.We propose a method for artistic augmentation of natural face images that enables training deep neural networks for landmark detection in artistic portraits. We utilize conventional facial landmarks datasets, and transform their content from natural images into “artistic face” images. In addition, we use a feature-based landmark correction step, to reduce the dependency between the different facial features, which is necessary due to position and shape variations of facial landmarks in artworks. To evaluate our landmark detection framework, we created an “Artistic-Faces” dataset, containing 160 artworks of various art genres, artists and styles, with a large variation in both geometry and texture. Using our method, we can detect facial features in artistic portraits and analyze their geometric style. This allows the definition of signatures for artistic styles of artworks and artists, that encode both the geometry and the texture style. It also allows us to present a geometric-aware style transfer method for portraits.

References:


    1. J. Alabort-i-Medina, E. Antonakos, J. Booth, P. Snape, and S. Zafeiriou. 2014. Menpo: A Comprehensive Platform for Parametric Image Alignment and Visual Deformable Models. In Proceedings of the ACM International Conference on Multimedia (MM ’14). ACM, New York, NY, USA, 679–682. Google ScholarDigital Library
    2. P.-N. Belhumeur, D.-W. Jacobs, D.-J. Kriegman, and N. Kumar. 2013. Localizing parts of faces using a consensus of exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 12 (2013), 2930–2940. Google ScholarDigital Library
    3. Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. 2013. Style and Abstraction in Portrait Sketching. ACM Transactions on Graphics 32(4) (SIGGRAPH Conference Proceedings) 32, 4, Article 55 (2013), 12 pages. Google ScholarDigital Library
    4. F.-L. Bookstein. 1989. Principal Warps: Thin-Plate Splines and the Decomposition of Deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence 11, 6 (1989), 567–585. Google ScholarDigital Library
    5. A. Bulat and G. Tzimiropoulos. 2016. Convolutional aggregation of local evidence for large pose face alignment. In Proceedings of the British Machine Vision Conference (BMVC).Google Scholar
    6. A. Bulat and G. Tzimiropoulos. 2017. How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1021–1030.Google Scholar
    7. Kaidi Cao, Jing Liao, and Lu Yuan. 2018. CariGANs: Unpaired Photo-to-caricature Translation. In SIGGRAPH Asia 2018 Technical Papers (SIGGRAPH Asia ’18). Article 244, 14 pages. Google ScholarDigital Library
    8. X. Cao, Y. Wei, F. Wen, and J. Sun. 2014. Face Alignment by Explicit Shape Regression. Proceedings of the International Journal of Computer Vision (IJCV) 107, 2 (2014), 177–190. Google ScholarDigital Library
    9. W.-T. Chu and W.-W. Li. 2017. Manga FaceNet: Face Detection in Manga based on Deep Neural Network. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR). 412–415. Google ScholarDigital Library
    10. T.-F. Cootes and C.-J. Taylor. 1992. Active Shape Models – `Smart Snakes’. In Proceedings of the British Machine Vision Conference (BMVC). 266–275.Google ScholarCross Ref
    11. P. Dollar, P. Welinder, and P. Perona. 2010. Cascaded Pose Regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1078–1085.Google Scholar
    12. X. Dong, Y. Yan, W. Ouyang, and Y. Yang. 2018. Style Aggregated Network for Facial Landmark Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 379–388.Google Scholar
    13. L.-G. Farkas, T.-A. Hreczko, J.-C. Kolar, and I.-R. Munro. 1985. Vertical and Horizontal Proportions of the Face in Young Adult North American Caucasians: Revision of Neoclassical Canons. Plastic and Reconstructive Surgery (PRS) 75, 3 (1985), 328–337.Google ScholarCross Ref
    14. L.-A. Gatys, A.-S. Ecker, and M. Bethge. 2015. A Neural Algorithm of Artistic Style. (2015). arXiv:arXiv:1508.06576Google Scholar
    15. G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. 2017. Exploring the structure of a Real-time, Arbitrary Neural Artistic Stylization Network. In Proceedings of the British Machine Vision Conference (BMVC).Google Scholar
    16. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Nets. In Proceedings of the Neural Information Processing Systems (NIPS). 2672–2680. Google ScholarDigital Library
    17. Jing Huo, Wenbin Li, Yinghuan Shi, Yang Gao, and Hujun Yin. 2018. WebCaricature: a benchmark for caricature recognition. In British Machine Vision Conference.Google Scholar
    18. P.-T. Jackson, A. Atapour-Abarghouei, S. Bonner, T. Breckon, and B. Obara. 2018. Style Augmentation: Data Augmentation via Style Randomization. (2018). arXiv:arXiv:1809.05375Google Scholar
    19. M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. 2015. Spatial Transformer Networks. In Advances in Neural Information Processing Systems (NIPS). 2017–2025. Google ScholarDigital Library
    20. S. Jha, N. Agarwal, and S. Agarwal. 2018. Bringing Cartoons to Life: Towards Improved Cartoon Face Detection and Recognition Systems. (2018). arXiv:arXiv:1804.01753Google Scholar
    21. Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song. 2017. Neural Style Transfer: A Review. (2017). arXiv:arXiv:1705.04058Google Scholar
    22. J. Johnson, A. Alahi, and L. Fei-Fei. 2016. Perceptual Losses for Realtime Style Transfer and Super-Resolution. In Proceedings of the European Conference on Computer Vision (ECCV). 694–711.Google Scholar
    23. Parneet Kaur, Hang Zhang, and Kristin J. Dana. 2017. Photo-Realistic Facial Texture Transfer. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (2017), 2097–2105.Google Scholar
    24. V. Kazemi and J. Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1867–1874. Google ScholarDigital Library
    25. Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758. Google ScholarDigital Library
    26. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2015).Google Scholar
    27. M. Kowalski, J. Naruniec, and T. Trzcinski. 2017. Deep Alignment Network: A Convolutional Neural Network for Robust Face Alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2034–2043.Google Scholar
    28. V. Le, J. Brandt, Z. Lin, L. Bourdev, and T.-S. Huang. 2012. Interactive facial feature localization. In Proceedings of the European Conference on Computer Vision (ECCV). 679–692. Google ScholarDigital Library
    29. C. Li and M. Wand. 2016. Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2479–2486.Google Scholar
    30. S. Li, X. Xu, L. Nie, and T.-S. Chua. 2017. Laplacian-steered Neural Style Transfer. In Proceedings of ACM on Multimedia Conference (ACM). 1716–1724. Google ScholarDigital Library
    31. J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou. 2017. A Deep Regression Architecture with Two-Stage Re-initialization for High Performance Facial Landmark Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3691–3700.Google Scholar
    32. M. Minear and D. Park. 2004. A Lifespan Database of Adult Facial Stimuli. Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc 36 (12 2004), 630–3.Google Scholar
    33. N.-V. Nguyen, C. Rigaud, and J.-C. Burie. 2017. Comic characters Detection using Deep Learning. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), Vol. 3. 42–46.Google ScholarCross Ref
    34. Toru Ogawa, Atsushi Otsubo, Rei Narita, Yusuke Matsui, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018. Object Detection for Comics using Manga109 Annotations. CoRR abs/1803.08670 (2018).Google Scholar
    35. S. Ren, X. Cao, Y. Wei, and J. Sun. 2016. Face Alignment via Regressing Local Binary Features. IEEE Transactions on Image Processing 25, 3 (March 2016), 1233–1245.Google ScholarDigital Library
    36. C. Sagonas, G. Tzimiropoulos, S. Zafeiriou, and M. Pantic. 2013. 300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 397–403. Google ScholarDigital Library
    37. J.-M. Saragih, S. Lucey, and J.-F. Cohn. 2011. Deformable Model Fitting by Regularized Landmark Mean-Shift. Proceedings of the International Journal of Computer Vision (IJCV) 91, 2 (2011), 200–215. Google ScholarDigital Library
    38. K. Schmid, D. Marx, and A. Samal. 2008. Computation of face attractiveness index based on neoclassic canons, symmetry and golden ratio. Pattern Recognition 41, 8 (2008), 2710–2717. Google ScholarDigital Library
    39. Ahmed A. S. Seleim, Mohamed A. Elgharib, and Linda Doyle. 2016. Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph. 35 (2016), 129:1–129:18. Google ScholarDigital Library
    40. Y. Shi, D. Deb, and A.-K. Jain. 2018. WarpGAN: Automatic Caricature Generation. (2018). arXiv:arXiv:1811.10100Google Scholar
    41. M. Stricker, O. Augereau, K. Kise, and M. Iwata. 2018. Facial Landmark Detection for Manga Images. (2018). arXiv:arXiv:1811.03214Google Scholar
    42. W. Sun and K. Kise. 2010. Similar Partial Copy Detection of Line Drawings Using a Cascade Classifier and Feature Matching. In Proceedings of the International Workshop on Computational Forensics (IWCF). 121–132. Google ScholarDigital Library
    43. Y. Sun, X. Wang, and X. Tang. 2013. Deep Convolutional Network Cascade for Facial Point Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3476–3483. Google ScholarDigital Library
    44. G. Trigeorgis, P. Snape, M. Nicolaou, E. Antonakos, and S. Zafeiriou. 2016. Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4177–4187.Google Scholar
    45. D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. 2016. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In Proceedings of the International Conference on Machine Learning (ICML). 1349–1357. Google ScholarDigital Library
    46. X. Xiong and F. De la Torre. 2013. Supervised Descent Method and its Applications to Face Alignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 532–539. Google ScholarDigital Library
    47. X. Glorot Y. and Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), Vol. 9. 249–256.Google Scholar
    48. H. Yanagisawa, D. Ishii, and H. Watanabe. 2014. Face Detection for Comic Images with Deformable Part Model. In Proceedings of the Image Electronics and Visual Computing Workshop (IIEEJ).Google Scholar
    49. J. Yang, Q. Liu, and K. Zhang. 2017. Stacked Hourglass Network for Robust Facial Landmark Localisation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2025–2033.Google Scholar
    50. X. Yu, F. Zhou, and M. Chandraker. 2016. Deep Deformation Network for Object Landmark Localization. In Proceedings of the European Conference on Computer Vision (ECCV). 52–70.Google Scholar
    51. H. Zhang, Q. Li, Z. Sun, and Y. Liu. 2018. Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection. IEEE Transactions on Information Forensics and Security 13, 10 (2018), 2409–2422.Google ScholarCross Ref
    52. K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. 2016. Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks. IEEE Signal Processing Letters 23, 10 (2016), 1499–1503.Google ScholarCross Ref
    53. J.-Y. Zhu, T. Park, P. Isola, and A.-A. Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2242–2251.Google ScholarCross Ref
    54. S. Zhu, C. Li, C. Change Loy, and X. Tang. 2015. Face Alignment by Coarse-to-Fine Shape Searching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4998–5006.Google Scholar
    55. S. Zhu, C. Li, C.-C. Loy, and X. Tang. 2016. Unconstrained Face Alignmentvia Cascaded Compositional Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3409–3417.Google Scholar
    56. X. Zhu and D. Ramana. 2012. Face detection, pose estimation, and landmark localization in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2879–2886. Google ScholarDigital Library


ACM Digital Library Publication:



Overview Page: