“Example-based synthesis of stylized facial animations” by Fišer, Jamriška, Simons, Shechtman, Lu, et al. …

  • ©Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sýkora

Conference:


Title:

    Example-based synthesis of stylized facial animations

Session/Category Title: Faces & Hair

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We introduce a novel approach to example-based stylization of portrait videos that preserves both the subject’s identity and the visual richness of the input style exemplar. Unlike the current state-of-the-art based on neural style transfer [Selim et al. 2016], our method performs non-parametric texture synthesis that retains more of the local textural details of the artistic exemplar and does not suffer from image warping artifacts caused by aligning the style exemplar with the target face. Our method allows the creation of videos with less than full temporal coherence [Ruder et al. 2016]. By introducing a controllable amount of temporal dynamics, it more closely approximates the appearance of real hand-painted animation in which every frame was created independently. We demonstrate the practical utility of the proposed solution on a variety of style exemplars and target videos.

References:


    1. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24.Google ScholarDigital Library
    2. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation By Example. ACM Transactions on Graphics 32, 4 (2013), 119.Google ScholarDigital Library
    3. Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth J. Carter, and Jessica K. Hodgins. 2013. Style and abstraction in portrait sketching. ACM Transactions on Graphics 32, 4 (2013), 55.Google ScholarDigital Library
    4. Hong Chen, Lin Liang, Ying-Qing Xu, Heung-Yeung Shum, and Nan-Ning Zheng. 2002. Example-Based Automatic Portraiture. In Proceedings of Asian Conference on Computer Vision. 171–178.Google Scholar
    5. Hong Chen, Ziqiang Liu, Chuck Rose, Yingqing Xu, Heung-Yeung Shum, and David Salesin. 2004. Example-Based Composite Sketching of Human Portraits. In Proceedings of International Symposium on Non-photorealistic Animation and Rendering. 95–102. Google ScholarDigital Library
    6. Hong Chen, Nanning Zheng, Lin Liang, Yan Li, Ying-Qing Xu, and Heung-Yeung Shum. 2002. PicToon: A Personalized Image-Based Cartoon System. In Proceedings of ACM International Conference on Multimedia. 171–178. Google ScholarDigital Library
    7. Doug DeCarlo and Anthony Santella. 2002. Stylization and Abstraction of Photographs. ACM Transactions on Graphics 21, 3 (2002), 769–776. Google ScholarDigital Library
    8. Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of Complex Image Appearance from Limited Exemplars. ACM Transactions on Graphics 34, 2 (2015), 22.Google ScholarDigital Library
    9. Steve DiPaola. 2007. Painterly rendered portraits from photographs using a knowledge-based approach. In Proceedings of SPIE Human Vision and Electronic Imaging, Vol. 6492. 33–43. Google ScholarCross Ref
    10. Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH Conference Proceedings. 341–346. Google ScholarDigital Library
    11. Jakub Fišer, Ondřej Jamriška, Michal Lukáč, Eli Shechtman, Paul Asente, Jingwan Lu, and Daniel Sýkora. 2016. StyLit: Illumination-Guided Example-Based Stylization of 3D Renderings. ACM Transactions on Graphics 35, 4 (2016), 92.Google ScholarDigital Library
    12. Jakub Fišer, Michal Lukáč, Ondřej Jamriška, Martin Čadík, Yotam Gingold, Paul Asente, and Daniel Sýkora. 2014. Color Me Noisy: Example-based Rendering of Hand-colored Animations with Temporal Noise Control. Computer Graphics Forum 33, 4 (2014), 1–10. Google ScholarDigital Library
    13. Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2414–2423. Google ScholarCross Ref
    14. Yihong Gong and Masao Sakauchi. 1995. Detection of Regions Matching Specified Chromatic Features. Computer Vision and Image Understanding 61, 2 (1995), 263–269. Google ScholarDigital Library
    15. Bruce Gooch, Erik Reinhard, and Amy Gooch. 2004. Human Facial Illustrations: Creation and Psychophysical Evaluation. ACM Transactions on Graphics 23, 1 (2004), 27–44. Google ScholarDigital Library
    16. James Hays and Irfan A. Essa. 2004. Image and Video Based Painterly Animation. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 113–120. Google ScholarDigital Library
    17. Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image Analogies. In SIGGRAPH Conference Proceedings. 327–340. Google ScholarDigital Library
    18. Aaron Jackson, Michel Valstar, and Georgios Tzimiropoulos. 2016. A CNN Cascade for Landmark Guided Semantic Part Segmentation. In Proceedings of ECCV 2016 Workshops, Geometry meets Deep Learning.Google ScholarCross Ref
    19. Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM Transactions on Graphics 34, 4 (2015), 92.Google ScholarDigital Library
    20. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In Proceedings of European Conference on Computer Vision. 694–711. Google ScholarCross Ref
    21. Scott F. Johnston. 2002. Lumo: Illumination for Cel Animation. In Proceedings of the 2Nd International Symposium on Non-photorealistic Animation and Rendering. 45-ff.Google ScholarDigital Library
    22. Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2 (2015), 349–360. Google ScholarDigital Library
    23. Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1867–1874. Google ScholarDigital Library
    24. Ira Kemelmacher-Shlizerman. 2016. Transfiguring Portraits. ACM Transactions on Graphics 35, 4 (2016), 94.Google ScholarDigital Library
    25. Vivek Kwatra, Irfan A. Essa, Aaron F. Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3 (2005), 795–802. Google ScholarDigital Library
    26. Jan Eric Kyprianidis, John Collomosse, Tinghuai Wang, and Tobias Isenberg. 2013. State of the “Art”: A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE Transactions on Visualization and Computer Graphics 19, 5 (2013), 866–885. Google ScholarDigital Library
    27. Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed-Form Solution to Natural Image Matting. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (2008), 228–242. Google ScholarDigital Library
    28. Hongliang Li, Guanghui Liu, and King Ngi Ngan. 2011. Guided Face Cartoon Synthesis. IEEE Transactions on Multimedia 13, 6 (2011), 1230–1239. Google ScholarDigital Library
    29. Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. SIFT Flow: Dense Correspondence across Scenes and Its Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5 (2011), 978–994. Google ScholarDigital Library
    30. Sifei Liu, Jimei Yang, Chang Huang, and Ming-Hsuan Yang. 2015. Multi-Objective Convolutional Learning for Face Labeling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
    31. Michal Lukáč, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. Brushables: Example-based Edge-aware Directional Texture Painting. Computer Graphics Forum 34, 7 (2015), 257–268. Google ScholarDigital Library
    32. Michal Lukáč, Jakub Fišer, Jean-Charles Bazin, Ondřej Jamriška, Alexander Sorkine-Hornung, and Daniel Sýkora. 2013. Painting by Feature: Texture Boundaries for Example-based Image Creation. ACM Transaction on Graphics 32, 4 (2013), 116.Google ScholarDigital Library
    33. Meng Meng, Mingtian Zhao, and Song Chun Zhu. 2010. Artistic paper-cut of human portraits. In Proceedings of ACM Multimedia. 931–934. Google ScholarDigital Library
    34. Gioacchino Noris, Daniel Sýkora, Stelian Coros, Brian Whited, Maryann Simmons, Alexander Hornung, Marcus Gross, and Robert Sumner. 2011. Temporal Noise Control for Sketchy Animation. In Proceedings of International Symposium on Non-photorealistic Animation and Rendering. 93–98. Google ScholarDigital Library
    35. Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, JoËlle Thollot, and David Salesin. 2008. Diffusion Curves: A Vector Representation for Smooth-Shaded Images. ACM Transactions on Graphics 27, 3 (2008), 92.Google ScholarDigital Library
    36. Javier Portilla and Eero P. Simoncelli. 2000. A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients. International Journal of Computer Vision 40, 1 (2000), 49–70. Google ScholarDigital Library
    37. Konstantinos Rematas, Tobias Ritschel, Mario Fritz, and Tinne Tuytelaars. 2014. Image-Based Synthesis and Re-synthesis of Viewpoints Guided by 3D Models.. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 3898–3905. Google ScholarDigital Library
    38. Lincoln Ritter, Wilmot Li, Brian Curless, Maneesh Agrawala, and David Salesin. 2006. Painting With Texture. In Proceedings of Eurographics Symposium on Rendering. 371–376.Google Scholar
    39. Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. 2016. Artistic Style Transfer for Videos. In Proceedings of German Conference Pattern Recognition. 26–36. Google ScholarCross Ref
    40. Scott Schaefer, Travis McPhail, and Joe Warren. 2006. Image Deformation Using Moving Least Squares. ACM Transactions on Graphics 25, 3 (2006), 533–540. Google ScholarDigital Library
    41. Ahmed Selim, Mohamed Elgharib, and Linda Doyle. 2016. Painting Style Transfer for Head Portraits Using Convolutional Neural Networks. ACM Transactions on Graphics 35, 4 (2016), 129.Google ScholarDigital Library
    42. Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian L. Price, Eli Shechtman, and Ian Sachs. 2016. Automatic Portrait Segmentation for Image Stylization. Computer Graphics Forum 35, 2 (2016), 93–102. Google ScholarDigital Library
    43. Yi-Chang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style Transfer for Headshot Portraits. ACM Transactions on Graphics 33, 4 (2014), 148.Google ScholarDigital Library
    44. Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014).Google Scholar
    45. Pawan Sinha, Benjamin Balas, Yuri Ostrovsky, and Richard Russell. 2006. Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About. Proc. IEEE 94, 11 (2006), 1948–1962. Google ScholarCross Ref
    46. Patrick Tresset and Frédéric F. Leymarie. 2005. Generative Portrait Sketching. In Proceedings of International Conference on Virtual Systems and Multimedia. 739–748.Google Scholar
    47. Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2013b. Transductive Face Sketch-Photo Synthesis. IEEE Transactions on Neural Networks and Learning Systems 24, 9 (2013), 1364–1376. Google ScholarCross Ref
    48. Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2014. A Comprehensive Survey to Face Hallucination. International Journal of Computer Vision 106, 1 (2014), 9–30. Google ScholarDigital Library
    49. Tinghuai Wang, John P. Collomosse, Andrew Hunter, and Darryl Greig. 2013a. Learnable Stroke Models for Example-based Portrait Painting. In Proceedings of British Machine Vision Conference. Google ScholarCross Ref
    50. Xiaogang Wang and Xiaoou Tang. 2009. Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 11 (2009), 1955–1967. Google ScholarDigital Library
    51. Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463–476. Google ScholarDigital Library
    52. Holger Winnemöller, Sven C. Olsen, and Bruce Gooch. 2006. Real-time video abstraction. ACM Transactions on Graphics 25, 3 (2006), 1221–1226. Google ScholarDigital Library
    53. Ming Yang, Shu Lin, Ping Luo, Liang Lin, and Hongyang Chao. 2010. Semantics-driven portrait cartoon stylization. In Proceedings of International Conference on Image Processing. 1805–1808. Google ScholarCross Ref
    54. Yue Yang, Hanli Zhao, Lihua You, Renlong Tu, Xueyi Wu, and Xiaogang Jin. 2015. Semantic portrait color transfer with internet images. Multimedia Tools and Applications (2015), 1–19.Google Scholar
    55. Kun Zeng, Mingtian Zhao, Caiming Xiong, and Song-Chun Zhu. 2009. From image parsing to painterly rendering. ACM Transactions on Graphics 29, 1 (2009), 2.Google ScholarDigital Library
    56. Yong Zhang, Weiming Dong, Oliver Deussen, Feiyue Huang, Ke Li, and Bao-Gang Hu. 2014. Data-driven Face Cartoon Stylization. In SIGGRAPH Asia Technical Briefs. 14. Google ScholarDigital Library
    57. Mingtian Zhao and Song-Chun Zhu. 2011. Portrait Painting Using Active Templates. In Proceedings of International Symposium on Non-Photorealistic Animation and Rendering. 117–124. Google ScholarDigital Library
    58. Hao Zhou, Zhanghui Kuang, and Kwan-Yee Kenneth Wong. 2012. Markov Weight Fields for face sketch synthesis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 1091–1097.Google Scholar
    59. Yang Zhou, Huajie Shi, Dani Lischinski, Minglun Gong, Johannes Kopf, and Hui Huang. 2017. Analysis and Controlled Synthesis of Inhomogeneous Textures. Computer Graphics Forum 36, 2 (2017). Google ScholarDigital Library

ACM Digital Library Publication: