“Optimizing UI layouts for deformable face-rig manipulation” by Kim and Singh

  • ©Joonho Kim and Karan Singh




    Optimizing UI layouts for deformable face-rig manipulation



    Complex deformable face-rigs have many independent parameters that control the shape of the object. A human face has upwards of 50 parameters (FACS Action Units), making conventional UI controls hard to find and operate. Animators address this problem by tediously hand-crafting in-situ layouts of UI controls that serve as visual deformation proxies, and facilitate rapid shape exploration. We propose the automatic creation of such in-situ UI control layouts. We distill the design choices made by animators into mathematical objectives that we optimize as the solution to an integer quadratic programming problem. Our evaluation is three-fold: we show the impact of our design principles on the resulting layouts; we show automated UI layouts for complex and diverse face rigs, comparable to animator handcrafted layouts; and we conduct a user study showing our UI layout to be an effective approach to face-rig manipulation, preferable to a baseline slider interface.


    1. Rinat Abdrashitov, Fanny Chevalier, and Karan Singh. 2020. Interactive Exploration and Refinement of Facial Expression Using Manifold Learning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 778–790. Google ScholarDigital Library
    2. Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz. 2011. Design Principles for Visual Communication. Commun. ACM 54, 4 (April 2011), 60–69. Google ScholarDigital Library
    3. Eric Allen and Kelly L. Murdock. 2008. Body Language: Advanced 3D Character Rigging (pap/cdr ed.). SYBEX Inc., USA.Google Scholar
    4. Norman I Badler, Brian A Barsky, and David Zeltzer. 1990. Making Them Move: Mechanics, Control & Animation of Articulated Figures. Routledge.Google Scholar
    5. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020), 15 pages. Google ScholarDigital Library
    6. Eric Allan Bier. 1987. Skitters and jacks: interactive 3D positioning tools. In Proceedings of the 1986 workshop on Interactive 3D graphics. ACM, 183–196.Google ScholarDigital Library
    7. Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. 2010. Polygon mesh processing. CRC press.Google Scholar
    8. Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103–112.Google ScholarDigital Library
    9. Ozan Cetinaslan and Verónica Orvalho. 2018. Direct Manipulation of Blendshapes Using a Sketch-Based Interface. In Proceedings of the 23rd International ACM Conference on 3D Web Technology (Poznań, Poland) (Web3D ’18). Association for Computing Machinery, New York, NY, USA, Article 14, 10 pages. Google ScholarDigital Library
    10. Pif Edwards, Chris Landreth, Mateusz Popławski, Robert Malinowski, Sarah Watling, Eugene Fiume, and Karan Singh. 2020. JALI-Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 60, 2 pages. Google ScholarDigital Library
    11. Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.Google Scholar
    12. Krzysztof Z Gajos, Mary Czerwinski, Desney S Tan, and Daniel S Weld. 2006. Exploring the design space for adaptive graphical user interfaces. In Proceedings of the working conference on Advanced visual interfaces. 201–208.Google ScholarDigital Library
    13. Sarah Gibson, Paul Beardsley, Wheeler Ruml, Thomas Kang, Brian Mirtich, Joshua Seims, William Freeman, Jessica Hodgins, Hanspeter Pfister, Joe Marks, et al. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. (1997).Google Scholar
    14. Michael Gleicher. 1992. Integrating constraints and direct manipulation. In Symposium on Interactive 3 D Graphics: Proceedings of the 1992 symposium on Interactive 3 D graphics, Vol. 1992. 171–174.Google ScholarDigital Library
    15. Martin Guay, Marie-Paule Cani, and Rémi Ronfard. 2013. The line of action: an intuitive interface for expressive character posing. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1–8.Google ScholarDigital Library
    16. Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 65–74.Google ScholarDigital Library
    17. Paul G Kry, Doug L James, and Dinesh K Pai. 2002. Eigenskin: real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 153–159.Google ScholarDigital Library
    18. Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.Google ScholarDigital Library
    19. Binh H. Le, Mingyang Zhu, and Zhigang Deng. 2013. Marker Optimization for Facial Motion Acquisition and Deformation. IEEE Transactions on Visualization and Computer Graphics 19, 11 (Nov. 2013), 1859–1871. Google ScholarDigital Library
    20. John P Lewis and Ken-ichi Anjyo. 2010. Direct manipulation blendshapes. IEEE Computer Graphics and Applications 30, 4 (2010), 42–50.Google ScholarDigital Library
    21. Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-Fly Correctives. ACM Trans. Graph. 32, 4, Article 42 (July 2013), 10 pages. Google ScholarDigital Library
    22. José Carlos Miranda, Xenxo Alvarez, João Orvalho, Diego Gutierrez, A Augusto Sousa, and Verónica Orvalho. 2011. Sketch express: facial expressions made easy. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling. ACM, 87–94.Google ScholarDigital Library
    23. Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. 2012. Symmetry in 3D Geometry: Extraction and Applications. In EUROGRAPHICS State-of-the-art Report. Google ScholarDigital Library
    24. Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. WIREs Data Mining and Knowledge Discovery 2, 1 (2012), 86–97. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/widm.53 Google ScholarCross Ref
    25. Victor Navone. 2020. Facial Animation for Feature Animated Films: Animating stylized facial expressions. https://www.thegnomonworkshop.com/tutorials/facial-animation-for-feature-animated-films (2020).Google Scholar
    26. Jason Osipa. 2010. Stop Staring: Facial Modeling and Animation Done Right (3rd ed.). SYBEX Inc., USA.Google ScholarDigital Library
    27. Rick Parent. 2012. Computer animation: algorithms and techniques. Newnes.Google Scholar
    28. Ryan Schmidt, Karan Singh, and Ravin Balakrishnan. 2008. Sketching and composing widgets for 3d manipulation. In Computer graphics forum, Vol. 27. Wiley Online Library, 301–310.Google Scholar
    29. Thomas W Sederberg and Scott R Parry. 1986. Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 151–160.Google ScholarDigital Library
    30. Mike Seymour. 2016. Put your (digital) game face on. (2016). https://www.fxguide.com/fxfeatured/put-your-digital-game-face-on/Google Scholar
    31. Mike Seymour. 2018. Making Thanos Face the Avengers. (2018). https://www.fxguide.com/fxfeatured/making-thanos-face-the-avengers/Google Scholar
    32. Mike Seymour. 2019. Bebyface in Bebylon. (2019). https://www.fxguide.com/fxfeatured/bebyface-in-bebylon/Google Scholar
    33. Karan Singh and Eugene Fiume. 1998. Wires: a geometric deformation technique. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 405–414.Google ScholarDigital Library
    34. Marc P Stevens, Robert C Zeleznik, and John F Hughes. 1994. An architecture for an extensible 3D interface toolkit. In Proceedings of the 7th annual ACM symposium on User interface software and technology. 59–67.Google ScholarDigital Library
    35. Tanasai Sucontphunt, Zhenyao Mo, Ulrich Neumann, and Zhigang Deng. 2008. Interactive 3D facial expression posing through 2D portrait manipulation. In Proceedings of graphics interface 2008. Canadian Information Processing Society, 177–184.Google Scholar
    36. Jerry O Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Transactions on Graphics-TOG 28, 5 (2009), 167.Google ScholarDigital Library
    37. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-Based Linear 3D Face Models. ACM Trans. Graph. 30, 4, Article 76 (July 2011), 10 pages. Google ScholarDigital Library
    38. Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. 2017. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV).Google Scholar
    39. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM transactions on graphics (TOG), Vol. 30. ACM.Google Scholar
    40. Lance Williams. 1990. Performance-driven facial animation. In ACM SIGGRAPH Computer Graphics, Vol. 24. ACM, 235–242.Google ScholarDigital Library
    41. Eduard Zell, JP Lewis, Junyong Noh, Mario Botsch, et al. 2017. Facial retargeting with automatic range of motion alignment. ACM Transactions on Graphics (TOG) 36, 4 (2017), 154.Google Scholar

ACM Digital Library Publication: