“Optimizing UI layouts for deformable face-rig manipulation” by Kim and Singh

Complex deformable face-rigs have many independent parameters that control the shape of the object. A human face has upwards of 50 parameters (FACS Action Units), making conventional UI controls hard to find and operate. Animators address this problem by tediously hand-crafting in-situ layouts of UI controls that serve as visual deformation proxies, and facilitate rapid shape exploration. We propose the automatic creation of such in-situ UI control layouts. We distill the design choices made by animators into mathematical objectives that we optimize as the solution to an integer quadratic programming problem. Our evaluation is three-fold: we show the impact of our design principles on the resulting layouts; we show automated UI layouts for complex and diverse face rigs, comparable to animator handcrafted layouts; and we conduct a user study showing our UI layout to be an effective approach to face-rig manipulation, preferable to a baseline slider interface.

1. Rinat Abdrashitov, Fanny Chevalier, and Karan Singh. 2020. Interactive Exploration and Refinement of Facial Expression Using Manifold Learning. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 778–790. Google ScholarDigital Library
2. Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz. 2011. Design Principles for Visual Communication. Commun. ACM 54, 4 (April 2011), 60–69. Google ScholarDigital Library
3. Eric Allen and Kelly L. Murdock. 2008. Body Language: Advanced 3D Character Rigging (pap/cdr ed.). SYBEX Inc., USA.Google Scholar
4. Norman I Badler, Brian A Barsky, and David Zeltzer. 1990. Making Them Move: Mechanics, Control & Animation of Articulated Figures. Routledge.Google Scholar
5. Stephen W. Bailey, Dalton Omens, Paul Dilorenzo, and James F. O’Brien. 2020. Fast and Deep Facial Deformations. ACM Trans. Graph. 39, 4, Article 94 (July 2020), 15 pages. Google ScholarDigital Library
6. Eric Allan Bier. 1987. Skitters and jacks: interactive 3D positioning tools. In Proceedings of the 1986 workshop on Interactive 3D graphics. ACM, 183–196.Google ScholarDigital Library
7. Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. 2010. Polygon mesh processing. CRC press.Google Scholar
8. Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103–112.Google ScholarDigital Library
9. Ozan Cetinaslan and Verónica Orvalho. 2018. Direct Manipulation of Blendshapes Using a Sketch-Based Interface. In Proceedings of the 23rd International ACM Conference on 3D Web Technology (Poznań, Poland) (Web3D ’18). Association for Computing Machinery, New York, NY, USA, Article 14, 10 pages. Google ScholarDigital Library
10. Pif Edwards, Chris Landreth, Mateusz Popławski, Robert Malinowski, Sarah Watling, Eugene Fiume, and Karan Singh. 2020. JALI-Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 60, 2 pages. Google ScholarDigital Library
11. Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.Google Scholar
12. Krzysztof Z Gajos, Mary Czerwinski, Desney S Tan, and Daniel S Weld. 2006. Exploring the design space for adaptive graphical user interfaces. In Proceedings of the working conference on Advanced visual interfaces. 201–208.Google ScholarDigital Library
13. Sarah Gibson, Paul Beardsley, Wheeler Ruml, Thomas Kang, Brian Mirtich, Joshua Seims, William Freeman, Jessica Hodgins, Hanspeter Pfister, Joe Marks, et al. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. (1997).Google Scholar
14. Michael Gleicher. 1992. Integrating constraints and direct manipulation. In Symposium on Interactive 3 D Graphics: Proceedings of the 1992 symposium on Interactive 3 D graphics, Vol. 1992. 171–174.Google ScholarDigital Library
15. Martin Guay, Marie-Paule Cani, and Rémi Ronfard. 2013. The line of action: an intuitive interface for expressive character posing. ACM Transactions on Graphics (TOG) 32, 6 (2013), 1–8.Google ScholarDigital Library
16. Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 65–74.Google ScholarDigital Library
17. Paul G Kry, Doug L James, and Dinesh K Pai. 2002. Eigenskin: real time large deformation character skinning in hardware. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation. 153–159.Google ScholarDigital Library
18. Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.Google ScholarDigital Library
19. Binh H. Le, Mingyang Zhu, and Zhigang Deng. 2013. Marker Optimization for Facial Motion Acquisition and Deformation. IEEE Transactions on Visualization and Computer Graphics 19, 11 (Nov. 2013), 1859–1871. Google ScholarDigital Library
20. John P Lewis and Ken-ichi Anjyo. 2010. Direct manipulation blendshapes. IEEE Computer Graphics and Applications 30, 4 (2010), 42–50.Google ScholarDigital Library
21. Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-Fly Correctives. ACM Trans. Graph. 32, 4, Article 42 (July 2013), 10 pages. Google ScholarDigital Library
22. José Carlos Miranda, Xenxo Alvarez, João Orvalho, Diego Gutierrez, A Augusto Sousa, and Verónica Orvalho. 2011. Sketch express: facial expressions made easy. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling. ACM, 87–94.Google ScholarDigital Library
23. Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. 2012. Symmetry in 3D Geometry: Extraction and Applications. In EUROGRAPHICS State-of-the-art Report. Google ScholarDigital Library
24. Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. WIREs Data Mining and Knowledge Discovery 2, 1 (2012), 86–97. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/widm.53 Google ScholarCross Ref
25. Victor Navone. 2020. Facial Animation for Feature Animated Films: Animating stylized facial expressions. https://www.thegnomonworkshop.com/tutorials/facial-animation-for-feature-animated-films (2020).Google Scholar
26. Jason Osipa. 2010. Stop Staring: Facial Modeling and Animation Done Right (3rd ed.). SYBEX Inc., USA.Google ScholarDigital Library
27. Rick Parent. 2012. Computer animation: algorithms and techniques. Newnes.Google Scholar
28. Ryan Schmidt, Karan Singh, and Ravin Balakrishnan. 2008. Sketching and composing widgets for 3d manipulation. In Computer graphics forum, Vol. 27. Wiley Online Library, 301–310.Google Scholar
29. Thomas W Sederberg and Scott R Parry. 1986. Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 151–160.Google ScholarDigital Library
30. Mike Seymour. 2016. Put your (digital) game face on. (2016). https://www.fxguide.com/fxfeatured/put-your-digital-game-face-on/Google Scholar
31. Mike Seymour. 2018. Making Thanos Face the Avengers. (2018). https://www.fxguide.com/fxfeatured/making-thanos-face-the-avengers/Google Scholar
32. Mike Seymour. 2019. Bebyface in Bebylon. (2019). https://www.fxguide.com/fxfeatured/bebyface-in-bebylon/Google Scholar
33. Karan Singh and Eugene Fiume. 1998. Wires: a geometric deformation technique. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 405–414.Google ScholarDigital Library
34. Marc P Stevens, Robert C Zeleznik, and John F Hughes. 1994. An architecture for an extensible 3D interface toolkit. In Proceedings of the 7th annual ACM symposium on User interface software and technology. 59–67.Google ScholarDigital Library
35. Tanasai Sucontphunt, Zhenyao Mo, Ulrich Neumann, and Zhigang Deng. 2008. Interactive 3D facial expression posing through 2D portrait manipulation. In Proceedings of graphics interface 2008. Canadian Information Processing Society, 177–184.Google Scholar
36. Jerry O Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Transactions on Graphics-TOG 28, 5 (2009), 167.Google ScholarDigital Library
37. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-Based Linear 3D Face Models. ACM Trans. Graph. 30, 4, Article 76 (July 2011), 10 pages. Google ScholarDigital Library
38. Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Theobalt Christian. 2017. MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction. In The IEEE International Conference on Computer Vision (ICCV).Google Scholar
39. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM transactions on graphics (TOG), Vol. 30. ACM.Google Scholar
40. Lance Williams. 1990. Performance-driven facial animation. In ACM SIGGRAPH Computer Graphics, Vol. 24. ACM, 235–242.Google ScholarDigital Library
41. Eduard Zell, JP Lewis, Junyong Noh, Mario Botsch, et al. 2017. Facial retargeting with automatic range of motion alignment. ACM Transactions on Graphics (TOG) 36, 4 (2017), 154.Google Scholar