“Synthesizing realistic facial expressions from photographs” by Pighin, Hecker, Lischinski, Szeliski and Salesin

We present new techniques for creating photorealistic textured 3D facial models from photographs of a human subject, and for creating smooth transitions between different facial expressions by morphing between these different models. Starting from several uncalibrated views of a human subject, we employ a user-assisted technique to recover the camera poses corresponding to the views as well as the 3D coordinates of a sparse set of chosen locations on the subject’s face. A scattered data interpolation technique is then used to deform a generic face mesh to fit the particular geometry of the subject’s face. Having recovered the camera poses and the facial geometry, we extract from the input images one or more texture maps for the model. This process is repeated for several facial expressions of a particular subject. To generate transitions between these facial expressions we use 3D shape morphing between the corresponding face models, while at the same time blending the corresponding textures. Using our technique, we have been able to generate highly realistic face models and natural looking animations.

1. Takaaki Akimoto, Yasuhito Suenaga, and Richard S. Wallace. Automatic Creation of 3D Facial Models. IEEE Computer Graphics and Applications, 13(5): 16-22, September 1993.
2. Alias I Wavefront, Toronto, Ontario. Alias V7.0, 1995.
3. Thaddeus Beier and Shawn Neely. Feature-based Image Metamorphosis. In SIGGRAPH 92 Conference Proceedings, pages 35-42. ACM SIGGRAPH, July 1992.
4. Philippe Bergeron and Pierre Lachapelle. Controlling Facial Expressions and Body Movements in the Computer-Generated Animated Short “Tony De Peltrie”. In SIGGRAPH 85 Advanced Computer Animation seminar notes. July 1985.
5. Christoph Bregler, Michele Covell, and Malcolm Slaney. Video Rewrite: Driving Visual Speech with Audio. In SIGGRAPH 97 Conference Proceedings, pages 353-360. ACM SIGGRAPH, August 1997.
6. David T. Chen, Andrei State, and David Banks. Interactive Shape Metamorphosis. In 1995 Symposium on Interactive 3D Graphics, pages 43-44. ACM SIGGRAPH, April 1995.
7. Chang S. Choi, Kiyoharu, Hiroshi Harashima, and Tsuyoshi Takebe. Analysis and Synthesis of Facial Image Sequences in Model-Based Image Coding. In IEEE Transactions on Circuits and Systems for Video Technology, volume 4, pages 257 – 275. June 1994.
8. Cyberware Laboratory, Inc, Monterey, California. 4020/RGB 3D Scanner with Color Digitizer, 1990.
9. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach. In SIGGRAPH 96 Conference Proceedings, pages 11-20. ACM SIGGRAPH, August 1996.
10. Eben Ostby, Pixar Animation Studios. Personal communication, January 1997.
11. Paul Ekman and Wallace V. Friesen. Unmasking the Face. A guide to recognizing emotions fron facial clues. Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1975.
12. Paul Ekman and Wallace V. Friesen. Manual for the Facial Action Coding System. Consulting Psychologists Press, Inc., Palo Alto, California, 1978.
13. Irfan Essa, Sumit Basu, Trevor Darrell, and Alex Pentland. Modeling, Tracking and Interactive Animation of Faces and Heads Using Input from Video. In Computer Animation Conference, pages 68-79. June 1996.
14. Gary Faigin. The Artist’s Complete Guide to Facial Expression. Watson-Guptill Publications, New York, 1990.
15. Olivier Faugeras. Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press, Cambridge, Massachusetts, 1993.
16. G. Golub and C. F. Van Loan. Matrix Computation, third edition. The John Hopkins University Press, Baltimore and London, 1996.
17. Brian Guenter, Cindy Grimm, Daniel Wood, Henrique Malvar, and Fr6d6ric Pighin. Making Faces. In SIGGRAPH 98 Conference Proceedings. ACM SIGGRAPH, July 1998.
18. Pat Hanrahan and Wolfgang Krueger. Reflection from Layered Surfaces Due to Subsurface Scattering. In SIGGRAPH 93 Conference Proceedings, volume 27, pages 165-174. ACM SIGGRAPH, August 1993.
19. Bright Star Technologies Inc. Beginning Reading Software. Sierra On- Line, Inc., 1993.
20. Horace H. S. Ip and Lijun Yin. Constructing a 3D Individualized Head Model from Two Orthogonal Views. The Visual Computer, 12:254- 266, 1996.
21. Gregory Ward J., Francis M. Rubinstein, and Robert D. Clear. A Ray Tracing Solution for Diffuse Interreflection. In SIGGRAPH 88 Conference Proceedings, volume 22, pages 85-92. August 1988.
22. James R. Kent, Wayne E. Carlson, and Richard E. Parent. Shape Transformation for Polyhedral Objects. In SIGGRAPH 92 Proceedings Conference, volume 26, pages 47-54. ACM SIGGRAPH, July 1992.
23. Rolf M. Koch, Markus H. Gross, Friedrich R. Carls, Daniel F. von Bfiren, George Fankhauser, and Yoav I. H. Parish. Simulating Facial Surgery Using Finite Element Methods. In SIGGRAPH 96 Conference Proceedings, pages 421-428. ACM SIGGRAPH, August 1996.
24. Tsuneya Kurihara and Kiyoshi Arai. A Transformation Method for Modeling and Animation of the Human Face from Photographs. In Nadia Magnenat Thalmann and Daniel Thalmann, editors, Computer Animation 91, pages 45-58. Springer-Verlag, Tokyo, 1991.
25. A. Lanitis, C. J. Taylor, and T. F. Cootes. A Unified Approach for Coding and Interpreting Face Images. In Fifth International Conference on Computer Vision (ICCV 95), pages 368-373. Cambridge, Massachusetts, June 1995.
26. C. L. Lawson and R. J. Hansen. Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs, 1974.
27. Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, and George Wolberg. Image Metamorphosis Using Snakes and Free-Form Deformations. In SIGGRAPH 95 Conference Proceedings, pages 439-448. ACM SIGGRAPH, August 1995.
28. Seung-Yong Lee, George Wolberg, Kyung-Yong Chwa, and Sung Yong Shin. Image Metamorphosis with Scattered Feature Constraints. IEEE Transactions on Visualization and Computer Graphics, 2(4), December 1996.
29. Yuencheng Lee, Demetri Terzopoulos, and Keith Waters. Realistic Modeling for Facial Animation. In SIGGRAPH 95 Conference Proceedings, pages 55-62. ACM SIGGRAPH, August 1995.
30. Yuencheng C. Lee, Demetri Terzopoulos, and Keith Waters. Constructing Physics-Based Facial Models of Individuals. In Proceedings of Graphics Interface 93, pages 1-8. May 1993.
31. Francis H. Moffitt and Edward M. Mikhail. Photogrammetry. Harper & Row, New York, 3 edition, 1980.
32. Shree K. Nayar, Katsushi Ikeuchi, and Takeo Kanade. Shape from Interreflections. International Journal of Computer Vision, 6:173-195, 1991.
33. Gregory M. Nielson. Scattered Data Modeling. IEEE Computer Graphics and Applications, 13(1):60-70, January 1993.
34. Frederic I. Parke. Computer Generated Animation of Faces. Proceedings ACM annual conference., August 1972.
35. Frederic I. Parke. A Parametric Model for Human Faces. PhD thesis, University of Utah, Salt Lake City, Utah, December 1974. UTEC-CSc- 75 -047.
36. Frederic I. Parke and Keith Waters. Computer Facial Animation. A K Peters, Wellesley, Massachusetts, 1996.
37. W.H. Press, B. R Flannery, S. A. Teukolsky, and W. T. Vetterling. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, Cambridge, England, second edition, 1992.
38. Karl Pulli, Michael Cohen, Tom Duchamp, Hugues Hoppe, Linda Shapiro, and Werner Stuetzle. View-based rendering: Visualizing real objects from scanned range and color data. In Proc. 8th Eurographics Workshop on Rendering. June 1997.
39. Steven M. Seitz and Charles R. Dyer. View Morphing. In SIGGRAPH 96 Conference Proceedings, Annual Conference Series, pages 21-30. ACM SIGGRAPH, August 1996.
40. Chester C. Slama, editor. Manual of Photogrammetry. American Society of Photogrammetry, Falls Church, Virginia, fourth edition, 1980.
41. Richard Szeliski and Sing Bing Kang. Recovering 3D Shape and Motion from Image Streams using Nonlinear Least Squares. Journal of Visual Communication and Image Representation, 5(1): 10-28, March 1994.
42. Richard Szeliski and Heung-Yeung Shum. Creating Full View Panoramic Image Mosaics and Texture-Mapped Models. In SIG- GRAPH 97 Conference Proceedings, pages 251-258. ACM SIG- GRAPH, August 1997.
43. Demetri Terzopoulos and Keith Waters. Physically-based Facial Modeling, Analysis, and Animation. Journal of Visualization and Computer Animation, 1 (4):73-80, March 1990.
44. Kristinn R. Th6risson. Gandalf: An Embodied Humanoid Capable of Real-Time Multimodal Dialogue with People. In First ACM International Conference on Autonomous Agents. 1997.
45. Michael W. Vannier, Jeffrey F. Marsh, and James O. Warren. Threedimentional Computer Graphics for Craniofacial Surgical Planning and Evaluation. In SIGGRAPH 83 Conference Proceedings, volume 17, pages 263-273. ACM SIGGRAPH, August 1983.
46. Keith Waters. A Muscle Model for Animating Three-Dimensional Facial Expression. In SIGGRAPH 87 Conference Proceedings), volume 21, pages 17-24. ACM SIGGRAPH, July 1987.
47. Lance Williams. Performance-Driven Facial Animation. In SIG- GRAPH 90 Conference Proceedings, volume 24, pages 235-242. August 1990.
48. Z. Zhang, K. Isono, and S. Akamatsu. Euclidean Structure from Uncalibrated Images Using Fuzzy Domain Knowledge: Application to Facial Images Synthesis. In Proc. International Conference on Computer Vision (ICCV’98). January 1998.