“Lip-sync animation from HMM using dynamic features” by Yotsukura, Kawamoto and Nakamura

This paper describes a method to create human-like talking-faces speech animation with naturalness and realism by mapping from speech information to a facial movement sequence using Hidden Markov Models (HMMs). In our previous research [Yotsukura and Morishima 2005b], we presented a speech animation method based on tri-viseme HMMs using static feature vectors (i.e. facial motion vectors). These HMMs consist of a 3-state left-to-right model and its output, where the demanded facial-motion vector sequences can be segmented for each HMM state. However, the output sequences’  trajectories are not smooth (step-like trajectories) because the out- put vectors compose the mean vectors on optimum HMM state. In  order to create smooth trajectories, we develop a new 3D facial animation system using an HMM-based method considering dynamic feature vectors (i.e. velocity facial motion vectors and acceleration facial motion vectors), which originally applied to 2D image syn- thesis [Sako et al. 2000] . Our method applies 3D facial objects and derives smooth motion trajectories that are similar to those of real  facial motion. Furthermore, the output can be mapped to many dif- ferent kinds of speech animation applications with 3D human-like characters.  

1. Sako, S., Tokuda, K., Masuko, T., Kobayashi, T., and Ki-Tamura, T. 2000. Hmm-based text-to-audio-visual speech synthesis. In ICSLP, vol. 3, 25–28.
2. Yotsukura, Tatsuo Amd Nakamura, S., and Morishima, S. 2005. Construction of audio-visual speech corpus using motion-capture system and corpus based facial animation. The IEICE Transaction on Information and System E 88-D, 11, 2377–2483.
3. Yotsukura, Tatsuo Amd Nakamura, S., and Morishima, S. 2005. Speech to talking heads system based on hidden markov models. In SIGGRAPH 2005 Poster, Full Conference DVD-ROM Disc 2, ACM, 028-yotsukura.pdf.