“Model Flowing: Capturing and tracking of deformable geometry”

In this sketch we present a new markerless deformable model capture system which is more accurate and controllable than previous methods. In recent years there has been considerable interest in applying computer vision methods to capture the geometry of de- formable objects [Zhang et al. 2004] such as human faces and cloth. For many applications, markerless methods are desirable since they allow the possibility of recovery of geometry, texture and lighting at the same time. Recent image-based methods employing optical flow have successfully been applied to facial animations in actual production [Borshukov et al. 2003].
We assume that the initial pose of the deforming object is known. Artists typically build the geometry for an object from a range scan of the object, computing the deformation for this geometry is ideal. After optical flow analysis of the image sequences from each camera, the 2D motion of each vertex can be used to triangulate the approximate movement of the vertex in 3D[Borshukov et al. 2003]. Unfortunately, since optical flow is an ill-posed problem and is computed independently for each image, this triangulation will likely have significant errors due to matching errors and smoothing in the original optical flow. We have modified the basic optical flow algorithm so that the movement of the 3D vertex is solved for directly and in all views simultaneously. This technique incorporates the epipolar constraint across the cameras, reducing the search space and resulting in higher accuracy and less drift.

Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for “the matrix reloaded”. In Proceedings of the SIGGRAPH 2003 Conference on Sketches & Applications, 1–1.]]
Lucas, B., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence ’81, 674–679.]]
Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: high resolution capture for modelling and animation. In In Proceedings of the 2004 SIGGRAPH Conference, 548–558.]]