“Video-based 3D motion capture through biped control” by Vondrak, Sigal, Hodgins and Jenkins

Marker-less motion capture is a challenging problem, particularly when only monocular video is available. We estimate human motion from monocular video by recovering three-dimensional controllers capable of implicitly simulating the observed human behavior and replaying this behavior in other environments and under physical perturbations. Our approach employs a state-space biped controller with a balance feedback mechanism that encodes control as a sequence of simple control tasks. Transitions among these tasks are triggered on time and on proprioceptive events (e.g., contact). Inference takes the form of optimal control where we optimize a high-dimensional vector of control parameters and the structure of the controller based on an objective function that compares the resulting simulated motion with input observations. We illustrate our approach by automatically estimating controllers for a variety of motions directly from monocular video. We show that the estimation of controller structure through incremental optimization and refinement leads to controllers that are more stable and that better approximate the reference motion. We demonstrate our approach by capturing sequences of walking, jumping, and gymnastics.

1. Baraff, D. 1996. Linear-time dynamics using Lagrange multipliers. In ACM SIGGRAPH. Google ScholarDigital Library
2. Bhat, K. S., Seitz, S. M., Popovic, J., and Khosla, P. 2002. Computing the physical parameters of rigid-body motion from video. In ECCV, 551–566. Google ScholarDigital Library
3. Bo, L., and Sminchisescu, C. 2010. Twin Gaussian processes for structured prediction. IJCV 87, 1–2. Google ScholarDigital Library
4. Brubaker, M. A., and Fleet, D. J. 2008. The Kneed Walker for human pose tracking. In IEEE CVPR.Google Scholar
5. Brubaker, M. A., Fleet, D. J., and Hertzmann, A. 2007. Physics-based person tracking using simplified lower body dynamics. In IEEE CVPR.Google Scholar
6. Brubaker, M. A., Sigal, L., and Fleet, D. J. 2009. Estimating contact dynamics. In ICCV.Google Scholar
7. Chu, D., Shapiro, A., Allen, B., and Faloutsos, P. 2007. A dynamic controller toolkit. In ACM SIGGRAPH Video Game Symposium (Sandbox), 21–26. Google ScholarDigital Library
8. Cline, M. 2002. Rigid Body Simulation with Contact and Constraints. Master’s thesis, The University of British Columbia.Google Scholar
9. Coros, S., Beaudoin, P., and van de Panne, M. 2009. Robust task-based control policies for physics-based characters. In ACM Transactions on Graphics, vol. 28. Google ScholarDigital Library
10. Crisis. 2006. http://crisis.sourceforge.net/.Google Scholar
11. Dempster, W. T. 1955. Space requirements of the seated operator: Geometrical, kinematic, and mechanical aspects of the body with special reference to the limbs. Tech. rep., Wright-Patterson Air Force Base 55–159.Google Scholar
12. Elgammal, A., Harwood, D., and Davis, L. 2000. Non-parametric model for background subtraction. In ECCV. Google ScholarDigital Library
13. Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., and Seidel, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In IEEE CVPR, 1746–1753.Google Scholar
14. Hansen, N. 2006. The CMA evolution strategy: A comparing review. Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms., 75–102.Google Scholar
15. Hodgins, J. K., Wooten, W. L., Brogan, D. C., and O’Brien, J. F. 1995. Animating human athletics. In ACM SIGGRAPH, 71–78. Google ScholarDigital Library
16. Kwon, T., and Hodgins, J. 2010. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Google ScholarDigital Library
17. Lee, Y., Kim, S., and Lee, J. 2010. Data-driven biped control. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
18. Liu, C. K., Hertzmann, A., and Popović, Z. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24, 3. Google ScholarDigital Library
19. Liu, L., Yin, K., van de Panne, M., Shao, T., and Xu, W. 2010. Sampling-based contact-rich motion control. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
20. Metaxas, D., and Terzopoulos, D. 1993. Shape and non-rigid motion estimation through physics-based synthesis. IEEE Transactions on PAMI 15, 6, 580–591. Google ScholarDigital Library
21. Muico, U., Lee, Y., Popovic, J., and Popovic, Z. 2009. Contact-aware nonlinear control of dynamic characters. ACM Transactions on Graphics 28, 3. Google ScholarDigital Library
22. Ngo, T., and Marks, J. 1993. Spacetime constraints revisited. In ACM SIGGRAPH. Google ScholarDigital Library
23. ODE. 2006. http://www.ode.org/.Google Scholar
24. Pang, J. S., and Facchinei, F. 2003. Finite-dimensional variational inequalities and complementarity problems (i). Springer.Google Scholar
25. Schaal, S., and Schweighofer, N. 2005. Computational motor control in humans and robots. Cur. Op. in Neurobio., 6.Google Scholar
26. Sigal, L., Balan, A., and Black, M. J. 2010. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision 87, 1–2, 4–27. Google ScholarDigital Library
27. Silva, M. D., Abe, Y., and Popovic, J. 2008. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics 27, 3. Google ScholarDigital Library
28. Sminchisescu, C., Kanaujia, A., and Metaxas, D. 2007. Bm3e: Discriminative density propagation for visual tracking. IEEE Transactions on PAMI 29, 11. Google ScholarDigital Library
29. Tsai, Y.-Y., Cheng, K. B., Lin, W.-C., Lee, J., and Lee, T.-Y. 2010. Real-time physics-based 3d biped character animation using an inverted pendulum model. IEEE Transactions on Visualization and Computer Graphics 16, 2, 325–337. Google ScholarDigital Library
30. Urtasun, R., J. Fleet, D., and Fua, P. 2006. Gaussian process dynamical models for 3d people tracking. In IEEE CVPR. Google ScholarDigital Library
31. Vondrak, M., Sigal, L., and Jenkins, O. C. 2008. Physical simulation for probabilistic motion tracking. In IEEE CVPR.Google Scholar
32. Wang, J., Fleet, D. J., and Hertzmann, A. 2009. Optimizing walking controllers. ACM Transactions on Graphics 28, 5. Google ScholarDigital Library
33. Wang, J., Fleet, D. J., and Hertzmann, A. 2010. Optimizing walking controllers for uncertain inputs and environments. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
34. Wei, X., and Chai, J. 2010. Videomocap: Modeling physically realistic human motion from monocular video sequences. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
35. Wren, C. R., and Pentland, A. 1998. Dynamic models of human motion. In Automatic Face and Gesture Recognition. Google ScholarDigital Library
36. Yin, K., Loken, K., and van de Panne, M. 2007. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics 26, 3. Google ScholarDigital Library
37. Yin, K., Coros, S., Beaudoin, P., and van de Panne, M. 2008. Continuation methods for adapting simulated skills. ACM Transactions on Graphics 27, 3. Google ScholarDigital Library