“Video-based 3D motion capture through biped control” by Vondrak, Sigal, Hodgins and Jenkins

  • ©Marek Vondrak, Leonid Sigal, Jessica K. Hodgins, and Odest Jenkins




    Video-based 3D motion capture through biped control



    Marker-less motion capture is a challenging problem, particularly when only monocular video is available. We estimate human motion from monocular video by recovering three-dimensional controllers capable of implicitly simulating the observed human behavior and replaying this behavior in other environments and under physical perturbations. Our approach employs a state-space biped controller with a balance feedback mechanism that encodes control as a sequence of simple control tasks. Transitions among these tasks are triggered on time and on proprioceptive events (e.g., contact). Inference takes the form of optimal control where we optimize a high-dimensional vector of control parameters and the structure of the controller based on an objective function that compares the resulting simulated motion with input observations. We illustrate our approach by automatically estimating controllers for a variety of motions directly from monocular video. We show that the estimation of controller structure through incremental optimization and refinement leads to controllers that are more stable and that better approximate the reference motion. We demonstrate our approach by capturing sequences of walking, jumping, and gymnastics.


    1. Baraff, D. 1996. Linear-time dynamics using Lagrange multipliers. In ACM SIGGRAPH. Google ScholarDigital Library
    2. Bhat, K. S., Seitz, S. M., Popovic, J., and Khosla, P. 2002. Computing the physical parameters of rigid-body motion from video. In ECCV, 551–566. Google ScholarDigital Library
    3. Bo, L., and Sminchisescu, C. 2010. Twin Gaussian processes for structured prediction. IJCV 87, 1–2. Google ScholarDigital Library
    4. Brubaker, M. A., and Fleet, D. J. 2008. The Kneed Walker for human pose tracking. In IEEE CVPR.Google Scholar
    5. Brubaker, M. A., Fleet, D. J., and Hertzmann, A. 2007. Physics-based person tracking using simplified lower body dynamics. In IEEE CVPR.Google Scholar
    6. Brubaker, M. A., Sigal, L., and Fleet, D. J. 2009. Estimating contact dynamics. In ICCV.Google Scholar
    7. Chu, D., Shapiro, A., Allen, B., and Faloutsos, P. 2007. A dynamic controller toolkit. In ACM SIGGRAPH Video Game Symposium (Sandbox), 21–26. Google ScholarDigital Library
    8. Cline, M. 2002. Rigid Body Simulation with Contact and Constraints. Master’s thesis, The University of British Columbia.Google Scholar
    9. Coros, S., Beaudoin, P., and van de Panne, M. 2009. Robust task-based control policies for physics-based characters. In ACM Transactions on Graphics, vol. 28. Google ScholarDigital Library
    10. Crisis. 2006. http://crisis.sourceforge.net/.Google Scholar
    11. Dempster, W. T. 1955. Space requirements of the seated operator: Geometrical, kinematic, and mechanical aspects of the body with special reference to the limbs. Tech. rep., Wright-Patterson Air Force Base 55–159.Google Scholar
    12. Elgammal, A., Harwood, D., and Davis, L. 2000. Non-parametric model for background subtraction. In ECCV. Google ScholarDigital Library
    13. Gall, J., Stoll, C., de Aguiar, E., Theobalt, C., Rosenhahn, B., and Seidel, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In IEEE CVPR, 1746–1753.Google Scholar
    14. Hansen, N. 2006. The CMA evolution strategy: A comparing review. Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms., 75–102.Google Scholar
    15. Hodgins, J. K., Wooten, W. L., Brogan, D. C., and O’Brien, J. F. 1995. Animating human athletics. In ACM SIGGRAPH, 71–78. Google ScholarDigital Library
    16. Kwon, T., and Hodgins, J. 2010. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Google ScholarDigital Library
    17. Lee, Y., Kim, S., and Lee, J. 2010. Data-driven biped control. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
    18. Liu, C. K., Hertzmann, A., and Popović, Z. 2005. Learning physics-based motion style with nonlinear inverse optimization. ACM Transactions on Graphics 24, 3. Google ScholarDigital Library
    19. Liu, L., Yin, K., van de Panne, M., Shao, T., and Xu, W. 2010. Sampling-based contact-rich motion control. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
    20. Metaxas, D., and Terzopoulos, D. 1993. Shape and non-rigid motion estimation through physics-based synthesis. IEEE Transactions on PAMI 15, 6, 580–591. Google ScholarDigital Library
    21. Muico, U., Lee, Y., Popovic, J., and Popovic, Z. 2009. Contact-aware nonlinear control of dynamic characters. ACM Transactions on Graphics 28, 3. Google ScholarDigital Library
    22. Ngo, T., and Marks, J. 1993. Spacetime constraints revisited. In ACM SIGGRAPH. Google ScholarDigital Library
    23. ODE. 2006. http://www.ode.org/.Google Scholar
    24. Pang, J. S., and Facchinei, F. 2003. Finite-dimensional variational inequalities and complementarity problems (i). Springer.Google Scholar
    25. Schaal, S., and Schweighofer, N. 2005. Computational motor control in humans and robots. Cur. Op. in Neurobio., 6.Google Scholar
    26. Sigal, L., Balan, A., and Black, M. J. 2010. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision 87, 1–2, 4–27. Google ScholarDigital Library
    27. Silva, M. D., Abe, Y., and Popovic, J. 2008. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics 27, 3. Google ScholarDigital Library
    28. Sminchisescu, C., Kanaujia, A., and Metaxas, D. 2007. Bm3e: Discriminative density propagation for visual tracking. IEEE Transactions on PAMI 29, 11. Google ScholarDigital Library
    29. Tsai, Y.-Y., Cheng, K. B., Lin, W.-C., Lee, J., and Lee, T.-Y. 2010. Real-time physics-based 3d biped character animation using an inverted pendulum model. IEEE Transactions on Visualization and Computer Graphics 16, 2, 325–337. Google ScholarDigital Library
    30. Urtasun, R., J. Fleet, D., and Fua, P. 2006. Gaussian process dynamical models for 3d people tracking. In IEEE CVPR. Google ScholarDigital Library
    31. Vondrak, M., Sigal, L., and Jenkins, O. C. 2008. Physical simulation for probabilistic motion tracking. In IEEE CVPR.Google Scholar
    32. Wang, J., Fleet, D. J., and Hertzmann, A. 2009. Optimizing walking controllers. ACM Transactions on Graphics 28, 5. Google ScholarDigital Library
    33. Wang, J., Fleet, D. J., and Hertzmann, A. 2010. Optimizing walking controllers for uncertain inputs and environments. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
    34. Wei, X., and Chai, J. 2010. Videomocap: Modeling physically realistic human motion from monocular video sequences. ACM Transactions on Graphics 29, 4. Google ScholarDigital Library
    35. Wren, C. R., and Pentland, A. 1998. Dynamic models of human motion. In Automatic Face and Gesture Recognition. Google ScholarDigital Library
    36. Yin, K., Loken, K., and van de Panne, M. 2007. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics 26, 3. Google ScholarDigital Library
    37. Yin, K., Coros, S., Beaudoin, P., and van de Panne, M. 2008. Continuation methods for adapting simulated skills. ACM Transactions on Graphics 27, 3. Google ScholarDigital Library

ACM Digital Library Publication:

Overview Page: