“Video-based characters: creating new human performances from a multi-view video database” by Xu, Liu, Stoll, Tompkin, Bharaj, et al. …

  • ©Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gaurav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz, and Christian Theobalt




    Video-based characters: creating new human performances from a multi-view video database



    We present a method to synthesize plausible video sequences of humans according to user-defined body motions and viewpoints. We first capture a small database of multi-view video sequences of an actor performing various basic motions. This database needs to be captured only once and serves as the input to our synthesis algorithm. We then apply a marker-less model-based performance capture approach to the entire database to obtain pose and geometry of the actor in each database frame. To create novel video sequences of the actor from the database, a user animates a 3D human skeleton with novel motion and viewpoints. Our technique then synthesizes a realistic video sequence of the actor performing the specified motion based only on the initial database. The first key component of our approach is a new efficient retrieval strategy to find appropriate spatio-temporally coherent database frames from which to synthesize target video frames. The second key component is a warping-based texture synthesis approach that uses the retrieved most-similar database frames to synthesize spatio-temporally coherent target video frames. For instance, this enables us to easily create video sequences of actors performing dangerous stunts without them being placed in harm’s way. We show through a variety of result videos and a user study that we can synthesize realistic videos of people, even if the target motions and camera views are different from the database content.


    1. Ballan, L., and Cortelazzo, G. M. 2008. Marker-less motion capture of skinned models in a four camera set-up using optical flow and silhouettes. In 3DPVT.Google Scholar
    2. Ballan, L., Brostow, G. J., Puwein, J., and Pollefeys, M. 2010. Unstructured video-based rendering: Interactive exploration of casually captured videos. ACM TOG (Proc. SIGGRAPH), 1–11. Google Scholar
    3. Baran, I., and Popovic, J. 2007. Automatic rigging and animation of 3d characters. ACM TOG (SIGGRAPH) 26, 3, 72. Google ScholarDigital Library
    4. Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM TOG (Proc. SIGGRAPH) 27, 3, 99. Google ScholarDigital Library
    5. Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. 2001. Unstructured lumigraph rendering. In SIGGRAPH, 425–432. Google Scholar
    6. Cagniart, C., Boyer, E., and Ilic, S. 2010. Free-form mesh tracking: a patch-based approach. In Proc. IEEE CVPR, 1–8.Google Scholar
    7. Carranza, J., Theobalt, C., Magnor, M., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. In ACM TOG (Proc. SIGGRAPH). Google Scholar
    8. Celly, B., and Zordan, V. 2004. Animated people textures. In Proc. of CASA, 331–338.Google Scholar
    9. Cheung, G. 2003. Visual Hull Construction, Alignment and Refinement for Human Kinematic Modeling, Motion Capture and Rendering. PhD thesis, Carnegie Mellon University. Google Scholar
    10. Cobzas, D., Yerex, K., and Jagersand, M. 2002. Dynamic textures for image-based rendering of fine-scale 3d structure and animation of non-rigid motion. In In Eurographics, 1067–7055.Google Scholar
    11. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM TOG (SIGGRAPH) 27, 1–10. Google ScholarDigital Library
    12. Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In SIGGRAPH, 11–20. Google Scholar
    13. Einarsson, P., Chabert, C.-F., Jones, A., Ma, W.-C., Lamond, B., im Hawkins, Bolas, M., Sylwan, S., and Debevec, P. 2006. Relighting human locomotion with flowed reflectance fields. In Proc. EGSR, 183–194. Google Scholar
    14. Flagg, M., Nakazawa, A., Zhang, Q., Kang, S. B., Ryu, Y. K., Essa, I., and Rehg, J. M. 2009. Human video textures. In Proc. of I3D, 199–206. Google Scholar
    15. Gall, J., Stoll, C., Aguiar, E., Theobalt, C., Rosenhahn, B., and Seidel, H.-P. 2009. Motion capture using joint skeleton tracking and surface estimation. In Proc. IEEE CVPR, 1746–1753.Google Scholar
    16. Gleicher, M. 1998. Retargetting motion to new characters. In SIGGRAPH ’98, 33–42. Google Scholar
    17. Hornung, A., and Kobbelt, L. 2009. Interactive pixel-accurate free viewpoint rendering from images with silhouette aware sampling. Comput. Graph. Forum 28, 8, 2090–2103.Google ScholarCross Ref
    18. Hornung, A., Dekkers, E., and Kobbelt, L. 2007. Character animation from 2d pictures and 3d motion data. ACM TOG 26, 1, 1:1–1:9. Google Scholar
    19. Huang, P., Hilton, A., and Starck, J. 2009. Human motion synthesis from 3d video. In Proc. CVPR, 1478–1485.Google Scholar
    20. Jain, A., Thormählen, T., Seidel, H.-P., and Theobalt, C. 2010. Moviereshape: tracking and reshaping of humans in videos. ACM TOG (Proc. SIGGRAPH Asia) 29, 148:1–148:10. Google Scholar
    21. Jimenez, J., Scully, T., Barbosa, N., Donner, C., Alvarez, X., Vieira, T., Matts, P., Orvalho, V., Gutierrez, D., and Weyrich, T. 2010. A practical appearance model for dynamic facial color. ACM TOG (Proc. SIGGRAPH Asia) 29, 141:1–141:10. Google Scholar
    22. Kemelmacher-Shlizerman, I., Sankar, A., Shechtman, E., and Seitz, S. M. 2010. Being john malkovich. In Proc. of ECCV, 341–353. Google ScholarDigital Library
    23. Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. ACM TOG (Proc. SIGGRAPH) 27, 3, 38:1–38:9. Google Scholar
    24. Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. SIGGRAPH ’00, 369–374. Google ScholarDigital Library
    25. Mori, G., Berg, A., Efros, A., Eden, A., and Malik, J. 2004. Video based motion synthesis by splicing and morphing. UC Berkeley Technical Reports, No. UCB/CSD-4-1337.Google Scholar
    26. Narayanan, P. J., Rander, P., and Kanade, T. 1998. Constructing virtual worlds using dense stereo. In Proc. of ICCV, 3–10. Google ScholarDigital Library
    27. Schaefer, S., McPhail, T., and Warren, J. D. 2006. Image deformation using moving least squares. ACM TOG (Proc. SIGGRAPH) 25, 3, 533–540. Google ScholarDigital Library
    28. Schödl, A., and Essa, I. 2002. Controlled animation of video sprites. In Proc. of SCA, 121–127. Google Scholar
    29. Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In SIGGRAPH, 489–498. Google Scholar
    30. Starck, J., Miller, G., and Hilton, A. 2005. Video-based character animation. In Proc. of SCA, 49–58. Google Scholar
    31. Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008. View and Time Interpolation in Image Space. Computer Graphics Forum (Proc. Pacific Graphics) 27, 7.Google Scholar
    32. Stoll, C., Gall, J., de Aguiar, E., Thrun, S., and Theobalt, C. 2010. Video-based reconstruction of animatable human characters. ACM TOG (Proc. SIGGRAPH Asia) 29, 139:1–139:10. Google Scholar
    33. Theobalt, C., Wuermlin, S., de Aguiar, E., and Nieder-berger, C. 2007. New trends in 3d video. In Eurographics Courses.Google Scholar
    34. Tung, T., Nobuhara, S., and Matsuyama, T. 2009. Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo. In Proc. IEEE ICCV, 1709–1716.Google Scholar
    35. Vlasic, D., Baran, I., Matusik, W., and Popović, J. 2008. Articulated mesh animation from multi-view silhouettes. ACM TOG (Proc. SIGGRAPH ’08). Google Scholar
    36. Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., and Matusik, W. 2009. Dynamic shape capture using multi-view photometric stereo. In ACM TOG (Proc. SIGGRAPH Asia ’09). Google Scholar
    37. Waschbüsch, M., Würmlin, S., and Gross, M. 2006. Interactive 3d video editing. Vis. Comput. 22, 631–641. Google ScholarDigital Library
    38. Weyrich, T., Pfister, H., and Gross, M. 2005. Rendering deformable surface reflectance fields. IEEE TVCG 11, 48–58. Google Scholar
    39. Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM TOG (Proc. SIGGRAPH) 24, 765–776. Google ScholarDigital Library
    40. Zhou, S., Fu, H., Liu, L., Cohen-Or, D., and Han, X. 2010. Parametric reshaping of human bodies in images. ACM TOG (Proc. SIGGRAPH) 29, 4, 126:1–126:10. Google Scholar
    41. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S. A. J., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM TOG (Proc. SIGGRAPH) 23, 3, 600–608. Google ScholarDigital Library

ACM Digital Library Publication: