“Temporally Coherent Completion of Dynamic Shapes” by Li, Luo, Vlasic∗, Peers, Popović, et al. …

  • ©Hao Li, Linjie Luo, Daniel Vlasic, Pieter Peers, Jovan Popović, Mark Pauly, and Szymon Rusinkiewicz




    Temporally Coherent Completion of Dynamic Shapes



    We present a novel shape completion technique for creating temporally coherent watertight surfaces from real-time captured dynamic performances. Because of occlusions and low surface albedo, scanned mesh sequences typically exhibit large holes that persist over extended periods of time. Most conventional dynamic shape reconstruction techniques rely on template models or assume slow deformations in the input data. Our framework sidesteps these requirements and directly initializes shape completion with topology derived from the visual hull. To seal the holes with patches that are consistent with the subject’s motion, we first minimize surface bending energies in each frame to ensure smooth transitions across hole boundaries. Temporally coherent dynamics of surface patches are obtained by unwarping all frames within a time window using accurate interframe correspondences. Aggregated surface samples are then filtered with a temporal visibility kernel that maximizes the use of nonoccluded surfaces. A key benefit of our shape completion strategy is that it does not rely on long-range correspondences or a template model. Consequently, our method does not suffer error accumulation typically introduced by noise, large deformations, and drastic topological changes. We illustrate the effectiveness of our method on several high-resolution scans of human performances captured with a state-of-the-art multiview 3D acquisition system.


    Ahmed, N., Theobalt, C., Dobrev, P., Seidel, H.-P., and Thrun, S. 2008. Robust fusion of dynamic shape and normal capture for high-quality reconstruction of time-varying geometry. In Proceedings of the IEEE Computer Vision and Pattern Recognition Conference.Google Scholar
    Alexa, M. 2003. Differential coordinates for local mesh morphing and deformation. Vis. Comput. 19, 2, 105–114.Google ScholarCross Ref
    Allen, B., Curless, B., and Popović, Z. 2002. Articulated body deformation from range scan data. ACM Trans. Graph. 21, 3, 612–619. Google ScholarDigital Library
    Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. Scape: Shape completion and animation of people. ACM Trans. Graph. 24, 3, 408–416. Google ScholarDigital Library
    Bay, H., Tuytelaars, T., and Van Gool, L. 2008. SURF: Speeded up robust features. Comput. Vis. Image Underst. 10, 3, 346–359. Google ScholarDigital Library
    Botsch, M. and Sorkine, O. 2008. On linear variational surface deformation methods. IEEE Trans. Vis. Comput. Graph. 14, 1, 213–230. Google ScholarDigital Library
    Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boubekeur, T. 2008. Markerless garment capture. ACM Trans. Graph. 27, 3, 99. Google ScholarDigital Library
    Brown, B. J. and Rusinkiewicz, S. 2007. Global non-rigid alignment of 3-d scans. ACM Trans. Graph. 26, 3, 21. Google ScholarDigital Library
    Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proceedings of the 8th European Conference on Computer Vision. 25–36.Google Scholar
    Carr, J. C., Fright, W. R., and Beatson, R. K. 1997. Surface interpolation with radial basis functions for medical imaging. IEEE Trans. Med. Imag. 16, 96–107.Google ScholarCross Ref
    Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. ACM Trans. Graph. 22, 3, 569–577. Google ScholarDigital Library
    Chang, W. and Zwicker, M. 2009. Range scan registration using reduced deformable models. Comput. Graph. Forum 28, 2, 447–456.Google ScholarCross Ref
    Chuang, M., Luo, L., Brown, B. J., Rusinkiewicz, S., and Kazhdan, M. 2009. Estimating the Laplace-Beltrami operator by restricting 3D functions. In Proceedings of the Symposium on Geometry Processing. Google ScholarDigital Library
    Corazza, S., Mündermann, L., Chaudhari, A., Demattio, T., Cobelli, C., and Andriacchi, T. P. 2006. A markerless motion capture system to study musculoskeletal biomechanics: Visual hull and simulated annealing approach. Ann. Biomed. Engin. 34, 6, 1019–1029.Google ScholarCross Ref
    Curless, B. and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proceedings of SIGGRAPH. Computer Graphics Proceedings, Annual Conference Series, 303–312. Google ScholarDigital Library
    Davis, J., Marschner, S. R., Garr, M., and Levoy, M. 2002. Filling holes in complex surfaces using volumetric diffusion. In Proceedings of the Symposium on 3D Data Processing, Visualization, and Transmission. 428–438.Google Scholar
    Davis, J., Nehab, D., Ramamoorthi, R., and Rusinkiewicz, S. 2005. Spacetime stereo: A unifying framework for depth from triangulation. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2, 296–302. Google ScholarDigital Library
    de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graph. 27, 3, 98. Google ScholarDigital Library
    de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H.-P. 2007. Marker-less deformable mesh tracking for human shape and motion capture. In Proceedings of the Conference on Computer Vision and Pattern Recognition.Google Scholar
    Gelfand, N., Rusinkiewicz, S., Ikemoto, L., and Levoy, M. 2003. Geometrically stable sampling for the icp algorithm. In Proceedings of the International Conference on 3D Digital Imaging and Modeling. 260.Google Scholar
    Held, M. 1998. Fist: Fast industrial-strength triangulation. Tech. rep.Google Scholar
    Ju, T. 2009. Fixing geometric errors on polygonal models: A survey. J. Comput. Sci. Technol. 24, 1, 19–29. Google ScholarDigital Library
    Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proceedings of the Symposium on Geometry Processing. Google ScholarDigital Library
    Kojekine, N., Savchenko, V., Senin, M., and Hagiwara, I. 2002. Real-time 3d deformations by means of compactly supported radial basis functions. In Proceedings of Eurographics Short Papers. 35–43.Google Scholar
    Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
    Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421–1430. Google ScholarDigital Library
    Liepa, P. 2003. Filling holes in meshes. In Proceedings of the Symposium on Geometry Processing. 200–205. Google ScholarDigital Library
    Mitra, N. J., Flory, S., Ovsjanikov, M., Gelfand, N., Guibas, L., and Pottmann, H. 2007. Dynamic geometry registration. In Proceedings of the Symposium on Geometry Processing. 173–182. Google ScholarDigital Library
    Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Trans. Graph. 24, 3, 536–543. Google ScholarDigital Library
    Pekelny, Y. and Gotsman, C. 2008. Articulated object reconstruction and markerless motion capture from depth video. Comput. Graph. Forum 27, 2, 399–408.Google ScholarCross Ref
    Sand, P., McMillan, L., and Popović, J. 2003. Continuous capture of skin deformation. ACM Trans. Graph. 22, 3, 578–586. Google ScholarDigital Library
    Sharf, A., Alcantara, D. A., Lewiner, T., Greif, C., Sheffer, A., Amenta, N., and Cohen-Or, D. 2008. Space-Time surface reconstruction using incompressible flow. ACM Trans. Graph. 27, 5, 1–10. Google ScholarDigital Library
    Sibson, R. and Stone, G. 1991. Computation of thin-plate splines. SIAM J. Sci. Statist. Comput. 12, 6, 1304–1313. Google ScholarDigital Library
    Starck, J. and Hilton, A. 2003. Model-based multiple view reconstruction of people. In Proceedings of the International Conference on Computer Vision. 915–922. Google ScholarDigital Library
    Süssmuth, J., Winter, M., and Greiner, G. 2008. Reconstructing animated meshes from time-varying point clouds. In Proceedings of the Symposium on Geometry Processing. 27, 5, 1469–1476. Google ScholarDigital Library
    Theobalt, C., Ahmed, N., Lensch, H., Magnor, M., and Seidel, H.-P. 2007. Seeing people in different light-joint shape, motion, and reflectance capture. IEEE Trans. Vis. Comput. Graph. 13, 4, 663–674. Google ScholarDigital Library
    Vlasic, D., Baran, I., Matusik, W., and Popović, J. 2008. Articulated mesh animation from multi-view silhouettes. ACM Trans. Graph. 27, 3, 97. Google ScholarDigital Library
    Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., and Matusik, W. 2009. Dynamic shape capture using multi-view photometric stereo. In SIGGRAPH Asia’09 ACM SIGGRAPH Asia 2009 Papers. 1–11. Google ScholarDigital Library
    Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., and Schilling, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3d scanner data. ACM Trans. Graph. 28, 2, 15. Google ScholarDigital Library
    Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., and Schilling, A. 2007. Reconstruction of deforming geometry from time-varying point clouds. In Proceedings of the Symposium on Geometry Processing. Google ScholarDigital Library
    Weise, T., Leibe, B., and Gool, L. V. 2007. Fast 3d scanning with automatic motion compensation. In Proceedings of the Conference on Computer Vision and Pattern Recognition.Google Scholar
    Zhang, L., Curless, B., Hertzmann, A., and Seitz, S. M. 2003. Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo. In Proceedings of the International Conference on Computer Vision. 618. Google ScholarDigital Library
    Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548–558. Google ScholarDigital Library
    Zheng, Q., Sharf, A., Tagliasacchi, A., Chen, B., Zhang, H., Sheffer, A., and Cohen-Or, D. 2010. Consensus skeleton for non-rigid space-time registration. Comput. Graph. Forum 29, 2, 635–644.Google ScholarCross Ref

ACM Digital Library Publication: