“Animation Cartography – Intrinsic Reconstruction of Shape and Motion” by Tevs, Berner, Wand, Ihrke, Bokeloh, et al. …

In this article, we consider the problem of animation reconstruction, that is, the reconstruction of shape and motion of a deformable object from dynamic 3D scanner data, without using user-provided template models. Unlike previous work that addressed this problem, we do not rely on locally convergent optimization but present a system that can handle fast motion, temporally disrupted input, and can correctly match objects that disappear for extended time periods in acquisition holes due to occlusion. Our approach is motivated by cartography: We first estimate a few landmark correspondences, which are extended to a dense matching and then used to reconstruct geometry and motion. We propose a number of algorithmic building blocks: a scheme for tracking landmarks in temporally coherent and incoherent data, an algorithm for robust estimation of dense correspondences under topological noise, and the integration of local matching techniques to refine the result. We describe and evaluate the individual components and propose a complete animation reconstruction pipeline based on these ideas. We evaluate our method on a number of standard benchmark datasets and show that we can obtain correct reconstructions in situations where other techniques fail completely or require additional user guidance such as a template model.

Ahmed, N., Theobalt, C., Rossl, C., Thrun, S., and Seidel, H.-P. 2008. Dense correspondence finding for parameterization-free animation reconstruction from video. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’08). 1–8.Google Scholar
Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22, 3. Google ScholarDigital Library
Anguelov, D., Srinivasan, P., Pang, H.-C., Koller, D., Thrun, S., and Davis, J. 2004. The correlated correspondence algorithm for unsupervised registration of nonrigid surfaces. In Proceedings of the Conference on Neural Information Processing Systems (NIPS’04).Google Scholar
Anuar, N. and Guskov, I. 2004. Extracting animated meshes with adaptive motion estimation. In Proceedings of the Vision, Modeling, and Visualization Workshop (VMV’04). 63–71.Google Scholar
Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-Scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33. Google ScholarDigital Library
Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 41:1-41:10. Google ScholarDigital Library
Bradley, D., Popa, T., Sheffer, A., Heidrich, W., and Boufekeur, T. 2008. Markerless garment capture. ACM Trans. Graph. 27, 3, 99. Google ScholarDigital Library
Bronstein, A. M., Bronstein, M. M., and Kimmel, R. 2006. Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching. Proc. Nat. Acad. Sci. 103, 5, 1168–1172.Google ScholarCross Ref
Bronstein, A. M., Bronstein, M. M., and Kimmel, R. 2009. Topology-Invariant similarity of nonrigid shapes. Int. J. Comput. Vis. 81, 3, 281–301. Google ScholarDigital Library
Bronstein, A. M., Bronstein, M. M., Kimmel, R., Mahmoudi, M., and Sapiro, G. 2010. A gromov-hausdorff framework with diffusion geometry for topologically-robust nonrigid shape matching. Int. J. Comput. Vis. 89, 266–286. Google ScholarDigital Library
Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-Viewpoint video of human actors. ACM Trans. Graph. 22, 569–577. Google ScholarDigital Library
Chang, W. and Zwicker, M. 2008. Automatic registration for articulated shapes. Comput. Graph. Forum 27, 5, 1459–1468. Google ScholarDigital Library
Chang, W. and Zwicker, M. 2009. Range scan registration using reduced deformable models. Comput. Graph. Forum 28, 2, 447–456.Google ScholarCross Ref
Chum, O. and Matas, J. 2005. Matching with PROSAC—Progressive sample consensus. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’05). Vol. 1. 220–226. Google ScholarDigital Library
Davis, J., Nehab, D., Ramamoorthi, R., and Rusinkiewicz, S. 2005. Spacetime stereo: A unifying framework for depth from triangularization. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2, 296–302. Google ScholarDigital Library
De Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graph. 27, 3, 98. Google ScholarDigital Library
Fang, Q., Gao, J., Guibas, L. J., de Silva, V., and Zhang, L. 2005. Glider: Gradient landmark-based distributed routing for sensor networks. In Proceedings of the 24th Conference of the IEEE Communications Society (InfoCom).Google Scholar
Gelfand, N. and Guibas, L. J. 2004. Shape segmentation using local slippage analysis. In Proceedings of the Symposium on Geometry Processing (SGP’04). 214–223. Google ScholarDigital Library
Häehnel, D., Thrun, S., and Burgard, W. 2003. An extension of the icp algorithm for modeling nonrigid objects with mobile robots. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). 915–920. Google ScholarDigital Library
Horn, B. K. P. 1987. Closed-Form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer. A4, 4, 629–642.Google ScholarCross Ref
Huang, Q.-X., Adams, B., Wicke, M., and Guibas, L. J. 2008. Nonrigid registration under isometric deformations. Comput. Graph. Forum 27, 5, 1449–1457. Google ScholarDigital Library
Huang, Q.-X., Flöry, S., Gelfand, N., Hofer, M., and Pottmann, H. 2006. Reassembling fractured objects by geometric matching. ACM Trans. Graph. 25, 3, 569–578. Google ScholarDigital Library
König, S. and Gumhold, S. 2008. Image-Based motion compensation for structured light scanning of dynamic scenes. Int. J. Int. Syst. Tech. Appl. 5, 3-4, 434–441. Google ScholarDigital Library
Leordenu, M. and Hebert, M. 2005. A spectral technique for correspondence problems using pairwise constraints. In Proceedings of the International Conference on Computer Vision (ICCV’05). Vol. 2, 1482–1489. Google ScholarDigital Library
Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5, 175. Google ScholarDigital Library
Li, H., Sumner, R. W., and Pauly, M. 2008. Global correspondence optimization for nonrigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421–1430. Google ScholarDigital Library
Liao, M., Zhang, Q., Wang, H., Yang, R., and Gong, M. 2009. Modeling deformable objects from a single depth camera. In Proceedings of the International Conference on Computer Vision (ICCV’09).Google Scholar
Lipman, Y. and Funkhouser, T. 2009. Möbius voting for surface correspondence. In Proceedings of the SIGGRAPH’09 Conference. ACM Press, New York, 1–12. Google ScholarDigital Library
Lucas, B. D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’81). 674–679. Google ScholarDigital Library
Mitra, N. J., Flory, S., Ovsjanikov, M., Gelfand, N., Guibas, L., and Pottsmann, H. 2007. Dynamic geometry registration. In Proceedings of the Symposium on Geometry Processing (SGP’07). 173–182. Google ScholarDigital Library
Ovsjanikov, M., Mérigot, Q., Mémoli, F., and Guibas, L. 2010. One point isometric matching with the heat kernel. In Proceedings of the Symposium on Geometry Processing (SGP’10). 1555–1564.Google Scholar
Park, S. I. and Hodgins, J. K. 2006. Capturing and animating skin deformation in human motion. ACM Trans. Graph. 25, 3, 881–889. Google ScholarDigital Library
Pekelny, Y. and Gotsman, C. 2008. Articulated object reconstruction and markerless motion capture from depth video. Comput. Graph. Forum 27, 2, 399–408.Google ScholarCross Ref
Popa, T., South-Dickenson, I., Bradley, D., Sheffer, A., and Heidrich, W. 2010. Globally consistent space-time reconstruction. Comput. Graph. Forum 29, 5, 1633–1642.Google ScholarCross Ref
Sand, P., McMillan, L., and Popović, J. 2003. Continuous capture of skin deformation. ACM Trans. Graph. 22, 3, 578–586. Google ScholarDigital Library
Sharf, A., Alcantara, D. A., Lewiner, T., Greif, C., Sheffer, A., Amenta, N., and Cohen-Or, D. 2008. Space-Time surface reconstruction using incompressible flow. ACM Trans. Graph. 27, 5, 110. Google ScholarDigital Library
Starck, J. and Hilton, A. 2007. Correspondence labeling for wide-timeframe free-form surface matching. In Proceedings of the International Conference on Computer Vision (ICCV’07). 1–8.Google Scholar
Süssmuth, J., Winter, M., and Greiner, G. 2008. Reconstructing animated meshes from time-varying point clouds. Comput. Graph. Forum 27, 5, 1469–1476. Google ScholarDigital Library
Tevs, A., Berner, A., Wand, M., Ihrke, I., and Seidel, H.-P. 2011. Intrinsic shape matching by planned landmark sampling. Comput. Graph. Forum 30, 543–552.Google ScholarCross Ref
Tevs, A., Bokeloh, M., Wand, M., Schilling, A., and Seidel, H.-P. 2009. Isometric registration of ambiguous and partial data. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’09). 1185–1192.Google Scholar
Varanasi, K., Zaharescu, A., Boyer, E., and Horaud, R. P. 2008. Temporal surface tracking using mesh evolution. In Proceedings of the European Conference on Computer Vision (ECCV’08). 30–43. Google ScholarDigital Library
Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., and Matusik, W. 2009. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph. 28, 5, 174. Google ScholarDigital Library
Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., and Schilling, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM Trans. Graph. 28, 2, 1–15. Google ScholarDigital Library
Wand, M., Jenke, P., Huang, Q., Bokeloh, M., Guibas, L., and Schilling, A. 2007. Reconstruction of deforming geometry from time-varying point clouds. In Proceedings of the Symposium on Geometric Processing (SGP’07). 49–58. Google ScholarDigital Library
Weise, T., Leife, B., and Gool, L. V. 2007. Fast 3D scanning with automatic motion compensation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’07). 1–8.Google Scholar
Würmlin, S., Lamboray, E., Staadt, O. G., and Gross, M. H. 2002. 3D video recorder. In Proceedings of the 10th Pacific Conference on Computer Graphics and Applications (PG’02). IEEE Computer Society, Washington, DC, 325. Google ScholarDigital Library
Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Space-Time faces: High resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548–558. Google ScholarDigital Library
Zheng, Q., Sharf, A., Tagliasacchi, A., Chen, B., Zhang, H., Sheffer, A., and Cohen-Or, D. 2010. Consensus skeleton for nonrigid space-time registration. Comput. Graph. Forum 29, 2, 635–644.Google ScholarCross Ref
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-Quality video view interpolation using a layered representation. In ACM SIGGRAPH’04 Papers. ACM, New York, 600–608. Google ScholarDigital Library