“Constraining dense hand surface tracking with elasticity” by Smith, Wu, Wen, Peluse, Sheikh, et al. …
Conference:
Type(s):
Title:
- Constraining dense hand surface tracking with elasticity
Session/Category Title: Hands and Faces
Presenter(s)/Author(s):
Abstract:
Many of the actions that we take with our hands involve self-contact and occlusion: shaking hands, making a fist, or interlacing our fingers while thinking. This use of of our hands illustrates the importance of tracking hands through self-contact and occlusion for many applications in computer vision and graphics, but existing methods for tracking hands and faces are not designed to treat the extreme amounts of self-contact and self-occlusion exhibited by common hand gestures. By extending recent advances in vision-based tracking and physically based animation, we present the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures, for both single hands and two hands. By constraining a vision-based tracking algorithm with a physically based deformable model, we obtain an algorithm that is robust to the ubiquitous self-interactions and massive self-occlusions exhibited by common hand gestures, allowing us to track two hand interactions and some of the most difficult possible configurations of a human hand.
References:
1. Pierre Alliez, David Cohen-Steiner, Mariette Yvinec, and Mathieu Desbrun. 2005. Variational Tetrahedral Meshing. ACM Trans. Graph. 24, 3 (2005), 617–625.Google ScholarDigital Library
2. S. Baek, K. I. Kim, and T. Kim. 2019. Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1067–1076.Google Scholar
3. David Baraff, Andrew Witkin, and Michael Kass. 2003. Untangling Cloth. ACM Trans. Graph. 22, 3 (July 2003), 862–870.Google ScholarDigital Library
4. Vincent Barrielle, Nicolas Stoiber, and Cédric Cagniart. 2016. BlendForces: A Dynamic Framework for Facial Animation. Comput. Graph. Forum 35, 2 (2016), 341–352.Google ScholarCross Ref
5. Thabo Beeler, Derek Bradley, Bernd Bickel, and Marku Gross. 2014. Medusa Performance Capture. https://studios.disneyresearch.com/medusa/.Google Scholar
6. Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. 2011. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Trans. Graph. 30, 4 (2011), 75:1–75:10.Google ScholarDigital Library
7. James R. Bergen, P. Anandan, Keith J. Hanna, and Rajesh Hingorani. 1992. Hierarchical Model-Based Motion Estimation. In Computer Vision — ECCV’92. Springer Berlin Heidelberg, Berlin, Heidelberg, 237–252.Google Scholar
8. James C. Bezdek and Richard J. Hathaway. 2003. Convergence of Alternating Optimization. Neural, Parallel Sci. Comput. 11, 4 (2003), 351–368.Google Scholar
9. Bernd Bickel, Moritz Bächer, Miguel A. Otaduy, Wojciech Matusik, Hanspeter Pfister, and Markus Gross. 2009. Capture and Modeling of Non-Linear Heterogeneous Soft Tissue. ACM Trans. Graph. 28, 3, Article 89 (July 2009), 9 pages.Google ScholarDigital Library
10. Jean-Yves Bouguet. 2001. Pyramidal Implementation of the Affine Lucas Kanade Feature Tracker Description of the Algorithm. Intel corporation 5, 1–10 (2001), 4.Google Scholar
11. Xiang Chen, Changxi Zheng, Weiwei Xu, and Kun Zhou. 2014. An Asymptotic Numerical Method for Inverse Elastic Shape Design. ACM Trans. Graph. 33, 4, Article 95 (July 2014), 11 pages.Google ScholarDigital Library
12. Matthew Cong, Michael Bao, Jane L. E, Kiran S. Bhat, and Ronald Fedkiw. 2015. Fully Automatic Generation of Anatomical Face Simulation Models. In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Association for Computing Machinery, New York, NY, USA, 175–183.Google ScholarDigital Library
13. Edilson de Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun. 2008. Performance Capture from Sparse Multi-View Video. ACM Trans. Graph. 27, 3 (2008), 1–10.Google ScholarDigital Library
14. G. D. Evangelidis and E. Z. Psarakis. 2008. Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 10 (2008), 1858–1865.Google ScholarDigital Library
15. François Faure, Sébastien Barbier, Jérémie Allard, and Florent Falipou. 2008. Image-Based Collision Detection and Response between Arbitrary Volume Objects. In Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, Goslar, DEU, 155–162.Google Scholar
16. Graham Fyffe, Andrew Jones, Oleg Alexander, Ryosuke Ichikari, and Paul Debevec. 2015. Driving High-Resolution Facial Scans with Video Performance Capture. ACM Trans. Graph. 34, 1 (2015), 8:1–8:14.Google ScholarDigital Library
17. G. Fyffe, K. Nagano, L. Huynh, S. Saito, J. Busch, A. Jones, H. Li, and P. Debevec. 2017. Multi-View Stereo on Consistent Face Topology. Comput. Graph. Forum 36, 2 (2017), 295–309.Google ScholarDigital Library
18. S. Galliani, K. Lasinger, and K. Schindler. 2015. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE, 873–881.Google Scholar
19. L. Ge, Z. Ren, Y. Li, Z. Xue, Y. Wang, J. Cai, and J. Yuan. 2019. 3D Hand Shape and Pose Estimation From a Single RGB Image. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 10825–10834.Google Scholar
20. Oliver Glauser, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019a. Deformation Capture via Self-Sensing Capacitive Arrays. ACM Trans. Graph. 38, 2 (2019), 16:1–16:16.Google ScholarDigital Library
21. Oliver Glauser, Shihao Wu, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019b. Interactive Hand Pose Estimation using a Stretch-Sensing Soft Glove. ACM Trans. Graph. 38, 4 (2019), 41:1–41:15.Google ScholarDigital Library
22. David Harmon, Etienne Vouga, Breannan Smith, Rasmus Tamstorf, and Eitan Grinspun. 2009. Asynchronous Contact Mechanics. ACM Trans. Graph. 28, 3, Article 87 (July 2009), 12 pages.Google ScholarDigital Library
23. Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, I. Laptev, and C. Schmid. 2019. Learning Joint Reconstruction of Hands and Manipulated Objects. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 11799–11808.Google Scholar
24. Bruno Heidelberger, Matthias Teschner, Richard Keiser, Matthias Müller, and Markus H Gross. 2004. Consistent Penetration Depth Estimation for Deformable Collision Response. In VMV, Vol. 4. 339–346.Google Scholar
25. G. Hirota, S. Fisher, A. State, C. Lee, and H. Fuchs. 2001. An Implicit Finite Element Method for Elastic Solids in Contact. In Proceedings of Computer Animation. 136–254.Google Scholar
26. Yixin Hu, Qingnan Zhou, Xifeng Gao, Alec Jacobson, Denis Zorin, and Daniele Panozzo. 2018. Tetrahedral Meshing in the Wild. ACM Trans. Graph. 37, 4, Article 60 (July 2018), 14 pages.Google ScholarDigital Library
27. Nikolaos Kyriazis Iason Oikonomidis and Antonis Argyros. 2011. Efficient Model-Based 3D Tracking of Hand Articulations using Kinect. In Proceedings of the British Machine Vision Conference. BMVA Press, 101.1–101.11.Google ScholarCross Ref
28. Geoffrey Irving, Craig Schroeder, and Ronald Fedkiw. 2007. Volume Conserving Finite Element Simulations of Deformable Models. ACM Trans. Graph. 26, 3 (July 2007), 13:1–13:6.Google ScholarDigital Library
29. G. Irving, J. Teran, and R. Fedkiw. 2004. Invertible Finite Elements for Robust Simulation of Large Deformation. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, Goslar, DEU, 131–140.Google Scholar
30. Shuangshuang Jin, Robert R. Lewis, and David West. 2005. A Comparison of Algorithms for Vertex Normal Computation. The Visual Computer 21, 1 (01 Feb 2005), 71–82.Google Scholar
31. Petr Kadleček, Alexandru-Eugen Ichim, Tiantian Liu, Jaroslav Křivánek, and Ladislav Kavan. 2016. Reconstructing Personalized Anatomical Models for Physics-Based Body Animation. ACM Trans. Graph. 35, 6, Article 213 (Nov. 2016), 13 pages.Google ScholarDigital Library
32. Petr Kadleček and Ladislav Kavan. 2019. Building Accurate Physics-Based Face Models from Data. Proc. ACM Comput. Graph. Interact. Tech. 2, 2, Article 15 (July 2019), 16 pages.Google ScholarDigital Library
33. Michael Kass, Andrew Witkin, and Demetri Terzopoulos. 1988. Snakes: Active Contour Models. International Journal of Computer Vision 1, 4 (1988), 321–331.Google ScholarCross Ref
34. J. P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Fred Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. In Eurographics 2014 – State of the Art Reports. The Eurographics Association.Google Scholar
35. Miles Macklin, Kenny Erleben, Matthias Müller, Nuttapong Chentanez, Stefan Jeschke, and Viktor Makoviychuk. 2019. Non-Smooth Newton Methods for Deformable Multi-Body Dynamics. ACM Trans. Graph. 38, 5, Article 140 (Oct. 2019), 20 pages.Google ScholarDigital Library
36. Aleka McAdams, Yongning Zhu, Andrew Selle, Mark Empey, Rasmus Tamstorf, Joseph Teran, and Eftychios Sifakis. 2011. Efficient Elasticity for Character Skinning with Contact and Collisions. ACM Trans. Graph. 30, 4, Article 37 (July 2011), 12 pages.Google ScholarDigital Library
37. Dimitri Metaxas and Demetri Terzopoulos. 1993. Shape and Nonrigid Motion Estimation through Physics-Based Synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence 15, 6 (1993), 580–591.Google ScholarDigital Library
38. Neil Molino, Robert Bridson, Joseph Teran, and Ronald Fedkiw. 2003. A Crystalline, Red Green Strategy for Meshing Highly Deformable Objects with Tetrahedra. In IMR. 103–114.Google Scholar
39. Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, and Christian Theobalt. 2019. Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera. ACM Trans. Graph. 38, 4 (2019), 49:1–49:13.Google ScholarDigital Library
40. Matthias Müller, Julie Dorsey, Leonard McMillan, Robert Jagnow, and Barbara Cutler. 2002. Stable Real-Time Deformations. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA ’02). Association for Computing Machinery, New York, NY, USA, 49–54.Google ScholarDigital Library
41. I. Oikonomidis, N. Kyriazis, and A. A. Argyros. 2011. Full DOF Tracking of a Hand Interacting with an Object by Modeling Occlusions and Physical Constraints. In 2011 International Conference on Computer Vision. IEEE, 2088–2095.Google Scholar
42. Dinesh K. Pai, Austin Rothwell, Pearson Wyder-Hodge, Alistair Wick, Ye Fan, Egor Larionov, Darcy Harrison, Debanga Raj Neog, and Cole Shing. 2018. The Human Touch: Measuring Contact with Real Human Soft Tissues. ACM Trans. Graph. 37, 4, Article 58 (July 2018), 12 pages.Google ScholarDigital Library
43. Zherong Pan, Hujun Bao, and Jin Huang. 2015. Subspace Dynamic Simulation Using Rotation-Strain Coordinates. ACM Trans. Graph. 34, 6, Article 242 (Oct. 2015), 12 pages.Google ScholarDigital Library
44. Olivier Rémillard and Paul G. Kry. 2013. Embedded Thin Shells for Wrinkle Simulation. ACM Trans. Graph. 32, 4, Article 50 (July 2013), 8 pages.Google ScholarDigital Library
45. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6 (2017), 245:1–245:17.Google ScholarDigital Library
46. J. Schulman, A. Lee, J. Ho, and P. Abbeel. 2013. Tracking Deformable Objects with Point Clouds. In 2013 IEEE International Conference on Robotics and Automation. IEEE, 1130–1137.Google Scholar
47. Agniva Sengupta, Romain Lagneau, Alexandre Krupa, Eric Marchand, and Maud Marchal. 2020. Simultaneous Tracking and Elasticity Parameter Estimation of Deformable Objects. In IEEE Int. Conf. on Robotics and Automation, ICRA’20. IEEE.Google Scholar
48. Hang Si. 2015. TetGen, a Delaunay-Based Quality Tetrahedral Mesh Generator. ACM Trans. on Mathematical Software 41, 2 (2015), 11:1–11:36.Google ScholarDigital Library
49. Eftychios Sifakis and Jernej Barbič. 2012. FEM Simulation of 3D Deformable Solids: A Practitioner’s Guide to Theory, Discretization and Model Reduction. In ACM SIGGRAPH 2012 Courses. Association for Computing Machinery, New York, NY, USA, 50.Google Scholar
50. T. Simon, H. Joo, I. Matthews, and Y. Sheikh. 2017. Hand Keypoint Detection in Single Images Using Multiview Bootstrapping. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 4645–4653.Google Scholar
51. Breannan Smith, Fernando De Goes, and Theodore Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2, Article 12 (2018), 12 pages.Google ScholarDigital Library
52. Breannan Smith, Fernando De Goes, and Theodore Kim. 2019. Analytic Eigensystems for Isotropic Distortion Energies. ACM Trans. Graph. 38, 1, Article 3 (Feb. 2019), 15 pages.Google ScholarDigital Library
53. Jason Smith and Scott Schaefer. 2015. Bijective Parameterization with Free Boundaries. ACM Trans. Graph. 34, 4, Article 70 (July 2015), 9 pages.Google ScholarDigital Library
54. Olga Sorkine and Marc Alexa. 2007. As-Rigid-As-Possible Surface Modeling. In Proceedings of the Fifth Eurographics Symposium on Geometry Processing. Eurographics Association, Goslar, DEU, 109–116.Google Scholar
55. S. Sridhar, A. Oulasvirta, and C. Theobalt. 2013. Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data. In 2013 IEEE International Conference on Computer Vision. IEEE, 2456–2463.Google Scholar
56. Richard Szeliski and Demetri Terzopoulos. 1991. Physically-Based and Probabilistic Models for Computer Vision. In Geometric Methods in Computer Vision, Vol. 1570. International Society for Optics and Photonics, Springer, 140–152.Google Scholar
57. D. J. Tan, T. Cashman, J. Taylor, A. Fitzgibbon, D. Tarlow, S. Khamis, S. Izadi, and J. Shotton. 2016. Fits Like a Glove: Rapid and Reliable Hand Shape Personalization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5610–5619.Google Scholar
58. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose and Correspondences. ACM Trans. Graph. 35, 4 (2016), 143:1–143:12.Google ScholarDigital Library
59. Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. 2017. Articulated Distance Fields for Ultra-Fast Tracking of Hands Interacting. ACM Trans. Graph. 36, 6 (2017), 244:1–244:12.Google ScholarDigital Library
60. J. Rafael Tena, Fernando De la Torre, and Iain Matthews. 2011. Interactive Region-Based Linear 3D Face Models. ACM Trans. Graph. 30, 4 (2011), 76:1–76:10.Google ScholarDigital Library
61. Joseph Teran, Eftychios Sifakis, Geoffrey Irving, and Ronald Fedkiw. 2005. Robust Quasistatic Finite Elements and Flesh Simulation. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Association for Computing Machinery, New York, NY, USA, 181–190.Google ScholarDigital Library
62. Demetri Terzopoulos, Andrew Witkin, and Michael Kass. 1987. Symmetry-Seeking Models and 3D Object Reconstruction. International Journal of Computer Vision 1, 3 (1987), 211–221.Google ScholarCross Ref
63. Demetri Terzopoulos, Andrew Witkin, and Michael Kass. 1988. Constraints on Deformable Models: Recovering 3D Shape and Nonrigid Motion. Artificial Intelligence 36, 1 (1988), 91–123.Google ScholarDigital Library
64. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-Meshes for Real-Time Hand Modeling and Tracking. ACM Trans. Graph. 35, 6 (2016), 222:1–222:11.Google ScholarDigital Library
65. Anastasia Tkach, Andrea Tagliasacchi, Edoardo Remelli, Mark Pauly, and Andrew Fitzgibbon. 2017. Online Generative Model Personalization for Hand Tracking. ACM Trans. Graph. 36, 6 (2017), 243:1–243:11.Google ScholarDigital Library
66. Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing Hands in Action using Discriminative Salient Points and Physics Simulation. International Journal of Computer Vision 118, 2 (2016), 172–193.Google ScholarDigital Library
67. Ingo Wald, Sven Woop, Carsten Benthin, Gregory S. Johnson, and Manfred Ernst. 2014. Embree: A Kernel Framework for Efficient CPU Ray Tracing. ACM Trans. Graph. 33, 4, Article 143 (July 2014), 8 pages.Google ScholarDigital Library
68. Bohan Wang, George Matcuk, and Jernej Barbič. 2019. Hand Modeling and Simulation Using Stabilized Magnetic Resonance Imaging. ACM Trans. Graph. 38, 4, Article 115 (July 2019), 14 pages.Google ScholarDigital Library
69. Bin Wang, Longhua Wu, KangKang Yin, Uri Ascher, Libin Liu, and Hui Huang. 2015. Deformation Capture and Modeling of Soft Objects. ACM Trans. Graph. 34, 4, Article 94 (July 2015), 12 pages.Google ScholarDigital Library
70. Huamin Wang and Yin Yang. 2016. Descent Methods for Elastic Body Simulation on the GPU. ACM Trans. Graph. 35, 6, Article 212 (Nov. 2016), 10 pages.Google ScholarDigital Library
71. Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional Pose Machines. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 4724–4732.Google Scholar
72. S. Weiss, R. Maier, D. Cremers, R. Westermann, and N. Thuerey. 2020. Correspondence-Free Material Reconstruction using Sparse Surface Constraints. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 4685–4694.Google Scholar
73. Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. 2016. An Anatomically-Constrained Local Deformation Model for Monocular Face Capture. ACM Trans. Graph. 35, 4 (2016), 115:1–115:12.Google ScholarDigital Library
74. Chenglei Wu, Takaaki Shiratori, and Yaser Sheikh. 2018. Deep Incremental Learning for Efficient High-Fidelity Face Tracking. ACM Trans. Graph. 37, 6 (2018), 234:1–234:12.Google ScholarDigital Library
75. Stefanie Wuhrer, Jochen Lang, Motahareh Tekieh, and Chang Shu. 2015. Finite Element Based Tracking of Deforming Surfaces. Graphical Models 77 (2015), 1–17.Google ScholarDigital Library
76. Shanxin Yuan, Guillermo Garcia-Hernando, Björn Stenger, Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee, Pavlo Molchanov, Jan Kautz, Sina Honari, Liuhao Ge, Junsong Yuan, Xinghao Chen, Guijin Wang, Fan Yang, Kai Akiyama, Yang Wu, Qingfu Wan, Meysam Madadi, Sergio Escalera, Shile Li, Dongheui Lee, Iason Oikonomidis, Antonis Argyros, and Tae-Kyun Kim. 2018. Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2636–2645.Google ScholarCross Ref
77. Yufeng Zhu, Robert Bridson, and Danny M. Kaufman. 2018. Blended Cured QuasiNewton for Distortion Optimization. ACM Trans. Graph. 37, 4, Article 40 (July 2018), 14 pages.Google ScholarDigital Library
78. Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, and Marc Stamminger. 2014. Real-Time Non-Rigid Reconstruction Using an RGB-D Camera. ACM Trans. Graph. 33, 4, Article 156 (July 2014), 12 pages.Google ScholarDigital Library
