“Simulation of Hand Anatomy Using Medical Imaging” by Zheng, Wang, Huang and Barbič
Conference:
Type(s):
Title:
- Simulation of Hand Anatomy Using Medical Imaging
Session/Category Title: Simulation of Everything
Presenter(s)/Author(s):
Abstract:
Precision modeling of the hand internal musculoskeletal anatomy has been largely limited to individual poses, and has not been connected into continuous volumetric motion of the hand anatomy actuating across the hand’s entire range of motion. This is for a good reason, as hand anatomy and its motion are extremely complex and cannot be predicted merely from the anatomy in a single pose. We give a method to simulate the volumetric shape of hand’s musculoskeletal organs to any pose in the hand’s range of motion, producing external hand shapes and internal organ shapes that match ground truth optical scans and medical images (MRI) in multiple scanned poses. We achieve this by combining MRI images in multiple hand poses with FEM multibody nonlinear elastoplastic simulation. Our system models bones, muscles, tendons, joint ligaments and fat as separate volumetric organs that mechanically interact through contact and attachments, and whose shape matches medical images (MRI) in the MRI-scanned hand poses. The match to MRI is achieved by incorporating pose-space deformation and plastic strains into the simulation. We show how to do this in a non-intrusive manner that still retains all the simulation benefits, namely the ability to prescribe realistic material properties, generalize to arbitrary poses, preserve volume and obey contacts and attachments. We use our method to produce volumetric renders of the internal anatomy of the human hand in motion, and to compute and render highly realistic hand surface shapes. We evaluate our method by comparing it to optical scans, and demonstrate that we qualitatively and quantitatively substantially decrease the error compared to previous work. We test our method on five complex hand sequences, generated either using keyframe animation or performance animation using modern hand tracking techniques.
References:
1. 3dMD. 2022. 3dMDHands. https://3dmd.com/.
2. R. Abdrashitov, S. Bang, D. Levin, K. Singh, and A. Jacobson. 2021. Interactive Modelling of Volumetric Musculoskeletal Anatomy. ACM Transactions on Graphics 40, 4 (2021).
3. B. Amberg, S. Romdhani, and T. Vetter. 2007. Optimal Step Nonrigid ICP Algorithms for Surface Registration. In Conf. on Computer Vision and Pattern Recognition (CVPR).
4. B. Angles, D. Rebain, M. Macklin, B. Wyvill, L. Barthe, JP Lewis, J. Von Der Pahlen, S. Izadi, J. Valentin, S. Bouaziz, and Tagliasacchi A. 2019. Viper: Volume invariant position-based elastic rods. Proc. of ACM on Computer Graphics and Interactive Techniques 2, 2 (2019), 1–26.
5. Artelys. 2019. Knitro. https://www.artelys.com/solvers/knitro/.
6. M. Bergou, M. Wardetzky, S. Robinson, B. Audoly, and E. Grinspun. 2008. Discrete Elastic Rods. ACM Transactions on Graphics (SIGGRAPH) 27, 3 (2008), 63:1–63:12.
7. S. Bouaziz, S. Martin, T. Liu, L. Kavan, and M. Pauly. 2014. Projective Dynamics: Fusing Constraint Projections for Fast Simulation. ACM Trans. Graph. 33, 4, Article 154 (2014), 154:1–154:11 pages.
8. C. Erolin. 2019. Hand Anatomy. University of Dundee, Centre for Anatomy and Human Identification. https://sketchfab.com/anatomy_dundee/collections/hand-anatomy.
9. S. Capell, M. Burkhart, B. Curless, T. Duchamp, and Z. Popović. 2005. Physically Based Rigging for Deformable Characters. In Symp. on Computer Animation (SCA). 301–310.
10. H. Chen, C. Chen, T. Yang, L. Kuo, IM Jou, F. Su, and Y. Sun. 2011. Model-based segmentation of flexor tendons from magnetic resonance images of finger joints. In Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society. 8009–8012.
11. K. Comley and N. A. Fleck. 2010. A micromechanical model for the Young’s modulus of adipose tissue. Int. J. of Solids and Structures 47, 21 (2010), 2982–2990.
12. A. H. Dicko, T. Liu, B. Gilles, L. Kavan, F. Faure, O. Palombi, and M.P. Cani. 2013. Anatomy Transfer. ACM Trans. on Graphics (SIGGRAPH) 32, 6 (2013), 188:1–188:8.
13. A. Dogadov, M. Alamir, C. Serviere, and F. Quaine. 2017. The biomechanical model of the long finger extensor mechanism and its parametric identification. J. of Biomechanics 58 (2017), 232–236.
14. A. Fedorov, R. Beichel, J. Kalpathy-Cramer, J. Finet, J.C. Fillion-Robin, S. Pujol, C. Bauer, D. Jennings, F. Fennessy, M. Sonka, J. Buatti, S. Aylward, J. V. Miller, S. Pieper, and R. Kikinis. 2012. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magnetic Resonance Imaging 30, 9 (2012), 1323–1341.
15. K. S. Fok and S. M. Chou. 2010. Development of a finger biomechanical model and its considerations. J. of Biomechanics 43, 4 (2010), 701–713.
16. A. K. Garland, D. S. Shah, and A. E. Kedgley. 2018. Wrist tendon moment arms: Quantification by imaging and experimental techniques. J. of Biomechanics 68 (2018), 136–140.
17. C. Garre, F. Hernández, A. Gracia, and M. A. Otaduy. 2011. Interactive simulation of a deformable hand for haptic rendering. In IEEE World Haptics Conf. 239–244.
18. M. Hadwiger, J. M. Kniss, C. Rezk-salama, D. Weiskopf, and K. Engel. 2006. Real-Time Volume Graphics. A. K. Peters, Ltd.
19. A.E. Ichim, P. Kadlecek, L. Kavan, and M. Pauly. 2017. Phace: Physics-based Face Modeling and Animation. ACM Trans. on Graphics (SIGGRAPH 2017) 36, 4 (2017).
20. A. Jacobson, Z. Deng, L. Kavan, and J. P. Lewis. 2014. Skinning: Real-time Shape Deformation. In ACM SIGGRAPH 2014 Courses.
21. P. Kadlecek, A.-E. Ichim, T. Liu, J. Krivanek, and L. Kavan. 2016. Reconstructing Personalized Anatomical Models for Physics-based Body Animation. ACM Trans. Graph. 35, 6 (2016).
22. A.I. Kapandji. 2009. The physiology of the joints, 6th Edition, Vol. 1: The Upper Limb. Elsevier Exclusive.
23. L. Kavan, S. Collins, J. Zara, and C. O’Sullivan. 2008. Geometric Skinning with Approximate Dual Quaternion Blending. ACM Trans. on Graphics 27, 4 (2008).
24. M. Keller, S. Zuffi, M. J. Black, and S. Pujades. 2022. OSSO: Obtaining Skeletal Shape from Outside. In Conf. on Computer Vision and Pattern Recognition (CVPR). 20492–20501.
25. J. Kim and N. S. Pollard. 2011. Fast simulation of skeleton-driven deformable body characters. ACM Trans. on Graphics (TOG) 30, 5 (2011), 121.
26. P. G. Kry, D. L. James, and D. K. Pai. 2002. EigenSkin: Real Time Large Deformation Character Skinning in Hardware. In In Symp. on Computer Animation (SCA).
27. T. Kugelstadt and E. Schömer. 2016. Position and Orientation Based Cosserat Rods. In Symp. on Computer Animation (SCA).
28. C. Kuok, T. Yang, B. Tsai, I. Jou, M. Horng, F. Su, Y. Sun, et al. 2020. Segmentation of finger tendon and synovial sheath in ultrasound image using deep convolutional neural network. Biomedical engineering online 19, 1 (2020), 1–25.
29. T. Kurihara and N. Miyata. 2004. Modeling deformable human hands from medical images. In Symp. on Computer Animation (SCA). 355–363.
30. Leal, Allan. 2018. autodiff. https://github.com/autodiff/autodiff/.
31. LeapMotion. 2017. https://www.leapmotion.com.
32. S. Lee, R. Yu, J. Park, M. Aanjaneya, E. Sifakis, and J. Lee. 2018. Dexterous manipulation and control with volumetric muscles. ACM Transactions on Graphics (SIGGRAPH 2018) 37, 4 (2018), 57:1–57:13.
33. S. H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive Biomechanical Modeling and Simulation of the Upper Body. ACM Trans. on Graphics 28, 4 (2009), 99:1–99:17.
34. J. P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformations: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proc. of ACM SIGGRAPH 2000. 165–172.
35. Y. Li, M. Wu, Y. Zhang, L. Xu, and J. Yu. 2021. PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging. arXiv preprint arXiv:2106.10893 (2021).
36. Y. Li, L. Zhang, Z. Qiu, Y. Jiang, N. Li, Y. Ma, Y. Zhang, L. Xu, and J. Yu. 2022. NIMBLE: a non-rigid hand model with bones and muscles. ACM Transactions on Graphics (SIGGRAPH 2022) 41, 4 (2022), 1–16.
37. L. Liu, K. Yin, B. Wang, and B. Guo. 2013. Simulation and control of skeleton-driven soft body characters. ACM Trans. on Graphics (SIGGRAPH Asia) 32, 6 (2013), 215.
38. M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. on Graphics (SIGGRAPH Asia 2015) 34, 6 (2015), 248:1–248:16.
39. C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J. Lee, W. Chang, W. Hua, M. Georg, and M Grundmann. 2019. MediaPipe: A Framework for Building Perception Pipelines.
40. A. McAdams, Y. Zhu, A. Selle, M. Empey, R. Tamstorf, J. Teran, and E. Sifakis. 2011. Efficient elasticity for character skinning with contact and collisions. ACM Trans. on Graphics (SIGGRAPH 2011) 30, 4 (2011).
41. N. Miyata, M. Kouch, M. Mochimaru, and T. Kurihara. 2005. Finger joint kinematics from MR images. In IEEE Int. Conf. on Intelligent Robots and Systems. 2750–2755.
42. V. Modi, L. Fulton, A. Jacobson, S. Sueda, and D. Levin. 2021. Emu: Efficient muscle simulation in deformation space. In Computer Graphics Forum, Vol. 40. 234–248.
43. M. Müller, B. Heidelberger, M. Teschner, and M. Gross. 2005. Meshless Deformations Based on Shape Matching. In Proc. of ACM SIGGRAPH 2005. 471–478.
44. N. Qian, J. Wang, F. Mueller, F. Bernard, V. Golyanik, and C. Theobalt. 2020. HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization. In Proc. of the European Conf. on Computer Vision (ECCV). Springer.
45. T. Rhee, J.P. Lewis, and U. Neumann. 2006. Real-Time Weighted Pose-Space Deformation on the GPU. In Proc. of Eurographics, Vol. 25.
46. J. Romero, D. Tzionas, and M. J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. on Graphics (SIGGRAPH Asia 2017) 36, 6 (2017), 245:1–245:17.
47. V. Roussellet, N. A. Rumman, F. Canezin, N. Mellado, K. Kavan, and L. Barthe. 2018. Dynamic implicit muscles for character skinning. Computers & Graphics 77 (2018), 227–239.
48. A. Rusu. 2011. Segmentation of bone structures in Magnetic Resonance Images for human hand skeletal kinematics modelling. Master’s thesis. German Aerospace Center.
49. P. Sachdeva, S. Sueda, S. Bradley, M. Fain, and D. K. Pai. 2015. Biomechanical Simulation and Control of Hands and Tendinous Systems. ACM Trans. Graph. 34, 4 (2015), 42:1–42:10.
50. S. Saito, Z. Zhou, and L. Kavan. 2015. Computational Bodybuilding: Anatomically-based Modeling of Human Bodies. ACM Trans. on Graphics (SIGGRAPH 2015) 34, 4 (2015).
51. E. Sifakis, I. Neverov, and R. Fedkiw. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Trans. on Graphics (SIGGRAPH 2005) 24, 3 (2005), 417–425.
52. B. Smith, F. De Goes, and T. Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2 (2018), 12:1–12:15.
53. G. Stillfried. 2015. Kinematic modelling of the human hand for robotics. Ph.D. Dissertation. Technische Universität München.
54. S. Sueda, A. Kaufman, and D. K. Pai. 2008. Musculotendon Simulation for Hand Animation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (2008).
55. Tissue. 2013. Weta Digital: Tissue Muscle and Fat Simulation System.
56. R. Vaillant, G. Guennebaud, L. Barthe, B. Wyvill, and M.P. Cani. 2014. Robust Iso-surface Tracking for Interactive Character Skinning. ACM Trans. on Graphics (SIGGRAPH Asia 2014) 33, 6 (2014), 189:1–137:11.
57. P. van der Smagt and G. Stillfried. 2008. Using MRI data to compute a hand kinematic model. In Conf. on Motion and Vibration Control (MOVIC).
58. P. Volino, N. Magnenat-Thalmann, and F. Faure. 2009. A Simple Approach to Nonlinear Tensile Stiffness for Accurate Cloth Simulation. ACM Trans. Graph. 28, 4, Article 105 (2009), 16 pages.
59. B. Wang, G. Matcuk, and J. Barbič. 2019. Hand Modeling and Simulation Using Stabilized Magnetic Resonance Imaging. ACM Trans. on Graphics (SIGGRAPH 2019) 38, 4 (2019).
60. B. Wang, G. Matcuk, and J. Barbič. 2020. Hand MRI dataset. http://www.jernejbarbic.com/hand-mri-dataset.
61. B. Wang, G. Matcuk, and J. Barbič. 2021. Modeling of Personalized Anatomy using Plastic Strains. ACM Trans. on Graphics (TOG) 40, 2 (2021).
62. D. L. Wilson, Q. Zhu, J. L. Duerk, J. M. Mansour, K. Kilgore, and P. E. Crago. 1999. Estimation of tendon moment arms from three-dimensional magnetic resonance images. Annals of Biomedical Engineering 27, 2 (1999), 247–256.
63. Wrap3. 2018. Nonlinear Iterative Closest Point mesh registration software. https://www.russian3dscanner.com.
64. FE Zajac. 1989. Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. Critical reviews in biomedical engineering 17, 4 (1989), 359–411.
65. Zygote. 2016. Zygote body. http://www.zygotebody.com.
