“Hand modeling and simulation using stabilized magnetic resonance imaging” by Wang, Matcuk and Barbic
Conference:
Type:
Title:
- Hand modeling and simulation using stabilized magnetic resonance imaging
Session/Category Title: Animation and Skinning
Presenter(s)/Author(s):
Abstract:
We demonstrate how to acquire complete human hand bone anatomy (meshes) in multiple poses using magnetic resonance imaging (MRI). Such acquisition was previously difficult because MRI scans must be long for high-precision results (over 10 minutes) and because humans cannot hold the hand perfectly still in non-trivial and badly supported poses. We invent a manufacturing process whereby we use lifecasting materials commonly employed in film special effects industry to generate hand molds, personalized to the subject, and to each pose. These molds are both ergonomic and encasing, and they stabilize the hand during scanning. We also demonstrate how to efficiently segment the MRI scans into individual bone meshes in all poses, and how to correspond each bone’s mesh to same mesh connectivity across all poses. Next, we interpolate and extrapolate the MRI-acquired bone meshes to the entire range of motion of the hand, producing an accurate data-driven animation-ready rig for bone meshes. We also demonstrate how to acquire not just bone geometry (using MRI) in each pose, but also a matching highly accurate surface geometry (using optical scanners) in each pose, modeling skin pores and wrinkles. We also give a soft tissue Finite Element Method simulation “rig”, consisting of novel tet meshing for stability at the joints, spatially varying geometric and material detail, and quality constraints to the acquired skeleton kinematic rig. Given an animation sequence of hand joint angles, our FEM soft tissue rig produces quality hand surface shapes in arbitrary poses in the hand range of motion. Our results qualitatively reproduce important features seen in the photographs of the subject’s hand, such as similar overall organic shape and fold formation.
References:
1. Agisoft. 2018. Photoscan, http://www.agisoft.com.Google Scholar
2. AljaSafe. 2018. SmoothOn Inc. www.smooth-on.com.Google Scholar
3. Amira. 2018. Amira for Life Sciences. https://www.fei.com/software/amira-3d-for-life-sciences/.Google Scholar
4. Artec3D. 2018. Spider Scanner, http://www.artec3d.com.Google Scholar
5. Noelle M. Austin. 2005. Chapter 9: The Wrist and Hand Complex”. In Levangie, Pamela K.; Norkin, Cynthia C. Joint Structure and Function: A Comprehensive Analysis (4th ed.). F. A. Davis Company.Google Scholar
6. J. Barbič and D. L. James. 2008. Six-DoF haptic rendering of contact between geometrically complex reduced deformable models. IEEE Trans. on Haptics 1, 1 (2008), 39–52. Google ScholarDigital Library
7. Pierre-Yves Baudin, Noura Azzabou, Pierre G Carlier, and Nikos Paragios. 2012. Automatic skeletal muscle segmentation through random walks and graph-based seed placement. In IEEE Int. Symp. on Biomedical Imaging (ISBI). 1036–1039.Google ScholarCross Ref
8. Paul J. Besl and N.D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239–256. Google ScholarDigital Library
9. Jean-Daniel Boissonnat and Steve Oudot. 2005. Provably good sampling and meshing of surfaces. Graphical Models 67, 5 (2005), 405 — 451. Google ScholarDigital Library
10. S. Capell, M. Burkhart, B. Curless, T. Duchamp, and Z. Popović. 2005. Physically Based Rigging for Deformable Characters. In Symp. on Computer Animation (SCA). 301–310. Google ScholarDigital Library
11. Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. 2008. MeshLab: an Open-Source Mesh Processing Tool. In Eurographics Italian Chapter Conference.Google Scholar
12. CyberGlove Systems. 2017. CyberGrasp. http://www.cyberglovesystems.com/cybergrasp.Google Scholar
13. Mary F Dempsey, Barrie Condon, and Donald M Hadley. 2002. MRI safety review. In Seminars in Ultrasound, CT and MRI, Vol. 23. Elsevier, 392–401.Google Scholar
14. C. M. Deniz, S. Xiang, S. Hallyburton, A. Welbeck, S. Honig, K. Cho, and G. Chang. 2017. Segmentation of the Proximal Femur from MR Images using Deep Convolutional Neural Networks. arXiv preprint arXiv:1704.06176 (2017).Google Scholar
15. D. E. Discher, D. J. Mooney, and P. W. Zandstra. 2009. Growth factors, matrices, and forces combine and control stem cells. Science 324, 5935 (2009), 1673–1677.Google Scholar
16. James S Duncan and Nicholas Ayache. 2000. Medical image analysis: Progress over two decades and the challenges ahead. IEEE Trans. on Pattern Analysis and Machine Intelligence 22, 1 (2000), 85–106. Google ScholarDigital Library
17. M. S. Farvid, T. W. K. Ng, D. C. Chan, P. H. R. Barrett, and G. F. Watts. 2005. Association of adiponectin and resistin with adipose tissue compartments, insulin resistance and dyslipidaemia. Diabetes, Obesity and Metabolism 7, 4 (2005), 406–413.Google ScholarCross Ref
18. A. Fenster and D. B. Downey. 1996. 3-D ultrasound imaging: a review. IEEE Engineering in Medicine and Biology Magazine 15, 6 (1996), 41–51.Google ScholarCross Ref
19. Carlos Garre, Fernando Hernández, Antonio Gracia, and Miguel A Otaduy. 2011. Interactive simulation of a deformable hand for haptic rendering. In IEEE World Haptics Conference (WHC). IEEE, 239–244.Google ScholarCross Ref
20. Benjamin Gilles and Nadia Magnenat-Thalmann. 2010. Musculoskeletal MRI segmentation using multi-resolution simplex meshes with medial representations. Medical image analysis 14, 3 (2010), 291–302.Google Scholar
21. Leo Grady. 2006. Random walks for image segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 11 (2006), 1768–1783. Google ScholarDigital Library
22. Agneta Gustus and Patrick van der Smagt. 2016. Evaluation of joint type modelling in the human hand. Journal of Biomechanics 49, 13 (2016), 3097 — 3100.Google ScholarCross Ref
23. Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D Twigg, and Kenrick Kin. 2018. Online optical marker-based hand tracking with deep labels. ACM Transactions on Graphics (SIGGRAPH 2018) 37, 4 (2018), 166. Google ScholarDigital Library
24. Hang Si. 2011. TetGen: A Quality Tetrahedral Mesh Generator and a 3D Delaunay Triangulator.Google Scholar
25. Yixin Hu, Qingnan Zhou, Xifeng Gao, Alec Jacobson, Denis Zorin, and Daniele Panozzo. 2018. Tetrahedral Meshing in the Wild. ACM Trans. on Graphics (SIGGRAPH 2018) 37, 4 (2018), 60:1–60:14. Google ScholarDigital Library
26. G. Irving, J. Teran, and R. Fedkiw. 2004. Invertible Finite Elements for Robust Simulation of Large Deformation. In Symp. on Computer Animation (SCA). 131–140. Google ScholarDigital Library
27. ITK-SNAP. 2018. ITK-SNAP. http://www.itksnap.org/pmwiki/pmwiki.php.Google Scholar
28. Alec Jacobson, Zhigang Deng, Ladislav Kavan, and JP Lewis. 2014. Skinning: Real-time Shape Deformation. In ACM SIGGRAPH 2014 Courses. Google ScholarDigital Library
29. A.I. Kapandji. 2009. The physiology of the joints, 6th Edition, Vol. 1: The Upper Limb. Elsevier Exclusive.Google Scholar
30. L. Kavan, S. Collins, J. Zara, and C. O’Sullivan. 2008. Geometric Skinning with Approximate Dual Quaternion Blending. ACM Trans. on Graphics 27, 4 (2008). Google ScholarDigital Library
31. Baris Kayalibay, Grady Jensen, and Patrick van der Smagt. 2017. CNN-based segmentation of medical imaging data. arXiv preprint arXiv:1701.03056 (2017).Google Scholar
32. Junggon Kim and Nancy S Pollard. 2011. Fast simulation of skeleton-driven deformable body characters. ACM Trans. on Graphics (TOG) 30, 5 (2011), 121. Google ScholarDigital Library
33. Jonathan P King, Dominik Bauer, Cornelia Schlagenhauf, Kai-Hung Chang, Daniele Moro, Nancy Pollard, and Stelian Coros. 2018. Design. Fabrication, and Evaluation of Tendon-Driven Multi-Fingered Foam Hands. In IEEE-RAS Int. Conf. on Humanoid Robots (Humanoids). 1–9.Google ScholarCross Ref
34. Paul G. Kry, Doug L. James, and Dinesh K. Pai. 2002. EigenSkin: Real Time Large Deformation Character Skinning in Hardware. In Proc. of the Symp. on Comp. Animation 2002. 153–160. Google ScholarDigital Library
35. Tsuneya Kurihara and Natsuki Miyata. 2004. Modeling deformable human hands from medical images. In Symp. on Computer Animation (SCA). 355–363. Google ScholarDigital Library
36. LeapMotion. 2017. https://www.leapmotion.com.Google Scholar
37. S. H. Lee, E. Sifakis, and D. Terzopoulos. 2009. Comprehensive Biomechanical Modeling and Simulation of the Upper Body. ACM Trans. on Graphics 28, 4 (2009), 99:1–99:17. Google ScholarDigital Library
38. J. P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformations: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation. In Proc. of ACM SIGGRAPH 2000. 165–172. Google ScholarDigital Library
39. Duo Li, Shinjiro Sueda, Debanga R Neog, and Dinesh K Pai. 2013. Thin Skin Elastodynamics. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 4 (2013), 49:1–49:9. Google ScholarDigital Library
40. Libin Liu, KangKang Yin, Bin Wang, and Baining Guo. 2013. Simulation and control of skeleton-driven soft body characters. ACM Trans. on Graphics (SIGGRAPH Asia 2013) 32, 6 (2013), 215. Google ScholarDigital Library
41. N. Magnenat-Thalmann, R. Laperrire, and D. Thalmann. 1988. Joint-dependent local deformations for hand animation and object grasping. In Proc. of Graphics Interface. Google ScholarDigital Library
42. Joe Mancewicz, Matt L. Derksen, Hans Rijpkema, and Cyrus A. Wilson. 2014. Delta Mush: Smoothing Deformations While Preserving Detail. In Proceedings of the Fourth Symposium on Digital Production (DigiPro ’14). 7–11. Google ScholarDigital Library
43. G. Marai, D. Laidlaw, J. Coburn, M. Upal, and J. Crisco. 2003. A 3D method for segmenting and registering carpal bones from CT volume images. In Proc. of Annual Meeting of the American Society of Biomechanics.Google Scholar
44. A. McAdams, Y. Zhu, A. Selle, M. Empey, R. Tamstorf, J. Teran, and E. Sifakis. 2011. Efficient elasticity for character skinning with contact and collisions. ACM Trans. on Graphics (SIGGRAPH 2011) 30, 4 (2011). Google ScholarDigital Library
45. Tim McInerney and Demetri Terzopoulos. 2008. Deformable models in medical image analysis: a survey. Medical Image Analysis 1, 2 (2008), 91–108.Google ScholarCross Ref
46. Fernand Meyer. 1992. Color image segmentation. In International Conf. on Image Processing and its Applications. IET, 303–306.Google Scholar
47. Aslan Miriyev, Kenneth Stack, and Hod Lipson. 2017. Soft material for soft actuators. Nature Communications 8, 596 (2017).Google Scholar
48. N. Miyata, M. Kouch, M. Mochimaru, and T. Kurihara. 2005. Finger joint kinematics from MR images. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems. 2750–2755.Google Scholar
49. NimbleVR. 2012. http://nimblevr.com.Google Scholar
50. RadiologyInfo. 2018. Radiation Dose in X-Ray and CT Exams. https://www.radiologyinfo.org/en/pdf/safety-xray.pdf.Google Scholar
51. Taehyun Rhee, J.P. Lewis, and Ulrich Neumann. 2006. Real-Time Weighted Pose-Space Deformation on the GPU. In Proc. of Eurographics 2006, Vol. 25.Google ScholarCross Ref
52. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. on Graphics (SIGGRAPH Asia 2017) 36, 6 (2017), 245:1–245:17. Google ScholarDigital Library
53. Alexandru Rusu. 2011. Segmentation of bone structures in Magnetic Resonance Images (MRI) for human hand skeletal kinematics modelling. Master’s thesis. German Aerospace Center.Google Scholar
54. Prashant Sachdeva, Shinjiro Sueda, Susanne Bradley, Mikhail Fain, and Dinesh K. Pai. 2015. Biomechanical Simulation and Control of Hands and Tendinous Systems. ACM Trans. Graph. 34, 4 (2015), 42:1–42:10. Google ScholarDigital Library
55. Cornelia Schlagenhauf, Dominik Bauer, Kai-Hung Chang, Jonathan P King, Daniele Moro, Stelian Coros, and Nancy Pollard. 2018. Control of tendon-driven soft foam robot hands. In IEEE-RAS Int. Conf. on Humanoid Robots (Humanoids). 1–7.Google ScholarCross Ref
56. Jérôme Schmid, Jinman Kim, and Nadia Magnenat-Thalmann. 2011. Robust statistical shape models for MRI bone segmentation in presence of small field of view. Medical image analysis 15, 1 (2011), 155–168.Google Scholar
57. Jérôme Schmid and Nadia Magnenat-Thalmann. 2008. MRI bone segmentation using deformable models and shape priors. In Int. Conf. on Medical Image Computing and Computer-Assisted Intervention. 119–126. Google ScholarDigital Library
58. Breannan Smith, Fernando De Goes, and Theodore Kim. 2018. Stable Neo-Hookean Flesh Simulation. ACM Trans. Graph. 37, 2 (2018), 12:1–12:15. Google ScholarDigital Library
59. Ole Vegard Solberg, Frank Lindseth, Hans Torp, Richard E. Blake, and Toril A. Nagelhus Hernes. 2007. Freehand 3D Ultrasound Reconstruction Algorithms: A Review. Ultrasound in Medicine and Biology 33, 7 (2007), 991 — 1009.Google ScholarCross Ref
60. Georg Stillfried. 2015. Kinematic modelling of the human hand for robotics. Ph.D. Dissertation. Technische Universität München.Google Scholar
61. Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. 2008. Musculotendon Simulation for Hand Animation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (2008). Google ScholarDigital Library
62. Tissue. 2013. Weta Digital: Tissue Muscle and Fat Simulation System.Google Scholar
63. Rodolphe Vaillant, Gäel Guennebaud, Loïc Barthe, Brian Wyvill, and Marie-Paule Cani. 2014. Robust Iso-surface Tracking for Interactive Character Skinning. ACM Trans. on Graphics (SIGGRAPH Asia 2014) 33, 6 (2014), 189:1–137:11. Google ScholarDigital Library
64. Patrick van der Smagt and Georg Stillfried. 2008. Using MRI data to compute a hand kinematic model. In Conf. on Motion and Vibration Control (MOVIC).Google Scholar
65. VTK. 2018. VTK. https://www.vtk.org/.Google Scholar
66. Andreas Wächter and Lorenz T. Biegler. 2006. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical Programming 106, 1 (01 Mar 2006), 25–57. Google ScholarDigital Library
67. Andrea Walther and Andreas Griewank. 2009. Getting Started with ADOL-C. In Combinatorial scientific computing. 181–202.Google Scholar
68. R. Y. Wang, S. Paris, and J. Popović. 2011. 6D Hands: Markerless Hand Tracking for Computer Aided Design. In ACM User Interface Software and Technology (UIST). Google ScholarDigital Library
69. Robert Y. Wang and Jovan Popović. 2009. Real-time hand-tracking with a color glove. ACM Trans. on Graphics (SIGGRAPH 2009) 28, 3 (2009), 63:1–63:8. Google ScholarDigital Library
70. Robert E. Watson. 2015. Lessons Learned from MRI Safety Events. Current Radiology Reports 3, 10 (12 Aug 2015), 37.Google Scholar
71. C. Wex, S. Arndt, A. Stoll, C. Bruns, and Y. Kupriyanova. 2015. Isotropic incompressible hyperelastic models for modelling the mechanical behaviour of biological tissues: a review. Biomedical Engineering / Biomedizinische Technik 60, 6 (2015), 577–592.Google ScholarCross Ref
72. Nkenge Wheatland, Yingying Wang, Huaguang Song, Michael Neff, Victor Zordan, and Sophie Jörg. 2015. State of the art in hand and finger modeling and animation. In Computer Graphics Forum, Vol. 34. 735–760. Google ScholarDigital Library
73. Wrap3. 2018. Nonlinear Iterative Closest Point mesh registration software. https://www.russian3dscanner.com.Google Scholar
74. Shanxin Yuan, Qi Ye, Björn Stenger, Siddhant Jain, and Tae-Kyun Kim. 2017. Bighand2. 2M benchmark: Hand pose dataset and state of the art analysis. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2605–2613.Google Scholar
75. Zygote. 2016. Zygote body. http://www.zygotebody.com.Google Scholar
