“Interactive hand pose estimation using a stretch-sensing soft glove” by Glauser, Wu, Panozzo, Hilliges and Sorkine-Hornung
Conference:
Type:
Title:
- Interactive hand pose estimation using a stretch-sensing soft glove
Session/Category Title: Capture Control
Presenter(s)/Author(s):
Abstract:
We propose a stretch-sensing soft glove to interactively capture hand poses with high accuracy and without requiring an external optical setup. We demonstrate how our device can be fabricated and calibrated at low cost, using simple tools available in most fabrication labs. To reconstruct the pose from the capacitive sensors embedded in the glove, we propose a deep network architecture that exploits the spatial layout of the sensor itself. The network is trained only once, using an inexpensive off-the-shelf hand pose reconstruction system to gather the training data. The per-user calibration is then performed on-the-fly using only the glove. The glove’s capabilities are demonstrated in a series of ablative experiments, exploring different models and calibration methods. Comparing against commercial data gloves, we achieve a 35% improvement in reconstruction accuracy.
References:
1. 2018 Bando. https://www.youtube.com/watch?v=2XskNHtarj0.Google Scholar
2. 2019 5DT Glove. http://www.5dt.com/data-gloves/.Google Scholar
3. 2019 CyberGlove. http://www.cyberglovesystems.com/.Google Scholar
4. 2019 HT2 Textile Glue. https://consumer.guetermann.com/en/product-finder/textile-glue-ht2.Google Scholar
5. 2019 HTC Vive Tracker. https://www.vive.com/eu/vive-tracker/.Google Scholar
6. 2019 Imerys ENSACO 250G. http://www.imerys-graphite-and-carbon.com/wordpress/wp-app/uploads/2014/04/Polymercompounds1.pdf.Google Scholar
7. 2019 Intel Real Sense SR3000. https://software.intel.com/en-us/realsense/sr300.Google Scholar
8. 2019 ManusVR glove. https://manus-vr.odoo.com/hardware.Google Scholar
9. 2019a. Sil-PoxySilicone Adhesive. https://www.smooth-on.com/product-line/sil-poxy/.Google Scholar
10. 2019b. Silbione RTV 4420. https://silicones.elkem.com/EN/ouroffer/Product/90060082/90060081/SILBIONE-RTV-4420-B-U1.Google Scholar
11. 2019 Stretchsense. https://www.youtube.com/watch?v=XJCqdpzxMME.Google Scholar
12. 2019 Vicon. http://www.vicon.com.Google Scholar
13. 2019 VPL Data Glove. https://www.britannica.com/technology/VPL-DataGlove.Google Scholar
14. Oluwaseun A. Araromi, Samuel Rosset, and Herbert R. Shea. 2015. High-Resolution, Large-Area Fabrication of Compliant Electrodes via Laser Ablation for Robust, Stretchable Dielectric Elastomer Actuators and Sensors. ACS Applied Materials & Interfaces 7, 32 (2015), 18046–18053.Google ScholarCross Ref
15. Asli Atalay, Vanessa Sanchez, Ozgur Atalay, Daniel M. Vogt, Florian Haufe, Robert J. Wood, and Conor J. Walsh. 2017. Batch Fabrication of Customizable Silicone-Textile Composite Capacitive Strain Sensors for Human Motion Tracking. Advanced Materials Technologies 2, 9 (2017).Google Scholar
16. Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Proc. ECCV. Springer, 640–653. Google ScholarDigital Library
17. Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. 2018. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proc. ECCV. Springer, Cham, 1–17.Google Scholar
18. Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proc. of the Conf. on Human Factors in Computing Systems. 2527–2530. Google ScholarDigital Library
19. Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 1504–1514. Google ScholarDigital Library
20. Jean-Baptiste Chossat, Yiwei Tao, Vincent Duchaine, and Yong-Lae Park. 2015. Wearable soft artificial skin for hand motion detection with embedded microfluidic strain sensing. In Proc. ICRA. IEEE, 2568–2573.Google ScholarCross Ref
21. Te-Shun Chou, Ashley Gadd, and Dave Knott. 2000. Hand-Eye: A Vision-Based Approach to Data Glove Calibration. In Proc. Human Interface Technologies.Google Scholar
22. Simone Ciotti, Edoardo Battaglia, Nicola Carbonaro, Antonio Bicchi, Alessandro Tognetti, and Matteo Bianchi. 2016. A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition. Sensors 16 (06 2016), 811.Google Scholar
23. James Connolly, Joan Condell, Brendan O’Flynn, Javier Torres Sanchez, and Philip Gardiner. 2018. IMU Sensor-Based Electronic Goniometric Glove for Clinical Finger Movement Analysis. IEEE Sensors Journal 18, 3 (Feb 2018), 1273–1281.Google Scholar
24. Laura Dipietro, Angelo M. Sabatini, and Paolo Dario. 2008. A Survey of Glove-Based Systems and Their Applications. Trans. Sys. Man Cyber Part C 38, 4 (2008), 461–482. Google ScholarDigital Library
25. Ali Erol, George Bebis, Mircea Nicolescu, Richard D Boyle, and Xander Twombly. 2007. Vision-based hand pose estimation: A review. Computer Vision and Image Understanding 108, 1 (2007), 52–73. Google ScholarDigital Library
26. Bin Fang, Fuchun Sun, Huaping Liu, and Di Guo. 2017. Development of a Wearable Device for Motion Capturing Based on Magnetic and Inertial Measurement Units. Scientific Programming 2017 (01 2017), 1–11. Google ScholarDigital Library
27. Max Fischer, Patrick van der Smagt, and Gerd Hirzinger. 1998. Learning techniques in a dataglove based telemanipulation system for the DLR hand. In Proc. ICRA, Vol. 2. IEEE, 1603–1608.Google Scholar
28. Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. 2017. 3D Convolutional Neural Networks for Efficient and Robust Hand Pose Estimation from Single Depth Images. In Proc. CVPR.Google ScholarCross Ref
29. Reinhard Gentner and Joseph Classen. 2008. Development and evaluation of a low-cost sensor glove for assessment of human finger movements in neurophysiological settings. Journal of neuroscience methods 178 (12 2008), 138–47.Google Scholar
30. Oliver Glauser, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019. Deformation Capture via Soft and Stretchable Sensor Arrays. ACM Transactions on Graphics 38 (03 2019), 1–16. Google ScholarDigital Library
31. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. of the Int. Conf. on Artificial Intelligence and Statistics. 249–256.Google Scholar
32. Weston Griffin, Ryan Findley, Michael Turner, and Mark Cutkosky. 2000. Calibration and mapping of a human hand for dexterous telemanipulation. In ASME IMECE Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems.Google ScholarCross Ref
33. Frank L Hammond, Yiğit Mengüç, and Robert J Wood. 2014. Toward a modular soft sensor-embedded glove for human hand motion and tactile pressure measurement. In Proc. IROS. IEEE, 4000–4007.Google ScholarCross Ref
34. Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D. Twigg, and Kenrick Kin. 2018. Online Optical Marker-based Hand Tracking with Deep Labels. ACM Trans. Graph. 37, 4 (July 2018), 166:1–166:10. Google ScholarDigital Library
35. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.Google ScholarCross Ref
36. Ronan Hinchet, Velko Vechev, Herbert Shea, and Otmar Hilliges. 2018. DextrES: Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic Brake. In Proc. UIST. Google ScholarDigital Library
37. Haiying Hu, Xiaohui Gao, Jiawei Li, Jie Wang, and Hong Liu. 2004. Calibrating human hand for teleoperating the HIT/DLR hand. In Proc. ICRA, Vol. 5. 4571–4576 Vol.5.Google Scholar
38. Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J Black, Otmar Hilliges, and Gerard Pons-Moll. 2018. Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time. ACM Trans. Graph. 37, 6 (2018). Google ScholarDigital Library
39. Umar Iqbal, Pavlo Molchanov, Thomas Breuel, Juergen Gall, and Jan Kautz. 2018. Hand Pose Estimation via Latent 2.5 D Heatmap Regression. arXiv preprint arXiv:1804.09534 (2018).Google Scholar
40. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proc. CVPR.Google Scholar
41. Lisa K Simone, Nappinnai Sundarrajan, Xun Luo, Yicheng Jia, and Derek Kamper. 2007. A low cost instrumented glove for extended monitoring and functional hand assessment. Journal of neuroscience methods 160 (04 2007), 335–48.Google Scholar
42. Ferenc Kahlesz, Gabriel Zachmann, and Reinhard Klein. 2004. ‘Visual-fidelity’ dataglove calibration. In Computer Graphics International, 2004. IEEE, 403–410. Google ScholarDigital Library
43. G. Drew Kessler, Larry F. Hodges, and Neff Walker. 1995. Evaluation of the CyberGlove As a Whole-hand Input Device. ACM Trans. Comput.-Hum. Interact. 2, 4 (1995), 263–283. Google ScholarDigital Library
44. David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, 167–176. Google ScholarDigital Library
45. Rebecca K Kramer, Carmel Majidi, Ranjana Sahai, and Robert J Wood. 2011. Soft curvature sensors for joint angle proprioception. In Proc. IROS. IEEE, 1919–1926.Google ScholarCross Ref
46. Bor-Shing Lin, I-Jung Lee, Shu-Yu Yang, Yi-Chiang Lo, Junghsi Lee, and Jean-Lon Chen. 2018. Design of an Inertial-Sensor-Based Data Glove for Hand Function Evaluation. Sensors 18 (05 2018), 1545.Google Scholar
47. Federico Lorussi, Enzo Pasquale Scilingo, Mario Tesconi, Alessandro Tognetti, and Danilo De Rossi. 2005. Strain sensing fabric for hand posture and gesture monitoring. IEEE transactions on information technology in biomedicine 9, 3 (2005), 372–381. Google ScholarDigital Library
48. Anil Menon, B. Barnes, R. Mills, Cynthia Bruyns, Xander Twombly, J. Smith, Kevin Montgomery, and Richard D. Boyle. 2003. Using Registration, Calibration, and Robotics to Build a More Accurate Virtual Reality Simulation for Astronaut Training and Telemedicine. In WSCG.Google Scholar
49. Hadrien O Michaud, Laurent Dejace, Séverine De Mulatier, and Stéphanie P Lacour. 2016. Design and functional evaluation of an epidermal strain sensing system for hand tracking. In Proc. IROS. 3186–3191.Google ScholarDigital Library
50. Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. Ganerated hands for real-time 3d hand tracking from monocular RGB. In Proc. CVPR. 49–59.Google ScholarCross Ref
51. Markus Oberweger and Vincent Lepetit. 2017. DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation. In International Conference on Computer Vision Workshops.Google ScholarCross Ref
52. Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Hands deep in deep learning for hand pose estimation. arXiv preprint arXiv:1502.06807 (2015).Google Scholar
53. Ben O’Brien, Todd Gisby, and Iain A Anderson. 2014. Stretch sensors for human body motion. In Proc. SPIE, Vol. 9056. 905618.Google Scholar
54. Timothy F O’Connor, Matthew E Fach, Rachel Miller, Samuel E Root, Patrick P Mercier, and Darren J Lipomi. 2017. The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics. PloS one 12, 7 (2017), e0179766.Google ScholarCross Ref
55. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. 2011a. Efficient model-based 3D tracking of hand articulations using Kinect. In Proc. BMVC.Google ScholarCross Ref
56. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. 2011b. Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In Proc. ICCV. Google ScholarDigital Library
57. Wookeun Park, Kyongkwan Ro, Suin Kim, and Joonbum Bae. 2017. A Soft Sensor-Based Three-Dimensional (3-D) Finger Motion Measurement System. Sensors (Basel, Switzerland) 17, 2 (02 2017), 420.Google Scholar
58. Panagiotis Polygerinos, Zheng Wang, Kevin C. Galloway, Robert J. Wood, and Conor J. Walsh. 2015. Soft robotic glove for combined assistance and at-home rehabilitation. Robotics and Autonomous Systems 73 (2015), 135 — 143. Google ScholarDigital Library
59. Adnan Rashid and Osman Hasan. 2018. Wearable technologies for hand joints monitoring for rehabilitation: A survey. Microelectronics Journal (02 2018).Google Scholar
60. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6 (2017). Google ScholarDigital Library
61. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Inte. Conf. on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
62. Hochung Ryu, Sangki Park, Jong-Jin Park, and Jihyun Bae. 2018. A knitted glove sensing system with compression strain for finger movements. Smart Materials and Structures 27, 5 (2018), 055016.Google ScholarCross Ref
63. T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling Always-available Input with Muscle-computer Interfaces. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’09). ACM, New York, NY, USA, 167–176. Google ScholarDigital Library
64. Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, et al. 2015. Accurate, robust, and flexible real-time hand tracking. In Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3633–3642. Google ScholarDigital Library
65. Zhong Shen, Juan Yi, Xiaodong Li, Mark Hin Pei Lo, Michael ZQ Chen, Yong Hu, and Zheng Wang. 2016. A soft stretchable bending sensor and data glove applications. IEEE Int. Conf. on Real-time Computing and Robotics (RCAR) 3, 1 (2016), 22.Google ScholarCross Ref
66. Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand keypoint detection in single images using multiview bootstrapping. In Proc. CVPR, Vol. 2.Google ScholarCross Ref
67. Ayan Sinha, Chiho Choi, and Karthik Ramani. 2016. Deephand: Robust hand pose estimation by completing a matrix imputed with deep features. In Proc. CVPR.Google ScholarCross Ref
68. Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges. 2018. Cross-modal Deep Variational Hand Pose Estimation. In Proc. CVPR.Google ScholarCross Ref
69. Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. 2013. Interactive Markerless Articulated Hand Motion Tracking using RGB and Depth Data. In Proc. ICCV. 8. Google ScholarDigital Library
70. Jan Steffen, Jonathan Maycock, Helge Ritter, Honghai Liu, and Daniel Schilberg. 2011. Robust Dataglove Mapping for Recording Human Hand Postures. In Intelligent Robotics and Applications. Google ScholarDigital Library
71. Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. 2015. Cascaded hand pose regression. In Proc. CVPR. 824–832.Google ScholarCross Ref
72. Andrea Tagliasacchi, Matthias Schroeder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust Articulated-ICP for Real-Time Hand Tracking. In Proc. SGP.Google ScholarCross Ref
73. Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. 2014. Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture. In Proc. CVPR. Google ScholarDigital Library
74. Danhang Tang, Jonathan Taylor, Pushmeet Kohli, Cem Keskin, Tae-Kyun Kim, and Jamie Shotton. 2015. Opening the black box: Hierarchical sampling optimization for estimating human hand pose. In Proc. ICCV. 3325–3333. Google ScholarDigital Library
75. Danhang Tang, Tsz-Ho Yu, and Tae-Kyun Kim. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proc. ICCV. 3224–3231. Google ScholarDigital Library
76. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking Through Joint, Continuous Optimization of Pose and Correspondences. ACM Trans. Graph. 35, 4 (2016), 143:1–143:12. Google ScholarDigital Library
77. Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. 2017. Articulated Distance Fields for Ultra-fast Tracking of Hands Interacting. ACM Trans. Graph. 36 (2017), 244:1–244:12. Google ScholarDigital Library
78. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-meshes for Real-time Hand Modeling and Tracking. ACM Trans. Graph. 35, 6 (2016), 222:1–222:11. Google ScholarDigital Library
79. Anastasia Tkach, Andrea Tagliasacchi, Edoardo Remelli, Mark Pauly, and Andrew Fitzgibbon. 2017. Online Generative Model Personalization for Hand Tracking. ACM Trans. Graph. 36, 6 (2017). Google ScholarDigital Library
80. Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph. 33, 5 (2014), 169. Google ScholarDigital Library
81. Hoang Truong, Shuo Zhang, Ufuk Muncuk, Phuc Nguyen, Nam Bui, Anh Nguyen, Qin Lv, Kaushik Chowdhury, Thang Dinh, and Tam Vu. 2018. CapBand: Battery-free Successive Capacitance Sensing Wristband for Hand Gesture Recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys ’18). ACM, New York, NY, USA, 54–67. Google ScholarDigital Library
82. Timo von Marcard, Bodo Rosenhahn, Michael J Black, and Gerard Pons-Moll. 2017. Sparse inertial poser: Automatic 3d human pose estimation from sparse imus. Comput. Graph. Forum 36, 2, 349–360. Google ScholarDigital Library
83. Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2017. Crossing Nets: Combining GANs and VAEs With a Shared Latent Space for Hand Pose Estimation. In Proc. CVPR.Google ScholarCross Ref
84. Chengde Wan, Angela Yao, and Luc Van Gool. 2016. Hand pose estimation from local surface normals. In European Conference on Computer Vision. Springer, 554–569.Google ScholarCross Ref
85. Robert Y. Wang and Jovan Popović. 2009. Real-time Hand-tracking with a Color Glove. ACM Trans. Graph. 28, 3 (2009), 63:1–63:8. Google ScholarDigital Library
86. Yingying Wang and Michael Neff. 2013. Data-driven Glove Calibration for Hand Motion Capture. In Proc. SCA. 15–24. Google ScholarDigital Library
87. Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 2016. 3D Hand Pose Tracking and Estimation Using Stereo Matching. arXiv preprint arXiv:1610.07214 (2016).Google Scholar
88. Yang Zheng, Yu Peng, Gang Wang, Xinrong Liu, Xiaotong Dong, and Jue Wang. 2016. Development and evaluation of a sensor glove for hand function assessment and preliminary attempts at assessing hand coordination. Measurement 93 (06 2016).Google Scholar
89. Jilin Zhou, François Malric, and Shervin Shirmohammadi. 2010. A New Hand-Measurement Method to Simplify Calibration in CyberGlove-Based Virtual Rehabilitation. IEEE Transactions on Instrumentation and Measurement 59, 10 (Oct 2010), 2496–2504.Google ScholarCross Ref
90. Christian Zimmermann and Thomas Brox. 2017. Learning to Estimate 3D Hand Pose from Single RGB Images. In Proc. ICCV. 4913–4921.Google ScholarCross Ref
