“Interactive hand pose estimation using a stretch-sensing soft glove” by Glauser, Wu, Panozzo, Hilliges and Sorkine-Hornung

  • ©Oliver Glauser, Shihao Wu, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung




    Interactive hand pose estimation using a stretch-sensing soft glove

Session/Category Title:   Capture Control



    We propose a stretch-sensing soft glove to interactively capture hand poses with high accuracy and without requiring an external optical setup. We demonstrate how our device can be fabricated and calibrated at low cost, using simple tools available in most fabrication labs. To reconstruct the pose from the capacitive sensors embedded in the glove, we propose a deep network architecture that exploits the spatial layout of the sensor itself. The network is trained only once, using an inexpensive off-the-shelf hand pose reconstruction system to gather the training data. The per-user calibration is then performed on-the-fly using only the glove. The glove’s capabilities are demonstrated in a series of ablative experiments, exploring different models and calibration methods. Comparing against commercial data gloves, we achieve a 35% improvement in reconstruction accuracy.


    1. 2018 Bando. https://www.youtube.com/watch?v=2XskNHtarj0.Google Scholar
    2. 2019 5DT Glove. http://www.5dt.com/data-gloves/.Google Scholar
    3. 2019 CyberGlove. http://www.cyberglovesystems.com/.Google Scholar
    4. 2019 HT2 Textile Glue. https://consumer.guetermann.com/en/product-finder/textile-glue-ht2.Google Scholar
    5. 2019 HTC Vive Tracker. https://www.vive.com/eu/vive-tracker/.Google Scholar
    6. 2019 Imerys ENSACO 250G. http://www.imerys-graphite-and-carbon.com/wordpress/wp-app/uploads/2014/04/Polymercompounds1.pdf.Google Scholar
    7. 2019 Intel Real Sense SR3000. https://software.intel.com/en-us/realsense/sr300.Google Scholar
    8. 2019 ManusVR glove. https://manus-vr.odoo.com/hardware.Google Scholar
    9. 2019a. Sil-PoxySilicone Adhesive. https://www.smooth-on.com/product-line/sil-poxy/.Google Scholar
    10. 2019b. Silbione RTV 4420. https://silicones.elkem.com/EN/ouroffer/Product/90060082/90060081/SILBIONE-RTV-4420-B-U1.Google Scholar
    11. 2019 Stretchsense. https://www.youtube.com/watch?v=XJCqdpzxMME.Google Scholar
    12. 2019 Vicon. http://www.vicon.com.Google Scholar
    13. 2019 VPL Data Glove. https://www.britannica.com/technology/VPL-DataGlove.Google Scholar
    14. Oluwaseun A. Araromi, Samuel Rosset, and Herbert R. Shea. 2015. High-Resolution, Large-Area Fabrication of Compliant Electrodes via Laser Ablation for Robust, Stretchable Dielectric Elastomer Actuators and Sensors. ACS Applied Materials & Interfaces 7, 32 (2015), 18046–18053.Google ScholarCross Ref
    15. Asli Atalay, Vanessa Sanchez, Ozgur Atalay, Daniel M. Vogt, Florian Haufe, Robert J. Wood, and Conor J. Walsh. 2017. Batch Fabrication of Customizable Silicone-Textile Composite Capacitive Strain Sensors for Human Motion Tracking. Advanced Materials Technologies 2, 9 (2017).Google Scholar
    16. Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Proc. ECCV. Springer, 640–653. Google ScholarDigital Library
    17. Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. 2018. Weakly-supervised 3d hand pose estimation from monocular rgb images. In Proc. ECCV. Springer, Cham, 1–17.Google Scholar
    18. Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proc. of the Conf. on Human Factors in Computing Systems. 2527–2530. Google ScholarDigital Library
    19. Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 1504–1514. Google ScholarDigital Library
    20. Jean-Baptiste Chossat, Yiwei Tao, Vincent Duchaine, and Yong-Lae Park. 2015. Wearable soft artificial skin for hand motion detection with embedded microfluidic strain sensing. In Proc. ICRA. IEEE, 2568–2573.Google ScholarCross Ref
    21. Te-Shun Chou, Ashley Gadd, and Dave Knott. 2000. Hand-Eye: A Vision-Based Approach to Data Glove Calibration. In Proc. Human Interface Technologies.Google Scholar
    22. Simone Ciotti, Edoardo Battaglia, Nicola Carbonaro, Antonio Bicchi, Alessandro Tognetti, and Matteo Bianchi. 2016. A Synergy-Based Optimally Designed Sensing Glove for Functional Grasp Recognition. Sensors 16 (06 2016), 811.Google Scholar
    23. James Connolly, Joan Condell, Brendan O’Flynn, Javier Torres Sanchez, and Philip Gardiner. 2018. IMU Sensor-Based Electronic Goniometric Glove for Clinical Finger Movement Analysis. IEEE Sensors Journal 18, 3 (Feb 2018), 1273–1281.Google Scholar
    24. Laura Dipietro, Angelo M. Sabatini, and Paolo Dario. 2008. A Survey of Glove-Based Systems and Their Applications. Trans. Sys. Man Cyber Part C 38, 4 (2008), 461–482. Google ScholarDigital Library
    25. Ali Erol, George Bebis, Mircea Nicolescu, Richard D Boyle, and Xander Twombly. 2007. Vision-based hand pose estimation: A review. Computer Vision and Image Understanding 108, 1 (2007), 52–73. Google ScholarDigital Library
    26. Bin Fang, Fuchun Sun, Huaping Liu, and Di Guo. 2017. Development of a Wearable Device for Motion Capturing Based on Magnetic and Inertial Measurement Units. Scientific Programming 2017 (01 2017), 1–11. Google ScholarDigital Library
    27. Max Fischer, Patrick van der Smagt, and Gerd Hirzinger. 1998. Learning techniques in a dataglove based telemanipulation system for the DLR hand. In Proc. ICRA, Vol. 2. IEEE, 1603–1608.Google Scholar
    28. Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. 2017. 3D Convolutional Neural Networks for Efficient and Robust Hand Pose Estimation from Single Depth Images. In Proc. CVPR.Google ScholarCross Ref
    29. Reinhard Gentner and Joseph Classen. 2008. Development and evaluation of a low-cost sensor glove for assessment of human finger movements in neurophysiological settings. Journal of neuroscience methods 178 (12 2008), 138–47.Google Scholar
    30. Oliver Glauser, Daniele Panozzo, Otmar Hilliges, and Olga Sorkine-Hornung. 2019. Deformation Capture via Soft and Stretchable Sensor Arrays. ACM Transactions on Graphics 38 (03 2019), 1–16. Google ScholarDigital Library
    31. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. of the Int. Conf. on Artificial Intelligence and Statistics. 249–256.Google Scholar
    32. Weston Griffin, Ryan Findley, Michael Turner, and Mark Cutkosky. 2000. Calibration and mapping of a human hand for dexterous telemanipulation. In ASME IMECE Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems.Google ScholarCross Ref
    33. Frank L Hammond, Yiğit Mengüç, and Robert J Wood. 2014. Toward a modular soft sensor-embedded glove for human hand motion and tactile pressure measurement. In Proc. IROS. IEEE, 4000–4007.Google ScholarCross Ref
    34. Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D. Twigg, and Kenrick Kin. 2018. Online Optical Marker-based Hand Tracking with Deep Labels. ACM Trans. Graph. 37, 4 (July 2018), 166:1–166:10. Google ScholarDigital Library
    35. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.Google ScholarCross Ref
    36. Ronan Hinchet, Velko Vechev, Herbert Shea, and Otmar Hilliges. 2018. DextrES: Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic Brake. In Proc. UIST. Google ScholarDigital Library
    37. Haiying Hu, Xiaohui Gao, Jiawei Li, Jie Wang, and Hong Liu. 2004. Calibrating human hand for teleoperating the HIT/DLR hand. In Proc. ICRA, Vol. 5. 4571–4576 Vol.5.Google Scholar
    38. Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J Black, Otmar Hilliges, and Gerard Pons-Moll. 2018. Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time. ACM Trans. Graph. 37, 6 (2018). Google ScholarDigital Library
    39. Umar Iqbal, Pavlo Molchanov, Thomas Breuel, Juergen Gall, and Jan Kautz. 2018. Hand Pose Estimation via Latent 2.5 D Heatmap Regression. arXiv preprint arXiv:1804.09534 (2018).Google Scholar
    40. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-To-Image Translation With Conditional Adversarial Networks. In Proc. CVPR.Google Scholar
    41. Lisa K Simone, Nappinnai Sundarrajan, Xun Luo, Yicheng Jia, and Derek Kamper. 2007. A low cost instrumented glove for extended monitoring and functional hand assessment. Journal of neuroscience methods 160 (04 2007), 335–48.Google Scholar
    42. Ferenc Kahlesz, Gabriel Zachmann, and Reinhard Klein. 2004. ‘Visual-fidelity’ dataglove calibration. In Computer Graphics International, 2004. IEEE, 403–410. Google ScholarDigital Library
    43. G. Drew Kessler, Larry F. Hodges, and Neff Walker. 1995. Evaluation of the CyberGlove As a Whole-hand Input Device. ACM Trans. Comput.-Hum. Interact. 2, 4 (1995), 263–283. Google ScholarDigital Library
    44. David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, 167–176. Google ScholarDigital Library
    45. Rebecca K Kramer, Carmel Majidi, Ranjana Sahai, and Robert J Wood. 2011. Soft curvature sensors for joint angle proprioception. In Proc. IROS. IEEE, 1919–1926.Google ScholarCross Ref
    46. Bor-Shing Lin, I-Jung Lee, Shu-Yu Yang, Yi-Chiang Lo, Junghsi Lee, and Jean-Lon Chen. 2018. Design of an Inertial-Sensor-Based Data Glove for Hand Function Evaluation. Sensors 18 (05 2018), 1545.Google Scholar
    47. Federico Lorussi, Enzo Pasquale Scilingo, Mario Tesconi, Alessandro Tognetti, and Danilo De Rossi. 2005. Strain sensing fabric for hand posture and gesture monitoring. IEEE transactions on information technology in biomedicine 9, 3 (2005), 372–381. Google ScholarDigital Library
    48. Anil Menon, B. Barnes, R. Mills, Cynthia Bruyns, Xander Twombly, J. Smith, Kevin Montgomery, and Richard D. Boyle. 2003. Using Registration, Calibration, and Robotics to Build a More Accurate Virtual Reality Simulation for Astronaut Training and Telemedicine. In WSCG.Google Scholar
    49. Hadrien O Michaud, Laurent Dejace, Séverine De Mulatier, and Stéphanie P Lacour. 2016. Design and functional evaluation of an epidermal strain sensing system for hand tracking. In Proc. IROS. 3186–3191.Google ScholarDigital Library
    50. Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2018. Ganerated hands for real-time 3d hand tracking from monocular RGB. In Proc. CVPR. 49–59.Google ScholarCross Ref
    51. Markus Oberweger and Vincent Lepetit. 2017. DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation. In International Conference on Computer Vision Workshops.Google ScholarCross Ref
    52. Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Hands deep in deep learning for hand pose estimation. arXiv preprint arXiv:1502.06807 (2015).Google Scholar
    53. Ben O’Brien, Todd Gisby, and Iain A Anderson. 2014. Stretch sensors for human body motion. In Proc. SPIE, Vol. 9056. 905618.Google Scholar
    54. Timothy F O’Connor, Matthew E Fach, Rachel Miller, Samuel E Root, Patrick P Mercier, and Darren J Lipomi. 2017. The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics. PloS one 12, 7 (2017), e0179766.Google ScholarCross Ref
    55. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. 2011a. Efficient model-based 3D tracking of hand articulations using Kinect. In Proc. BMVC.Google ScholarCross Ref
    56. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros. 2011b. Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In Proc. ICCV. Google ScholarDigital Library
    57. Wookeun Park, Kyongkwan Ro, Suin Kim, and Joonbum Bae. 2017. A Soft Sensor-Based Three-Dimensional (3-D) Finger Motion Measurement System. Sensors (Basel, Switzerland) 17, 2 (02 2017), 420.Google Scholar
    58. Panagiotis Polygerinos, Zheng Wang, Kevin C. Galloway, Robert J. Wood, and Conor J. Walsh. 2015. Soft robotic glove for combined assistance and at-home rehabilitation. Robotics and Autonomous Systems 73 (2015), 135 — 143. Google ScholarDigital Library
    59. Adnan Rashid and Osman Hasan. 2018. Wearable technologies for hand joints monitoring for rehabilitation: A survey. Microelectronics Journal (02 2018).Google Scholar
    60. Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Trans. Graph. 36, 6 (2017). Google ScholarDigital Library
    61. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Inte. Conf. on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
    62. Hochung Ryu, Sangki Park, Jong-Jin Park, and Jihyun Bae. 2018. A knitted glove sensing system with compression strain for finger movements. Smart Materials and Structures 27, 5 (2018), 055016.Google ScholarCross Ref
    63. T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling Always-available Input with Muscle-computer Interfaces. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST ’09). ACM, New York, NY, USA, 167–176. Google ScholarDigital Library
    64. Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, et al. 2015. Accurate, robust, and flexible real-time hand tracking. In Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3633–3642. Google ScholarDigital Library
    65. Zhong Shen, Juan Yi, Xiaodong Li, Mark Hin Pei Lo, Michael ZQ Chen, Yong Hu, and Zheng Wang. 2016. A soft stretchable bending sensor and data glove applications. IEEE Int. Conf. on Real-time Computing and Robotics (RCAR) 3, 1 (2016), 22.Google ScholarCross Ref
    66. Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand keypoint detection in single images using multiview bootstrapping. In Proc. CVPR, Vol. 2.Google ScholarCross Ref
    67. Ayan Sinha, Chiho Choi, and Karthik Ramani. 2016. Deephand: Robust hand pose estimation by completing a matrix imputed with deep features. In Proc. CVPR.Google ScholarCross Ref
    68. Adrian Spurr, Jie Song, Seonwook Park, and Otmar Hilliges. 2018. Cross-modal Deep Variational Hand Pose Estimation. In Proc. CVPR.Google ScholarCross Ref
    69. Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. 2013. Interactive Markerless Articulated Hand Motion Tracking using RGB and Depth Data. In Proc. ICCV. 8. Google ScholarDigital Library
    70. Jan Steffen, Jonathan Maycock, Helge Ritter, Honghai Liu, and Daniel Schilberg. 2011. Robust Dataglove Mapping for Recording Human Hand Postures. In Intelligent Robotics and Applications. Google ScholarDigital Library
    71. Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. 2015. Cascaded hand pose regression. In Proc. CVPR. 824–832.Google ScholarCross Ref
    72. Andrea Tagliasacchi, Matthias Schroeder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust Articulated-ICP for Real-Time Hand Tracking. In Proc. SGP.Google ScholarCross Ref
    73. Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. 2014. Latent Regression Forest: Structured Estimation of 3D Articulated Hand Posture. In Proc. CVPR. Google ScholarDigital Library
    74. Danhang Tang, Jonathan Taylor, Pushmeet Kohli, Cem Keskin, Tae-Kyun Kim, and Jamie Shotton. 2015. Opening the black box: Hierarchical sampling optimization for estimating human hand pose. In Proc. ICCV. 3325–3333. Google ScholarDigital Library
    75. Danhang Tang, Tsz-Ho Yu, and Tae-Kyun Kim. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proc. ICCV. 3224–3231. Google ScholarDigital Library
    76. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking Through Joint, Continuous Optimization of Pose and Correspondences. ACM Trans. Graph. 35, 4 (2016), 143:1–143:12. Google ScholarDigital Library
    77. Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. 2017. Articulated Distance Fields for Ultra-fast Tracking of Hands Interacting. ACM Trans. Graph. 36 (2017), 244:1–244:12. Google ScholarDigital Library
    78. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-meshes for Real-time Hand Modeling and Tracking. ACM Trans. Graph. 35, 6 (2016), 222:1–222:11. Google ScholarDigital Library
    79. Anastasia Tkach, Andrea Tagliasacchi, Edoardo Remelli, Mark Pauly, and Andrew Fitzgibbon. 2017. Online Generative Model Personalization for Hand Tracking. ACM Trans. Graph. 36, 6 (2017). Google ScholarDigital Library
    80. Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph. 33, 5 (2014), 169. Google ScholarDigital Library
    81. Hoang Truong, Shuo Zhang, Ufuk Muncuk, Phuc Nguyen, Nam Bui, Anh Nguyen, Qin Lv, Kaushik Chowdhury, Thang Dinh, and Tam Vu. 2018. CapBand: Battery-free Successive Capacitance Sensing Wristband for Hand Gesture Recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys ’18). ACM, New York, NY, USA, 54–67. Google ScholarDigital Library
    82. Timo von Marcard, Bodo Rosenhahn, Michael J Black, and Gerard Pons-Moll. 2017. Sparse inertial poser: Automatic 3d human pose estimation from sparse imus. Comput. Graph. Forum 36, 2, 349–360. Google ScholarDigital Library
    83. Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2017. Crossing Nets: Combining GANs and VAEs With a Shared Latent Space for Hand Pose Estimation. In Proc. CVPR.Google ScholarCross Ref
    84. Chengde Wan, Angela Yao, and Luc Van Gool. 2016. Hand pose estimation from local surface normals. In European Conference on Computer Vision. Springer, 554–569.Google ScholarCross Ref
    85. Robert Y. Wang and Jovan Popović. 2009. Real-time Hand-tracking with a Color Glove. ACM Trans. Graph. 28, 3 (2009), 63:1–63:8. Google ScholarDigital Library
    86. Yingying Wang and Michael Neff. 2013. Data-driven Glove Calibration for Hand Motion Capture. In Proc. SCA. 15–24. Google ScholarDigital Library
    87. Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang. 2016. 3D Hand Pose Tracking and Estimation Using Stereo Matching. arXiv preprint arXiv:1610.07214 (2016).Google Scholar
    88. Yang Zheng, Yu Peng, Gang Wang, Xinrong Liu, Xiaotong Dong, and Jue Wang. 2016. Development and evaluation of a sensor glove for hand function assessment and preliminary attempts at assessing hand coordination. Measurement 93 (06 2016).Google Scholar
    89. Jilin Zhou, François Malric, and Shervin Shirmohammadi. 2010. A New Hand-Measurement Method to Simplify Calibration in CyberGlove-Based Virtual Rehabilitation. IEEE Transactions on Instrumentation and Measurement 59, 10 (Oct 2010), 2496–2504.Google ScholarCross Ref
    90. Christian Zimmermann and Thomas Brox. 2017. Learning to Estimate 3D Hand Pose from Single RGB Images. In Proc. ICCV. 4913–4921.Google ScholarCross Ref

ACM Digital Library Publication:

Overview Page: