“Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks” by Tompson, Stein, LeCun and Perlin

  • ©

Conference:


Type(s):


Title:

    Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks

Session/Category Title:   Animating Characters

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and finally an inverse kinematics stage for stable real-time pose recovery. As one possible application of this pipeline, we show state-of-the-art results for real-time puppeteering of a skinned hand-model.

References:


    1. 3GEAR. 2014. 3gear sytems hand-tracking development platform. http://www.threegear.com/.
    2. B. Allen, B. Curless, and Z. Popovic. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22, 3, 587–594.
    3. L. Ballan, A. Taneja, J. Gall, L. Van Gool, and M. Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Proceedings of the 12th European Conference on Computer Vision. 640–653.
    4. Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222–1239.
    5. D. A. Butler, S. Izadi, O. Hilliges, D. Molyneaux, S. Hodges, and D. Kim. 2012. Shake’n’sense: Reducing interference for overlapping structured light depth cameras. In Proceedings of the ACM Annual Conference on Human Factors in Computing Systems. 1933–1936.
    6. R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A matlab-like environment for machine learning. http://ronan.collobert.com/pub/matos/2011_torch7_nipsw.pdf.
    7. C. Couprie, C. Farabet, L. Najman, and Y. Lecun. 2013. Indoor semantic segmentation using depth information. In Proceedings of the International Conference on Learning Representations.
    8. A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly. 2007. Vision-based hand pose estimation: A review. Comput. Vis. Image Understand. 108, 1–2, 52–73.
    9. C. Farabet, C. Couprie, L. Najman, and Y. Lecun. 2013. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8, 1915–1929.
    10. B. K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer. 4, 4, 629–642.
    11. K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. Lecun. 2009. What is the best multi-stage architecture for object recognition? In Proceedings of the 12th IEEE International Conference on Computer Vision. 2146–2153.
    12. M. Jiu, C. Wolf, G. W. Taylor, and A. Baskurt. 2013. Human body part estimation from depth images via spatially-constrained deep learning. Pattern Recogn. Lett. (to appear).
    13. C. Keskin, F. Kirac, Y. Kara, and L. Akarun. 2011. Real time hand pose estimation using depth sensors. In Proceedings of the IEEE International Computer Vision Workshops. 1228–1234.
    14. C. Keskin, F. Kirac, Y. E. Kara, and L. Akarun. 2012. Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In Proceedings of the 12th European Conference on Computer Vision. Vol. 6. Springer, 852–863.
    15. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Neural Information Processing Systems Conference. P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., 1106–1114.
    16. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11, 2278–2324.
    17. Y. Lecun, F. J. Huang, and L. Bottou. 2004. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2. 97–104.
    18. H. Li, R. W. Sumner, and M. Pauly. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421–1430.
    19. H. Li, J. Yu, Y. Ye, and C. Bregler. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4.
    20. S. Melax, L. Keselman, and S. Orsten. 2013. Dynamics based 3D skeletal hand tracking. In Proceedings of the ACM Symposium on Interactive 3D Graphics and Games.
    21. J. Nagi, F. Ducatelle, G. Di Caro, D. Ciresan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and L. Gambardella. 2011. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications. 342–347.
    22. S. J. Nowlan and J. C. Platt. 1995. A convolutional neural network hand tracker. In Proceedings of the Neural Information Processing Systems Conference. 901–908.
    23. I. Oikonomidis, N. Kyriazis, and A. Argyros. 2011. Efficient model-based 3D tracking of hand articulations using kinect. In Proceedings of the British Machine Vision Conference.
    24. M. Osadchy, Y. Lecun, M. L. Miller, and P. Perona. 2005. Synergistic face detection and pose estimation with energy-based model. In Proceedings of the Neural Information Processing Systems Conference. 1017–1024.
    25. J. M. Rehg and T. Kanade. 1994. Visual tracking of high dof articulated structures: An application to human hand tracking. In Proceedings of the 3rd European Conference on Computer Vision. 35–46.
    26. M. Saric. 2011. Libhand: A library for hand articulation. http://www.libhand. org/.
    27. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. 2011. Real-time human pose recognition in parts from single depth images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1297–1304.
    28. M. Stein, J. Tompson, X. Xiao, C. Hendeee, H. Ishii, and K. Perlin. 2012. Arcade: A system for augmenting gesture-based computer graphic presentations. In Proceedings of the ACM SIGGRAPH Computer Animation Festival. 77–77.
    29. G. W. Taylor, I. Spiro, C. Bregler, and R. Fergus. 2011. Learning invariance through imitation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2729–2736.
    30. P. Tseng. 1995. Fortified-descent simplicial search method: A general approach. SIAM J. Optim. 10, 1, 269–288.
    31. R. Wang, S. Paris, and J. Popovic. 2011. 6d hands: Markerless tracking for computer aided design. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. 549–558.
    32. R. Y. Wang. and J. Popovic. 2009. Real-time hand-tracking with a color glove. ACM Trans. Graph. 28, 3.
    33. T. Weise, H. Li, L. Van Gool, and M. Pauly. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation.
    34. T. Yasuda, K. Ohkura, and Y. Matsumura. 2010. Extended PSO with partial randomization for large scale multimodal problems. In Proceedings of the World Automation Congress. 1–6.
    35. W. Zhao, J. Chai, and Y.-Q. Xu. 2012. Combining marker-based MOCAP and RGB-d camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 33–42.

ACM Digital Library Publication:


Overview Page: