“Online generative model personalization for hand tracking”
Conference:
Type(s):
Title:
- Online generative model personalization for hand tracking
Session/Category Title: Hands and Bodies
Presenter(s)/Author(s):
Abstract:
We present a new algorithm for real-time hand tracking on commodity depth-sensing devices. Our method does not require a user-specific calibration session, but rather learns the geometry as the user performs live in front of the camera, thus enabling seamless virtual interaction at the consumer level. The key novelty in our approach is an online optimization algorithm that jointly estimates pose and shape in each frame, and determines the uncertainty in such estimates. This knowledge allows the algorithm to integrate per-frame estimates over time, and build a personalized geometric model of the captured user. Our approach can easily be integrated in state-of-the-art continuous generative motion tracking software. We provide a detailed evaluation that shows how our approach achieves accurate motion tracking for real-time applications, while significantly simplifying the workflow of accurate hand performance capture. We also provide quantitative evaluation datasets at http://gfx.uvic.ca/datasets/handy
References:
1. Irene Albrecht, Jörg Haber, and Hans-Peter Seidel. 2003. Construction and animation of anatomically based human hand models. In Proc. Symp. on Computer Animation (SCA).
2. Brian Anderson and John Moore. 1979. Optimal filtering. Englewood Cliffs.
3. Bradley M Bell and Frederick W Cathey. 1993. The iterated Kalman filter update as a Gauss-Newton method. In IEEE Trans. on Automatic Control.
4. R Louis Bellaire, Edward W Kamen, and Serena M Zabin. 1995. New nonlinear iterated filter with applications to target tracking. In SPIE Intl. Symposium on Optical Science, Engineering, and Instrumentation.
5. Federica Bogo, Michael J Black, Matthew Loper, and Javier Romero. 2015. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In Proc. Intl. Conf. on Comp. Vision (ICCV).
6. Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online modeling for realtime facial animation. In ACM Trans. on Graphics (Proc. SIGGRAPH).
7. Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-time high-fidelity facial performance capture. In ACM Trans. on Graphics (Proc. SIGGRAPH).
8. Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, and Kun Zhou. 2016. Real-time facial animation with image-based dynamic avatars. In ACM Trans. on Graphics (Proc. SIGGRAPH).
9. Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3D hand pose estimation from monocular video. In Pattern Analysis and Machine Intelligence (PAMI).
10. Paul Ekman and Wallace V Friesen. 1977. Facial Action Coding System. Consulting Psychologists Press, Stanford University, Palo Alto.
11. Jinwei Gu, Xiaodong Yang, Shalini De Mello, and Jan Kautz. 2017. Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Network. In Proc. Computer Vision and Pattern Recognition (CVPR). Cross Ref
12. Jindřich Havlík and Ondřej Straka. 2015. Performance evaluation of iterated extended Kalman filter with variable step-length. In Journal of Physics: Conf. Series. Cross Ref
13. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and others. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proc. ACM User Interface Software and Technology.
14. Sameh Khamis, Jonathan Taylor, Jamie Shotton, Cem Keskin, Shahram Izadi, and Andrew Fitzgibbon. 2015. Learning an efficient model of hand shape variation from depth images. In Proc. Computer Vision and Pattern Recognition (CVPR). Cross Ref
15. Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-fly Correctives. In ACM Trans. on Graphics (Proc. SIGGRAPH).
16. Alexandros Makris and A Argyros. 2015. Model-based 3D hand tracking with online hand shape adaptation. In Proc. British Machine Vision Conf. (BMVC).
17. Jorge Nocedal and Stephen Wright. 2006. Numerical optimization. Springer.
18. Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Hands deep in deep learning for hand pose estimation. In Proc. Computer Vision Winter Workshop.
19. Kaare Brandt Petersen, Michael Syskind Pedersen, and others. 2008. The matrix cookbook. Technical University of Denmark.
20. Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and robust hand tracking from depth. In Proc. Computer Vision and Pattern Recognition (CVPR).
21. Edoardo Remelli, Anastasia Tkach, Andrea Tagliassachi, and Mark Pauly. 2017. Low-Dimensionality Calibration through Local Anisotropic for Scaling for Robust Hand Model Personalization. In Proc. Intl. Conf. on Comp. Vision (ICCV). Cross Ref
22. Taehyun Rhee, Ulrich Neumann, and John P Lewis. 2006. Human hand modeling from surface anatomy. In Proc. Symposium on Interactive 3D graphics and games.
23. Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, and others. 2015. Accurate, robust, and flexible real-time hand tracking. In Proc. ACM Special Interest Group on Computer-Human Interaction (CHI).
24. Martin A Skoglund, Gustaf Hendeby, and Daniel Axehill. 2015. Extended Kalman filter modifications based on an optimization view point. In Intl. Conf. on Inf. Fusion.
25. Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. 2015. Fast and Robust Hand Tracking Using Detection-Guided Optimization. In Proc. Computer Vision and Pattern Recognition (CVPR). Cross Ref
26. Hauke Strasdat, José MM Montiel, and Andrew J Davison. 2012. Visual SLAM: why filter?. In Image and Vision Computing.
27. James S Supancic, Grégory Rogez, Yi Yang, Jamie Shotton, and Deva Ramanan. 2015. Depth-based hand pose estimation: data, methods, and challenges. In Proc. Intl. Conf. on Comp. Vision (ICCV).
28. Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust Articulated-ICP for Real-Time Hand Tracking. In Computer Graphics Forum (Proc. Symposium on Geometry Processing).
29. David J. Tan, Thomas Cashman, Jonathan Taylor, Andrew Fitzgibbon, Daniel Tarlow, Sameh Khamis, Shahram Izadi, and Jamie Shotton. 2016. Fits like a glove: Rapid and reliable hand shape personalization. In Proc. Computer Vision and Pattern Recognition (CVPR). Cross Ref
30. Danhang Tang, Jonathan Taylor, Pushmeet Kohli, Cem Keskin, Tae-Kyun Kim, and Jamie Shotton. 2015. Opening the black box: Hierarchical sampling optimization for estimating human hand pose. In Proc. Intl. Conf. on Comp. Vision (ICCV).
31. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, and others. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. In ACM Trans. on Graphics (Proc. SIGGRAPH).
32. Jonathan Taylor, Richard Stebbing, Varun Ramakrishna, Cem Keskin, Jamie Shotton, Shahram Izadi, Aaron Hertzmann, and Andrew Fitzgibbon. 2014. User-specific hand modeling from monocular depth sequences. In Proc. Computer Vision and Pattern Recognition (CVPR).
33. Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015. Real-time expression transfer for facial reenactment. ACM Trans. Graph. 34, 6 (2015), 183–1.
34. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-meshes for real-time hand modeling and tracking. In ACM Trans. on Graphics (Proc. SIGGRAPH Asia).
35. Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. In ACM Trans. on Graphics (TOG).
36. Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip Torr, Shahram Izadi, and Cem Keskin. 2016. Learning to navigate the energy landscape. In Int. Conf. on 3D Vision (3DV). Cross Ref
37. Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM Trans. on Graphics (Proc. SIGGRAPH).
38. Greg Welch and Gary Bishop. 1995. An introduction to the Kalman filter.
