“Soli: ubiquitous gesture sensing with millimeter wave radar”

  • ©Mustafa Karagozler, Ivan Poupyrev, Jaime Lien, Carsten Schwesig, Nick Gillian, Patrick Amihood, Hakim Raja, and Erik Olson

Conference:


Type:


Title:

    Soli: ubiquitous gesture sensing with millimeter wave radar

Session/Category Title: USER INTERFACES

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    This paper presents Soli, a new, robust, high-resolution, low-power, miniature gesture sensing technology for human-computer interaction based on millimeter-wave radar. We describe a new approach to developing a radar-based sensor optimized for human-computer interaction, building the sensor architecture from the ground up with the inclusion of radar design principles, high temporal resolution gesture tracking, a hardware abstraction layer (HAL), a solid-state radar chip and system architecture, interaction models and gesture vocabularies, and gesture recognition. We demonstrate that Soli can be used for robust gesture recognition and can track gestures with sub-millimeter accuracy, running at over 10,000 frames per second on embedded hardware.

References:


    1. Axelsson, S. R. 2004. Noise radar using random phase and frequency modulation. IEEE Transactions on Geoscience and Remote Sensing 42, 11, 2370–2384.Google ScholarCross Ref
    2. Azevedo, S., and McEwan, T. 1996. Micropower impulse radar. Science and Technology Review, 17–29.Google Scholar
    3. Barnes, S. B. 1997. Douglas Carl Engelbart: Developing the underlying concepts for contemporary computing. IEEE Annals of the History of Computing 19, 3, 16–26. Google ScholarDigital Library
    4. Baum, C. E., Rothwell, E. J., Chen, K.-M., and Nyquist, D. P. 1991. The singularity expansion method and its application to target identification. Proceedings of the IEEE 79, 10, 1481–1492.Google ScholarCross Ref
    5. Benezeth, Y., Jodoin, P. M., Emile, B., Laurent, H., and Rosenberger, C. 2008. Review and evaluation of commonly-implemented background subtraction algorithms. In ICPR 2008.Google Scholar
    6. Bishop, C. M. 2006. Pattern recognition. Machine Learning. Google ScholarDigital Library
    7. Bolt, R. a. 1980. Put-that-there. SIGGRAPH ’80, 262–270. Google ScholarDigital Library
    8. Breiman, L. 2001. Random forests. Machine Learning 45, 1, 5–32. Google ScholarDigital Library
    9. Brookner, E. 1985. Phased-array radars. Scientific American 252, 2, 94–102.Google ScholarCross Ref
    10. Brown, L. 1999. A Radar History of World War II: Technical and Military Imperatives. Institute of Physics Publishing.Google Scholar
    11. Brunnbauer, M., Meyer, T., Ofner, G., Mueller, K., and Hagen, R. 2008. Embedded wafer level ball grid array (eWLB). In IEMT 2008, IEEE, 1–6.Google Scholar
    12. Card, S., Moran, T., and Newell, A. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Associates. Google ScholarDigital Library
    13. Chan, L., Chen, C.-h. H. Y.-l., and Yang, S. 2015. Cyclops: Wearable and single-piece full-body gesture input devices. In CHI 2015, ACM, 3001–3009. Google ScholarDigital Library
    14. Comscore Inc. 2014. The US mobile app report. Tech. rep.Google Scholar
    15. Cooperstock, J. R., Fels, S. S., Buxton, W., and Smith, K. C. 1997. Reactive environments. Communications of the ACM 40, 9 (Sep), 65–73. Google ScholarDigital Library
    16. Dardas, N. H., and Georganas, N. D. 2011. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement 60, 11, 3592–3607.Google ScholarCross Ref
    17. Dietz, P., and Leigh, D. 2001. Diamond Touch. In UIST ’01, 219.Google Scholar
    18. Dorfmuller-Ulhaas, K., and Schmalstieg, D. 2001. Finger tracking for interaction in augmented environments. ISMAR 2001, 55–64. Google ScholarDigital Library
    19. Duffner, S., Berlemont, S., Lefebvre, G., and Garcia, C. 2014. 3D gesture classification with convolutional neural networks. In ICASSP 2014, IEEE, 5432–5436.Google Scholar
    20. FCC, 2016. FCC online table of frequency allocations.Google Scholar
    21. Geng, S., Kivinen, J., Zhao, X., and Vainikainen, P. 2009. Millimeter-wave propagation channel characterization for short-range wireless communications. IEEE Transactions on Vehicular Technology 58, 1 (Jan), 3–13.Google Scholar
    22. Gillian, N., Knapp, R. B., and O’Modhrain, S. 2011. Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping. In Proceedings of the 2011 International Conference on New Interfaces for Musical Expression (NIME 11), Oslo, Norway.Google Scholar
    23. Gillian, N. 2011. Gesture Recognition for Musician Computer Interaction. PhD thesis, School of Music and Sonic Arts, Queen’s University Belfast.Google Scholar
    24. Gustafson, S., Holz, C., and Baudisch, P. 2011. Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. UIST ’11, 283–292. Google ScholarDigital Library
    25. Guyon, I., and Elisseeff, A. 2003. An introduction to variable and feature selection. The Journal of Machine Learning Research 3. Google ScholarDigital Library
    26. Hansen, C. J. 2011. WiGiG: Multi-gigabit wireless communications in the 60 GHz band. IEEE Wireless Communications 18, 6, 6–7.Google ScholarCross Ref
    27. Harrison, C., Tan, D., and Morris, D. 2010. Skinput: Appropriating the body as an input surface. CHI 2010, 453–462. Google ScholarDigital Library
    28. Holleis, P., Schmidt, A., Paasovaara, S., Puikkonen, A., and Häkkilä, J. 2008. Evaluating capacitive touch input on clothes. In MobileHCI ’08, ACM Press, New York, New York, USA, 81–90. Google ScholarDigital Library
    29. Holz, C., and Wilson, A. 2011. Data miming: Inferring spatial object descriptions from human gesture. CHI 2011, 811–820. Google ScholarDigital Library
    30. Hussain, M. G. 1998. Ultra-wideband impulse radar — an overview of the principles. IEEE Aerospace and Electronic Systems Magazine 13, 9, 9–14.Google ScholarCross Ref
    31. Ji, S., Xu, W., Yang, M., and Yu, K. 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1, 221–231. Google ScholarDigital Library
    32. Junker, H., Amft, O., Lukowicz, P., and Tröster, G. 2008. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41, 6, 2010–2024. Google ScholarDigital Library
    33. Keller, J. B. 1962. Geometrical theory of diffraction. JOSA 52, 2, 116–130.Google ScholarCross Ref
    34. Kellogg, B., Talla, V., and Gollakota, S. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), 303–316. Google ScholarDigital Library
    35. Keskin, C., Kiraç, F., Kara, Y. E., and Akarun, L. 2013. Real time hand pose estimation using depth sensors. In Consumer Depth Cameras for Computer Vision. Springer, 119–137.Google Scholar
    36. Khoshelham, K., and Elberink, S. O. 2012. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors 12, 2, 1437–1454.Google ScholarCross Ref
    37. Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. 2012. Digits. UIST ’12, 167–176. Google ScholarDigital Library
    38. Knott, E. F. 2012. Radar Cross Section Measurements. Springer Science & Business Media.Google Scholar
    39. Kurakin, A., Zhang, Z., and Liu, Z. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), 1975–1979.Google Scholar
    40. Lee, S., and Buxton, W. 1985. A multi-touch three dimensional touch-sensitive tablet. In CHI’85, 21–25. Google ScholarDigital Library
    41. Levanon, N. 2000. Multifrequency complementary phase-coded radar signal. IEE Proceedings – Radar, Sonar and Navigation 147, 6, 276–284.Google ScholarCross Ref
    42. LoPresti, L., LaCascia, M., Sclaroff, S., and Camps, O. 2015. Gesture modeling by Hanklet-based hidden Markov model. In Computer Vision — ACCV 2014, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., vol. 9005 of Lecture Notes in Computer Science. Springer International Publishing, 529–546.Google Scholar
    43. Melgarejo, P., Zhang, X., Ramanathan, P., and Chu, D. 2014. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 541–551. Google ScholarDigital Library
    44. Molchanov, P., Gupta, S., Kim, K., and Kautz, J. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1–7.Google Scholar
    45. Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Multi-sensor system for driver’s hand-gesture recognition. In IEEE Conference on Automatic Face and Gesture Recognition.Google Scholar
    46. Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Short-range FMCW monopulse radar for hand-gesture sensing. In IEEE Radar Conference (RadarCon 2015), IEEE, 1491–1496.Google Scholar
    47. Nasr, I., Karagozler, E., Poupyrev, I., and Trotta, S. 2015. A highly integrated 60-GHz 6-channel transceiver chip in 0.35 um SiGe technology for smart sensing and short-range communications. In IEEE CSICS 2015, IEEE, 1–4.Google Scholar
    48. Nasr, I., Jungmaier, R., Baheti, A., Noppeney, D., Bal, J. S., Wojnowski, M., Karagozler, E., Raja, H., Lien, J., Poupyrev, I., and Trotta, S. 2016. A highly integrated 60 GHz 6-channel transceiver with antenna in package for smart sensing and short range communications. submitted to IEEE Journal of Solid State Circuits.Google Scholar
    49. Nathanson, F. E., Reilly, J. P., and Cohen, M. N. 1991. Radar Design Principles — Signal Processing and the Environment, vol. 91.Google Scholar
    50. Novelda. Novelda Xethru NVA620x IC.Google Scholar
    51. Otero, M. 2005. Application of a continuous wave radar for human gait recognition. Proceedings of SPIE 5809, 538–548.Google ScholarCross Ref
    52. Paradiso, J., Abler, C., Hsiao, K.-y., and Reynolds, M. 1997. The Magic Carpet: Physical sensing for immersive environments. In CHI’97 Extended Abstracts on Human Factors in Computing Systems, ACM, 277–278. Google ScholarDigital Library
    53. Paradiso, J. A. 1999. The Brain Opera technology: New instruments and gestural sensors for musical interaction and performance. Journal of New Music Research 28, 2, 130–149.Google ScholarCross Ref
    54. Park, J.-I., and Kim, K.-T. 2010. A comparative study on ISAR imaging algorithms for radar target identification. Progress In Electromagnetics Research 108, 155–175.Google ScholarCross Ref
    55. Potter, L. C., Chiang, D.-M., Carriere, R., and Gerry, M. J. 1995. A GTD-based parametric model for radar scattering. IEEE Transactions on Antennas and Propagation 43, 10, 1058–1067.Google ScholarCross Ref
    56. Pu, Q., Gupta, S., Gollakota, S., and Patel, S. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, ACM, 27–38. Google ScholarDigital Library
    57. Rahman, T., Adams, A. T., Zhang, M., and Choudhury, T. 2015. DoppleSleep: A contactless unobtrusive sleep sensing system using short-range doppler radar. In Ubicomp ’15, ACM. Google ScholarDigital Library
    58. Ralston, T. S., Charvat, G. L., and Peabody, J. E. 2010. Real-time through-wall imaging using an ultrawideband MIMO phased array radar system. In IEEE International Symposium on Phased Array Systems and Technology (ARRAY), IEEE.Google Scholar
    59. Rekimoto, J. 2001. GestureWrist and GesturePad: Unobtrusive wearable interaction devices. ISWC 2001, 21–27. Google ScholarDigital Library
    60. Richards, M. A., Scheer, J., Holm, W. A., et al. 2010. Principles of modern radar: basic principles. SciTech Pub.Google Scholar
    61. Romero, J., Kjellström, H., Ek, C. H., and Kragic, D. 2013. Non-parametric hand pose estimation with object context. Image and Vision Computing 31, 8, 555–564. Google ScholarDigital Library
    62. Russell, D., Streitz, N., and Winograd, T. 2005. Building disappearing computers. Communications of the ACM 48, 3, 42–48. Google ScholarDigital Library
    63. Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. A. 2009. Enabling always-available input with muscle-computer interfaces. UIST 2009, 167–176. Google ScholarDigital Library
    64. Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., Freedman, D., Kohli, P., Krupka, E., Fitzgibbon, A., and Izadi, S. 2015. Accurate, robust, and flexible real-time hand tracking. CHI 2015, 3633–3642. Google ScholarDigital Library
    65. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1, 116–124. Google ScholarDigital Library
    66. Shuey, D., Bailey, D., and Morrissey, T. P. 1986. PHIGS: A standard, dynamic, interactive graphics interface. IEEE Computer Graphics and Applications 6, 8, 50–57. Google ScholarDigital Library
    67. Skolnik, M. I. 1962. Introduction to radar. Radar Handbook 2.Google Scholar
    68. Smith, C., and Goggans, P. 1993. Radar target identification. IEEE Antennas and Propagation Magazine 35, 2 (April), 27–38.Google ScholarCross Ref
    69. Smith, J., White, T., Dodge, C., Paradiso, J., Gershenfeld, N., and Allport, D. 1998. Electric field sensing for graphical interfaces. IEEE Computer Graphics and Applications 18, June, 54–59. Google ScholarDigital Library
    70. Smulders, P. 2002. Exploiting the 60 GHz band for local wireless multimedia access: Prospects and future directions. IEEE Communications Magazine 40, 1, 140–147. Google ScholarDigital Library
    71. Song, J., Sörös, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. 2014. In-air gestures around unmodified mobile devices. UIST 2014, 319–329. Google ScholarDigital Library
    72. Storcheus, D., Rostamizadeh, A., and Kumar, S. 2015. A survey of modern questions and challenges in feature extraction. In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges, NIPS, 1–18.Google Scholar
    73. Stove, A. G. 1992. Linear FMCW radar techniques. In IEEE Proceedings on Radar and Signal Processing, vol. 139, IET, 343–350.Google ScholarCross Ref
    74. Strickon, J., and Paradiso, J. 1998. Tracking hands above large interactive surfaces with a low-cost scanning laser rangefinder. CHI’98, 231–232. Google ScholarDigital Library
    75. Sutherland, I. E., Blackwell, A., and Rodden, K. 1963. Sketchpad: A man-machine graphical communication system. Tech. Rep. TK296, Lincoln Laboratory, MIT, Lexington, MA.Google ScholarDigital Library
    76. Wan, Q., Li, Y., Li, C., and Pal, R. 2014. Gesture recognition for smart home applications using portable radar sensors. In 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), IEEE, 6414–6417.Google Scholar
    77. Wang, Y., and Fathy, A. E. 2011. Micro-doppler signatures for intelligent human gait recognition using a UWB impulse radar. In IEEE Antennas and Propagation Society, AP-S International Symposium (Digest), 2103–2106.Google Scholar
    78. Wang, R. Y., and Popović, J. 2009. Real-time hand-tracking with a color glove. In SIGGRAPH 2009, ACM Press, New York, New York, USA, 1. Google ScholarDigital Library
    79. Watson-Watt, R. 1945. Radar in war and in peace. Nature 155, 3935, 319–324.Google Scholar
    80. Wei, T., and Zhang, X. 2015. mTrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, ACM, 117–129. Google ScholarDigital Library
    81. Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D. 2013. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 5, 6380–6393.Google ScholarCross Ref
    82. Weigel, M., Lu, T., Bailly, G., Oulasvirta, A., Majidi, C., and Steimle, J. 2015. iSkin: Flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In CHI 2015, ACM, 1–10. Google ScholarDigital Library
    83. Wu, J., Konrad, J., and Ishwar, P. 2013. Dynamic time warping for gesture-based user identification and authentication with Kinect. In ICASSP 2013, 2371–2375.Google Scholar
    84. Yatani, K., and Truong, K. N. 2012. BodyScope: A wearable acoustic sensor for activity recognition. In Ubicomp ’12, ACM, 341–350. Google ScholarDigital Library
    85. Zhai, S., Milgram, P., and Buxton, W. 1996. The influence of muscle groups on performance of multiple degree-of-freedom input. CHI ’96, 308–315. Google ScholarDigital Library
    86. Zhuang, Y., Song, C., Wang, A., Lin, F., Li, Y., Gu, C., Li, C., and Xu, W. 2015. SleepSense: Non-invasive sleep event recognition using an electromagnetic probe. In IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN 2015), 1–6.Google ScholarCross Ref

ACM Digital Library Publication:


Overview Page: