“CoLux: multi-object 3D micro-motion analysis using speckle imaging” by Smith, Desai, Agarwal and Gupta

  • ©Brandon Smith, Pratham Desai, Vishal Agarwal, and Mohit Gupta




    CoLux: multi-object 3D micro-motion analysis using speckle imaging

Session/Category Title: Imaginative Imaging




    We present CoLux, a novel system for measuring micro 3D motion of multiple independently moving objects at macroscopic standoff distances. CoLux is based on speckle imaging, where the scene is illuminated with a coherent light source and imaged with a camera. Coherent light, on interacting with optically rough surfaces, creates a high-frequency speckle pattern in the captured images. The motion of objects results in movement of speckle, which can be measured to estimate the object motion. Speckle imaging is widely used for micro-motion estimation in several applications, including industrial inspection, scientific imaging, and user interfaces (e.g., optical mice). However, current speckle imaging methods are largely limited to measuring 2D motion (parallel to the sensor image plane) of a single rigid object. We develop a novel theoretical model for speckle movement due to multi-object motion, and present a simple technique based on global scale-space speckle motion analysis for measuring small (5–50 microns) compound motion of multiple objects, along all three axes. Using these tools, we develop a method for measuring 3D micro-motion histograms of multiple independently moving objects, without tracking the individual motion trajectories. In order to demonstrate the capabilities of CoLux, we develop a hardware prototype and a proof-of-concept subtle hand gesture recognition system with a broad range of potential applications in user interfaces and interactive computer graphics.


    1. E. Archbold and A. E. Ennos. 1972. Displacement measurement from double exposure laser photographs. Optical Acta (1972).Google Scholar
    2. J. Bertolotti, E. G. van Putten, C. Blum, A. Lagendijk, W. L. Vos, and A. P. Mosk. 2012. Non-invasive imaging through opaque scattering layers. Nature 491 (2012). Google ScholarCross Ref
    3. O. Cossairt, N. Matsuda, and M. Gupta. 2014. Digital refocusing with incoherent holography. In IEEE ICCP. Google ScholarCross Ref
    4. J. C. Dainty. 1975. Laser Speckle and Related Phenomena. Springer. Google ScholarCross Ref
    5. A. Davis, M. Rubinstein, N. Wadhwa, G. Mysore, F. Durand, and W. T. Freeman. 2014. The Visual Microphone: Passive Recovery of Sound from Video. ACM Trans. Graph. (2014).Google Scholar
    6. L. Ek and N. E. Molin. 1971. Detection of the nodal lines and the amplitude of vibration by speckle interferometry. Opt. Commun. 2 (1971). Google ScholarCross Ref
    7. J. Engel, T. Schöps, and D. Cremers. 2014. LSD-SLAM: Large-Scale Direct Monocular SLAM. In Proc. ECCV. Google ScholarCross Ref
    8. M. Françon. 1979. Laser Speckle and Applications in Optics. Academic Press.Google Scholar
    9. I. Freund. 1990. Looking Through Walls and Around Corners. Physica A: Statistical Mechanics and its Applications (1990).Google Scholar
    10. J. García, Z. Zalevsky, P. García-Martínez, C. Ferreira, M. Teicher, and Y. Beiderman. 2008. Three-dimensional mapping and range measurement by means of projected speckle patterns. Applied Aptics (June 2008).Google Scholar
    11. W J Godinez and K. Rohr. 2015. Tracking multiple particles in fluorescence time-lapse microscopy images via probabilistic data association. IEEE Trans Med Imaging (2015).Google Scholar
    12. J. W. Goodman. 2000. Statistical Optics. Wiley-Interscience.Google Scholar
    13. J. W. Goodman. 2007. Speckle Phenomena in Optics: Theory and Applications. Roberts and Company Publishers.Google Scholar
    14. D. A. Gregory. 1976. Basic physical principles of defocused speckle photography: A tilt topology inspection technique. Optics and Laser Technology (1976).Google Scholar
    15. D. A. Gregory. 1978. Topological Speckle and Surface Inspection. In Speckle Metrology. Chapter 8.Google Scholar
    16. S. Gupta, D. Morris, S. Patel, and D. Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proc. ACM CHI. Google ScholarDigital Library
    17. P. Jacquot and P. K. Rastogi. 1979. Speckle motions induced by rigid-body movements in freespace geometry: an explicit investigation and extension to new cases. Applied Optics (1979).Google Scholar
    18. M. L. Jakobsen, H. T. Yura, and S. G. Hanson. 2012. Spatial filtering velocimetry of objective speckles for measuring out-of-plane motion. Applied Optics 51, 9 (2012). Google ScholarCross Ref
    19. K. Jo, M. Gupta, and S. Nayar. 2015. SpeDo: 6 DOF Ego-Motion Sensor Using Speckle Imaging. In Proc. ICCV. Google ScholarDigital Library
    20. B. Judkewitz, R. Horstmeyer, I. M. Vellekoop, I. N. Papadopoulos, and C. Yang. 2015. Translation correlations in anisotropically scattering media. Nature Physics (2015).Google Scholar
    21. J. C. Silveira Jacques Junior, S. R. Musse, and C. R. Jung. 2010. Crowd Analysis Using Computer Vision Techniques. IEEE Signal Processing Magazine 27, 5 (2010). Google ScholarCross Ref
    22. K. Kapinchev, A. Bradu, F. Barnes, and A. Podoleanu. 2015. GPU Implementation of Cross-Correlation for Image Generation in Real Time. In Proc. ICSPCS. Google ScholarCross Ref
    23. O. Katz, P. Heidmann, M. Fink, and S. Gigan. 2014. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nature Photonics 8 (2014). Google ScholarCross Ref
    24. O. Katz, E. Small, and Y. Silberberg. 2012. Looking around corners and through thin turbid layers in real time with scattered incoherent light. Nature Photonics 6 (2012). Google ScholarCross Ref
    25. J. Lien, N. Gillian, M. E. Karagozler, P. Amihood, C. Schwesig, E. Olson, H. Raja, and I. Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph. (2016).Google ScholarDigital Library
    26. S. Rollie and K. Sundmacher. 2010. Tracking the clustering dynamics in ternary particle mixtures by flow cytometry. Powder Technology 202 (2010). Google ScholarCross Ref
    27. J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore. 2013. Real-time human pose recognition in parts from single depth images. Commun. ACM (2013).Google Scholar
    28. S. K. Sinha. 1988. Improving the accuracy and resolution of particle image or laser speckle velocimetry. Experiments in Fluids (1988).Google Scholar
    29. S.W. Smith. 2002. Digital Signal Processing: A Practical Guide for Engineers and Scientists. California Technical Publishing.Google Scholar
    30. P. Synnergren. 1997. Measurement of three-dimensional displacement fields and shape using electronic speckle photography. Optical Engineering (1997).Google Scholar
    31. H. J. Tiziani. 1972. A study of the use of laser speckle to measure small tilts. Opt. Commun. 5 (1972).Google Scholar
    32. H. J. Tiziani. 1978. Vibration Analysis and Deformation Measurement. In Speckle Metrology. Chapter 5. Google ScholarCross Ref
    33. N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman. 2013. Phase-Based Video Motion Processing. ACM Trans. Graph. (2013).Google Scholar
    34. S. Wang, J. Song, J. Lien, I. Poupyrev, and O. Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. In Proc. ACM UIST. Google ScholarDigital Library
    35. F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler. 2013. Analysis of the Accuracy and Robustness of the Leap Motion Controller. Sensors (2013).Google Scholar
    36. H.-Y. Wu, M. Rubinstein, E. Shih, J. Guttag, F. Durand, and W. T. Freeman. 2012. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Trans. Graph. (2012).Google Scholar
    37. C. Xu, N. Ashwin, X. Zhang, and L. Cheng. 2015. Estimate Hand Poses Efficiently from Single Depth Images. IJCV (2015).Google Scholar
    38. Z. Zalevsky, Y. Beiderman, I. Margalit, S. Gingold, M. Teicher, V. Mico, and J. Garcia. 2009. Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern. Optics Express (2009).Google Scholar
    39. C. Zhao, K.-Y. Chen, M. T. I. Aumi, S. Patel, and M. S. Reynolds. 2014. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using GSM Signal. In Proc. ACM UIST. Google ScholarDigital Library
    40. J. Zizka, A. Olwal, and R. Raskar. 2011. SpeckleSense: Fast, Precise, Low-cost and Compact Motion Sensing using Laser Speckle. In Proc. ACM UIST. Google ScholarDigital Library

ACM Digital Library Publication: