“Blending camera and 77 GHz radar sensing for equitable, robust plethysmography” by Vilesov, Chari, Armouti, Harish, Kulkarni, et al. …

With the resurgence of non-contact vital sign sensing due to the COVID-19 pandemic, remote heart-rate monitoring has gained significant prominence. Many existing methods use cameras; however previous work shows a performance loss for darker skin tones. In this paper, we show through light transport analysis that the camera modality is fundamentally biased against darker skin tones. We propose to reduce this bias through multi-modal fusion with a complementary and fairer modality – radar. Through a novel debiasing oriented fusion framework, we achieve performance gains over all tested baselines and achieve skin tone fairness improvements over the RGB modality. That is, the associated Pareto frontier between performance and fairness is improved when compared to the RGB modality. In addition, performance improvements are obtained over the radar-based method, with small trade-offs in fairness. We also open-source the largest multi-modal remote heart-rate estimation dataset of paired camera and radar measurements with a focus on skin tone representation.

1. Øyvind Aardal, Yoann Paichard, Sverre Brovoll, Tor Berger, Tor Sverre Lande, and Svein-Erik Hamran. 2012. Physical working principles of medical radar. IEEE Transactions on Biomedical Engineering 60, 4 (2012), 1142–1149.Google ScholarCross Ref
2. Md Atiqur Rahman Ahad, Upal Mahbub, and Tauhidur Rahman. 2021. Contactless Human Activity Analysis. Springer.Google Scholar
3. Mostafa Alizadeh, George Shaker, João Carlos Martins De Almeida, Plinio Pelegrini Morita, and Safeddin Safavi-Naeini. 2019. Remote monitoring of human vital signs using mm-wave FMCW radar. IEEE Access 7 (2019), 54958–54968.Google ScholarCross Ref
4. Sarah Alotaibi and William AP Smith. 2017. A biophysical 3D morphable model of face appearance. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 824–832.Google ScholarCross Ref
5. R. Rox Anderson and John A. Parrish. 1981. The Optics of Human Skin. Journal of Investigative Dermatology 77, 1 (1981), 13–19. Google ScholarCross Ref
6. Consumer Technology Association. 2018. Physical Activity Monitoring for Heart Rate, ANSI/CTA-2065.Google Scholar
7. Yunhao Ba, Zhen Wang, Kerim Doruk Karinca, Oyku Deniz Bozkurt, and Achuta Kadambi. 2021. Overcoming Difficulty in Obtaining Dark-skinned Subjects for Remote-PPG by Synthetic Augmentation. arXiv preprint arXiv:2106.06007 (2021).Google Scholar
8. Guha Balakrishnan, Fredo Durand, and John Guttag. 2013. Detecting pulse from head motions in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3430–3437.Google ScholarDigital Library
9. Carina Barbosa Pereira, Michael Czaplik, Vladimir Blazek, Steffen Leonhardt, and Daniel Teichmann. 2018. Monitoring of cardiorespiratory signals using thermal imaging: a pilot study on healthy human subjects. Sensors 18, 5 (2018), 1541.Google ScholarCross Ref
10. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. Nips tutorial 1 (2017), 2017.Google Scholar
11. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016), 4349–4357.Google Scholar
12. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.Google Scholar
13. Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang. 2013. Controlling attribute effect in linear regression. In 2013 IEEE 13th international conference on data mining. IEEE, 71–80.Google ScholarCross Ref
14. Weixuan Chen and Daniel McDuff. 2018. Deepphys: Video-based physiological measurement using convolutional attention networks. In Proceedings of the European Conference on Computer Vision (ECCV). 349–365.Google ScholarDigital Library
15. Gerard De Haan and Vincent Jeanne. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering 60, 10 (2013), 2878–2886.Google ScholarCross Ref
16. Amy Diane Droitcour. 2006. Non-contact measurement of heart and respiration rates with a single-chip microwave doppler radar. Stanford University.Google Scholar
17. Amy D Droitcour, Olga Boric-Lubecke, Victor M Lubecke, Jenshan Lin, and Gregory TA Kovacs. 2004. Range correlation and I/Q performance benefits in single-chip silicon Doppler radars for noncontact cardiopulmonary monitoring. IEEE Transactions on Microwave Theory and Techniques 52, 3 (2004), 838–848.Google ScholarCross Ref
18. Davide Giavarina. 2015. Understanding bland altman analysis. Biochemia medica 25, 2 (2015), 141–151.Google Scholar
19. Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. arXiv preprint arXiv:2109.00412 (2021).Google Scholar
20. S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 553–560. ISSN: 1063-6919. Google ScholarCross Ref
21. Christophe Hurter and Daniel McDuff. 2017. Cardiolens: remote physiological monitoring in a mixed reality environment. In ACM siggraph 2017 emerging technologies. 1–2.Google Scholar
22. Takanori Igarashi, Ko Nishino, and Shree K Nayar. 2007. The appearance of human skin: A survey. Now Publishers Inc.Google Scholar
23. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. PMLR, 448–456.Google Scholar
24. Manjit Kaur and Dilbag Singh. 2020. Fusion of medical images using deep belief networks. Cluster Computing 23, 2 (2020), 1439–1453.Google ScholarDigital Library
25. Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. 2019. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9012–9020.Google ScholarCross Ref
26. Yoonkyoung Kim, Yosep Park, Jinman Kim, and Eui Chul Lee. 2018. Remote heart rate monitoring method using infrared thermal camera. International Journal of Engineering Research and Technology 11, 3 (2018), 493–500.Google Scholar
27. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
28. Eugene Lee, Evan Chen, and Chen-Yi Lee. 2020. Meta-rppg: Remote heart rate estimation using a transductive meta-learner. In European Conference on Computer Vision. Springer, 392–409.Google ScholarDigital Library
29. Magdalena Lewandowska, Jacek Rumiński, Tomasz Kocejko, and Jędrzej Nowak. 2011. Measuring pulse rate with a webcam—a non-contact method for evaluating cardiac activity. In 2011 federated conference on computer science and information systems (FedCSIS). IEEE, 405–410.Google Scholar
30. James C Lin. 1975. Noninvasive microwave measurement of respiration. Proc. IEEE 63, 10 (1975), 1530–1530.Google ScholarCross Ref
31. Wenjie Lv, Wangdong He, Xipeng Lin, and Jungang Miao. 2021. Non-Contact Monitoring of Human Vital Signs Using FMCW Millimeter Wave Radar in the 120 GHz Band. Sensors 21, 8 (2021), 2732.Google ScholarCross Ref
32. Ewa Magdalena Nowara, Tim K Marks, Hassan Mansour, and Ashok Veeraraghavan. 2018. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 1272–1281.Google ScholarCross Ref
33. Kenta Matsumura, Sogo Toda, and Yuji Kato. 2020. RGB and near-infrared light reflectance/transmittance photoplethysmography for measuring heart rate during motion. IEEE Access 8 (2020), 80233–80242.Google ScholarCross Ref
34. Cardiac Monitors. 2002. Heart Rate Meters, and Alarms. ANSI/AAMI Standard EC13 (2002).Google Scholar
35. Toshiaki Negishi, Shigeto Abe, Takemi Matsui, He Liu, Masaki Kurosawa, Tetsuo Kirimoto, and Guanghao Sun. 2020. Contactless vital signs measurement system using RGB-thermal image sensors and its clinical screening test on patients with seasonal influenza. Sensors 20, 8 (2020), 2171.Google ScholarCross Ref
36. Benjamin W Nelson and Nicholas B Allen. 2019. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth and uHealth 7, 3 (2019), e10828.Google Scholar
37. Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. 2019. Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing 29 (2019), 2409–2423.Google ScholarDigital Library
38. Ewa M Nowara, Daniel McDuff, and Ashok Veeraraghavan. 2020. A meta-analysis of the impact of skin tone and gender on non-contact photoplethysmography measurements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 284–285.Google Scholar
39. Prashanth Pai and Mohammed Zafar Ali Khan. 2008. Comparison of SC and MRC receiver complexity for three antenna diversity systems. In 2008 24th Biennial Symposium on Communications. IEEE, 302–305.Google ScholarCross Ref
40. Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. 2010. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE transactions on biomedical engineering 58, 1 (2010), 7–11.Google Scholar
41. Lingyun Ren, Lingqin Kong, Farnaz Foroughian, Haofei Wang, Paul Theilmann, and Aly E Fathy. 2017. Comparison study of noncontact vital signs detection using a Doppler stepped-frequency continuous-wave radar and camera-based imaging photoplethysmography. IEEE Transactions on Microwave Theory and Techniques 65, 9 (2017), 3519–3529.Google ScholarCross Ref
42. Lingyun Ren, Haofei Wang, Krishna Naishadham, Ozlem Kilic, and Aly E Fathy. 2016. Phase-based methods for heart rate detection using UWB impulse Doppler radar. IEEE Transactions on Microwave Theory and Techniques 64, 10 (2016), 3319–3331.Google ScholarCross Ref
43. Silonie Sachdeva et al. 2009. Fitzpatrick skin typing: Applications in dermatology. Indian journal of dermatology, venereology and leprology 75, 1 (2009), 93.Google Scholar
44. Vikas Singh, Nishchal K Verma, Zeeshan Ul Islam, and Yan Cui. 2019. Feature learning using stacked autoencoder for shared and multimodal fusion of medical images. In Computational Intelligence: Theories, Applications and Future Directions-Volume I. Springer, 53–66.Google Scholar
45. Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. 2021. PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE Journal of Biomedical and Health Informatics 25, 5 (2021), 1373–1384.Google ScholarCross Ref
46. Nitish Srivastava, Ruslan Salakhutdinov, et al. 2012. Multimodal Learning with Deep Boltzmann Machines.. In NIPS, Vol. 1. Citeseer, 2.Google Scholar
47. Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. 2008. Remote plethysmographic imaging using ambient light. Optics express 16, 26 (2008), 21434–21445.Google Scholar
48. Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In 2018 ieee/acm international workshop on software fairness (fairware). IEEE, 1–7.Google Scholar
49. Vytautas Vizbara. 2013. Comparison of green, blue and infrared light in wrist and forehead photoplethysmography. BIOMEDICAL ENGINEERING 2016 17, 1 (2013).Google Scholar
50. Wenjin Wang, Albertus C den Brinker, Sander Stuijk, and Gerard De Haan. 2016. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering 64, 7 (2016), 1479–1491.Google ScholarCross Ref
51. Zhen Wang, Yunhao Ba, Pradyumna Chari, Oyku Bozkurt, Gianna Brown, Parth Patwa, Niranjan Vaddi, Laleh Jalilian, and Achuta Kadambi. 2022. Synthetic Generation of Face Videos with Plethysmograph Physiology. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.Google ScholarCross Ref
52. Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. 2020. Towards fairness in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8919–8928.Google ScholarCross Ref
53. Fokko P Wieringa, Frits Mastik, and Antonius FW van der Steen. 2005. Contactless multiple wavelength photoplethysmographic imaging: A first step toward “SpO 2 camera” technology. Annals of biomedical engineering 33, 8 (2005), 1034–1041.Google ScholarCross Ref
54. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM transactions on graphics (TOG) 31, 4 (2012), 1–8.Google Scholar
55. Shuqiong Wu, Takuya Sakamoto, Kentaro Oishi, Toru Sato, Kenichi Inoue, Takeshi Fukuda, Kenji Mizutani, and Hiroyuki Sakai. 2019. Person-specific heart rate estimation with ultra-wideband radar using convolutional neural networks. IEEE Access 7 (2019), 168484–168494.Google ScholarCross Ref
56. T Wu. 2003. Ppgi: New development in noninvasive and contactless diagnosis of dermal perfusion using near infrared light. J. GCPD eV 7, 1 (2003), 17–24.Google Scholar
57. Ting Wu, Vladimir Blazek, and Hans Juergen Schmitt. 2000. Photoplethysmography imaging: a new noninvasive and noncontact method for mapping of the dermal perfusion changes. In Optical Techniques and Instrumentation for the Measurement of Blood Composition, Structure, and Dynamics, Vol. 4163. International Society for Optics and Photonics, 62–70.Google ScholarCross Ref
58. Zitong Yu, Xiaobai Li, and Guoying Zhao. 2019. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. arXiv preprint arXiv:1905.02419 (2019).Google Scholar
59. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics. PMLR, 962–970.Google Scholar
60. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23, 10 (2016), 1499–1503.Google ScholarCross Ref
61. Tianyue Zheng, Zhe Chen, Shujie Zhang, Chao Cai, and Jun Luo. 2021. MoRe-Fi: Motion-robust and Fine-grained Respiration Monitoring via Deep-Learning UWB Radar. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 111–124.Google ScholarDigital Library