“Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands”
Conference:
Experience Type(s):
Title:
- Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands
Organizer(s)/Presenter(s):
Description:
We present a technique for dynamically projecting 3D content onto human hands with short perceived motion-to-photon latency. Computing the pose and shape of human hands accurately and quickly is a challenging task due to their articulated and deformable nature. We combine a slower 3D coarse estimation of the hand pose with high speed 2D correction steps which improve the alignment of the projection to the hands, increase the projected surface area, and reduce perceived latency. Since our approach leverages a full 3D reconstruction of the hands, any arbitrary texture or reasonably performant effect can be applied, which was not possible before. We conducted two user studies to assess the benefits of using our method. The results show subjects are less sensitive to latency artifacts and perform faster and with more ease a given associated task over the naïve approach of directly projecting rendered frames from the 3D pose estimation. We demonstrate several novel use cases and applications.
References:
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:https://arXiv.org/abs/2303.08774 (2023).
Google Scholar
[2]
Clément Baticle. 2016. “Aliens vs. Humans”. https://www.shadertoy.com/view/XldGDN. Accessed: 2024-03-20.
Google Scholar
[3]
Amit H. Bermano, Markus Billeter, Daisuke Iwai, and Anselm Grundhöfer. 2017. Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum 36, 2 (2017), 311–323. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13128
Crossref
Google Scholar
[4]
Yuta Halvorson, Takumi Saito, and Naoki Hashimoto. 2022. Robust tangible projection mapping with multi-view contour-based object tracking. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 756–757.
Crossref
Google Scholar
[5]
Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2018. Midas projection: Markerless and modelless dynamic projection mapping for material representation. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1–12.
Google Scholar
[6]
Hao-Lun Peng, Shin’Ya Nishida, and Yoshihiro Watanabe. 2023. Studying user perceptible misalignment in simulated dynamic facial projection mapping. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 493–502.
Crossref
Google Scholar
[7]
Hao-Lun Peng and Yoshihiro Watanabe. 2021. High-speed dynamic projection mapping onto human arm with realistic skin deformation. Applied Sciences 11, 9 (2021), 3753.
Google Scholar
[8]
Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. 2020. FleXeen: Visually Manipulating Perceived Fabric Bending Stiffness in Spatial Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 26, 2 (2020), 1433–1439.
Crossref
Google Scholar
[9]
Tomohiro Sueishi and Masatoshi Ishikawa. 2021. Ellipses Ring Marker for High-Speed Finger Tracking. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. 1–5.
Digital Library
Google Scholar
[10]
Ultraleap. 2024. “Leap Motion Controller, San Francisco, CA, USA”. https://www.ultraleap.com/product/. Accessed: 2024-05-15.
Google Scholar
[11]
Yoshihiro Watanabe and Masatoshi Ishikawa. 2019. High-speed and high-brightness color single-chip DLP projector using high-power led-based light sources. In 26th International Display Workshops, IDW 2019. International Display Workshops, 1350–1352.
Crossref
Google Scholar
[12]
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3836–3847.
Crossref
Google Scholar
