“MIDAS projection: markerless and modelless dynamic projection mapping for material representation”
Conference:
Type(s):
Title:
- MIDAS projection: markerless and modelless dynamic projection mapping for material representation
Session/Category Title: Mixed reality
Presenter(s)/Author(s):
Moderator(s):
Abstract:
The visual appearance of an object can be disguised by projecting virtual shading as if overwriting the material. However, conventional projection-mapping methods depend on markers on a target or a model of the target shape, which limits the types of targets and the visual quality. In this paper, we focus on the fact that the shading of a virtual material in a virtual scene is mainly characterized by surface normals of the target, and we attempt to realize markerless and modelless projection mapping for material representation. In order to deal with various targets, including static, dynamic, rigid, soft, and fluid objects, without any interference with visible light, we measure surface normals in the infrared region in real time and project material shading with a novel high-speed texturing algorithm in screen space. Our system achieved 500-fps high-speed projection mapping of a uniform material and a tileable-textured material with millisecond-order latency, and it realized dynamic and flexible material representation for unknown objects. We also demonstrated advanced applications and showed the expressive shading performance of our technique.
References:
1. Pablo Fernández Alcantarilla, Jesús Nuevo, and Adrien Bartoli. 2013. Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces. In Proceedings of the British Machine Vision Conference (BMVC ’13). 1–10.Google ScholarCross Ref
2. Daniel G. Aliaga, Yu Hong Yeung, Alvin J. Law, Behzad Sajadi, and Aditi Majumder. 2012. Fast High-Resolution Appearance Editing Using Superimposed Projections. ACM Transactions on Graphics 31, 2 (2012), 13:1–13:12. Google ScholarDigital Library
3. Toshiyuki Amano. 2012. Shading illusion: A novel way for 3-D representation on paper media. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR ’12). 1–6.Google ScholarCross Ref
4. Toshiyuki Amano, Kenichi Komura, Takashi Sasabuchi, Shota Nakano, and Shunya Yamashita. 2012. Appearance Control for Human Material Perception Manipulation. In Proceedings of the International Conference on Pattern Recognition (ICPR ’12). 13–16.Google Scholar
5. Hirotaka Asayama, Daisuke Iwai, and Kosuke Sato. 2018. Fabricating Diminishable Visual Markers for Geometric Registration in Projection Mapping. IEEE Transactions on Visualization and Computer Graphics 24, 2 (2018), 1091–1102.Google ScholarCross Ref
6. Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. 2001. Dynamic Shader Lamps: Painting on Movable Objects. In Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR ’01). 207–216. Google ScholarDigital Library
7. Amit Bermano, Markus Billeter, Daisuke Iwai, and Anselm Grundhöfer. 2017. Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum 36, 2 (2017), 311–323. Google ScholarDigital Library
8. Amit Bermano, Philipp Brüschweiler, Anselm Grundhöfer, Daisuke Iwai, Bernd Bickel, and Markus Gross. 2013. Augmenting Physical Avatars Using Projector-based Illumination. ACM Transactions on Graphics 32, 6 (2013), 189:1–189:10. Google ScholarDigital Library
9. James F. Blinn. 1977. Models of Light Reflection for Computer Synthesized Pictures. ACM SIGGRAPH Computer Graphics 11, 2 (1977), 192–198. Google ScholarDigital Library
10. Gunnar Farneback. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and simultaneous segmentation of the motion field. In Proceedings of the IEEE International Conference on Computer Vision (ICCV ’01). 171–177.Google ScholarCross Ref
11. Chi-Wing Fu and Man-Kang Leung. 2005. Texture Tiling on Arbitrary Topological Surfaces Using Wang Tiles. In Proceedings of the Sixteenth Eurographics Conference on Rendering Techniques (EGSR ’05). 99–104. Google ScholarDigital Library
12. Jonas Garding. 1992. Shape from Texture for Smooth Curved Surfaces. Journal of Mathematical Imaging and Vision 2, 4 (1992), 327–350.Google ScholarCross Ref
13. James J. Gibson. 1950. The Perception of the Visual World. Houghton Mifflin.Google Scholar
14. Anselm Grundhöfer and Daisuke Iwai. 2018. Recent Advances in Projection Mapping Algorithms, Hardware and Applications. Computer Graphics Forum 37, 2 (2018), 653–675.Google ScholarCross Ref
15. Felix Heide, Wolfgang Heidrich, Matthias Hullin, and Gordon Wetzstein. 2015. Doppler Time-of-flight Imaging. ACM Transactions on Graphics 34, 4 (2015), 36:1–36:11. Google ScholarDigital Library
16. Carlos Hernandez, George Vogiatzis, Gabriel J. Brostow, Bjorn Stenger, and Roberto Cipolla. 2007. Non-rigid Photometric Stereo with Colored Lights. In Proceedings of the IEEE International Conference on Computer Vision (ICCV ’07). 1–8.Google ScholarCross Ref
17. Berthold K. P. Horn. 1986. Robot Vision. The MIT Press. Google ScholarDigital Library
18. Berthold K. P. Horn. 1990. Height and Gradient from Shading. International Journal of Computer Vision 5, 1 (1990), 37–75. Google ScholarDigital Library
19. Yunpu Hu, Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2017. Robust 6-DOF motion sensing for an arbitrary rigid body by multi-view laser Doppler measurements. Optical Express 25, 24 (2017), 30371–30387.Google ScholarCross Ref
20. Matthias B. Hullin, Hendrik P. A. Lensch, Ramesh Raskar, Hans-Peter Seidel, and Ivo Ihrke. 2011. Dynamic Display of BRDFs. Computer Graphics Forum 30, 2 (2011), 475–483.Google ScholarCross Ref
21. Shingo Kagami and Koichi Hashimoto. 2015. Sticky Projection Mapping: 450-fps Tracking Projection Onto a Moving Planar Surface. In Proceedings of the SIGGRAPH Asia, Emerging Technologies (SA ’15). 23:1–23:3. Google ScholarDigital Library
22. Alvin J. Law, Daniel G. Aliaga, Aditi Majumder, and Zygmunt Pizlo. 2011. Perceptually Based Appearance Modification for Compliant Appearance Editing. Computer Graphics Forum 30, 8 (2011), 2288–2300.Google ScholarCross Ref
23. Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI ’81). 674–679. Google ScholarDigital Library
24. Leo Miyashita, Kota Ishihara, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2016. ZoeMatrope: A System for Physical Material Design. ACM Transactions on Graphics 35, 4 (2016), 66:1–66:11. Google ScholarDigital Library
25. Leo Miyashita, Ryota Yonezawa, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2015. 3D Motion Sensing of Any Object Without Prior Knowledge. ACM Transactions on Graphics 34, 6 (2015), 218:1–218:11. Google ScholarDigital Library
26. Yasushi Nagamune. 2014. Image capturing device and image capturing method. (Sept. 16 2014). US Patent 8,836,795.Google Scholar
27. Akihiro Nakamura, Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2018. RIFNOM: 3D Rotaion-Invariant Features on Normal Maps. In Proceedings of the Eurographics, Poster (EG ’18).Google Scholar
28. Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2017. Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker. IEEE Transactions on Visualization and Computer Graphics 23, 3 (2017), 1235–1248. Google ScholarDigital Library
29. Albert Ng, Julian Lepinski, Daniel Wigdor, Steven Sanders, and Paul Dietz. 2012. Designing for Low-latency Direct-touch Input. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST ’12). 453–464. Google ScholarDigital Library
30. Kohei Okumura, Hiromasa Oku, and Masatoshi Ishikawa. 2013. Active Projection AR using High-speed Optical Axis Control and Appearance Estimation Algorithm. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME ’13). 1–6.Google Scholar
31. Emil Praun, Adam Finkelstein, and Hugues Hoppe. 2000. Lapped Textures. In Proceedings of the ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’00). 465–470. Google ScholarDigital Library
32. Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader Lamps: Animating Real Objects With Image-Based Illumination. In Proceedings of the Eurographics Workshop on Rendering Techniques (Eurographics ’01). 89–102. Google ScholarDigital Library
33. Christoph Resch, Peter Keitler, and Gudrun Klinker. 2016. Sticky Projections-A Model-Based Approach to Interactive Shader Lamps Tracking. IEEE Transactions on Visualization and Computer Graphics 22, 3 (2016), 1291–1301. Google ScholarDigital Library
34. Edward Rosten, Reid Porter, and Tom Drummond. 2010. Faster and Better: A Machine Learning Approach to Corner Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 1 (2010), 105–119. Google ScholarDigital Library
35. Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the International Conference on Computer Vision (ICCV ’11). 2564–2571. Google ScholarDigital Library
36. Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. 2009. Fast Point Feature Histograms (FPFH) for 3D Registration. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ’09). 1848–1853. Google ScholarDigital Library
37. Reza Sabzevari, Eren Alak, and Davide Scaramuzza. 2015. PHENOM: Interest Points on Photometric Normal Maps. In Proceedings of the Eurographics, Poster (EG ’15).Google Scholar
38. Samuele Salti, Federico Tombari, and Luigi Di Stefano. 2014. SHOT: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding 125, Supplement C (2014), 251–264.Google ScholarCross Ref
39. Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollh”ofer, Sharam Izadi, Marc Stamminger, and Bauer Frank. 2015. Real-Time Pixel Luminance Optimization for Dynamic Multi-Projection Mapping. ACM Transactions on Graphics 34, 6 (2015), 237:1–237:11. Google ScholarDigital Library
40. Peter-Pike J. Sloan, William Martin, Amy Gooch, and Bruce Gooch. 2001. The Lit Sphere: A Model for Capturing NPR Shading from Art. In Proceedings of the Graphics Interface (GI ’01). 143–150. Google ScholarDigital Library
41. Tomohiro Sueishi, Hiromasa Oku, and Masatoshi Ishikawa. 2017. Lumipen 2: Dynamic Projection Mapping with Mirror-based Robust High-speed Tracking against Illumination Changes. PRESENCE: Teleoperators and Virtual Environments 25, 4 (2017), 299–321. Google ScholarDigital Library
42. Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. 2007. Microfacet Models for Refraction Through Rough Surfaces. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR ’07). 195–206. Google ScholarDigital Library
43. Gregory J. Ward. 1992. Measuring and Modeling Anisotropic Reflection. ACM SIGGRAPH Computer Graphics 26, 2 (1992), 265–272. Google ScholarDigital Library
44. Yoshihiro Watanabe, Toshiyuki Kato, and Masatoshi Ishikawa. 2017. Extended Dot Cluster Marker for High-speed 3D Tracking in Dynamic Projection Mapping. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR ’17). 52–61.Google ScholarCross Ref
45. Yoshihiro Watanabe, Gaku Narita, Sho Tatsuno, and Masatoshi Ishikawa. 2015. High-speed 8-bit Image Projector at 1,000 fps with 3 ms Delay. In Proceedings of the International Display Workshops 2015 (IDW ’15). 1064–1065.Google Scholar
46. Li-Yi Wei and Marc Levoy. 2001. Texture Synthesis over Arbitrary Manifold Surfaces. In Proceedings of the ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’01). 355–360. Google ScholarDigital Library
47. Robert J. Woodham. 1994. Gradient and curvature from the photometric-stereo method, including local confidence estimation. Journal of the Optical Society of America A 11, 11 (1994), 3050–3068.Google ScholarCross Ref
