“Space Fusion: Context-Aware Interaction Using 3D Scene Parsing”
Conference:
Experience Type(s):
Title:
- Space Fusion: Context-Aware Interaction Using 3D Scene Parsing
Description:
Context-aware interaction (interaction varying correspondingly to scene semantics or object categories) is an important element to make mixed reality experience more immersive and realistic. However, few mixed reality applications provide this kind of interaction since it is difficult to recognize all objects in the real world scene densely as 3D. In this work, we present a 3D scene parsing system by combining semantic segmentation with visual Simultaneous Localization and Mapping (SLAM). This system can reconstruct and recognize the real indoor scene as dense point cloud with categorical labels in real-time. We also present a context-aware mixed reality application that utilizes the parsing system. Users can import their own room into the mixed reality world, and enjoy interaction with a virtual robot in their room through a head-mounted display (HMD). The virtual robot behaves correspondingly to the real object’s category. Therefore, our 3D scene parsing realizes context-aware interaction.
References:
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431–3440.
[2] Microsoft. 2016. RoboRaid – Microsoft Store. https://www.microsoft.eom/en-us/p/roboraid/9nblggh5fv3j Accessed: 2018-06-28.
[3] Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger. 2016. ElasticFusion: Real-time dense SLAM and light source estimation. The International Journal of Robotics Research 35, 14 (2016), 1697–1716.
