“Dimix: A Cross-Dimensional Mixed Reality System Based on Latent Diffusion Model” by Taniguchi
Conference:
Type(s):
Entry Number: 06
Title:
- Dimix: A Cross-Dimensional Mixed Reality System Based on Latent Diffusion Model
Presenter(s):
Description:
Dimix is a system that integrates 2D images generated by Latent Diffusion Models (LDMs) [et al. 2022] with 3D and interactive Mixed Reality (MR) experiences. This system extracts regions from MR scenes based on panoramic images, adds optional mask images, and applies image-inpainting using LDMs to output an LDMs-generated image (LDMs image) that naturally extends the scene. Furthermore, by performing depth estimation on the images and reconstructing them as 3D scenes, the LDMs image can be treated in the same manner as 3D objects. This enables both basic features like occlusion and collision detection and highly interactive operations such as ink painting, achieving high immersion and realism. Additionally, the system incorporates real-time object detection to constrain the inpainting area, making the LDMs image more convincing. All processing is performed in real-time, allowing users to interact with the world in 3D without waiting for loading or any other preparation by simply uploading their preferred panorama image to the application.
To our best knowledge, Dimix is the first system to seamlessly integrate LDMs images into interactive and three-dimensional MR experiences. Users can immerse themselves in an unprecedented space where the real world seamlessly blends with a world generated by a neural network, offering a glimpse into the future of MR experiences.
References:
- Georgios et al.2021. Pano3D: A Holistic Benchmark and a Solid Baseline for 360deg Depth Estimation. (2021).
- Robin et al.2022. High-Resolution Image Synthesis with Latent Diffusion Models. (2022).
- Simon Green. 2010. Particle Simulation using CUDA. (2010).
- Daiki Taniguchi. 2021. Garage: GPU particle based AR contents for futuristic experience. (2021), 1–2.
