“3Dlite: towards commodity 3D scanning for content creation” by Huang, Dai, Guibas and Niessner
Conference:
Type(s):
Title:
- 3Dlite: towards commodity 3D scanning for content creation
Session/Category Title: 3D Scenes
Presenter(s)/Author(s):
Abstract:
We present 3DLite1, a novel approach to reconstruct 3D environments using consumer RGB-D sensors, making a step towards directly utilizing captured 3D content in graphics applications, such as video games, VR, or AR. Rather than reconstructing an accurate one-to-one representation of the real world, our method computes a lightweight, low-polygonal geometric abstraction of the scanned geometry. We argue that for many graphics applications it is much more important to obtain high-quality surface textures rather than highly-detailed geometry. To this end, we compensate for motion blur, auto-exposure artifacts, and micro-misalignments in camera poses by warping and stitching image fragments from low-quality RGB input data to achieve high-resolution, sharp surface textures. In addition to the observed regions of a scene, we extrapolate the scene geometry, as well as the mapped surface textures, to obtain a complete 3D model of the environment. We show that a simple planar abstraction of the scene geometry is ideally suited for this completion task, enabling 3DLite to produce complete, lightweight, and visually compelling 3D scene models. We believe that these CAD-like reconstructions are an important step towards leveraging RGB-D scanning in actual content creation pipelines.
References:
1. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics-TOG 28, 3 (2009), 24.
2. Fausto Bernardini, Ioana M. Martin, and Holly Rushmeier. 2001. High-quality texture reconstruction from multiple scans. IEEE Transactions on Visualization and Computer Graphics 7, 4 (2001), 318–332.
3. Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017. Patch-Based Optimization for Image-Based Texture Mapping. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4 (2017).
4. Yuri Boykov, Olga Veksler, and Ramin Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence 23, 11 (2001), 1222–1239.
5. Jiawen Chen, Dennis Bautembach, and Shahram Izadi. 2013. Scalable real-time volumetric surface reconstruction. ACM Transactions on Graphics (TOG) 32, 4 (2013), 113.
6. L Paul Chew. 1987. Constrained delaunay triangulations. In Proceedings of the third annual symposium on Computational geometry. ACM, 215–222.
7. Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. 2015. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5556–5565.
8. Frederique Crete, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. 2007. The blur effect: perception and estimation with a new no-reference perceptual blur metric. In Electronic Imaging 2007. International Society for Optics and Photonics, 64920I–64920I.
9. Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing 13, 9 (2004), 1200–1212.
10. Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 303–312.
11. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017a. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Cross Ref
12. Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt. 2017b. BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration. ACM Transactions on Graphics 2017 (TOG) (2017).
13. Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. 2017c. Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Cross Ref
14. Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, 4 (2012), 82–1.
15. Xavier Decoret, Franĉois Sillion, Gernot Schaufler, and Julie Dorsey. 1999. Multi-layered impostors for accelerated rendering. In Computer Graphics Forum, Vol. 18. Wiley Online Library, 61–73.
16. Mingsong Dou, Li Guan, Jan-Michael Frahm, and Henry Fuchs. 2012. Exploring high-level plane primitives for indoor 3D reconstruction with a hand-held RGB-D camera. In Asian Conference on Computer Vision. Springer, 94–108.
17. M. Dzitsiuk, J. Sturm, R. Maier, L. Ma, and D. Cremers. 2017. De-noising, Stabilizing and Completing 3D Reconstructions On-the-go using Plane Priors. In International Conference on Robotics and Automation (ICRA).
18. Pedro F Felzenszwalb and Daniel P Huttenlocher. 2004. Efficient graph-based image segmentation. International journal of computer vision 59, 2 (2004), 167–181.
19. Chen Feng, Yuichi Taguchi, and Vineet R Kamat. 2014. Fast plane extraction in organized point clouds using agglomerative hierarchical clustering. In Robotics and Automation (ICRA), 2014 IEEE International Conference on. IEEE, 6218–6225. Cross Ref
20. Yoav HaCohen, Eli Shechtman, Dan B Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM transactions on graphics (TOG) 30, 4 (2011), 70.
21. Maciej Halber and Thomas Funkhouser. 2017. Fine-To-Coarse Global Registration of RGB-D Scans. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Cross Ref
22. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable Inside-Out Image-Based Rendering. 35, 6 (2016), 231:1–231:11.
23. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 559–568.
24. Andrew Edie Johnson and Sing Bing Kang. 1999. Registration and integration of textured 3D data. Image and vision computing 17, 2 (1999), 135–147.
25. Maik Keller, Damien Lefloch, Martin Lambers, Shahram Izadi, Tim Weyrich, and Andreas Kolb. 2013. Real-time 3d reconstruction in dynamic scenes using point-based fusion. In 3DTV-Conference, 2013 International Conference on. IEEE, 1–8.
26. David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91–110.
27. Peter J Neugebauer and Konrad Klein. 1999. Texturing 3d models of real world objects from multiple unregistered photographic views. In Computer Graphics Forum, Vol. 18. Wiley Online Library, 245–256.
28. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-Time Dense Surface Mapping and Tracking. In Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on. IEEE, 127–136.
29. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Realtime 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG) 32, 6 (2013), 169.
30. Eyal Ofek, Erez Shilat, Ari Rappoport, and Michael Werman. 1997. Multiresolution Textures from Image Sequences. IEEE Comput. Graph. Appl. 17, 2 (1997), 18–29.
31. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2536–2544. Cross Ref
32. Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 313–318.
33. Kari Pulli, Simo Piiroinen, Tom Duchamp, and Werner Stuetzle. 2005. Projective surface matching of colored 3D scans. In 3-D Digital Imaging and Modeling, 2005. 3DIM 2005. Fifth International Conference on. IEEE, 531–538.
34. Kari Pulli and Linda G Shapiro. 2000. Surface reconstruction and display from range and color data. Graphical Models 62, 3 (2000), 165–201.
35. Claudio Rocchini, Paolo Cignoni, Claudio Montani, and Roberto Scopigno. 1999. Multiple textures stitching and blending on 3D objects. In Rendering TechniquesâĂŹ 99. Springer, 119–130.
36. François Sillion, George Drettakis, and Benoit Bodelet. 1997. Efficient Impostor Manipulation for Real-Time Visualization of Urban Scenery. In Computer Graphics Forum, Vol. 16. Wiley Online Library.
37. Denis Simakov, Yaron Caspi, Eli Shechtman, and Michal Irani. 2008. Summarizing visual data using bidirectional similarity. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 1–8. Cross Ref
38. Ioannis Stamos and PE Allen. 2000. 3–D model construction using range and image data. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, Vol. 1. IEEE, 531–536. Cross Ref
39. Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. 2016. Deep Video Deblurring. arXiv preprint arXiv:1611.08387 (2016).
40. Yuichi Taguchi, Yong-Dian Jian, Srikumar Ramalingam, and Chen Feng. 2013. Point-plane SLAM for hand-held 3D sensors. In Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 5182–5189. Cross Ref
41. Cuong T Vu, Thien D Phan, and Damon M Chandler. 2012. S3: A Spectral and Spatial Measure of Local Perceived Sharpness in Natural Images. IEEE Transactions on Image Processing 21, 3 (2012), 934–945.
42. Tuanfeng Y Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas Guibas, and Niloy J Mitra. 2016. Unsupervised texture transfer from images to model collections. ACM Transactions on Graphics (TOG) 35, 6 (2016), 177.
43. Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker, and Andrew Davison. 2015. ElasticFusion: Dense SLAM without a pose graph. Robotics: Science and Systems.
44. Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2016. High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis. CoRR abs/1611.09969 (2016). http://arxiv.org/abs/1611.09969
45. Edward Zhang, Michael F Cohen, and Brian Curless. 2016. Emptying, refurnishing, and relighting indoor spaces. ACM Transactions on Graphics (TOG) 35, 6 (2016), 174.
46. Yizhong Zhang, Weiwei Xu, Yiying Tong, and Kun Zhou. 2015. Online structure analysis for real-time indoor scene reconstruction. ACM Transactions on Graphics (TOG) 34, 5 (2015), 159.
47. Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4 (2014), 155.
