“Reconstructing scenes with mirror and glass surfaces” by Whelan, Goesele, Lovegrove, Straub, Green, et al. …

  • ©Thomas Whelan, Michael Goesele, Steven J. Lovegrove, Julian Straub, Simon G. Green, Richard Szeliski, Steven ButterField, Shobit Verma, and Richard Newcombe

Conference:


Type:


Entry Number: 102

Title:

    Reconstructing scenes with mirror and glass surfaces

Session/Category Title: 3D Capture

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Planar reflective surfaces such as glass and mirrors are notoriously hard to reconstruct for most current 3D scanning techniques. When treated naïvely, they introduce duplicate scene structures, effectively destroying the reconstruction altogether. Our key insight is that an easy to identify structure attached to the scanner—in our case an AprilTag—can yield reliable information about the existence and the geometry of glass and mirror surfaces in a scene. We introduce a fully automatic pipeline that allows us to reconstruct the geometry and extent of planar glass and mirror surfaces while being able to distinguish between the two. Furthermore, our system can automatically segment observations of multiple reflective surfaces in a scene based on their estimated planes and locations. In the proposed setup, minimal additional hardware is needed to create high-quality results. We demonstrate this using reconstructions of several scenes with a variety of real mirrors and glass.

References:


    1. Sameer Agarwal, Keir Mierle, and Others. 2018. Ceres Solver, http://ceres-solver.org. (2018).Google Scholar
    2. N. Arvanitopoulos, R. Achanta, and S. Süsstrunk. 2017. Single Image Reflection Suppression. In CVPR 2017. 1752–1760.Google Scholar
    3. J. Balzer, D. Acevedo-Feliz, S. Soatto, S. Höfer, M. Hadwiger, and J. Beyerer. 2014. Cavlectometry: Towards Holistic Reconstruction of Large Mirror Objects. In 2nd International Conference on 3D Vision (3DV). 448–455. Google ScholarDigital Library
    4. J. Balzer, S. Höfer, and J. Beyerer. 2011. Multiview specular stereo reconstruction of large mirror surfaces. In CVPR 2011. 2537–2544. Google ScholarDigital Library
    5. L. G. Brown. 1992. A Survey of Image Registration Techniques. Computing Surveys 24, 4 (December 1992), 325–376. Google ScholarDigital Library
    6. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3D: Learning from RGB-D Data in Indoor Environments. In 5th International Conference on 3D Vision (3DV).Google ScholarCross Ref
    7. Tongbo Chen, Michael Goesele, and Hans-Peter Seidel. 2006. Mesostructure from Specularity. In CVPR 2006, Vol. 2. 1825–1832. Google ScholarDigital Library
    8. Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
    9. A. DelPozo and S. Savarese. 2007. Detecting Specular Surfaces on Natural Images. In CVPR 2007.Google Scholar
    10. Yuanyuan Ding and Jingyi Yu. 2008. Recovering shape characteristics on near-flat specular surfaces. In CVPR 2008.Google Scholar
    11. J. Engel, V. Koltun, and D. Cremers. 2018. Direct Sparse Odometry. PAMI 40, 3 (2018), 611–625.Google ScholarCross Ref
    12. A. Fasano, M. Callieri, P. Cignoni, and R. Scopigno. 2003. Exploiting mirrors for laser stripe 3D scanning. In 3DIM 2003. 243–250.Google Scholar
    13. Paul Foster, Zhenghong Sun, Jong Jin Park, and Benjamin Kuipers. 2013. VisAGGE: Visible angle grid for glass environments. In CVPR 2013. 2213–2220.Google ScholarCross Ref
    14. C. Godard, P. Hedman, W. Li, and G. J. Brostow. 2015. Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled Environments. In 3rd International Conference on 3D Vision (3DV). 19–27. Google ScholarDigital Library
    15. Ivo Ihrke, Kiriakos N. Kutulakos, Hendrik P. A. Lensch, Marcus Magnor, and Wolfgang Heidrich. 2010. Transparent and Specular Object Reconstruction. Computer Graphics Forum 29, 8 (2010), 2400–2426.Google ScholarCross Ref
    16. B. Jacquet, C. Häne, K. Köser, and M. Pollefeys. 2013. Real-World Normal Map Capture for Nearly Flat Reflective Surfaces. In CVPR 2013. 713–720. Google ScholarDigital Library
    17. Jun Jiang, Renato Miyagusuku, Atsushi Yamashita, and Hajime Asama. 2017. Glass Confidence Maps Building Based on Neural Networks Using Laser Range-Finders for Mobile Robots. In IEEE/SICE International Symposium on System Integration.Google Scholar
    18. O. Kähler, V. Adrian Prisacariu, C. Yuheng Ren, X. Sun, P. Torr, and D. Murray. 2015. Very High Frame Rate Volumetric Integration of Depth Images on Mobile Devices. TVCG 21, 11 (Nov 2015), 1241–1250. Google ScholarDigital Library
    19. J. Kannala and S. S. Brandt. 2006. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. PAMI 28, 8 (Aug 2006), 1335–1340. Google ScholarDigital Library
    20. P.-F. Käshammer and A. Nüchter. 2015. Mirror identification and correction of 3D point clouds. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 40, 5 (2015), 109.Google Scholar
    21. U Klank, D. Carton, and M. Beetz. 2011. Transparent object detection and reconstruction on a mobile platform. In ICRA 2011. 5971–5978.Google Scholar
    22. Rainer Koch, Stefan May, Patrick Murmann, and Andreas Nüchter. 2017b. Identification of Transparent and Specular Reflective Material in Laser Scans to Discriminate Affected Measurements for Faultless Robotic SLAM. Journal of Robotics and Autonomous Systems (JRAS) 87 (2017), 296–312.Google ScholarCross Ref
    23. R. Koch, S. May, and A. Nüchter. 2017a. Effective distinction of transparent and specular reflective objects in point clouds of a multi-echo laser scanner. In ICAR 2017. 566–571.Google Scholar
    24. Brian Kulis and Michael I. Jordan. 2011. Revisiting k-means: New algorithms via Bayesian nonparametrics. arXiv preprint arXiv.1111.0352 (2011). Google ScholarDigital Library
    25. J. P. Lewis. 1995. Fast Normalized Cross-Correlation. In Vision Interface ’95. Canadian Image Processing and Pattern Recognition Society.Google Scholar
    26. M. Liu, R. Hartley, and M. Salzmann. 2015. Mirror Surface Reconstruction from a Single Image. PAMI 37, 4 (April 2015), 760–773.Google ScholarDigital Library
    27. William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. In SIGGRAPH. ACM, 163–169. Google ScholarDigital Library
    28. D. Miyazaki, M. Kagesawa, and K. Ikeuchi. 2004. Transparent surface modeling from a pair of polarization images. PAMI 26, 1 (2004), 73–82. Google ScholarDigital Library
    29. R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós. 2015. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics 31, 5 (Oct 2015), 1147–1163.Google ScholarDigital Library
    30. R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR 2011. 127–136. Google ScholarDigital Library
    31. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D Reconstruction at Scale Using Voxel Hashing. ACM Trans. Graph. 32, 6, Article 169 (Nov. 2013), 11 pages. Google ScholarDigital Library
    32. Edwin Olson. 2011. AprilTag: A robust and flexible visual fiducial system. In ICRA 2011. 3400–3407.Google ScholarCross Ref
    33. Rui Rodrigues, João P. Barreto, and Urbano Nunes. 2010. Camera Pose Estimation Using Images of Planar Mirror Reflections. In ECCV 2010. 382–395. Google ScholarDigital Library
    34. YiChang Shih, D. Krishnan, F. Durand, and W. T. Freeman. 2015. Reflection removal using ghosting cues. In CVPR 2015. 3193–3201.Google Scholar
    35. Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based Rendering for Scenes with Reflections. ACM Trans. Graph. 31, 4, Article 100 (July 2012), 10 pages. Google ScholarDigital Library
    36. Julian Straub, Trevor Campbell, Jonathan P How, and John W Fisher. 2015. Small-variance nonparametric clustering on the hypersphere. In CVPR 2015. 334–342.Google ScholarCross Ref
    37. Marco Tarini, Hendrik P.A. Lensch, Michael Goesele, and Hans-Peter Seidel. 2005. 3D acquisition of mirroring objects using striped patterns. Graphical Models 67, 4 (2005), 233 — 259. Google ScholarDigital Library
    38. Markus Unger, Thomas Pock, and Horst Bischof. 2008a. Interactive globally optimal image segmentation. Technical Report ICG-TR-08/02. Graz University of Technology.Google Scholar
    39. Markus Unger, Thomas Pock, Werner Trobin, Daniel Cremers, and Horst Bischof. 2008b. TVSeg – Interactive Total Variation Based Image Segmentation. In BMVC 2008.Google Scholar
    40. J. Wang and E. Olson. 2016. AprilTag 2: Efficient and robust fiducial detection. In IROS 2016. 4193–4198.Google Scholar
    41. Qiaosong Wang, Haiting Lin, Yi Ma, Sing Bing Kang, and Jingyi Yu. 2015. Automatic Layer Separation using Light Field Imaging. CoRR abs/1506.04721 (2015). arXiv:1506.04721 http://arxiv.org/abs/1506.04721Google Scholar
    42. Sven Wanner and Bastian Goldluecke. 2013. Reconstructing Reflective and Transparent Surfaces from Epipolar Plane Images. In GCPR 2013.Google ScholarCross Ref
    43. Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A computational approach for obstruction-free photography. ACM Trans. Graph. 34, 4 (2015), 79:1–79:11. Google ScholarDigital Library
    44. Shao-Wen Yang and Chieh-Chih Wang. 2008. Dealing with laser scanner failure: Mirrors and windows. In ICRA 2008. 3009–3015.Google Scholar
    45. S. W Yang and C. C. Wang. 2011. On Solving Mirror Reflection in LIDAR Sensing. IEEE/ASME Transactions on Mechatronics 16, 2 (April 2011), 255–265.Google ScholarCross Ref
    46. Y. Zhang, M. Ye, D. Manocha, and R. Yang. 2017. 3D Reconstruction in the Presence of Glass and Mirrors by Acoustic and Visual Fusion. PAMI (2017).Google Scholar
    47. Zhengyou Zhang. 2000. A Flexible New Technique for Camera Calibration. PAMI 22, 11 (Nov. 2000), 1330–1334. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: