“CurveFusion: reconstructing thin structures from RGBD sequences”
Conference:
Type(s):
Title:
- CurveFusion: reconstructing thin structures from RGBD sequences
Session/Category Title: Acquiring and editing, geometry via RGB (D) images
Presenter(s)/Author(s):
Moderator(s):
Abstract:
We introduce CurveFusion, the first approach for high quality scanning of thin structures at interactive rates using a handheld RGBD camera. Thin filament-like structures are mathematically just 1D curves embedded in R3, and integration-based reconstruction works best when depth sequences (from the thin structure parts) are fused using the object’s (unknown) curve skeleton. Thus, using the complementary but noisy color and depth channels, CurveFusion first automatically identifies point samples on potential thin structures and groups them into bundles, each being a group of a fixed number of aligned consecutive frames. Then, the algorithm extracts per-bundle skeleton curves using L1 axes, and aligns and iteratively merges the L1 segments from all the bundles to form the final complete curve skeleton. Thus, unlike previous methods, reconstruction happens via integration along a data-dependent fusion primitive, i.e., the extracted curve skeleton. We extensively evaluate CurveFusion on a range of challenging examples, different scanner and calibration settings, and present high fidelity thin structure reconstructions previously just not possible from raw RGBD sequences.
References:
1. Samir Aroudj, Patrick Seemann, Fabian Langguth, Stefan Guthe, and Michael Goesele. 2017. Visibility-Consistent Thin Surface Reconstruction Using Multi-Scale Kernels. ACM SIGGRAPH Asia 36, 6 (2017), 187:1–187:13. Google ScholarDigital Library
2. Paul J. Besl and Neil D. McKay. 1992. A Method for Registration of 3-D Shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2 (Feb. 1992), 239–256. Google ScholarDigital Library
3. Y-P Cao, T Ju, J Xu, and S-M Hu. 2017. Extracting Sharp Features from RGB-D Images. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 138–152.Google Scholar
4. Sungjoon Choi, Q. Y. Zhou, and V. Koltun. 2015. Robust reconstruction of indoor scenes. In IEEE CVPR. 5556–5565.Google Scholar
5. Brian Curless and Marc Levoy. 1996. A Volumetric Method for Building Complex Models from Range Images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96). ACM, New York, NY, USA, 303–312. Google ScholarDigital Library
6. Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration. ACM TOG 36, 3, Article 24 (May 2017), 18 pages. Google ScholarDigital Library
7. Charlotte Delmas, Marie-Odile Berger, Erwan Kerrien, Cyril Riddell, Yves Trousset, René Anxionnat, and Serge Bracard. 2015. Three-dimensional curvilinear device reconstruction from two fluoroscopic views. In SPIE, Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, Vol. 9415. San Diego, CA, France, 94150F.Google Scholar
8. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A Density-based Algorithm for Discovering Clusters a Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD’96). AAAI Press, 226–231. http://dl.acm.org/citation.cfm?id=3001460.3001507 Google ScholarDigital Library
9. Ricardo Fabbri and Benjamin Kimia. 2010. 3D curve sketch: Flexible curve-based stereo reconstruction and calibration. In IEEE CVPR. 1538–1545.Google Scholar
10. Huazhu Fu, Dong Xu, and Stephen Lin. 2017. Object-based multiple foreground segmentation in RGBD video. IEEE Transactions on Image Processing 26, 3 (2017), 1418–1427. Google ScholarDigital Library
11. Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. 2012. RGB-D Mapping: Using Kinect-style Depth Cameras for Dense 3D Modeling of Indoor Environments. Int. J. Rob. Res. 31, 5 (April 2012), 647–663. Google ScholarDigital Library
12. Manuel Hofer, Michael Maurer, and Horst Bischof. 2014. Improving Sparse 3D Models for Man-Made Environments Using Line-Based 3D Reconstruction. In International Conference on 3D Vision (3DV). Google ScholarDigital Library
13. Hui Huang, Shihao Wu, Daniel Cohen-Or, Minglun Gong, Hao Zhang, Guiqing Li, and Baoquan Chen. 2013. L1-medial Skeleton of Point Cloud. In ACM SIGGRAPH. ACM, New York, NY, USA, Article 65, 8 pages. Google ScholarDigital Library
14. Arjun Jain, Christian Kurz, Thorsten Thormählen, and Hans-Peter Seidel. 2010. Exploiting Global Connectivity Constraints for Reconstruction of 3D Line Segment from Images. In IEEE CVPR. San Francisco, CA.Google Scholar
15. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In Proceedings of the Fourth Eurographics Symposium on Geometry Processing (SGP ’06). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 61–70. http://dl.acm.org/citation.cfm?id=1281957.1281965 Google ScholarDigital Library
16. M. Keller, D. Lefloch, M. Lambers, S. Izadi, T. Weyrich, and A. Kolb. 2013. Real-Time 3D Reconstruction in Dynamic Scenes Using Point-Based Fusion. In 2013 International Conference on 3D Vision – 3DV 2013. 1–8. Google ScholarDigital Library
17. Johannes Kopf, Michael F. Cohen, Dani Lischinski, and Matt Uyttendaele. 2007. Joint Bilateral Upsampling. ACM Trans. Graph. 26, 3, Article 96 (July 2007). Google ScholarDigital Library
18. Guo Li, Ligang Liu, Hanlin Zheng, and Niloy J. Mitra. 2010. Analysis, Reconstruction and Manipulation using Arterial Snakes. In ACM SIGGRAPH Asia, Vol. 29. Article 152, 10 pages. Google ScholarDigital Library
19. Shiwei Li, Yao Yao, Tian Fang, and Long Quan. 2018. Reconstructing Thin Structures of Manifold Surfaces by Integrating Spatial Curves. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2887–2896.Google ScholarCross Ref
20. Lingjie Liu, Duygu Ceylan, Cheng Lin, Wenping Wang, and Niloy J. Mitra. 2017. Image-based Reconstruction of Wire Art. ACM SIGGRAPH 36, 4, Article 63 (July 2017), 11 pages. Google ScholarDigital Library
21. Tobias Martin, Juan Montes, Jean-Charles Bazin, and Tiberiu Popa. 2014. Topology-aware Reconstruction of Thin Tubular Structures. In SIGGRAPH Asia 2014 Technical Briefs (SA ’14). ACM, New York, NY, USA, Article 12, 4 pages. Google ScholarDigital Library
22. R. MurArtal and J. D. Tardós. 2017. ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Transactions on Robotics 33, 5 (Oct 2017), 1255–1262.Google Scholar
23. R. A. Newcombe, D. Fox, and S. M. Seitz. 2015. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In IEEE CVPR. 343–352.Google Scholar
24. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In IEEE ISMAR (ISMAR ’11). IEEE Computer Society, Washington, DC, USA, 127–136. Google ScholarDigital Library
25. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013. Real-time 3D Reconstruction at Scale Using Voxel Hashing. ACM SIGGRAPH Asia 32, 6, Article 169 (Nov. 2013), 11 pages. Google ScholarDigital Library
26. Irina Nurutdinova and Andrew Fitzgibbon. 2015. Towards Pointless Structure from Motion: 3D Reconstruction and Camera Parameters from General 3D Curves. In IEEE ICCV. IEEE Computer Society, Washington, DC, USA, 2363–2371. Google ScholarDigital Library
27. Xiao Pan, Yuanfeng Zhou, Feng Li, and Caiming Zhang. 2017. Superpixels of RGB-D images for indoor scenes based on weighted geodesic driven metric. IEEE Transactions on Visualization and Computer Graphics 23, 10 (2017), 2342–2356.Google ScholarDigital Library
28. Point Cloud Library PCL. 2018. Kinfu. https://github.com/PointCloudLibrary/pcl/tree/master/gpu/kinfu. (2018).Google Scholar
29. Oxford PTAM. 2018. PTAM-GPL. https://github.com/Oxford-PTAM/PTAM-GPL. (2018).Google Scholar
30. Dushyant Rao, Soon-Jo Chung, and Seth Hutchinson. 2012. CurveSLAM: An approach for vision-based navigation without point features. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 4198–4204.Google ScholarCross Ref
31. Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. 2012. Coherent Spatiotemporal Filtering, Upsampling and Rendering of RGBZ Videos. CGF (Proc. of EUROGRAPHICS) 31, 2 (May 2012). Google ScholarDigital Library
32. Szymon Rusinkiewicz, Olaf Hall-Holt, and Marc Levoy. 2002. Real-time 3D Model Acquisition. ACM SIGGRAPH 21, 3 (July 2002), 438–446. Google ScholarDigital Library
33. Nikolay Savinov, Christian Hane, Lubor Ladicky, and Marc Pollefeys. 2016. Semantic 3D Reconstruction With Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint. In IEEE CVPR.Google Scholar
34. A. Tabb. 2013. Shape from Silhouette Probability Maps: Reconstruction of Thin Objects in the Presence of Silhouette Extraction and Calibration Error. In 2013 IEEE Conference on Computer Vision and Pattern Recognition. 161–168. Google ScholarDigital Library
35. Alex Teichman, Stephen Miller, and Sebastian Thrun. 2013. Unsupervised Intrinsic Calibration of Depth Sensors via SLAM.. In Robotics: Science and Systems, Vol. 248. 3.Google Scholar
36. B. Ummenhofer and T. Brox. 2013. Point-Based 3D Reconstruction of Thin Objects. In IEEE ICCV. 969–976. Google ScholarDigital Library
37. Anil Usumezbas, Ricardo Fabbri, and Benjamin B. Kimia. 2016. From Multi-view Image Curves to 3D Drawings. In ECCV. 70–87.Google Scholar
38. T. Weise, T. Wismer, B. Leibe, and L. Van Gool. 2009. In-hand scanning with online loop closure. In IEEE ICCV Workshops. 1630–1637.Google Scholar
39. Changchang Wu. 2011. VisualSFM: A Visual Structure from Motion System. (2011). http://ccwu.me/vsfm/Google Scholar
40. Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, and Christian Theobalt. 2014. Real-time Shading-based Refinement for Consumer Depth Cameras. ACM Trans. Graph. 33, 6, Article 200 (Nov. 2014), 10 pages. Google ScholarDigital Library
41. Yi Jun Xiao and Youfu Li. 2005. Optimized stereo reconstruction of free-form space curves based on a nonuniform rational B-spline model. J. Opt. Soc. Am. A 22, 9 (Sep 2005), 1746–1762.Google ScholarCross Ref
42. Kangxue Yin, Hui Huang, Hao Zhang, Minglun Gong, Daniel Cohen-Or, and Baoquan Chen. 2014. Morfit: Interactive Surface Reconstruction from Incomplete Point Clouds with Curve-driven Topology and Geometry Control. In ACM SIGGRAPH Asia. ACM, New York, NY, USA, Article 202, 12 pages. Google ScholarDigital Library
43. K. Yücer, C. Kim, A. Sorkine-Hornung, and O. Sorkine-Hornung. 2016a. Depth from Gradients in Dense Light Fields for Object Reconstruction. In 2016 Fourth International Conference on 3D Vision (3DV). 249–257.Google Scholar
44. Kaan Yücer, Alexander Sorkine-Hornung, Oliver Wang, and Olga Sorkine-Hornung. 2016b. Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction. ACM TOG 35, 3, Article 22 (March 2016), 15 pages. Google ScholarDigital Library
45. Ming Zeng, Fukai Zhao, Jiaxiang Zheng, and Xinguo Liu. 2013. Octree-based Fusion for Realtime 3D Reconstruction. Graph. Models 75, 3 (May 2013), 126–136. Google ScholarDigital Library
46. Qian-Yi Zhou and Vladlen Koltun. 2013. Dense Scene Reconstruction with Points of Interest. ACM SIGGRAPH 32, 4, Article 112 (July 2013), 8 pages. Google ScholarDigital Library
47. Q. Y. Zhou, S. Miller, and V. Koltun. 2013. Elastic Fragments for Dense Scene Reconstruction. In IEEE ICCV. 473–480. Google ScholarDigital Library
48. Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2015. Shading-based Refinement on Volumetric Signed Distance Functions. ACM Trans. Graph (Proc. SIGGRAPH) 34, 4, Article 96 (July 2015), 14 pages. Google ScholarDigital Library
