“Vid2Curve: simultaneous camera motion estimation and thin structure reconstruction from an RGB video” by Wang, Liu, Chen, Chu, Theobalt, et al. …

  • ©Peng Wang, Lingjie Liu, Nenglun Chen, Hung-Kuo Chu, Christian Theobalt, and Wenping Wang

Conference:


Type:


Title:

    Vid2Curve: simultaneous camera motion estimation and thin structure reconstruction from an RGB video

Session/Category Title: Motion and Matching

Presenter(s)/Author(s):


Abstract:


    Thin structures, such as wire-frame sculptures, fences, cables, power lines, and tree branches, are common in the real world. It is extremely challenging to acquire their 3D digital models using traditional image-based or depth-based reconstruction methods, because thin structures often lack distinct point features and have severe self-occlusion. We propose the first approach that simultaneously estimates camera motion and reconstructs the geometry of complex 3D thin structures in high quality from a color video captured by a handheld camera. Specifically, we present a new curve-based approach to estimate accurate camera poses by establishing correspondences between featureless thin objects in the foreground in consecutive video frames, without requiring visual texture in the background scene to lock on. Enabled by this effective curve-based camera pose estimation strategy, we develop an iterative optimization method with tailored measures on geometry, topology as well as self-occlusion handling for reconstructing 3D thin structures. Extensive validations on a variety of thin structures show that our method achieves accurate camera pose estimation and faithful reconstruction of 3D thin structures with complex shape and topology at a level that has not been attained by other existing reconstruction methods.

References:


    1. Samir Aroudj, Patrick Seemann, Fabian Langguth, Stefan Guthe, and Michael Goesele. 2017. Visibility-Consistent Thin Surface Reconstruction Using Multi-Scale Kernels. ACM SIGGRAPH Asia 36, 6 (2017), 187:1–187:13.Google Scholar
    2. Xue Bai, Jue Wang, David Simons, and Guillermo Sapiro. 2009. Video SnapCut: Robust Video Object Cutout Using Localized Classifiers. ACM SIGGRAPH 28, 3, Article 70 (2009), 11 pages.Google Scholar
    3. M. Berger, A. Tagliasacchi, L. Seversky, P. Alliez, J. Levine, A. Sharf, and C. Silva. 2014. State of the Art in Surface Reconstruction from Point Clouds. In CGF Eurographics – State of the Art Reports. Google ScholarCross Ref
    4. Rikard Berthilsson, Kalle Astrom, and Anders Heyden. 2001. Reconstruction of General Curves, Using Factorization and Bundle Adjustment. IJCV 41, 3 (2001), 171–182.Google ScholarDigital Library
    5. Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-Sweep: Extracting Editable Objects from a Single Photo. ACM SIGGRAPH Asia, Article 195 (2013), 10 pages.Google Scholar
    6. Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. 2017. BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration. ACM TOG 36, 3, Article 24 (2017), 18 pages.Google Scholar
    7. R. Fabbri and B. Kimia. 2010. 3D Curve Sketch: Flexible Curve-based Stereo Reconstruction and Calibration. In IEEE CVPR. 1538–1545.Google Scholar
    8. Ricardo Fabbri and Benjamin B. Kimia. 2016. Multiview Differential Geometry of Curves. IJCV 120, 3 (2016), 324–346.Google ScholarDigital Library
    9. Xinyi Fan, Linguang Zhang, Benedict Brown, and Szymon Rusinkiewicz. 2016. Automated View and Path Planning for Scalable Multi-object 3D Scanning. ACM SIGGRAPH Asia, Article 239 (2016), 13 pages.Google Scholar
    10. Yasutaka Furukawa and Jean Ponce. 2010. Accurate, Dense, and Robust Multiview Stereopsis. IEEE PAMI 32, 8 (2010), 1362–1376.Google ScholarDigital Library
    11. M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. M. Seitz. 2007. Multi-View Stereo for Community Photo Collections. In IEEE ICCV. 1–8.Google Scholar
    12. Manuel Hofer, Michael Maurer, and Horst Bischof. 2014. Improving Sparse 3D Models for Man-Made Environments Using Line-Based 3D Reconstruction. In International Conference on 3D Vision (3DV).Google ScholarDigital Library
    13. Wei Hong, Yi Ma, and Yizhou Yu. 2004. Reconstruction of 3-D Symmetric Curves from Perspective Images without Discrete Features. In ECCV. 533–545.Google Scholar
    14. Kai-Wen Hsiao, Jia-Bin Huang, and Hung-Kuo Chu. 2018. Multi-view Wire Art. ACM SIGGRAPH Asia 37, 6, Article 242 (2018), 11 pages.Google Scholar
    15. Hui Huang, Shihao Wu, Daniel Cohen-Or, Minglun Gong, Hao Zhang, Guiqing Li, and Baoquan Chen. 2013. L1-medial Skeleton of Point Cloud. ACM SIGGRAPH, Article 65 (2013), 8 pages.Google Scholar
    16. Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. 2018. DeepMVS: Learning Multi-view Stereopsis. In IEEE CVPR.Google Scholar
    17. Arjun Jain, Christian Kurz, Thorsten Thormählen, and Hans-Peter Seidel. 2010. Exploiting Global Connectivity Constraints for Reconstruction of 3D Line Segment from Images. In IEEE CVPR.Google Scholar
    18. Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. In SGP. 61–70.Google Scholar
    19. Vladislav Kraevoy, Alla Sheffer, and Michiel van de Panne. 2009. Modeling from Contour Drawings. In CGF Eurographics. ACM, 37–44.Google Scholar
    20. Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. 2016. Fast Optical Flow Using Dense Inverse Search. In ECCV. Springer, 471–488.Google Scholar
    21. Joseph B Kruskal. 1956. On the Shortest Spanning Subtree of a Graph and the Traveling Salesman Problem. Proceedings of the American Mathematical society 7, 1 (1956), 48–50.Google ScholarCross Ref
    22. Andreas Kuhn, Heiko Hirschmüller, Daniel Scharstein, and Helmut Mayer. 2017. A TV Prior for High-quality Scalable Multi-view Stereo Reconstruction. IJCV 124, 1 (2017), 2–17.Google ScholarDigital Library
    23. Guo Li, Ligang Liu, Hanlin Zheng, and Niloy J. Mitra. 2010. Analysis, Reconstruction and Manipulation using Arterial Snakes. ACM SIGGRAPH Asia 29, 6, Article 152 (2010), 10 pages.Google Scholar
    24. Shiwei Li, Yao Yao, Tian Fang, and Long Quan. 2018. Reconstructing Thin Structures of Manifold Surfaces by Integrating Spatial Curves. In IEEE CVPR. 2887–2896.Google Scholar
    25. Lingjie Liu, Duygu Ceylan, Cheng Lin, Wenping Wang, and Niloy J. Mitra. 2017. Image-based Reconstruction of Wire Art. ACM SIGGRAPH 36, 4, Article 63 (2017), 11 pages.Google Scholar
    26. Lingjie Liu, Nenglun Chen, Duygu Ceylan, Christian Theobalt, Wenping Wang, and Niloy J. Mitra. 2018. CurveFusion: Reconstructing Thin Structures from RGBD Sequences. ACM SIGGRAPH Asia 37, 6 (2018).Google Scholar
    27. Tobias Martin, Juan Montes, Jean-Charles Bazin, and Tiberiu Popa. 2014. Topology-aware Reconstruction of Thin Tubular Structures. In SIGGRAPH Asia Technical Briefs. Article 12, 4 pages.Google Scholar
    28. Ravish Mehra, Qingnan Zhou, Jeremy Long, Alla Sheffer, Amy Gooch, and Niloy J Mitra. 2009. Abstraction of Man-made Shapes. In ACM SIGGRAPH Asia 2009 papers. 1–10.Google Scholar
    29. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In IEEE ISMAR. 127–136.Google Scholar
    30. Irina Nurutdinova and Andrew Fitzgibbon. 2015. Towards Pointless Structure from Motion: 3D Reconstruction and Camera Parameters from General 3D Curves. In IEEE ICCV. 2363–2371.Google Scholar
    31. Helmut Pottmann, Stefan Leopoldseder, and Michael Hofer. 2002. Approximation with Active B-spline Curves and Surfaces. In 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings. IEEE, 8–25.Google Scholar
    32. Dushyant Rao, Soon-Jo Chung, and Seth Hutchinson. 2012. CurveSLAM: an Approach for Vision-based Navigation Without Point Features. In International Conference on Intelligent Robots and Systems. 4198–4204.Google ScholarCross Ref
    33. Nikolay Savinov, Christian Hane, Lubor Ladicky, and Marc Pollefeys. 2016. Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint. In IEEE CVPR.Google Scholar
    34. Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In IEEE CVPR.Google Scholar
    35. Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In ECCV.Google Scholar
    36. Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. In ACM SIGGRAPH. 835–846.Google ScholarDigital Library
    37. Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. 2012. A Benchmark for the Evaluation of RGB-D SLAM Systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 573–580.Google ScholarCross Ref
    38. A. Tabb. 2013. Shape from Silhouette Probability Maps: Reconstruction of Thin Objects in the Presence of Silhouette Extraction and Calibration Error. In IEEE CVPR. 161–168. Google ScholarDigital Library
    39. A. Tabb and H. Medeiros. 2017. A Robotic Vision System to Measure Tree Traits. In International Conference on Intelligent Robots and Systems (IROS). 6005–6012.Google Scholar
    40. Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. 1999. Bundle Adjustment—a Modern Synthesis. In International Workshop on Vision Algorithms. Springer, 298–372.Google Scholar
    41. B. Ummenhofer and T. Brox. 2013. Point-Based 3D Reconstruction of Thin Objects. In IEEE ICCV. 969–976. Google ScholarDigital Library
    42. Anil Usumezbas, Ricardo Fabbri, and Benjamin B. Kimia. 2016. From Multiview Image Curves to 3D Drawings. In ECCV. 70–87.Google Scholar
    43. Oliver Whyte, Josef Sivic, Andrew Zisserman, and Jean Ponce. 2012. Non-uniform Deblurring for Shaken Images. IJCV 98, 2 (2012), 168–186.Google ScholarDigital Library
    44. Shihao Wu, Wei Sun, Pinxin Long, Hui Huang, Daniel Cohen-Or, Minglun Gong, Oliver Deussen, and Baoquan Chen. 2014. Quality-driven Poisson-guided Autoscanning. ACM SIGGRAPH Asia, Article 203 (2014), 12 pages.Google Scholar
    45. Yi Jun Xiao and Youfu Li. 2005. Optimized Stereo Reconstruction of Free-form Space Curves Based on a Nonuniform Rational B-spline Model. J. Opt. Soc. Am. A 22, 9 (2005), 1746–1762.Google ScholarCross Ref
    46. Baoxuan Xu, William Chang, Alla Sheffer, Adrien Bousseau, James McCrae, and Karan Singh. 2014. True2Form: 3D Curve Networks from 2D Sketches via Selective Regularization. (2014).Google Scholar
    47. Feilong Yan, Andrei Sharf, Wenzhen Lin, Hui Huang, and Baoquan Chen. 2014. Proactive 3D Scanning of Inaccessible Parts. ACM SIGGRAPH Asia, Article 157 (2014), 8 pages.Google Scholar
    48. Kangxue Yin, Hui Huang, Hao Zhang, Minglun Gong, Daniel Cohen-Or, and Baoquan Chen. 2014. Morfit: Interactive Surface Reconstruction from Incomplete Point Clouds with Curve-driven Topology and Geometry Control. ACM SIGGRAPH Asia, Article 202 (2014), 12 pages.Google ScholarDigital Library
    49. K. Yücer, C. Kim, A. Sorkine-Hornung, and O. Sorkine-Hornung. 2016a. Depth from Gradients in Dense Light Fields for Object Reconstruction. In International Conference on 3D Vision (3DV). 249–257.Google Scholar
    50. Kaan Yücer, Alexander Sorkine-Hornung, Oliver Wang, and Olga Sorkine-Hornung. 2016b. Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction. ACM TOG 35, 3, Article 22 (2016), 15 pages.Google Scholar

ACM Digital Library Publication:


Overview Page: