“Patch-based optimization for image-based texture mapping”

Image-based texture mapping is a common way of producing texture maps for geometric models of real-world objects. Although a high-quality texture map can be easily computed for accurate geometry and calibrated cameras, the quality of texture map degrades significantly in the presence of inaccuracies. In this paper, we address this problem by proposing a novel global patch-based optimization system to synthesize the aligned images. Specifically, we use patch-based synthesis to reconstruct a set of photometrically-consistent aligned images by drawing information from the source images. Our optimization system is simple, flexible, and more suitable for correcting large misalignments than other techniques such as local warping. To solve the optimization, we propose a two-step approach which involves patch search and vote, and reconstruction. Experimental results show that our approach can produce high-quality texture maps better than existing techniques for objects scanned by consumer depth cameras such as Intel RealSense. Moreover, we demonstrate that our system can be used for texture editing tasks such as hole-filling and reshuffling as well as multiview camouflage.

1. Ehsan Aganj, Pascal Monasse, and Renaud Keriven. 2010. Multi-view Texturing of Imprecise Mesh. In ACCV. 468–476. Google ScholarDigital Library
2. S. H. Baek, I. Choi, and M. H. Kim. 2016. Multiview Image Completion with Space Structure Propagation. In CVPR. 488–496. Google ScholarCross Ref
3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: a randomized correspondence algorithm for structural image editing. ACM TOG 28, Article 24 (2009), 11 pages. Issue 3.Google Scholar
4. Pierre Bénard, Forrester Cole, Michael Kass, Igor Mordatch, James Hegarty, Martin Sebastian Senn, Kurt Fleischer, Davide Pesare, and Katherine Breeden. 2013. Stylizing Animation by Example. ACM TOG 32, 4, Article 119 (2013), 12 pages.Google Scholar
5. Fausto Bernardini, Ioana M. Martin, and Holly Rushmeier. 2001. High-Quality Texture Reconstruction from Multiple Scans. IEEE TVCG 7, 4 (2001), 318–332. Google ScholarDigital Library
6. Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured Lumigraph Rendering. In SIGGRAPH. 425–432. Google ScholarDigital Library
7. M. Corsini, M. Dellepiane, F. Ganovelli, R. Gherardi, A. Fusiello, and R. Scopigno. 2013. Fully Automatic Registration of Image Sets on Approximate Geometry. IJCV 102, 1–3 (2013), 91–111.Google ScholarDigital Library
8. Massimiliano Corsini, Matteo Dellepiane, Federico Ponchio, and Roberto Scopigno. 2009. Image-to-Geometry Registration: a Mutual Information Method exploiting Illumination-related Geometric Properties. CGF 28, 7 (2009), 1755–1764. Google ScholarCross Ref
9. Frederique Crete-Roffet, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. 2007. The Blur Effect: Perception and Estimation with a New No-Reference Perceptual Blur Metric. In HVIE.Google Scholar
10. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-based Approach. In SIGGRAPH. 11–20.Google Scholar
11. M. Dellepiane, R. Marroquim, M. Callieri, P. Cignoni, and R. Scopigno. 2012. Flow-Based Local Optimization for Image-to-Geometry Projection. IEEE TVCG 18, 3 (2012), 463–474. Google ScholarDigital Library
12. M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. 2008. Floating Textures. CGF 27, 2 (2008), 409–418. Google ScholarCross Ref
13. Thomas Franken, Matteo Dellepiane, Fabio Ganovelli, Paolo Cignoni, Claudio Montani, and Roberto Scopigno. 2005. Minimizing user intervention in registering 2D images to 3D models. The Visual Computer 21, 8 (2005), 619–628. Google ScholarCross Ref
14. Ran Gal, Yonatan Wexler, Eyal Ofek, Hugues Hoppe, and Daniel Cohen-Or. 2010. Seamless Montage for Texturing Models. CGF 29, 2 (2010), 479–486. Google ScholarCross Ref
15. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable Inside-Out Image-Based Rendering. ACM TOG 35, 6 (2016), 231:1–231:11.Google Scholar
16. Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. UIST (2011), 559–568.Google Scholar
17. Ondřej Jamriška, Jakub Fišer, Paul Asente, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2015. LazyFluids: Appearance Transfer for Fluid Animations. ACM TOG 34, 4, Article 92 (2015), 10 pages.Google Scholar
18. Nima Khademi Kalantari, Eli Shechtman, Connelly Barnes, Soheil Darabi, Dan B Goldman, and Pradeep Sen. 2013. Patch-based High Dynamic Range Video. ACM TOG 32, 6 (2013).Google Scholar
19. V. Lempitsky and D. Ivanov. 2007. Seamless Mosaicing of Image-Based Texture Maps. In CVPR. 1–6. Google ScholarCross Ref
20. Hendrik P.A. Lensch, Wolfgang Heidrich, and Hans-Peter Seidel. 2001. A Silhouette-Based Algorithm for Texture Registration and Stitching. Graphical Models 63, 4 (2001), 245 — 262. Google ScholarDigital Library
21. Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In ISMAR. 127–136.Google Scholar
22. Eyal Ofek, Erez Shilat, Ari Rappoport, and Michael Werman. 1997. Multiresolution Textures from Image Sequences. IEEE Computer Graphics and Applications 17, 2 (1997), 18–29. Google ScholarDigital Library
23. A. Owens, C. Barnes, A. Flint, H. Singh, and W. Freeman. 2014. Camouflaging an Object from Many Viewpoints. In CVPR. 2782–2789. Google ScholarDigital Library
24. Frédéric Pighin, Jamie Hecker, Dani Lischinski, Richard Szeliski, and David H. Salesin. 1998. Synthesizing Realistic Facial Expressions from Photographs. In SIGGRAPH. 75–84. Google ScholarDigital Library
25. Kari Pulli and Linda G. Shapiro. 2000. Surface Reconstruction and Display from Range and Color Data. Graphical Models 62, 3 (2000), 165 — 201. Google ScholarDigital Library
26. Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. In CVPR. 519–528. Google ScholarDigital Library
27. Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B Goldman, and Eli Shechtman. 2012. Robust Patch-Based HDR Reconstruction of Dynamic Scenes. ACM TOG 31, 6 (2012).Google Scholar
28. Eli Shechtman, Alex Rav-Acha, Michal Irani, and Steve Seitz. 2010. Regenerative Morphing. In CVPR. 615–622. Google ScholarCross Ref
29. D. Simakov, Y. Caspi, E. Shechtman, and M. Irani. 2008. Summarizing visual data using bidirectional similarity. In CVPR. 1–8. Google ScholarCross Ref
30. Sudipta N. Sinha, Drew Steedly, Richard Szeliski, Maneesh Agrawala, and Marc Pollefeys. 2008. Interactive 3D Architectural Modeling from Unordered Photo Collections. ACM TOG 27, 5, Article 159 (2008), 10 pages.Google Scholar
31. Ioannis Stamos and Peter K. Allen. 2002. Geometry and Texture Recovery of Scenes of Large Scale. Computer Vision and Image Understanding 88, 2 (2002), 94–118. Google ScholarDigital Library
32. T. Thonat, E. Shechtman, S. Paris, and G. Drettakis. 2016. Multi-View Inpainting for Image-Based Scene Editing and Rendering. In IEEE 3DV. 351–359. Google ScholarCross Ref
33. Yochay Tzur and Ayellet Tal. 2009. FlexiStickers: Photogrammetric Texture Mapping Using Casual Images. ACM TOG 28, 3, Article 45 (2009), 10 pages.Google Scholar
34. Luiz Velho and Jonas Sossai Jr. 2007. Projective Texture Atlas Construction for 3D Photography. The Visual Computer 23, 9 (2007), 621–629. Google ScholarDigital Library
35. Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let there be color! Large-scale texturing of 3D reconstructions. In ECCV. Springer, 836–850. Google ScholarCross Ref
36. Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-Time Completion of Video. IEEE PAMI 29, 3 (2007), 463–476. Google ScholarDigital Library
37. Qian-Yi Zhou and Vladlen Koltun. 2014. Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras. ACM TOG 33, 4, Article 155 (2014), 10 pages.Google Scholar