“Appearance Modeling via Proxy-to-Image Alignment” by Huang, Xie, Ma, Lischinski, Gong, et al. …

  • ©Hui Huang, Ke Xie, Lin Ma, Daniel (Dani) Lischinski, Minglun Gong, Xin Tong, and Daniel Cohen-Or

Conference:


Type:


Title:

    Appearance Modeling via Proxy-to-Image Alignment

Session/Category Title: Layers, Glints and Surface Microstructure

Presenter(s)/Author(s):


Abstract:


    Endowing 3D objects with realistic surface appearance is a challenging and time-demanding task, as real-world surfaces typically exhibit a plethora of spatially variant geometric and photometric detail. Not surprisingly, computer artists commonly use images of real-world objects as an inspiration and a reference for their digital creations. However, despite two decades of research on image-based modeling, there are still no tools available for automatically extracting the detailed appearance (microgeometry and texture) of a 3D surface from a single image. In this article, we present a novel user-assisted approach for quickly and easily extracting a nonparametric appearance model from a single photograph of a reference object.

    The extraction process requires a user-provided proxy, whose geometry roughly approximates that of the object in the image. Since the proxy is just a rough approximation, it is necessary to align and deform it so as to match the reference object. The main contribution of this work is a novel technique to perform such an alignment, which enables accurate joint recovery of geometric detail and reflectance. The correlations between the recovered geometry at various scales and the spatially varying reflectance constitute a nonparametric appearance model. Once extracted, the appearance model may then be applied to various 3D shapes, whose large-scale geometry may differ considerably from that of the original reference object. Thus, our approach makes it possible to construct an appearance library, allowing users to easily enrich detail-less 3D shapes with realistic geometric detail and surface texture.

References:


    1. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3, 24:1–24:11. Google ScholarDigital Library
    2. Jonathan T. Barron and Jitendra Malik. 2015. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 8, 1670–1687.Google ScholarDigital Library
    3. Nicolas Bonneel, Michiel Van de Panne, Sylvain Lefebvre, and George Drettakis. 2010. Proxy-guided texture synthesis for rendering natural scenes. In Proceedings of the Conference on Vision, Modeling, and Visualization.Google Scholar
    4. Adrien Bousseau, Sylvain Paris, and Frédo Durand. 2009. User-assisted intrinsic images. ACM Transactions on Graphics 28, 5, 130:1–130:10. Google ScholarDigital Library
    5. Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics 32, 6, 195:1–195:10. Google ScholarDigital Library
    6. Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B. Goldman, and Pradeep Sen. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics 31, 4, 82–91. Google ScholarDigital Library
    7. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 1996 SIGGRAPH Conference. 11–20. Google ScholarDigital Library
    8. Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015. Synthesis of complex image appearance from limited exemplars. ACM Transactions on Graphics 34, 2, 22:1–22:14. Google ScholarDigital Library
    9. J.-M. Dischler, K. Maritaud, and D. Ghazanfarpour. 2002. Coherent bump map recovery from a single texture image. In Proceedings of the Canadian Conference on Graphics Interface. 201–208.Google Scholar
    10. Piotr Dollár and C. Lawrence Zitnick. 2013. Structured forests for fast edge detection. In Proceedings of the International Conference on Computer Vision. 1841–1848. Google ScholarDigital Library
    11. Yue Dong, Xin Tong, Fabio Pellacini, and Baining Guo. 2011. AppGen: Interactive material modeling from a single image. ACM Transactions on Graphics 30, 6, 146:1–146:10. Google ScholarDigital Library
    12. Hui Fang and John C. Hart. 2004. Textureshop: Texture synthesis as a photograph editing tool. ACM Transactions on Graphics 23, 3, 354–359. Google ScholarDigital Library
    13. Berthold Horn. 1986. Robot Vision. MIT Press, Cambridge, NY. Google ScholarDigital Library
    14. Qixing Huang, Hai Wang, and Vladlen Koltun. 2015. Single-view reconstruction via joint analysis of image and shape collections. ACM Transactions on Graphics 34, 4, 87:1–87:10. Google ScholarDigital Library
    15. Thouis R. Jones, Frédo Durand, and Mathieu Desbrun. 2003. Non-iterative, feature-preserving mesh smoothing. ACM Transactions on Graphics 22, 3, 943–949. Google ScholarDigital Library
    16. Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015. Self tuning texture optimization. Computer Graphics Forum 34, 2, 349–359. Google ScholarDigital Library
    17. Erum Arif Khan, Erik Reinhard, Roland W. Fleming, and Heinrich H. Bülthoff. 2006. Image-based material editing. ACM Transactions on Graphics 25, 3, 654–663. Google ScholarDigital Library
    18. Natasha Kholgade, Tomas Simon, Alexei Efros, and Yaser Sheikh. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Transactions on Graphics 33, 4, 127:1–127:12. Google ScholarDigital Library
    19. Vladislav Kraevoy, Alla Sheffer, and Michiel van de Panne. 2009. Modeling from contour drawings. In Proceedings of the Eurographics Symposium on Sketch-Based Interfaces and Modeling. 37–44. Google ScholarDigital Library
    20. Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005. Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3, 795–802. Google ScholarDigital Library
    21. Tom Mertens, Jan Kautz, Jiawen Chen, Philippe Bekaert, and Frédo Durand. 2006. Texture transfer using geometry correlation. In Proceedings of the Eurographics Symposium on Rendering, Vol. 273. 273–284. Google ScholarDigital Library
    22. Mark Meyer, Mathieu Desbrun, Peter Schröder, and Alan H. Barr. 2003. Discrete differential-geometry operators for triangulated 2-manifolds. In Visualization and Mathematics III. Springer, 35–57.Google Scholar
    23. Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2nd ed.). Springer-Verlag, New York, NY.Google Scholar
    24. Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. 2001. Image-based modeling and photo editing. In Proceedings of the 2001 SIGGRAPH Conference. 433–442. Google ScholarDigital Library
    25. Manuel M. Oliveira. 2002. Image-based modeling and rendering techniques: A survey. RITA—Revista de Informática Teórica e Aplicada 9, 2, 37–66.Google Scholar
    26. Lawrence R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77, 257–286.Google ScholarCross Ref
    27. Konstantinos Rematas, Chuong Nguyen, Mario Fritz, and Tinne Tuytelaars. 2017. Novel views of objects from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 8, 1576–1590.Google ScholarDigital Library
    28. Olga Sorkine and Marc Alexa. 2007. As-rigid-as-possible surface modeling. Computer Graphics Forum 4, 109–116. Google ScholarDigital Library
    29. Hao Su, Qixing Huang, Niloy J. Mitra, Yangyan Li, and Leonidas Guibas. 2014. Estimating image depth using shape collections. ACM Transactions on Graphics 33, 4, 37:1–37:11. Google ScholarDigital Library
    30. Tuanfeng Y. Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas Guibas, and Niloy J. Mitra. 2016. Unsupervised texture transfer from images to model collections. ACM Trans.actions on Graphics 35, 6, 177:1–177:13. Google ScholarDigital Library
    31. Xi Wang, Lifeng Wang, Ligang Liu, Shimin Hu, and Baining Guo. 2003. Interactive modeling of tree bark. In Proceedings of the Pacific Conference on Computer Graphics and Applications.83–90. Google ScholarDigital Library
    32. Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009. State of the art in example-based texture synthesis. In Eurographics State of the Art (EG-STAR) Reports. 93–117.Google Scholar
    33. Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, and Christian Theobalt. 2014. Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics 33, 6, 200:1–200:10. Google ScholarDigital Library
    34. Hongzhi Wu, Zhaotian Wang, and Kun Zhou. 2016. Simultaneous localization and appearance estimation with a consumer RGB-D camera. IEEE Transactions on Visualization and Computer Graphics 22, 8, 2012–2023. Google ScholarDigital Library
    35. Kai Xu, Hao Zhang, Andrea Tagliasacchi, Ligang Liu, Guo Li, Min Meng, and Yueshan Xiong. 2009. Partial intrinsic reflectional symmetry of 3D shapes. ACM Transactions on Graphics 28, 5, 138:1–138:10. Google ScholarDigital Library
    36. Kai Xu, Hanlin Zheng, Hao Zhang, Daniel Cohen-Or, Ligang Liu, and Yueshan Xiong. 2011. Photo-inspired model-driven 3D object modeling. ACM Transactions on Graphics 30, 4, 80:1–80:10. Google ScholarDigital Library
    37. Su Xue, Jiaping Wang, Xin Tong, Qionghai Dai, and Baining Guo. 2008. Image-based material weathering. Computer Graphics Forum 27, 2, 617–626.Google ScholarCross Ref
    38. Lap-Fai Yu, Sai-Kit Yeung, Yu-Wing Tai, and Stephen Lin. 2013. Shading-based shape refinement of RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1415–1422. Google ScholarDigital Library
    39. Steve Zelinka, Hui Fang, Michael Garland, and John C. Hart. 2005. Interactive material replacement in photographs. In Proceedings of the Canadian Conference on Graphics Interface. 227–232. Google ScholarDigital Library
    40. Youyi Zheng, Xiang Chen, Ming-Ming Cheng, Kun Zhou, Shi-Min Hu, and Niloy J. Mitra. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM Transactions on Graphics 31, 4, 99:1–99:11. Google ScholarDigital Library
    41. Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics 33, 4, 155:1–155:10. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: