“Unsupervised texture transfer from images to model collections”
Conference:
Type(s):
Title:
- Unsupervised texture transfer from images to model collections
Session/Category Title: All About Seeing
Presenter(s)/Author(s):
Abstract:
Large 3D model repositories of common objects are now ubiquitous and are increasingly being used in computer graphics and computer vision for both analysis and synthesis tasks. However, images of objects in the real world have a richness of appearance that these repositories do not capture, largely because most existing 3D models are untextured. In this work we develop an automated pipeline capable of transporting texture information from images of real objects to 3D models of similar objects. This is a challenging problem, as an object’s texture as seen in a photograph is distorted by many factors, including pose, geometry, and illumination. These geometric and photometric distortions must be undone in order to transfer the pure underlying texture to a new object — the 3D model. Instead of using problematic dense correspondences, we factorize the problem into the reconstruction of a set of base textures (materials) and an illumination model for the object in the image. By exploiting the geometry of the similar 3D model, we reconstruct certain reliable texture regions and correct for the illumination, from which a full texture map can be recovered and applied to the model. Our method allows for large-scale unsupervised production of richly textured 3D models directly from image data, providing high quality virtual objects for 3D scene design or photo editing applications, as well as a wealth of data for training machine learning algorithms for various inference tasks in graphics and vision.
References:
1. Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., and Sivic, J. 2014. Seeing 3D chairs: Exemplar part-based 2D–3D alignment using a large dataset of CAD models. In CVPR.
2. Averkiou, M., Kim, V. G., and Mitra, N. J. 2016. Autocorrelation descriptor for efficient co-alignment of 3d shape collections. Computer Graphics Forum 35, 1, 261–271.
3. Barron, J. T., and Malik, J. 2015. Shape, illumination, and reflectance from shading. TPAMI.
4. Bell, S., Bala, K., and Snavely, N. 2014. Intrinsic images in the wild. ACM Trans. on Graphics (SIGGRAPH) 33, 4.
5. Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012.
6. Choy, C. B., Xu, D., Gwak, J., Chen, K., and Savarese, S. 2016. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. arXiv preprint arXiv:1604.00449.
7. Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., and Sen, P. 2012. Image Melding: Combining inconsistent images using patch-based synthesis. (Proc. SIGGRAPH) 31, 4, 82:1–82:10.
8. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In CVPR.
9. Dosovitskiy, A., Tobias Springenberg, J., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In Proceedings of the IEEE CVPR, 1538–1546.
10. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proc. ACM CGI, ACM, 341–346.
11. Fish*, N., Averkiou*, M., van Kaick, O., Sorkine-Hornung, O., Cohen-Or, D., and Mitra, N. J. 2014. Meta-representation of shape families. ACM SIGGRAPH.
12. Guillaumin, M., Küttel, D., and Ferrari, V. 2014. Imagenet auto-annotation with segmentation propagation. IJCV 110, 3, 328–348.
13. Hu, R., van Kaick, O., Wu, B., Huang, H., Shamir, A., and Zhang, H. 2016. Learning how objects function via co-analysis of interactions. ACM SIGGRAPH 35, 4.
14. Huang, Q., Wang, H., and Koltun, V. 2015. Single-view reconstruction via joint analysis of image and shape collections. ACM Trans. Graph. 34, 4 (July), 87:1–87:10.
15. Hueting, M., Ovsjanikov, M., and Mitra, N. 2015. Crosslink: Joint understanding of image and 3d model collections through shape and camera pose variations. ACM SIGGRAPH Asia 2015.
16. Hunter, D. R. 2004. Mm algorithms for generalized bradley-terry models. Annals of Statistics, 384–406.
17. Kholgade, N., Simon, T., Efros, A. A., and Sheikh, Y. 2014. 3D object manipulation in a single photograph using stock 3D models. ACM Trans. Graph. 33, 4.
18. Kuettel, D., Guillaumin, M., and Ferrari, V. 2012. Segmentation propagation in imagenet. In ECCV, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds., 459–473.
19. Lim, J. J., Khosla, A., and Torralba, A. 2014. FPM: fine pose parts-based model with 3d CAD models. In ECCV, 478–493.
20. Liu, C., Yuen, J., and Torralba, A. 2011. Sift flow: Dense correspondence across scenes and its applications. PAMI 33, 5.
21. Liu, T., Hertzmann, A., Li, W., and Funkhouser, T. 2015. Style compatibility for 3D furniture models. ACM SIGGRAPH 34, 4.
22. Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV 60, 2, 91–110.
23. Noh, H., Hong, S., and Han, B. 2015. Learning deconvolution network for semantic segmentation. In Proc. ICCV, 1520–1528.
24. Su, H., Huang, Q., Mitra, N. J., Li, Y., and Guibas, L. 2014. Estimating image depth using shape collections. SIGGRAPH.
25. Su, H., Wang, F., Yi, L., and Guibas, L. 2014. 3d-assisted image feature synthesis for novel views of an object. arXiv preprint arXiv:1412.0003.
26. Su, H., Qi, C. R., Li, Y., and Guibas, L. J. 2015. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views. In Proc. ICCV, 2686–2694.
27. Tatarchenko, M., Dosovitskiy, A., and Brox, T. 2015. Single-view to multi-view: Reconstructing unseen views with a convolutional network. CoRR abs/1511.06702.
28. Vallet, B., and Lvy, B. 2009. What you seam is what you get. Tech. rep., INRIA – ALICE Project Team.
29. Wang, Y., Gong, M., Wang, T., Cohen-Or, D., Zhang, H., and Chen, B. 2013. Projective analysis for 3d shape segmentation. ACM Trans. Graph. 32, 6 (Nov.), 192:1–192:12.
30. Wei, L.-Y., Lefebvre, S., Kwatra, V., and Turk, G. 2009. State of the art in example-based texture synthesis. In EG-STAR.
31. Zhang, Z., Ganesh, A., Liang, X., and Ma, Y. 2012. TILT: transform invariant low-rank textures. IJCV 99, 1, 1–24.
32. Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM Transactions on Graphics 31, 4, 99:1–99:11.
