“Reflectance modeling by neural texture synthesis”

  • ©Miika Aittala, Timo Aila, and Jaakko Lehtinen

Conference:


Type:


Title:

    Reflectance modeling by neural texture synthesis

Session/Category Title: MATERIALS

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    We extend parametric texture synthesis to capture rich, spatially varying parametric reflectance models from a single image. Our input is a single head-lit flash image of a mostly flat, mostly stationary (textured) surface, and the output is a tile of SVBRDF parameters that reproduce the appearance of the material. No user intervention is required. Our key insight is to make use of a recent, powerful texture descriptor based on deep convolutional neural network statistics for “softly” comparing the model prediction and the examplars without requiring an explicit point-to-point correspondence between them. This is in contrast to traditional reflectance capture that requires pointwise constraints between inputs and outputs under varying viewing and lighting conditions. Seen through this lens, our method is an indirect algorithm for fitting photorealistic SVBRDFs. The problem is severely ill-posed and non-convex. To guide the optimizer towards desirable solutions, we introduce a soft Fourier-domain prior for encouraging spatial stationarity of the reflectance parameters and their correlations, and a complementary preconditioning technique that enables efficient exploration of such solutions by L-BFGS, a standard non-linear numerical optimizer.

References:


    1. Aittala, M., Weyrich, T., and Lehtinen, J. 2015. Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. 34, 4, 110:1–110:13. Google ScholarDigital Library
    2. Barron, J., and Malik, J. 2015. Shape, illumination, and reflectance from shading. IEEE TPAMI (to appear).Google Scholar
    3. Boivin, S., and Gagalowicz, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. ACM SIGGRAPH, 107–116. Google ScholarDigital Library
    4. Clark, R., 2010. Crazybump. http://www.crazybump.com, Last access: 16 Jan 2016.Google Scholar
    5. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proc. ACM SIGGRAPH, 145–156. Google ScholarDigital Library
    6. Dong, Y., Wang, J., Tong, X., Snyder, J., Lan, Y., Ben-Ezra, M., and Guo, B. 2010. Manifold bootstrapping for SVBRDF capture. ACM Trans. Graph. 29, 4, 98:1–98:10. Google ScholarDigital Library
    7. Dong, Y., Tong, X., Pellacini, F., and Guo, B. 2011. Appgen: interactive material modeling from a single image. ACM Trans. Graph. 30, 6, 146:1–146:10. Google ScholarDigital Library
    8. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proc. ACM SIGGRAPH, 341–346. Google ScholarDigital Library
    9. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In Proc. International Conference on Computer Vision (ICCV ’99), vol. 2, 1033–1038. Google ScholarDigital Library
    10. Galerne, B., Gousseau, Y., and Morel, J.-M. 2010. Random phase textures: Theory and synthesis. IEEE Transactions in Image Processing 20, 1, 257–267. Google ScholarDigital Library
    11. Gatys, L. A., Ecker, A. S., and Bethge, M. 2015. A neural algorithm of artistic style. CoRR abs/1508.06576.Google Scholar
    12. Gatys, L. A., Ecker, A. S., and Bethge, M. 2015. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems 28. Google ScholarDigital Library
    13. Graham, P., Tunwattanapong, B., Busch, J., Yu, X., Jones, A., Debevec, P., and Ghosh, A. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum 32, 2pt3, 335–344.Google Scholar
    14. Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. In Proc. ACM SIGGRAPH, 229–238. Google ScholarDigital Library
    15. Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In Proc. ACM SIGGRAPH, 327–340. Google ScholarDigital Library
    16. Huang, L., Yang, Y., Deng, Y., and Yu, Y 2015. Densebox: Unifying landmark localization with end to end object detection. CoRR abs/1509.04874.Google Scholar
    17. Kaspar, A., Neubert, B., Lischinski, D., Pauly, M., and Kopf, J. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2. Google ScholarDigital Library
    18. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 3, 795–802. Google ScholarDigital Library
    19. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. In Proc. IEEE, 2278–2324.Google Scholar
    20. Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and Seidel, H.-P. 2003. Image-based reconstruction of spatial appearance and geometric detail. ACM Trans. Graph. 22, 2, 234–257. Google ScholarDigital Library
    21. Ngan, A., Durand, F., and Matusik, W. 2005. Experimental analysis of BRDF models. In Proc. Eurographics Symposium on Rendering, 117–226. Google ScholarDigital Library
    22. Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision 40, 1, 49–70. Google ScholarDigital Library
    23. Rabin, J., Peyré, G., Delon, J., and Bernot, M. 2011. Wasserstein barycenter and its application to texture mixing. In Proc. Scale Space and Variational Methods in Computer Vision (SSVM), vol. 6667, 435–446. Google ScholarDigital Library
    24. Razavian, A. S., Azizpour, H., Sullivan, J., and Carls-son, S. 2014. CNN features off-the-shelf: An astounding baseline for recognition. In Proc. CVPR. Google ScholarDigital Library
    25. Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556.Google Scholar
    26. Super, B., and Bovik, A. 1995. Shape from texture using local spectral moments. IEEE TPAMI 17, 4. Google ScholarDigital Library
    27. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. 2015. Going deeper with convolutions. In Proc. CVPR.Google Scholar
    28. Tong, X., Zhang, J., Liu, L., Wang, X., Guo, B., and Shum, H.-Y. 2002. Synthesis of bidirectional texture functions on arbitrary surfaces. ACM Trans. Graph. 21, 3, 665–672. Google ScholarDigital Library
    29. Vedaldi, A., and Lenc, K. 2014. MatConvNet – convolutional neural networks for MATLAB. CoRR abs/1412.4564.Google Scholar
    30. Wang, J., Zhao, S., Tong, X., Snyder, J., and Guo, B. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Trans. Graph. 27, 3, 41:1–41:9. Google ScholarDigital Library
    31. Wang, C.-P., Snavely, N., and Marschner, S. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. ACM Trans. Graph. 30, 6, 172:1–172:12. Google ScholarDigital Library
    32. Wei, L.-Y., Han, J., Zhou, K., Bao, H., Guo, B., and Shum, H.-Y. 2008. Inverse texture synthesis. ACM Trans. Graph. 27, 3, 52:1–52:9. Google ScholarDigital Library
    33. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25, 3, 1013–1024. Google ScholarDigital Library
    34. Weyrich, T., Lawrence, J., Lensch, H., Rusinkiewicz, S., and Zickler, T. 2009. Principles of appearance acquisition and representation. Foundations and Trends in Computer Graphics and Vision 4, 2, 75–191. Google ScholarDigital Library
    35. Zickler, T., Ramamoorthi, R., Enrique, S., and Belhumeur, P. N. 2006. Reflectance sharing: predicting appearance from a sparse set of images of a known shape. IEEE TPAMI 28, 8, 1287–1302. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: