“Synthesis of Complex Image Appearance From Limited Exemplars” by Diamanti, Barnes, Shechtman, Paris and Sorkine-Hornung

  • ©

Conference:


Type(s):


Title:

    Synthesis of Complex Image Appearance From Limited Exemplars

Session/Category Title:   Image Similarity & Search

Presenter(s)/Author(s):


Moderator(s):


Abstract:


    Editing materials in photos opens up numerous opportunities like turning an unappealing dirt ground into luscious grass and creating a comfortable wool sweater in place of a cheap t-shirt. However, such edits are challenging. Approaches such as 3D rendering and BTF rendering can represent virtually everything, but they are also data intensive and computationally expensive, which makes user interaction difficult. Leaner methods such as texture synthesis are more easily controllable by artists, but also more limited in the range of materials that they handle, for example, grass and wool are typically problematic because of their non-Lambertian reflectance and numerous self-occlusions. We propose a new approach for editing of complex materials in photographs. We extend the texture-by-numbers approach with ideas from texture interpolation. The inputs to our method are coarse user annotation maps that specify the desired output, such as the local scale of the material and the illumination direction. Our algorithm then synthesizes the output from a discrete set of annotated exemplars. A key component of our method is that it can cope with missing data, interpolating information from the available exemplars when needed. This enables production of satisfying results involving materials with complex appearance variations such as foliage, carpet, and fabric from only one or a couple of exemplar photographs.

References:


    1. C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
    2. C. Barnes, E. Shechtman, D. Goldman, and A. Finkelstein. 2010. The generalized patchmatch correspondence algorithm. In Proceedings of the European Conference on Computer Vision (ECCV’10). 29–43. Google ScholarDigital Library
    3. J. T. Barron and J. Malik. 2012. Color constancy, intrinsic images, and shape estimation. In Proceedings of the European Conference on Computer Vision (ECCV’12). A. W. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds., Lecture Notes in Computer Science, vol. 7575, Springer, 57–70. Google ScholarDigital Library
    4. P. Bhat, B. Curless, M. Cohen, and C. L. Zitnick. 2008. Fourier analysis of the 2D screened poisson equation for gradient domain problems. In Proceedings of the European Conference on Computer Vision (ECCV’08). 114–128. Google ScholarDigital Library
    5. N. Bonneel, M. Van De Panne, S. Lefebvre, and G. Drettakis. 2010. Proxy-guided texture synthesis for rendering natural scenes. In Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV’10). 87–95.Google Scholar
    6. D. Comaniciu and P. Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Mach. Intell. 24, 5, 603–619. Google ScholarDigital Library
    7. K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Trans. Graph. 18, 1, 1–34. Google ScholarDigital Library
    8. S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and P. Sen. 2012. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31, 4. Google ScholarDigital Library
    9. A. A. Efros and T. K. Leung. 1999. Texture synthesis by nonparametric sampling. In Proceeding of the International Conference on Computer Vision (ICCV’99). Vol. 2. 1033–1038. Google ScholarDigital Library
    10. C. Eisenacher, S. Lefebvre, and M. Stamminger. 2008. Texture synthesis from photographs. Comput. Graph. Forum 27, 2, 419–428.Google ScholarCross Ref
    11. H. Fang and J. C. Hart. 2004. Textureshop: Texture synthesis as a photograph editing tool. ACM Trans. Graph. 23, 3, 354–359. Google ScholarDigital Library
    12. C. Han, E. Risser, R. Ramamoorthi, and E. Grinspun. 2008. Multiscale texture synthesis. ACM Trans. Graph. 27, 3. Google ScholarDigital Library
    13. A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’01). 327–340. Google ScholarDigital Library
    14. W. Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.Google Scholar
    15. M. K. Johnson, K. Dale, S. Avidan, H. Pfister, W. T. Freeman, and W. Matusik. 2011. CG2Real: Improving the realism of computer generated images using a large collection of photographs. IEEE Trans. Visual. Comput. Graph. 17, 9, 1273–1285. Google ScholarDigital Library
    16. J. Kautz, S. Boulos, and F. Durand. 2007. Interactive editing and modeling of bidirectional texture functions. ACM Trans. Graph. 26, 3. Google ScholarDigital Library
    17. V. Kwatra, I. A. Essa, A. F. Bobick, and N. Kwatra. 2005. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 3, 795–802. Google ScholarDigital Library
    18. S. Lefebvre and H. Hoppe. 2005. Parallel controllable texture synthesis. ACM Trans. Graph. 24, 3, 777–786. Google ScholarDigital Library
    19. S. Lefebvre and H. Hoppe. 2006. Appearance-space texture synthesis. ACM Trans. Graph. 25, 3, 541–548. Google ScholarDigital Library
    20. D. Lepage and J. Lawrence. 2011. Material matting. ACM Trans. Graph. 30, 6, 144. Google ScholarDigital Library
    21. X. Liu, Y. Yu, and H.-Y. Shum. 2001. Synthesizing bidirectional texture functions for real-world surfaces. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’01). ACM Press, New York, 97–106. Google ScholarDigital Library
    22. J. Lu, A. S. Georghiades, A. Glaser, H. Wu, L.-Y. Wei, B. Guo, J. Dorsey, and H. E. Rushmeier. 2007. Context-aware textures. ACM Trans. Graph. 26, 1. Google ScholarDigital Library
    23. W. Matusik, M. Zwicker, and F. Durand. 2005. Texture design using a simplicial complex of morphable textures. ACM Trans. Graph. 24, 3, 787–794. Google ScholarDigital Library
    24. D. M. Mount and S. Arya. 1998. ANN: Library for approximate nearest neighbour searching. http://www.cs.umd.edu/-mount/ANN.Google Scholar
    25. A. Ngan and F. Durand. 2006. Statistical acquisition of texture appearance. In Proceedings of the 17th Eurographics Conference on Rendering Techniques (EGSR’06). 31–40. Google ScholarDigital Library
    26. A. Orzan, A. Bousseau, H. Winnemoller, P. Barla, J. Thollot, and D. Salesin. 2008. Diffusion curves: A vector representation for smooth-shaded images. ACM Trans. Graph. 27, 3. Google ScholarDigital Library
    27. H. Park, H. Byun, and C.-H. Kim. 2013. Multi-exemplar inhomogeneous texture synthesis. Comput. Graph. 37, 1–2, 54–64. Google ScholarDigital Library
    28. E. Risser, C. Han, R. Dahyot, and E. Grinspun. 2010. Synthesizing structured image hybrids. ACM Trans. Graph. 29, 4. Google ScholarDigital Library
    29. A. Rosenberger, D. Cohen-Or, and D. Lischinski. 2009. Layered shape synthesis: Automatic generation of control maps for non-stationary textures. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
    30. R. Ruiters, C. Schwartz, and R. Klein. 2013. Example-based interpolation and synthesis of bidirectional texture functions. Comput. Graph. Forum 32, 2.Google ScholarCross Ref
    31. J. Wang, X. Tong, S. Lin, M. Pan, C. Wang, H. Bao, B. Guo, and H.-Y. Shum. 2006. Appearance manifolds for modeling time-variant appearance of materials. ACM Trans. Graph. 25, 3, 754–761. Google ScholarDigital Library
    32. L.-Y. Wei and M. Levoy. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’00). 479–488. Google ScholarDigital Library
    33. J. Zhang, K. Zhou, L. Velho, B. Guo, and H.-Y. Shum. 2003. Synthesis of progressively-variant textures on arbitrary surfaces. ACM Trans. Graph. 22, 3, 295–302. Google ScholarDigital Library

ACM Digital Library Publication:


Overview Page: