“Image analogies” by Hertzmann, Jacobs, Oliver, Curless and Salesin

This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface.

1. Sunil Arya, David M. Mount, Nathan S. Netanyahu, Ruth Silverman, and Angela Y. Wu. An Optimal Algorithm for Approximate Nearest Neighbor Searching in Fixed Dimensions. Journal of the ACM, 45(6):891-923, 1998. Source code available from http://www.cs.umd.edu/~mount/ANN.
2. Michael Ashikhmin. Synthesizing Natural Textures. 2001 ACM Symposium on Interactive 3D Graphics, pages 217-226, March 2001.
3. Michael F. Barnsley, Lyman P. Hurd, and Louisa F. Anson. Fractal Image Compression. A.K. Peters Ltd, 1993.
4. Jeremy S. De Bonet. Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. Proceedings of SIGGRAPH 97, pages 361-368, August 1997.
5. Matthew Brand. Voice Puppetry. Proceedings of SIGGRAPH 99, pages 21-28, August 1999.
6. Matthew Brand and Aaron Hertzmann. Style machines. Proceedings of SIG- GRAPH 2000, pages 183-192, July 2000.
7. Christoph Bregler, Michele Covell, and Malcolm Slaney. Video Rewrite: Driving Visual Speech with Audio. Proceedings of SIGGRAPH 97, pages 353-360, August 1997.
8. Ian Buck, Adam Finkelstein, Charles Jacobs, Allison Klein, David H. Salesin, Joshua Seims, Richard Szeliski, and Kentaro Toyama. Performance-driven handdrawn animation. NPAR 2000: First International Symposium on Non Photorealistic Animation and Rendering, pages 101-108, June 2000.
9. Kenneth Castleman. Digital Image Processing. Prentice-Hall, 1996.
10. Cassidy J. Curtis, Sean E. Anderson, Joshua E. Seims, Kurt W. Fleischer, and David H. Salesin. Computer-Generated Watercolor. Proceedings of SIGGRAPH 97, pages 421-430, August 1997.
11. Alexei Efros and Thomas Leung. Texture Synthesis by Non-parametric Sampling. 7th IEEE International Conference on Computer Vision, 1999.
12. Alexei A. Efros and William T. Freeman. Quilting for Texture Synthesis and Transfer. Proceedings of SIGGRAPH 2001, August 2001.
13. Alex Eilhauer, Alice Pritikin, Dylan Weed, and Steven J. Gortler. Combining Textures and Pictures with Specialized Texture Synthesis, 2000. http://www.people.fas.harvard.edu/ ~pritikin/cs/graphics/.
14. Chris Eliasmith. Dictionary of Philosophy of Mind. http://artsci.wustl.edu/ ~philos/MindDict/.
15. T.G. Evans. A program for the solution of geometric analogy intelligence test questions. In M. Minsky, editor, Semantic Information Processing. MIT Press, 1968.
16. James D. Foley, Andries van Dam, Steven K. Feiner, and John F. Hughes. Computer Graphics, Principles and Practice, Second Edition. Addison-Wesley, 1990.
17. W. T. Freeman, E. C. Pasztor, and O. T. Carmichael. Learning Low- Level Vision. Intl. J. Computer Vision, 40(1):25-47, 2000. See also http://www.merl.com/reports/TR2000-05/.
18. William T. Freeman, Joshua B. Tenenbaum, and Egon Pasztor. An examplebased approach to style translation for line drawings. Technical Report TR99-11, MERL, February 1999.
19. D. Gentner. Structure mapping: A theoretical framework for analogy. Cognitive Science, 7(2):155-170, 1983.
20. Allen Gersho and Robert M. Gray. Vector Quantization and Signal Compression. Kluwer Academic Publishers, 1992.
21. Paul E. Haeberli. Paint By Numbers: Abstract Image Representations. In Computer Graphics (SIGGRAPH ’90 Proceedings), volume 24, pages 207-214, August 1990.
22. J. Hamel and T. Strothotte. Capturing and re-using rendition styles for nonphotorealistic rendering. Computer Graphics Forum, 18(3):173-182, September 1999.
23. David J. Heeger and James R. Bergen. Pyramid-Based Texture Analysis/Synthesis. Proceedings of SIGGRAPH 95, pages 229-238, August 1995.
24. Aaron Hertzmann. Painterly Rendering with Curved Brush Strokes of Multiple Sizes. In SIGGRAPH 98 Conference Proceedings, pages 453-460, July 1998.
25. Aaron Hertzmann. Algorithms for Rendering in Artistic Styles. PhD thesis, New York University, May 2001.
26. Aaron Hertzmann and Denis Zorin. Illustrating smooth surfaces. Proceedings of SIGGRAPH 2000, pages 517-526, July 2000.
27. Youichi Horry, Ken ichi Anjyo, and Kiyoshi Arai. Tour Into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image. Proceedings of SIGGRAPH 97, pages 225-232, August 1997.
28. William James. The Principles of Psychology. 1890.
29. Bela Julesz. Textons, the elements of texture perception, and their interactions. Nature, 290:91-97, 1981.
30. Allison W. Klein, Wilmot W. Li, Michael M. Kazhdan, Wagner T. Correa, Adam Finkelstein, and Thomas A. Funkhouser. Non-photorealistic virtual environments. Proceedings of SIGGRAPH 2000, pages 527-534, July 2000.
31. Arthur Koestler. The Act of Creation. Picador, London, 1964.
32. Michael A. Kowalski, Lee Markosian, J. D. Northrup, Lubomir Bourdev, Ronen Barzel, Loring S. Holden, and John Hughes. Art-Based Rendering of Fur, Grass, and Trees. Proceedings of SIGGRAPH 99, pages 433-438, August 1999.
33. G. Lakoff and M. Johnson. Metaphors we live by. University of Chicago Press, Chicago, IL, 1980.
34. Thomas Leung and Jitendra Malik. Recognizing surfaces using threedimensional textons. 7th IEEE International Conference on Computer Vision, September 1999.
35. Jitendra Malik, Serge Belongie, Jianbo Shi, and Thomas Leung. Textons, Contours, and Regions: Cue Integration in Image Segmentation. 7th IEEE International Conference on Computer Vision, September 1999.
36. Barbara J. Meier. Painterly Rendering for Animation. In SIGGRAPH 96 Conference Proceedings, pages 477-484, August 1996.
37. Pietro Perona and Jitendra Malik. Scale-Space and Edge Detection using Anisotropic Diffusion. IEEE Trans. on Pattern Analysis and Machine Intelligence, 12:629-639, December 1990.
38. Ferdinand Petrie and John Shaw. The Big Book of Painting Nature in Watercolor. Watson-Guptill Publications, 1990.
39. Kris Popat and Rosalind W. Picard. Cluster-based probability model and its application to image and texture processing. IEEE Trans. on Image Processing, 6(2):268-284, February 1997.
40. J. Portilla and E. P. Simoncelli. A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients. International Journal of Computer Vision, 40(1):49-71, December 2000.
41. Michael P. Salisbury, Sean E. Anderson, Ronen Barzel, and David H. Salesin. Interactive Pen-And-Ink Illustration. In Proceedings of SIGGRAPH ’94 (Orlando, Florida, July 24-29, 1994), pages 101-108, July 1994.
42. Michael P. Salisbury, Michael T. Wong, John F. Hughes, and David H. Salesin. Orientable Textures for Image-Based Pen-and-Ink Illustration. In SIGGRAPH 97 Conference Proceedings, pages 401-406, August 1997.
43. Arno Schodl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video Textures. Proceedings of SIGGRAPH 2000, pages 489-498, July 2000.
44. K. Schunn and K. Dunbar. Priming, Analogy and Awareness in complex reasoning. Memory and Cognition, 24:271-284, 1996.
45. Eero P. Simoncelli and William T. Freeman. The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation. Proc. 2nd Int’l Conf on Image Processing, October 1995.
46. Oleg Veryovka and John W. Buchanan. Comprehensive Halftoning of 3D Scenes. Computer Graphics Forum, 18(3):13-22, September 1999.
47. Oleg Veryovka and John W. Buchanan. Halftoning With Image-Based Dither Screens. Graphics Interface ’99, pages 167-174, June 1999.
48. B. Wandell. Foundations of Vision. Sinauer Associates Inc., 1995.
49. Li-Yi Wei and Marc Levoy. Fast Texture Synthesis Using Tree-Structured Vector Quantization. Proceedings of SIGGRAPH 2000, pages 479-488, July 2000.
50. Georges Winkenbach and David H. Salesin. Computer-Generated Pen-And-Ink Illustration. In Proceedings of SIGGRAPH ’94 (Orlando, Florida, July 24-29, 1994), pages 91-100, July 1994.
51. Georges Winkenbach and David H. Salesin. Rendering Parametric Surfaces in Pen and Ink. In SIGGRAPH 96 Conference Proceedings, pages 469-476, August 1996.
52. P.H. Winston. Learning and Reasoning by Analogy. Communications of the ACM, (23) 12, December 1980.
53. Daniel N. Wood, Adam Finkelstein, John F. Hughes, Craig E. Thayer, and David H. Salesin. Multiperspective panoramas for cel animation. Proceedings of SIGGRAPH 97, pages 243-250, August 1997.
54. Song Chun Zhu, Ying Nian Wu, and David Mumford. Filters, Random fields, And Maximum Entropy: Towards a Unified Theory for Texture Modeling. International Journal of Computer Vision, 12(2):1-20, March/April 1998.