“Stylization and abstraction of photographs”

Good information design depends on clarifying the meaningful structure in an image. We describe a computational approach to stylizing and abstracting photographs that explicitly responds to this design goal. Our system transforms images into a line-drawing style using bold edges and large regions of constant color. To do this, it represents images as a hierarchical structure of parts and boundaries computed using state-of-the-art computer vision. Our system identifies the meaningful elements of this structure using a model of human perception and a record of a user’s eye movements in looking at the photo; the system renders a new image using transformations that preserve and highlight these visual elements. Our method thus represents a new alternative for non-photorealistic rendering both in its visual style, in its approach to visual form, and in its techniques for interaction.

1. AGRAWALA, M., AND STOLTE, C. 2001. Rendering effective route maps: improving usability through generalization. In Proc. of ACM SIGGRAPH 2001, 241-249. Google Scholar
2. AHUJA, N. 1996. A transform for multiscale image segmentation by integrated edge and region detection. IEEE Trans. on Pattern Analysis and Machine Intelligence 18, 12, 1211-1235. Google Scholar
3. BURT, P., AND ADELSON, E. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. on Communications 31, 4, 532-540.Google Scholar
4. CAMPBELL, F., AND ROBSON, J. 1968. Application of Fourier analysis to the visibility of gratings. Journal of Physiology 197, 551-566.Google Scholar
5. CHRISTOUDIAS, C., GEORGESCU, B., AND MEER, P. 2002. Synergism in low level vision. In Proc. ICPR 2002. Google Scholar
6. COMANICIU, D., AND MEER, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence 24, 5. Google Scholar
7. CURTIS, C. 1999. Non-photorealistic animation. In ACM SIGGRAPH 1999 Course Notes #17 (Section 9).Google Scholar
8. DEUSSEN, O., AND STROTHOTTE, T. 2000. Computer-generated pen-and-ink illustration of trees. In Proc. of ACM SIGGRAPH 2000, 13-18. Google Scholar
9. DUCHOWSKI, A., AND VERTEGAAL, R. 2000. Eye-based interaction in graphical systems: Theory and practice. In ACM SIGGRAPH 2000 Course Notes #5.Google Scholar
10. DUCHOWSKI, A. 2000. Acuity-matching resolution degradation through wavelet coefficient scaling. IEEE Trans. on Image Processing 9, 8 (Aug.), 1437-1440. Google Scholar
11. DURAND, F., OSTROMOUKHOV, V., MILLER, M., DURANLEAU, F., AND DORSEY, J. 2001. Decoupling strokes and high-level attributes for interactive traditional drawing. In Proceedings of the 12th Eurographics Workshop on Rendering, 71-82. Google Scholar
12. FINKELSTEIN, A., AND SALESIN, D. 1994. Multiresolution curves. In Proc. of ACM SIGGRAPH 94, 261-268. Google Scholar
13. FOLEY, J., VAN DAM, A., FEINER, S., AND HUGHES, J. 1997. Computer Graphics: Principles and Practice, 2nd edition. Addison Wesley. Google Scholar
14. GOOCH, B., AND GOOCH, A. 2001. Non-Photorealistic Rendering. A K Peters. Google Scholar
15. GOOCH, A. A., GOOCH, B., SHIRLEY, P., AND COHEN, E. 1998. A non-photorealistic lighting model for automatic technical illustration. In Proc. of ACM SIGGRAPH 98, 447-452. Google Scholar
16. HAEBERLI, P. 1990. Paint by numbers: Abstract image representations. In Proc. of ACM SIGGRAPH 90, 207-214. Google Scholar
17. HANDFORD, M. 1987. Where’s Waldo? Little, Brown and Company.Google Scholar
18. HENDERSON, J. M., AND HOLLINGWORTH, A. 1998. Eye movements during scene viewing: An overview. In Eye Guidance in Reading and Scene Perception, G. Underwood, Ed. Elsevier Science Ltd., 269-293.Google Scholar
19. HERMAN, I., AND DUKE, D. 2001. Minimal graphics. IEEE Computer Graphics and Applications 21, 6, 18-21. Google Scholar
20. HERTZMANN, A., AND ZORIN, D. 2000. Illustrating smooth surfaces. In Proc. of ACM SIGGRAPH 2000, 517-526. Google Scholar
21. HERTZMANN, A. 1998. Painterly rendering with curved brush strokes of multiple sizes. In Proc. of ACM SIGGRAPH 98, 453-460. Google Scholar
22. HERTZMANN, A. 2001. Paint by relaxation. In Computer Graphics International, 47-54. Google Scholar
23. HOFFMAN, D. D. 1998. Visual intelligence: how we create what we see. Norton.Google Scholar
24. JUST, M. A., AND CARPENTER, P. A. 1976. Eye fixations and cognitive processes. Cognitive Psychology 8, 441-480.Google Scholar
25. KELLY, D. 1984. Retinal inhomogenity: I. spatiotemporal contrast sensitivity. Journal of the Optical Society of America A 74, 1, 107-113.Google Scholar
26. KOENDERINK, J. J., M. A. BOUMAN, A. B. D. M., AND SLAPPENDEL, S. 1978. Perimetry of contrast detection thresholds of moving spatial sine wave patterns. II. the far peripheral visual field (eccentricity 0-50). Journal of the Optical Society of America A 68, 6, 850-854.Google Scholar
27. KOENDERINK, J. J. 1984. The structure of images. Biological Cybernetics 50, 363-370.Google Scholar
28. KOENDERINK, J. J. 1984. What does the occluding contour tell us about solid shape? Perception 13, 321-330.Google Scholar
29. KOWALSKI, M. A., MARKOSIAN, L., NORTHRUP, J. D., BOURDEV, L., BARZEL, R., HOLDEN, L. S., AND HUGHES, J. 1999. Art-based rendering of fur, grass, and trees. In Proc. of ACM SIGGRAPH 99, 433-438. Google Scholar
30. LEYTON, M. 1992. Symmetry, causality, mind. MIT Press.Google Scholar
31. LINDEBERG, T. 1994. Scale-Space Theory in Computer Vision. Kluwer Academic Publishers. Google Scholar
32. LITWINOWICZ, P. 1997. Processing images and video for an impressionist effect. In Proc. of ACM SIGGRAPH 97, 407-414. Google Scholar
33. MACKWORTH, N., AND MORANDI, A. 1967. The gaze selects informative details within pictures. Perception and Psychophysics 2, 547-552.Google Scholar
34. MANNOS, J. L., AND SAKRISON, D. J. 1974. The effects of a visual fidelity criterion on the encoding of images. IEEE Trans. on Information Theory 20, 4, 525-536.Google Scholar
35. MARKOSIAN, L., KOWALSKI, M. A., TRYCHIN, S. J., BOURDEV, L. D., GOLDSTEIN, D., AND HUGHES, J. F. 1997. Real-time nonphotorealistic rendering. In Proc. of ACM SIGGRAPH 97, 415-420. Google Scholar
36. MARR, D. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W. H. Freeman, San Francisco. Google Scholar
37. MEER, P., AND GEORGESCU, B. 2001. Edge detection with embedded confidence. IEEE Trans. on Pattern Analysis and Machine Intelligence 23, 12, 1351-1365. Google Scholar
38. PATTANAIK, S. N., FERWERDA, J. A., FAIRCHILD, M. D., AND GREENBERG, D. P. 1998. A multiscale model of adaptation and spatial vision for realistic image display. In Proc. of ACM SIGGRAPH 98, 287-298. Google Scholar
39. REDDY, M. 2001. Perceptually optimized 3D graphics. IEEE Computer Graphics and Applications 21, 5 (September/October), 68-75. Google Scholar
40. REGAN, D. 2000. Human Perception of Objects: Early Visual Processing of Spatial Form Defined by Luminance, Color, Texture, Motion and Binocular Disparity. Sinauer.Google Scholar
41. ROVAMO, J., AND VIRSU, V. 1979. An estimation and application of the human cortical magnification factor. Experimental Brain Research 37, 495-510.Google Scholar
42. SAITO, T., AND TAKAHASHI, T. 1990. Comprehensible rendering of 3-D shapes. In Proc. of ACM SlGGRAPH 90, 197-206. Google Scholar
43. SANTELLA, A., AND DECARLO, D. 2002. Abstracted painterly renderings using eye-tracking data. In Proc. of the Second International Symp. on Non-photorealistic Animation and Rendering (NPAR). Google Scholar
44. SHIRAISHI, M., AND YAMAGUCHI, Y. 2000. An algorithm for automatic painterly rendering based on local source image approximation. In Proc. of the First International Symp. on Non-photorealistic Animation and Rendering (NPAR), 53-58. Google Scholar
45. SIBERT, L. E., AND JACOB, R. J. K. 2000. Evaluation of eye gaze interaction. In Proc. CHI 2000, 281-288. Google Scholar
46. TRUCCO, E., AND VERRI, A. 1998. Introductory Techniques for 3-D Computer Vision. Prentice-Hall. Google Scholar
47. TUFTE, E. R. 1990. Envisioning Information. Graphics Press. Google Scholar
48. VERTEGAAL, R. 1999. The gaze groupware system: Mediating joint attention in mutiparty communication and collaboration. In Proc. CHI ’99, 294-301. Google Scholar
49. WINKENBACH, G., AND SALESIN, D. H. 1994. Computer-generated pen-and-ink illustration. In Proc. of ACM SIGGRAPH 94, 91-100. Google Scholar
50. YARBUS, A. L. 1967. Eye Movements and Vision. Plenum Press.Google Scholar
51. ZEKI, S. 1999. Inner Vision: An Exploration of Art and the Brain. Oxford Univ. Press.Google Scholar