“Image inpainting” by Bertalmio, Sapiro, Caselles and Ballester

Inpainting, the technique of modifying an image in an undetectable form, is as ancient as art itself. The goals and applications of inpainting are numerous, from the restoration of damaged paintings and photographs to the removal/replacement of selected objects. In this paper, we introduce a novel algorithm for digital inpainting of still images that attempts to replicate the basic techniques used by professional restorators. After the user selects the regions to be restored, the algorithm automatically fills-in these regions with information surrounding them. The fill-in is done in such a way that isophote lines arriving at the regions’ boundaries are completed inside. In contrast with previous approaches, the technique here introduced does not require the user to specify where the novel information comes from. This is automatically done (and in a fast way), thereby allowing to simultaneously fill-in numerous regions containing completely different structures and surrounding backgrounds. In addition, no limitations are imposed on the topology of the region to be inpainted. Applications of this technique include the restoration of old photographs and damaged film; removal of superimposed text like dates, subtitles, or publicity; and the removal of entire objects from the image like microphones or wires in special effects.

1. S. Walden. The Ravished Image. St. Martin’s Press, New York, 1985.
2. G. Emile-Male. The Restorer’s Handbook of Easel Painting. Van Nostrand Reinhold, New York, 1976.
3. D. King. The Commissar Vanishes. Henry Holt and Company, 1997.
4. A.C. Kokaram, R.D. Morris, W.J. Fitzgerald, P.J.W. Rayner. Detection of missing data in image sequences. IEEE Transac-tions on Image Processing 11(4), 1496-1508, 1995.
5. A.C. Kokaram, R.D. Morris, W.J. Fitzgerald, P.J.W. Rayner. Interpolation of missing data in image sequences. IEEE Trans-actions on Image Processing 11(4), 1509-1519, 1995.
6. C. Braverman. Photoshop retouching handbook. IDG Books Worldwide, 1998.
7. A. Hirani and T. Totsuka. Combining Frequency and spatial domain information for fast interactive image noise removal. Computer Graphics, pp. 269-276, SIGGRAPH 96, 1996.
8. A. Efros and T. Leung, “Texture synthesis by non-parametric sampling,” Proc. IEEE International Conference Computer Vi-sion, pp. 1033-1038, Corfu, Greece, September 1999.
9. D.Heeger andJ.Bergen. Pyramid based texture analy-sis/ synthesis. Computer Graphics, pp. 229-238, SIGGRAPH 95, 1995.
10. E. Simoncelli and J. Portilla. Texture characterization via joint statistics of wavelet coefficient magnitudes. 5th IEEE Int’l Conf. on Image Processing, Chicago, IL. Oct 4-7, 1998.
11. M. Nitzberg, D. Mumford, and T. Shiota, Filtering, Segmen-tation, and Depth, Springer-Verlag, Berlin, 1993.
12. S. Masnou and J.M. Morel. Level-lines based disocclusion. 5th IEEE Int’l Conf. on Image Processing, Chicago, IL. Oct 4-7, 1998.
13. C. Kenney and J. Langan. A new image processing primitive: reconstructing images from modified flow fields. University of California Santa Barbara Preprint, 1999.
14. P. Perona and J. Malik Scale-space and edge detection using anisotropic diffusion. IEEE-PAMI 12, pp. 629-639, 1990.
15. L. Alvarez, P.L. Lions, J.M. Morel. Image selective smoothing and edge detection by nonlinear diffusion.SIAMJ.Numer. Anal. 29, pp. 845-866, 1992.
16. S. Osher and J. Sethian. Fronts propagating with curvature dependent speed: algorithms based on Hamilton-Jacobi for-mulations. Journal of Computational Physics, 79:12-49, 1988.
17. A. Marquina and S. Osher. Explicit algorithms for a new time dependent model based on level set motion for nonlinear de-bluring and noise removal. UCLA CAM Report 99-5, January 1999.
18. L. Rudin, S. Osher and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D, 60, pp. 259-268, 1992.
19. S. Osher, personal communication, October 1999.
20. H. K. Zhao, T. Chan, B. Merriman, and S. Osher, “A varia-tional level-set approach to multiphase motion,” J. of Compu-tational Physics 127, pp. 179-195, 1996.
21. A. Bertozzi The mathematics of moving contact lines in thin liquid films. Notices Amer. Math. Soc., Volume 45, Number 6, pp. 689-697, June/July 1998.
22. J. Tumblin and G. Turk, “LCIS: A boundary hierarchy for detail-preserving contrast reduction,” Computer Graphics, pp. 83-90, SIGGRAPH 99, 1999.
23. T. Chan and J. Shen, “Mathematical models for local deter-ministic inpaintings,” UCLA CAM TR 00-11, March 2000.
24. C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-in by joint interpolation of vector fields and grey levels,” University of Minnesota IMA TR, April 2000.