Image Inpainting

Introduction:  Inpainting is an artistic synonym for image interpolation, and has been circulated among museum restoration artists for a long time. The notion of digital inpainting was firstintroduced in the paper by Bertalmio-Sapiro-Caselles-Ballester (SIGGRAPH 2000). Smart digital inpainting models, techniques, and algorithms have broad applications in image interpolation, photo restoration, zooming and super-resolution, primal-sketch based perceptual image compression and coding, and the error concealment of (wireless) image transmission, etc.

Our approach
 is primarily based on the Bayesian (or equivalently, Helmholtz) philosophy of vision: an optimal guess of the complete ideal image from its incomplete and distorted data crucially relies on the answers to the two questions:  (A) (Prior Model) what do we mean by ``images?"  and (B) (Data Model) how have the observed data been generated from (or connected to) the original ideal image? We have developed our variational/PDE models and algorithms along this line of philosophy.

Star Image Inpainting is featured in the  Science News  (Vol. 161/19, 05/11/2002).  Please read the article Filling in Blanks   by Dr. Ivars Peterson based on the AMS annual conference at San Diego, Jan 2002.

* Inpainting & Fundamental Problem of I.P. pdf & ps . SIAM News, invited, 2003.
* BV Image Model and Inpainting of Noisy Blurred Images (March, 2002)
* Mumford-Shah-Euler Image Model for Digital Inpainting (Sept, 2001)
* Local Inpainting Models and TV (total variation) Inpainting (March, 2000)
* Non-Texture Inpainting by Curvature-Driven Diffusions (CDD) (Sept, 2000)
* Euler's Elastica and Curvature Based Inpainting (April, 2001)
* Landmark Based Inpainting from Multiple Views (March, 2002)

BV Image Model and Inpainting of Noisy Blurred Images

From the abstract: What we believe images are determines how we take actions in image and lower-level vision analysis. In the Bayesian framework, it is manifest in the importance of a good image prior model. This paper intends to give a concise overview on the vision foundation, mathematical theory, computational algorithms, and various classical as well as unexpected new applications of the BV (bounded variation) image model, first introduced into image processing by Rudin, Osher, and Fatemi in 1992 [Physica D, 60:259-268]. (By Chan and Shen , March, 2002: On the Role of the BV Image Model in Image Restoration . Dedicated to Stan Osher on the occasion of his 60th birthday.)

Mumford-Shah-Euler Image Model and Inpainting

From the abstract: Image inpainting is an image restoration problem, in which image models play a critical role, as demonstrated by Chan, Kang, and Shen's recent inpainting schemes based on the Total Variation and elastica image models. In this paper, we propose two novel inpainting models based on the Mumford-Shah image model, and its high order correction -- the Mumford-Shah-Euler image model. We also present their efficient numerical realization based on the Gamma-convergence approximation of Ambrosio and Tortorelli, and a conjecture of De Giorgi. (By Selim Esedoglu and Jianhong Shen , European J. Appl. Math., 13, pp. 353-370, 2002.)

Local Inpainting Models and TV (total variation) Inpainting

As an ancient painting gets older, on certain regions, the pigments start to fall off the canvas, and the painting becomes incomplete. The human work of filling in the missing parts of the painting is called "inpainting," as first introduced to image processing by Bertalmio, Sapiro, Caselles and Ballester at University of Minnesota. Digital inpainting has much wider applications in image processing and computer vision. Inspired by the work of Bertalmio et al. (1999), we intend to develop general inpainting models for non-texture images. In smooth regions, inpaintings are connected to the harmonic and biharmonic extensions and inpainting orders are defined and analyzed. For inpaintings involving the recovery of edges, we propose a variational model that is closely connected to "Total Variational" denoising and debluring model, invented by Rudin, Osher, and Fatemi (Phys. D, 50, 1992). Such a model and its algorithm intrinsically combine the denoising and inpainting processes. In other words, our inpainting scheme is robust to noise, and thus insensitive to pixel values. This work is also closely related to disocclusion in computer vision by Nitzbeg, Mumford, and Shiota (1993), and by Masnou and Morel (1998). (from CAM 00-11 Abstract, March 2000) Report by Chan and Shen [SIAM Appl. Math, 62(3), 1019-1043, 2001]. (Wonder how this famous Kanizsa's "Entangled Man" illusion is related to the inpainting model? Find the answer in the paper.)

Zoom-in : We apply both the digital TV zoom-in and harmonic zoom-in to the test image "Lamp". It is clear that the TV zoom-in model produces much better visual results in terms of edge sharpness and boundary regularity.

Inpainting of Primal Sketches (David Marr): In the very beginning of Computer Vision and Artificial Intelligence, David Marr (MIT), inspired by the human vision system, inquired the possibilty of reconstructing images only based on their primal sketches. Our inpainting approach may provide a partial answer. The lower right image is inpainted from the grey primal sketch (which we call the edge tube ) on top of it. This lossy edge decoding scheme certainly mollifies the original image function, but does successfully catch the essential visual information in the original images.
[SIAM J. Appl. Math., 62(3), 1019-1043, 2001]

Edge decoding

Non-Texture Inpainting by Curvature-Driven Diffusions (CDD)

We propose a new inpainting model based on the diffusion mechanism as inspired by our previous work on the TV inpainting, aiming at realizing the Connectivity Principle in vision psychology. Since in this new diffusion model, the conductivity coefficient depends on the curvature of the isophotes, we call such new diffusions Curvature-Driven Diffusions(CDD), as contrast to other diffusion models (such as the celebrated Perona-Malik) prevailing in image and vision analysis. (from CAM 00-35 Introduction, Sep 2000) Report by Chan and Shen . [J. Visual Communication and Image Representation, 12(4), 436-449, 2001]

Euler's Elastica and Curvature Based Inpaintings

Image inpainting is a special image restoration problem for which image prior models play a crucial role. Euler's elastica was first introduced by Mumford to computer vision as a curve prior model. By functionalizing the elastica energy, Masnou and Morel proposed an elastica based variational inpainting model. The current paper is intended to develop its mathematical foundation, study its properties and connections to the earlier works of Bertalmio, Sapiro, Caselles and Ballester and Chan and Shen and construct computational schemes that are based on numerical PDE's, instead of the dynamical programming algorithm, which imposes topological constraints on inpainting domains. (from CAM 01-12, April 2001). Report by Chan, Kang and Shen [SIAM J. Appl. Math., 63(2): pp.564-592, 2002]

Landmark Based Inpainting from Multiple Views

From the abstract : Most existing inpainting algorithms are local in nature and extrapolate information from neighboring pixels into the inpainting regions. In this paper, we are interested in the inpainting problem where the missing region are so large that these local inpainting methods fail. As an alternative to the local principle, we assume that there are other images with related global information to enable a reasonable inpainting. These additional images could be from a movie sequence, an image of the same object from a different time and a different viewpoint, or an image of a similar object.

Our method has roughly three phases: landmark matching, interpolation, and copying. For the landmark matching, modified shape context information is used to exploit the global information. Then matched information is interpolated (and regularized) using thin plate splines. Finally, we copy the information from one image to another. Using landmark matching and interpolation, allows the missing regions to be significantly larger compared to the local inpainting methods, and can be used when the object is distorted from one image to another. The experimental results are promising. (from CAM 02-11, March 2002) Report by Kang, Chan and Soatto [submitted to IEEE PAMI 2002]. Short version is also available at CAM 02-31, Proceedings of 3DPVT, June 2002

Reports on Image Restorations  

Tony Chan , Stanley Osher , Jianhong (JACKIE) Shen , David Strong , Peter Blomgren , C. K. Wong, Sung Ha Kang