Image Inpainting
Introduction: Inpainting
is an artistic synonym for image interpolation, and
has been circulated among
museum restoration artists
for a long time. The notion of digital inpainting was
firstintroduced in the paper by Bertalmio-Sapiro-Caselles-Ballester
(SIGGRAPH 2000). Smart digital inpainting models, techniques, and algorithms
have broad applications in image interpolation, photo restoration, zooming
and super-resolution, primal-sketch based perceptual image compression and
coding, and the error concealment of (wireless) image transmission, etc.
Our approach is primarily based on the Bayesian (or equivalently, Helmholtz) philosophy of vision: an optimal guess of the complete ideal image from its incomplete and distorted data crucially relies on the answers to the two questions: (A) (Prior Model) what do we mean by ``images?" and (B) (Data Model) how have the observed data been generated from (or connected to) the original ideal image? We have developed our variational/PDE models and algorithms along this line of philosophy.
Our approach is primarily based on the Bayesian (or equivalently, Helmholtz) philosophy of vision: an optimal guess of the complete ideal image from its incomplete and distorted data crucially relies on the answers to the two questions: (A) (Prior Model) what do we mean by ``images?" and (B) (Data Model) how have the observed data been generated from (or connected to) the original ideal image? We have developed our variational/PDE models and algorithms along this line of philosophy.
Image Inpainting is featured in the
Science News
(Vol. 161/19, 05/11/2002). Please
read the article
Filling in Blanks
by Dr. Ivars Peterson based on the AMS annual conference
at San Diego, Jan 2002. * BV Image Model and Inpainting of Noisy Blurred Images (March, 2002)
* Mumford-Shah-Euler Image Model for Digital Inpainting (Sept, 2001)
* Local Inpainting Models and TV (total variation) Inpainting (March, 2000)
* Non-Texture Inpainting by Curvature-Driven Diffusions (CDD) (Sept, 2000)
* Euler's Elastica and Curvature Based Inpainting (April, 2001)
* Landmark Based Inpainting from Multiple Views (March, 2002)
BV Image Model and Inpainting of Noisy Blurred Images 
From the abstract: What
we believe images are determines how we take actions in image and lower-level
vision analysis. In the Bayesian framework, it is manifest in the importance
of a good image prior model. This paper intends to give a concise overview
on the vision foundation, mathematical theory, computational algorithms,
and various classical as well as unexpected new applications of the BV (bounded
variation) image model, first introduced into image processing by Rudin,
Osher, and Fatemi in 1992 [Physica D, 60:259-268]. (By
Chan and Shen
, March, 2002: On the Role of the BV Image Model in Image Restoration
. Dedicated to Stan Osher on the occasion of his 60th birthday.) 
|
|
Mumford-Shah-Euler Image Model and Inpainting
From the abstract: Image
inpainting is an image restoration problem, in which image models play
a critical role, as demonstrated by Chan, Kang, and Shen's recent inpainting
schemes based on the Total Variation and elastica image
models. In this paper, we propose two novel inpainting models based on the
Mumford-Shah image model, and its high order correction -- the Mumford-Shah-Euler
image model. We also present their efficient numerical realization based
on the Gamma-convergence approximation of Ambrosio and Tortorelli, and a
conjecture of De Giorgi. (By Selim Esedoglu
and Jianhong Shen , European J. Appl. Math., 13, pp. 353-370, 2002.)
|
|
Local Inpainting Models and TV (total variation) Inpainting
As an ancient painting gets older, on certain regions, the pigments
start to fall off the canvas, and the painting becomes incomplete. The
human work of filling in the missing parts of the painting is called
"inpainting,"
as first introduced to image processing by Bertalmio, Sapiro, Caselles
and Ballester at University of Minnesota. Digital inpainting has much wider
applications in image processing and computer vision. Inspired by the work
of Bertalmio et al. (1999), we intend to develop general inpainting models
for non-texture images. In smooth regions, inpaintings are connected to
the harmonic and biharmonic extensions and inpainting orders are defined
and analyzed. For inpaintings involving the recovery of edges, we propose
a variational model that is closely connected to "Total Variational"
denoising
and debluring model, invented by Rudin, Osher, and Fatemi (Phys. D, 50, 1992).
Such a model and its algorithm intrinsically combine the denoising and
inpainting processes. In other words, our inpainting scheme is robust to
noise, and thus insensitive to pixel values. This work is also closely
related to disocclusion in computer vision by Nitzbeg, Mumford, and Shiota
(1993), and by Masnou and Morel (1998). (from CAM 00-11 Abstract, March 2000)
Report by Chan and Shen
[SIAM Appl. Math, 62(3), 1019-1043, 2001]. (Wonder how this famous
Kanizsa's
"Entangled Man" illusion is related to the inpainting model? Find the answer
in the paper.)
|
|
|
|
|
|
Non-Texture Inpainting by Curvature-Driven Diffusions (CDD)
We propose a new inpainting model based on the diffusion mechanism as
inspired by our previous work on the TV inpainting, aiming at realizing
the Connectivity Principle in vision psychology. Since in this new
diffusion model, the conductivity
coefficient depends on the curvature of the isophotes, we call such new
diffusions Curvature-Driven Diffusions(CDD), as contrast to
other diffusion models (such as the celebrated Perona-Malik)
prevailing in image and vision analysis. (from
CAM 00-35 Introduction, Sep 2000)
Report by Chan and Shen
. [J. Visual Communication and Image Representation, 12(4), 436-449, 2001]
|
|
|
Euler's Elastica and Curvature Based Inpaintings
| Image inpainting is a special image restoration problem for which image prior models play a crucial role. Euler's elastica was first introduced by Mumford to computer vision as a curve prior model. By functionalizing the elastica energy, Masnou and Morel proposed an elastica based variational inpainting model. The current paper is intended to develop its mathematical foundation, study its properties and connections to the earlier works of Bertalmio, Sapiro, Caselles and Ballester and Chan and Shen and construct computational schemes that are based on numerical PDE's, instead of the dynamical programming algorithm, which imposes topological constraints on inpainting domains. (from CAM 01-12, April 2001). Report by Chan, Kang and Shen [SIAM J. Appl. Math., 63(2): pp.564-592, 2002] |
|
Landmark Based Inpainting from Multiple Views
From the abstract : Most existing inpainting algorithms are local in nature and extrapolate information from neighboring pixels into the inpainting regions. In this paper, we are interested in the inpainting problem where the missing region are so large that these local inpainting methods fail. As an alternative to the local principle, we assume that there are other images with related global information to enable a reasonable inpainting. These additional images could be from a movie sequence, an image of the same object from a different time and a different viewpoint, or an image of a similar object.
Our method has roughly three phases: landmark matching, interpolation, and copying. For the landmark matching, modified shape context information is used to exploit the global information. Then matched information is interpolated (and regularized) using thin plate splines. Finally, we copy the information from one image to another. Using landmark matching and interpolation, allows the missing regions to be significantly larger compared to the local inpainting methods, and can be used when the object is distorted from one image to another. The experimental results are promising. (from CAM 02-11, March 2002) Report by Kang, Chan and Soatto [submitted to IEEE PAMI 2002]. Short version is also available at CAM 02-31, Proceedings of 3DPVT, June 2002
Reports on Image Restorations
People