ExInpaint plugin - Exemplar-Based Image Inpainting

[Fizick]

Exemplar-Based Image Inpainting - removing large objects from images.

Selected objects has been replaced by a visually plausible background that mimics the appearance of the source region. Unlike other inpainting tools (logo removers), this filter does not blur image "holes". It tries to replace holes by similar texture elements (patches, exemplars, blocks) from frame image area, taking into account the image special points (with estimated confidence and priority). Block inpainting is started at hole border and propagates to its internal area, with isophote as a direction.

Algorithm is based on article: Object Removal by Exemplar-Based Inpainting. A. Criminisi, P. Perez, K. Toyama. In Proc. Conf. Comp. Vision Pattern Rec., Madison, WI, Jun 2003. http://research.microsoft.com/vision/cambridge/papers/Criminisi_cvpr03.pdf

Code is based on project of Qiushuang Zhang
http://www.cc.gatech.edu/grads/q/qszhang/project/inpainting.htm

First version of ExInpaint plugin is slow, not optimized, especially for large radius.

It is spatial filter developed for still images. Only current frame is used. So, we can see temporal instability in this version.

Potentially it may be used in video for:
-film restoration (spots and scratches removal);
-logo removal;
-filling of empty areas (borders) after motion stabilization (compensation);
-filling damaged areas after VHS capture;
-frame increasing without resizing :-)

It can be implemented differently, may be as a part of MVTools. Current version is mostly for experiment and testing.

For logo removing and similar tasks you can also try XLogo, AVSInpaint, Greycstoration inpainting filters.

[Mug Funky]

this is pretty amazing actually :|

i think i might hack this to do scratch concealment.

[edit]

seems to be RGB only?

it works really well! try this:

stackhorizontal(last.crop(0,0,64,last.height),last.blankclip(width=4,color=$0000ff),last.crop(68,0,652,last.height))#blankclip(width=652,color=$000000))

exinpaint(last,last,radius=16,xsize=3,ysize=3,color=$0000ff)

the stackhorizontal is to simulate a film scratch. it could be replaced by an automated scratch detector using some masks.

[Selur]

little question:
Am I overlooking something? (can't find a avisynth filter, just a download for papers and a command line executeable)

btw. tested the executeable with a little smaple I tool form google picture search:
Original: http://www.postimage.org/image.php?v=aVV6qzA
Masked: http://www.postimage.org/image.php?v=aVV6Ay9
Result: http://www.postimage.org/image.php?v=aVV6Fxr

Cu Selur

[foxyshadis]

Fizick's plugins (http://avisynth.org.ru/fizick.html)

[Selur]

thanks :)

http://avisynth.org.ru/ seems to be down atm, would be nice if someone who got the filter&co could up it to megaupload or something like it. ;)
-> works again :)

[Konrad Klar]

@Fizick
Thanks for plugin.
Two questions:
1. Does it make difference in speed which colorspace is used (RGB24, RGB32, YV12)?
2. Is there any chance in near future for multithreaded version of ExInpaint? It does not work (at least for me) with MT() nor with SetMTMode(). Maybe because it takes more than one clip as argument.

[Fizick]

1.1 You can measure it yourself :)
I almost finished next speed-optimized version (with SSE MMX).
RGB32, RGB24, YV12 are ready. But YUY2 optimizing is not trivial task in MMX. :(
2.1. I do not develep m.t. filters (do not have hardware).
2.2. exinpaint can be used with some tricks with mask.
2.3. But according to algo the filter inpaints one block per a time (iteration).

[Fizick]

I found that I developed version 0.2 at 26 January 2008 :)

-even patch sizes instead of odd (with some approriate changes)
-assembler MMX optimization of PatchTexture
-use SAD instead of SSD
-local max priority updating
-option to estimate search radius automatically
-added YUY2 format and YUV24
-added dilation of mask

But I am afraid this version is not finally finished (beta or even alpha stage).

[TSchniede]

Some sort of overlapping for block prevention would be nice too.

And some addition for the documentation: the color for yv12-mode is based on the binary representation of the mask color NOT the converted color of RGB. So $FFFFFF is quite colorful instead of white.

Example:

enlargedclip=clip.converttorgb().addborders(8,8,8,8,color=$FFFFFF).converttoyv12(matrix="pc.601")
exinpaint(enlargedclip,enlargedclip,color=$FF8080)

remark: the color conversion introduces a 1-pixel line around the image which will not get inpainted.This is only for demonstration.

[AnnaFan777]

Thank you, it's very promising.

Hope the temporal instability can be fixed soon.

btw, the border of inpaint area look sharper, somewhat
noticeble if you look carefully, now if only we can
blend the adjacent pixels a bit to make it more natural.

[Fizick]

Today I found that I forget to release version 0.2 beta in april (again!)
Prabably becourse of not many responces....

So, released now.

[mikeytown2]

I've used it a couple of times on photos, thanks for the update!

[Revgen]

I'm trying to figure out how to remove a logo with this tool. I've yet to figure out how.

Any clues or tips would be appreciated.

Here's a sample with a logo I'm trying to remove.

http://www.sendspace.com/file/5atrwo

[mikeytown2]

U try this?
http://avisynth.org/mediawiki/Rm_logo

ExInpaint is commented out, but you can uncomment it and try it out.

[Fizick]

Here is the message i have got recently from Jean (i have no clear answer to it):

... i'd like to ask you a bit about
exinpaint which i found in your plugins list. I really like that tool
and it produces good output; However, i wanted to ask you, how an
implementation into MVTools could look like (i'm new to avisynth, so
i'm not very skilled yet)

I used exinpaint to do the same in your example with some borders,
sadly every frame the the inpainting is changed, even if the edges are
very similar. Thats why i think an implementation into mvtools could be
very useful, i just have no idea how.

Let's imagine we have a video with 800x600px. Now we add 20px border at
the top, and 20px border at the bottom. Now i use exinpaint with 24px
top and 24px bottom to get the border's inpainted (produces a good
result so far) though on the next frame this inpainting changed while
the rest of the frame did not change. (there's maybe only 10% in the
whole frame changing, the only thing i can see is some dust in the
middle and a girl moving the mouth cuz of talking, - the dust doesn't
reach any of the borders, it's really in the middle) so, what i would
need, would be something to "stabilize" the inpainting. An idea i'd
have would be:


- Create a Mask in the same size around the mask (yeah, i'm
complicated explainer) so if we have a border of 24 px at the top, we
make a mask of it with 28 px which gets inpainted. Now we create a
second mask BELOW of it with 28px in which we check for motion - if
there's no motion use the inpainting of the previous frame. That's just
an example if we do something at the top. Same functionality should be
right, left and at the bottom, though individually (because, while at
the top nothing might change, at the bottom something could change, etc)

You think that could work? And you got any idea how to implement
something like that?

[Didée]

A black/white decision (either there is motion, or there is not) doesn't sound very promising. The forseeable result is false locking when the frame is slightly moving, with the funky garbling randomly kicking in when the holy motion threshold happens to be exceeded.

The better question is why ExInpaint delivers vastly different inpainting results when hardly anything changes between the frames ...

[chani]

Hey,

thanks for posting fizzick, that was me, with the mail, i couldn't post earlier here, because of the "you have to be 5 days registered to write"-rule. However, back to the problem/idea:

I also talked with manao and kuukunen on IRC, it came out that what i wanted to do is most likely not possible. Because i simply can't take the result of the previous frame (at least i have no idea how, i tried with imagewrite and imageread, though that didnt work because the image is only read once, and not on every request) i tried with trim, duplicateframe.. So.. There's some additional magic needed i guess :-)

Some people might say that this is useless at all, because it wouldn't give good results, however, the result's i've seen in my testing clip were good (especially if you just want to do that at borders. The less you want to add to the borders the better - This might be useful for turning 4/3 into 16/9 video. My idea was just to make masks around the borders, so that we have a left, top, bottom and right mask, and just checking whether there's motion in a mask - if there is, do inpainting, if not, use the previously inpainted one. Dunno. I weren't able to get something like that working and i was told it's impossible, so i gave up yet. Though if anyone else has any idea about it ... i'd be interested.

[Mug Funky]

i think the algo would need to be rewritten almost completely - we'd be adding a new dimension to it.

the idea of using mvtools' vector field (and associated confidence levels, scene detection, etc etc) is pretty exciting though. the algorithm could track it's choices of exemplars (is this correct terminology?) along a contiguous scene, and update them locally when there's large enough changes, then possibly revert back if the scene returns to how it was earlier...

i have no idea how to do this :)

exinpaint already works beautifully for removing film scratches, but must be combined with a very heavy temporal blur (mv'd of course) to become invisible. also it can't handle very thick scratches without becoming apparent.

i've saved the edges of quite a few films where the camera mag was unkind to them - those black vertical lines just vanish and leave no trace if you zero in well enough. i love it when clients think you've just performed an act of magic.

[Floatingshed]

This looks promising, can somebody please give me a few pointers to use it to remove film scatches? Thanks.