idea for mvtools based logo-removal

[E-Male]

ok, i got an other idea that i most likely won't be able to implement myself
this time it's about logo-removal (non-transparent logos)
i'll skip the logo detection/selection part and go to the actual removal
for static scenes the classic interploration from the edge pixels is fine
but when we have for example a pan that interploration looks bad IMO
so i think we could try something like that:

-get the motion vectors of the frame (or a part of it around the logo)
(forward and backward, 1 frame in both cases)
-get the "global motion" from them (i hope that term is right)
[i'd just averagre all but those that are of 0 length, also we must ignore those starting or ending in the logo area!!]
-now we change all vectors, using their original starting blocks but giving them the length and direction of the global motion
-now we check which of them point into the logo-area
-and finally we do this check for every block in the logo area:
if we have no vector pointing to the block leave it as it is (classicly interplorated or whatever)
if a forward (x)or a backward vector is pointing there, move the block there
if a forward and a backward vector is pointing there, move both blocks there and average them

so on pans (best case) the logo could be totally gone

i'll just leave that idea open for discussion here

if no-one else wants to do it and no-one has input in why it won't work at all i'll put it on my much too long list (read: next year)

[scharfis_brain]

this idea already has been implemented by MSU:
http://www.compression.ru/video/subtitles_removal/index_en.html
but it is for VDub only :(

[E-Male]

i know that one, it's vdub only (as oyu said) and AFAIK not open-source
having it in avisynth, for YV12 and under GPL would be nice

[Didée]

Thought about it several times, too. But (I think that) with the current featureset of MVTools, it is not possible to implement this idea: What lacks is the possibility to make a motion estimation with a "hole" in the clip, i.e. to not consider a given region of the source, and to interpolate vectors for this area from the vectors in the neighborhood, etc...


scharfis_brain:

Yep, they have ... something. Do you know of an example for a successful removal, apart from their demo clip?

[E-Male]

the "hole"-problem, yes, i've thought about that and in my first here post you get my primitive solution
maybe i can get into some more detail later

[Didée]

Yes I've read your thoughts above, and they're valid.

Still, there are no filters or commands in MVTools that would allow you any sort of manipulation of the motion vectors themselves ... at least not from the script level. You let search for vectors, you get vectors. After that, you're incapable of action.

If at all, one could try Depan: Blurrrrr the Logo, make de-panned clips, and copy the logo area from those onto the original. (Or the same with MVTools and *very* high vector coherence.)
But then some sort of decision is needed to check if the compensated data is valid at all - e.g. it could still contain parts of the logo (if the panning is slow), or could be b0rked in other ways.

I don't want to discourage you in any way. In contrary. It's just that we're thinking of how to cut down the tree, but have neither saw nor axe. We have a swiss army knife, but that's too small for a tree ...

[scharfis_brain]

pseudo Logo-quadrants. X==Logo, A..G intact image around it.
ABC
DXE
FGH

instead of blurring the logo (X), replace it with a blend of B, D, E & G
smooth the transitions to the intact image by overlaying it with a falling opacity.
Then run a motion vector search over it.

The motion vectors should be come out fine on pure background motion when no differentiol movements are present in the affected area.

Didèe, what do you think?

[Didée]

Grab an image editor and demonstrate this replacement for one single frame containing detail :)

[scharfis_brain]

Uhm. Me and image editing?
I wish I could!

I could try to script this...

[E-Male]

i know it's not possible on script-level (currently)
but i think it's possible in c with the mvtools code
i'm just hoping for someone more experienced than me to do it

[Manao]

As Didee said, the area with the logo must not be taken into account when doing the motion estimation.

I'd say a huge blur, only applied to that area, should do. It will still pin down the motion vectors around that area, but not as much as the real logo.

From now, the 'real' work begins. Motion interpolators - which aren't working quite as I would like because I kinda lost my interest in them ( too complicated - too slow ) - should be used to compensate both previous and next frame on the current one.

Then, of course, you're getting 3 clips : the blurred one ( which we will shamelessly assume to be the best spatial reconstruction ), and the two temporal ones.

Now, you just need a filter to decide, for each pixels, which is the best one ( simplest one : a median / an average ).

It can be improved by replacing the blur by an real inpainting filter. And, of course, it won't work because the motion interpolators aren't finished ( since, well, pure temporal interpolation is bad, so it also need an inpainter )

Oh, and I almost forgot : all that trouble will only be useful when there is motion in the clip, which isn't as frequent as it seems, alas.

So, all in one, efforts should be put into writing a good inpainter, and after that, why not, trying to create a temporal delogoer.

[morsa]

Anyone interested can take the inpainting module from restoreinpaint.
It is a .cpp.
At least by this time Bernard has converted everything to a modular structure.
So getting code from it should be much more easier than before.

[MfA]

It's only a bootstrap problem, after the first frame of a sequence you use the logo-removed frame as a reference.

[IanB]

Consider the rate at which the pan is happening in arbitary units of "Logo Width's per Frame". If it is less than 1 you will need to parse more frames to completely fill the logo'ed area.

i.e. if the pan is 0.25 LogoWidths per frame you will need to parse 4 frames.

IanB

[E-Male]

i'd always get blocks from the next and previouse frame
so you can, an the best case, average from 2 sources