InpaintDelogo - advanced logo removal script

[VoodooFX]

##================================================================================================##
#                                                                                                  #
#             ____                  _       __  ____       __                                      #
#            /  _/___  ____  ____ _(_)___  / /_/ __ \___  / /___  ____  ____                       #
#            / // __ \/ __ \/ __ `/ / __ \/ __/ / / / _ \/ / __ \/ __ `/ __ \                      #
#          _/ // / / / /_/ / /_/ / / / / / /_/ /_/ /  __/ / /_/ / /_/ / /_/ /                      #
#         /___/_/ /_/ .___/\__,_/_/_/ /_/\__/_____/\___/_/\____/\__, /\____/                       #
#                  /_/                                         /____/                              #
#                                                                                                  #
##================================================================================================##


#  InpaintDelogo is an advanced logo and watermark removal function using inpainting and deblending
# with an adjustable fine process to hide artifacts and get best delogo results.
# Can remove opaque, transparent, semi-transparent and some dynamic logos from video.

#  Inpainting refers to the application of sophisticated algorithms to reconstruct of lost
# or deteriorated parts of images or videos. 


##================================================================================================##

Someone asked for my delogo script, so here it is: InpaintDelogo() is user friendly script for AvsInpaint plugin (plugin created by Wolfgang Boiger).
Why another script for AvsInpaint!? Because other scripts - rm_logo & InpaintFunc are both semi-broken, slow inpainting, with subpar to unusable results.

AvsInpaint is a great plugin, questionably the best tool atm to remove logos. With InpaintDelogo you can expect better delogo results than from any other tool.
InpaintDelogo can save your time by generating a logo mask automatically, get Color/Alpha masks for deblending in a second, 3x times faster inpainting than with previous user friendly scripts.

Issues with previous scripts for AvsInpaint:

InpaintFunc Deblend: Results from poor to unusable. As in practice you can't get good masks by analyzing random frames. 
InpaintFunc Both:    Broken. Results same as Inpaint, simply there is no code to do it.
InpaintFunc Inpaint: Works. No controls. Very slow.
InpaintFunc PP:      Subpar. Broken code creates edges and PP=3 can't solve it.

rm_logo Deblend:     Results from poor to unusable. Same as InpaintFunc.
rm_logo Both:        Broken. Does both but can't do PP as there is no masks to do it.
rm_logo Inpaint:     Works. Very slow.
rm_logo PP:          Works (not for both).

Benchmarks (in brackets - fps with all new features On. rm_logo is modded for speed or it wouldn't be able to run test at all, both rm_logo & InpaintFunc runs at ~same speed. ):

                 InpaintDelogo v1.11      rm_logo v0.6_mod/InpaintFunc v1.15
180x180(pre cut):
Deblend             702 fps (153 fps)         587 Fps
Inpaint              33 fps                    10 fps
Both                 47 fps                    10 fps

1920x1080:
Deblend             121 fps (60 fps)          121 Fps
Inpaint              12 fps                     4 fps

Basic short guide ( things you need before starting InpaintDelogo() ):

0) Read help/manual(in script) about parameters mentioned below.
1) Run this function to get "Loc" values: InpaintLoc(Loc="100,100,-100,-100") .
Adjust "Loc" crop parameters around logo (aka "Left,Top,-Right,-Bottom"). Use even(mod2) numbers.
2) Get frames numbers for manual "Analyze" (you can skip it for slower & lower quality job).
3) Prepare the "mask" manually, or start the function with "Automask=1".
4) Now you can add this stuff from above to InpaintDelogo() function and run it.

Basic parameters:

InpaintDelogo( mask="c:\test\mymask.bmp",
\ Automask=0, aMix=0, Loc="100,100,-100,-100",
\ Mode="Inpaint",
\ Analyze=1, FrB=0, FrW=0, FrS=0)

Full parameters :
(not recommended to keep it all in your script if you don't know what you are doing,
you can, as script will silently ignore illogical param combos or will give some defined error msg.)

InpaintDelogo( mask="c:\test\mymask.bmp",
\ Automask=0, aMix=0, Loc="100,100,-100,-100",
\ Mode="Inpaint", Analyze=1, FrB=0, FrW=0, FrS=0,
\ Inflate=1, Deep=2, ReAnalyze=0,
\ Interp=2, IntSpd=1, InterpM=0, iTune=0, iTinflate=0,
\ dPP=3, dPPm=3,
\ oPP=5, oPPm=3,
\ diPP=5, diPPm=3,
\ EdgePP=0, EdgePos=0, EdgeWide=1,
\ Turbo=0, prePP=0, KillShape=0, oTune=0, oTinflate=0,
\ DynMask=0, DynTune=0, DynInflate=0, maskPatch="", KillNoise=0,
\ Dyn3Seq=300, ClpBlend=0, DynPostTune=0, Dyn3buffer=1,
\ GrainPP=0,
\ Sharpness=45, PreBlur=6.5, PostBlur=4.0, Radius=8.0,
\ Show=0, mShow=0, rSize=4)

Version history:

# Version 1.11 : 2019/08/31.  First public release.
# Version 1.12 : 2019/09/06.  Various tweaks. New parameters. "Inflate" and "Deep" changed.
# Version 1.13 : 2019/12/08.  Allow -ve LogoX,LogoY, and +ve LogoW,LogoH in Loc string (by StainlessS).
# Version 1.14 : [non public] "Fr1"/"Fr2" are renamed to "FrB"/"FrW". "GrainPP" is off by default.
# Version 1.15 : [non public] New mode for "InterpM", (-1) takes mask from new "iTune" parameter.
# Version 1.16 : [non public] New "diPP" & "diPPm" parameters.
# Version 1.17 : [non public] New "iTinflate" & "oTinflate" parameters. Interp/iTune mask to "mShow".
# Version 1.18 : [non public] New "EdgePP", "EdgePos" & "EdgeWide" parameters. EdgeMask to "mShow".
# Version 1.19 : [non public] New "IntSpd" parameter for "Interp" & the slow modes for "EdgePP".
# Version 1.20 : [non public] "Turbo" presets moved from soft to hard. iTune & Edge masks to "Show".
# Version 1.21 : [non public] Added support for the mask clips and a dynamic mask. 
#                             Added basic auto generation of a dynamic mask (aka "DynMask=2").
#                             New parameters: "DynMask", "DynInflate", "DynTune", "KillNoise", "maskPatch".
# Version 1.22 : [non public] Optimized speed for a dynamic mask, MaskTools2 required.
# Version 1.23 : [non public] Added auto generation of a dynamic mask with ClipBlend (aka "DynMask=3").
#                             New parameters: "Dyn3Seq", "ClpBlend", "DynPostTune", "Dyn3buffer".  
# Version 1.24 : 2020/11/28.  Various tweaks. "Show" 3 & 4. Updated the manual with the new stuff.

Requirements:

# Requirements   : Avisynth+ ( https://github.com/AviSynth/AviSynthPlus ) or Avisynth v2.6.
#                  AvsInpaint ( https://github.com/pinterf/AvsInpaint ).
#
# Optional requirements:
#                : GrainFactory3 (for "GrainPP", http://avisynth.nl/index.php/GrainFactory3 ).
#                : MaskTools2    (for "DynMask>0", https://github.com/pinterf/masktools ).
#                : ClipBlend     (for "DynMask=3", http://avisynth.nl/index.php/ClipBlend ).
#                : RequestLinear (for "DynMask=3", http://avisynth.nl/index.php/TIVTC ).

Link to AvsInpaint's Doom9 thread:
https://forum.doom9.org/showthread.php?t=133682

Download links:

[VoodooFX]

Comparisons with rm_logo and InpaintFunc:

Easy logo to deblend:
Original (zoomed in few times):


Deblended: (1-InpaintDelogo, 2-InpaintFunc, 3-rm-delogo)




Inpainted: (1-InpaintDelogo, 2-InpaintFunc, 3-rm-delogo)





Both logo:
1-Original, 2-InpaintDelogo, 3-rm_logo:





Hard logo to deblend:
Deblended:
1-Original, 2-InpaintDelogo, 3-rm_logo/InpaintFunc(they look similar):

[johnmeyer]

Interesting work. Thanks for doing that.

I've spent a lot of time using Delogo inside of Virtualdub, and even created a YouTube tutorial for it:

Tutorial on Delogo

I did that tutorial in response to a long thread about how to eliminate a sensor defect from a camera. This is a considerably easier job than logo/bug removal and one that Delogo was able to do with near-perfection.

I have not used any AVISynth logo removal tools, but I have used all the features of Delogo, including training it for removal of semi-transparent bugs, and specifying frame ranges for TV shows where the logo comes and goes, or moves around.

For me, the biggest need in advancing the state of the art is the blending algorithm. When the background is uniform and dark, such as the edges of a stage performance where the talent is brightly lit, but the wings of the stage are dark, the removal can be perfect. However, if the video was shot outdoors in the daylight, and the camera pans, you can get some pretty annoying results. For some videos, I just give up and leave it as is because the artifacts from the removal are worse than looking at the bug.

The problem is that the inpainting is nothing more than a blur function.

I know nothing about the technology of the existing AVISynth DLL you are using because I haven't used it, but the example above of the "Hawe Knho" shows the Inpaint blending looking pretty similar to Delogo. This is not meant as a criticism, but instead as a way to lead to my main question:

Is there a reason why I would want to use this instead of continuing to use Delogo? What are its advantages?

I am always looking for something that works either better or faster. Delogo is plenty fast, and it is also easy to use because it lets me tune the mask parameter interactively using the VD "preview" display. I want something that can make the logo disappear, with almost no residual evidence that it was there.

Is this possible?

I think it might be. If I were going to do development in this area, I'd look at how RemoveDirt does its painting. I just did a restoration of one of the most spectacularly degraded movies I've ever dealt with, a one-of-a-kind 16mm print of a 1963 AFL football game. It had been badly stored, and the emulsion stuck to the backing on the adjacent film in the reel, so once every revolution of the big film reel, there were huge (1/3 of the frame) chunks of mottled image. Where the emulsion had been completely removed, and there was no image, I had to reconstruct by hand from adjacent frames. However, when it was just a big stain, I used a heavily modified version of RemoveDirtMC to fix it, and when it worked, the reconstructed frame was near-perfect.

I haven't posted the results of this work, because it is copyrighted, but I did post a very short before/after of some Polavision restoration. This instant movie film from Polaroid had much of the same problem, because the film was developed in the cassette and then stayed there for 40+ years and, because of the residual chemicals, stuck together and, when unwound after all those years, the emulsion would come off, resulting in a black spot on one frame, and a matching blank spot a few frames later. Here is a few "before/after" frames from that work. You'll need to freeze at a random frame to see the results of what RemoveDirt was able to do. I did no manual work at all; it was all done by RemoveDirt.

Before After Polavision

Whatever inpainting algorithms RemoveDirt uses might provide some ideas for how to do better logo removal.

[StainlessS]

I think that you will find that RemoveDirt uses temporal info from adjacent frames to fix spots/dirt, logo is entirely different being also on adjacent frames.

Looks nice VDX.

[johnmeyer]

Originally Posted by StainlessS

I think that you will find that RemoveDirt uses temporal info from adjacent frames to fix spots/dirt, logo is entirely different being also on adjacent frames.

Thanks, StainlessS. I wondered to myself as I was writing that last post whether that might be the case and, if that is the secret, you are right: it would be mostly useless for logo removal.

Having said that, in theory (maybe not in practice) you could use motion estimation from adjacent frames to predict what should be under the logo in the current frame and come up with a better replacement. It might work for my example of what happens when the camera pans. However, in practice, it probably wouldn't be useful, and would be of no help at all unless there was motion going through/under the logo.

P.S. The other area for improvement would be semi-transparent logos. In that case you DO have information on what is behind the logo. My experience, even with Delogo, is that if the logo is really faint, so you can see lots of detail behind it, you can get remarkably good, artifact-free removal.

[VoodooFX]

Originally Posted by johnmeyer

Is there a reason why I would want to use this instead of continuing to use Delogo? What are its advantages?
I am always looking for something that works either better or faster.

Delogo is pretty outdated, there were tools that perform way better, and InpaintDelogo does better than those other ones. Let me know if you find tool/script doing better job than InpaintDelogo.

Originally Posted by johnmeyer

The problem is that the inpainting is nothing more than a blur function.

You are mixing up something, it's not a "blur" function or a temporal thing like RemoveDirt.
Here is good starting point to read about it - Inpainting

Originally Posted by johnmeyer

Having said that, in theory (maybe not in practice) you could use motion estimation from adjacent frames to predict what should be under the logo in the current frame...

It sounds good only in theory, in practice it would be very complicated task to implement something to fit various cases of motion, actually it was implemented and was called "msu" something.

[StainlessS]

John, the S_Exlogo() thing I did [ based on DeKafka() ], is purely blurring the logo [using data from spatially outside of the logo], but not the AvsInpaint.dll based scripts [as is this one],
however, S_Exlogo() is easy to set up and goes like the clappers, and so long as you are easily pleased, is quite good [a man on a galloping horse might not notice any remaining logo].

MC based delogo [as with mvtools] is doomed to failure, methinks.
[neural network stuff might at some future date be much better than today, but would likely be a somewhat unstable [although maybe less so if 3D neural networks],
ExInpaint [another dll] uses something along those lines [purely spatial, I think] and is a little temporally unstable but good for single frames (see the demo image in docs)]

EDIT: Think ExInpaint is using the "Exemplar-based image inpainting" in above link, seem to remember the word 'Exemplar' being used in the docs [and source for the plugin name].

from above VDX link

Exemplar-based image inpainting attempts to automate the clone tool process. It fills "holes" in the image by searching for similar patches in a nearby source
region of the image, and copying the pixels from the most similar patch into the hole. By performing the fill at the patch level as opposed to the pixel level, the
algorithm reduces blurring artifacts caused by prior techniques.

EDIT:

it was implemented and was called "msu" something.

MSU Subtitle and Logo Remover:- http://forum.doom9.org/showthread.php?t=95955

Originally Posted by krieger2005

Use this

global VDub = "C:\Programme\AviSynth 2.5\VDub\plugins"

LoadVirtualdubPlugin(VDub+"\MSU_DeLogo.vdf",       "_MSUDeLogo")

The open VirtualDub. Add to the filters the MSUDelogo-Filter. Tweak it, so it remove your Logo fine. Go to
FILE ->
SAVE PROCESSING SETTINGS...

and save them to a file. Open The file in a texteditor and look for line like this one:

VirtualDub.video.filters.Add("MSU Subtitle&Logo Remover");
VirtualDub.video.filters.instance[0].Config(0, 0);

Now copy allparameters in Config and use the to Call the Filter in Avisynth this way:

ConvertToRGB32._MSUDeLogo(0, 0)

I tried the filter but does not get good results with it. It is also very slow. Have you better results with this filter?

Many MSU Vdub filters do NOT export some of the settings, [described in code above as Config] and so cannot be used properley in Avisynth, [settings only exist when set up within VDub, and
cannot be saved and reused at a later date, even within VDub], dont know if MSU_Delogo is one of those which do not save ALL settings.

EDIT: Dont know/recall what method AvsInpaint dll is using [the one used by this script].

[VoodooFX]

Originally Posted by StainlessS

the S_Exlogo() thing I did [ based on DeKafka() ], is purely blurring the logo...

I was thinking to add Very Fast mode where s_Exlogo could do post-processing, could be useful for big logos where there is no point to use slow inpainting. But forgot about it as I didn't wrote that to todo list .
I strikethrough that list today, only one line left - "moving logo", it should be doable in theory but could be over my head, maybe I'll replace it with "very fast mode".

Originally Posted by StainlessS

MSU Subtitle and Logo Remover...

I looked at MSU site, it is called MSU Logo Remover 2.xx, MC implementation is broken, no open source...

Originally Posted by StainlessS

Dont know/recall what method AvsInpaint dll is using [the one used by this script].

Ideas are implemented from these papers:

Folkmar Bornemann, Tom März: Fast Image Inpainting Based on Coherence Transport.
Joachim Weickert: Coherence-Enhancing Shock Filters.
Alexandru Telea: An Image Inpainting Technique Based on the Fast Marching Method.
James Sethian: Fast Marching Methods.

[StainlessS]

I should go back to my current mod for s_Exlogo(), It stalled a bit when I bumped into a bug in r2772, fixed now in bugfix avs+ test build (r2915), but been a bit busy of late.

I've seen some delogo stuff that was somewhat like s_ExLogo(), but seems to take most data from mostly one side (inner most long side), but partially mixed with other sides,
it in some cases was way better than s_ExLogo, I could not figure out how it worked. [sending you a link, not suitable clip for general viewing].

[VoodooFX]

Originally Posted by StainlessS

I've seen some delogo stuff that was somewhat like s_ExLogo(), but seems to take most data from mostly one side (inner most long side), but partially mixed with other sides,
it in some cases was way better than s_ExLogo, I could not figure out how it worked. [sending you a link, not suitable clip for general viewing].

That looks pretty simple, take 2 pixels at the top resize them to the size of the hole. I don't see any interpolation from other sides.

[StainlessS]

Oh, simple as that, I guess I was seeing stuff that wern't there.
Thanks awfully

EDIT: sometimes less is more.

[VoodooFX]

I just tested it to be sure. Sometimes in original I see some discrepancies, but I don't have any logical explanation, maybe some compression/resize artifacts maybe broken delogo code. Like this one:

2x Size
1) original (SD), arrows to some artifacts, they appear and disappear without any apparent reason, but mostly it looks same as "3".
2) s_exLogo, delogo of "original", sides and bottom is blocked from interpolation, 4px left at top, "Amount=256".
3) 2px from the top resized.

[manolito]

You guys made me curious so I did some tests comparing s_ExLogo to the old LogoAway plugin by Krzysztof Wojdon. I did not test InpaintDelogo for several reasons (too complicated, I hate plugins with dozens of params, have to select frames and create masks manually, requires AVS+, no option for logos over black borders). Here are the results:
https://www.sendspace.com/file/8nvhi6

For s_ExLogo I used the default parameters, for LogoAway the UGLARM method was used with the blur slider set to max (all the way to the right).



LogoAway also has a much faster XYBlur mode which creates results very similar to s_ExLogo. But UGLARM does look so much better in almost every situation that I have been using this mode exclusively over the last couple of years.

Now I would like to challenge VooDooFX if his script can create a better looking result. My requirements are pretty simple:

1. Just 1 mode for opaque logos. I do not care for semi-transparent logos.
2. No manual mask creation. All I want to provide to the plugin are the logo coordinates.
3. I do not mind if the logo area gets blurred strongly. All I want is that my attention is not taken away from the movie itself, so the delogo'd area should be as unobtrusive as possible, mainly I hate when this area shows strong flickering.
4. Easy to use and reasonably fast.

Tough challenge, I know. For now I will stick with LogoAway, s_ExLogo is no serious contender IMO.


Cheers
manolito

[StainlessS]

Mani, for s_ExLogo, sPow=1.0 (linear) should be about same as your XYBlur mode [I think].

[VoodooFX]

Originally Posted by manolito

Now I would like to challenge VooDooFX...

Do you want some fast preset? I'll look what can be done. Can you PM link to that unresized logo?

[manolito]

Originally Posted by VoodooFX

Do you want some fast preset? I'll look what can be done. Can you PM link to that unresized logo?

Of course faster is better, if the result does not suffer too much.

For the "unresized logo" I don't know what you mean. The original source is a German DVB-T2 capture which means 1080p @ 50fps. Applying logo removal at this resolution is forbiddingly slow, and since my target format is SD (704 x 396) I perform my logo removal after resizing which is way faster.

So you need to use the "Source.mkv" file in my upload as your source for the logo removal.

[VoodooFX]

It's just for personal tests, logo is "both" and I don't have much logos to test that mode on. If you have source then I would be happy with just the logo cut with surrounding of ~80 px (no need black bar), ~5-10 mins, ~Lagarith YV12, should include frame where logo is in a black background.

[VoodooFX]

New version 1.12 attached at original post.
Various tweaks for speed and quality. Changed "Inflate", "Deep" now is not re-analyzed on every load, selectable number of passes for "Deep".
New "Turbo" presets parameter for fast and blurry logo removal, few other parameters.

v1.12 Deblend, 2x zoom:


v1.11 Deblend, 2x zoom:

Originally Posted by manolito

You guys made me curious so I did some tests comparing s_ExLogo to the old LogoAway plugin by Krzysztof Wojdon. I did not test InpaintDelogo for several reasons (too complicated, I hate plugins with dozens of params, have to select frames and create masks manually, requires AVS+, no option for logos over black borders). Here are the results:
https://www.sendspace.com/file/8nvhi6

For s_ExLogo I used the default parameters, for LogoAway the UGLARM method was used with the blur slider set to max (all the way to the right).

[url=https://i.postimg.cc/9fFcJcFF/LogoAway.png[/url]

LogoAway also has a much faster XYBlur mode which creates results very similar to s_ExLogo. But UGLARM does look so much better in almost every situation that I have been using this mode exclusively over the last couple of years.

Now I would like to challenge VooDooFX if his script can create a better looking result. My requirements are pretty simple:

1. Just 1 mode for opaque logos. I do not care for semi-transparent logos.
2. No manual mask creation. All I want to provide to the plugin are the logo coordinates.
3. I do not mind if the logo area gets blurred strongly. All I want is that my attention is not taken away from the movie itself, so the delogo'd area should be as unobtrusive as possible, mainly I hate when this area shows strong flickering.
4. Easy to use and reasonably fast.

Tough challenge, I know. For now I will stick with LogoAway, s_ExLogo is no serious contender IMO.


Cheers
manolito

a) "too complicated" - It's relatively simple.
b) "dozens of params" - It's not mandatory to use them all, you can use only 2 parameters if you want. LogoAway has dozens parameters too...
c) "have to select frames and create masks manually" - Not mandatory. Quote from op - "can save your time by generating a logo mask automatically".
d) "requires AVS+" - Not mandatory.
e) "no option for logos over black borders" - Does it prevent you to delogo? You said it has too many params already.

1) There is already such mode.
2) Mask generation is automatic already.
3) You can do that already. Why you don't want to use deblend? It would remove your logo almost without a trace.
4) It is easy already. Added new fast "Turbo" presets parameter in v1.12.



Whole comparison video: https://we.tl/t-kcAI66C8GQ

[manolito]

Thanks for these comparisons...

Especially the "Turbo=1" mode impresses me. In most cases it looks just as good or even better than the slower "Turbo=3" mode. This mode is something I would really be interested in...

The speed comparison with LogoAway is a little unfair. I use an "Active Area" approach where VDub just processes the actual logo. It was inspired by Didées DeLogo script, but a little bit expanded and added exception handling with the help of gavino. Makes it much faster.

Your new version is not approved yet by a mod, please upload it somewhere else.

For the AVS+ requirement this is what you say in the first post of this thread:

Requirement:
Avisynth+ r2915 or newer ( https://github.com/pinterf/AviSynthPlus )

Please tell me what I have to change under classic AVS 2.60.

And then I would like to know how you can preview your results. For LogoAway it is easy: In VDub you can visually see the different results when you change any of the parameters. In AviSynth would I have to use AVSP(mod) to interactively see the results when changing parameters?

And for treating logos which are partly over letterboxed (or pillarboxed) areas do you have any special treatment for such logos? LogoAway and s_ExLogo do have provisions for such logos.

[manono]

Originally Posted by manolito

Your new version is not approved yet by a mod...

I'll do it immediately.

Please tell me what I have to change under classic AVS 2.60.

Me too. I have no interest in AviSynth+ yet and do a lot of logo removal using InPaintFunc.