InpaintDelogo - advanced logo removal & hardcoded subtitles extraction

[Reel.Deel]

Good job VoodooFX.

Your result vs "official" video (https://www.youtube.com/watch?v=saah9UTwaUI):

Frame 774 (688):
https://i.imgur.com/PcMvXF4.png https://i.ibb.co/N33vHQY/Clattanoia-Ox-T-688.png

Frame 1297 (1244):
https://i.imgur.com/K7FFllp.png https://i.ibb.co/PTLV7s2/Clattanoia-Ox-T-1244.png

[VoodooFX]

Good job VoodooFX.

Top delogo is not good there [for reason posted], it will be better in next version.
Bottom delogo could be better with this conditional feature (https://github.com/AviSynth/AviSynthPlus/issues/298), currently the detailed frames suffer a bit in favor of better looking the solid background frames.

That frame is not very "detailed" but in the middle I added how frame looks if "in favor of the detailed frames" case:

https://i.imgur.com/K7FFllp.png https://i.imgur.com/NZ4W7mB.png https://i.ibb.co/PTLV7s2/Clattanoia-Ox-T-1244.png

with this Blue Logo, i can't got logo mask correctly
Told you that video is not good for test/masking.
Learn to create basic masks yourself, it's MS Paint level skill... there are hundreds image editing programs, even Avs can be used to reproduce Mona Lisa. :D

[StainlessS]

Bottom delogo could be better with this conditional feature,
[Feature request] masked MinMaxDifference

Hows bout,

****************************************
******* MASKED Luma Y Functions ********
****************************************

Compile time/runtime functions, Planar, YUY2, RGB24 & RGB32. (RGB internally converted to YUV-Y).

The compiletime/runtime clip functions share some common characteristics:-

The 'n' arg is an optional frame number and defaults to 'current_frame' if not specified.
The x,y,w,h, coords specify the source rectangle under scrutiny and are specified as for Crop(), the default 0,0,0,0 is full frame.
If 'interlaced' is true, then every other line is ommited from the scan, so if eg y=1, then scanlines 1,3,5,7 etc are scanned,
if eg y=4 then scanlines 4,6,8,10 etc are scanned. The 'h' coord specifies the full height scan ie same whether interlaced is true
or false, although it will not matter if the 'h' coord is specified as eg odd or even, internally the height 'h' is reduced by 1
when interlaced=true and 'h' is even.

Matrix: Conversion matrix for conversion of RGB to YUV-Y Luma. 0=REC601 : 1=REC709 : 2 = PC601 : 3 = PC709,
Default = (Width > 1100 OR Height>600) then 3(PC709) else 2(PC601). YUV not used

The optional mask clip, governs which pixels are processed by the functions. Where a luma pixel in selected area of the the mask clip is
in range "MaskMin" to "MaskMax" inclusive, then those pixels will be processed. The Mask must also be Planar but not
necessarily the same colorspace as clip c, also must be same dimensions and have at least the same number of frames as clip c.
Calling without mask clip OR with MaskMin=0,MaskMax=255 will effectively ignore the mask and scan full x,y,w,h area.



RT_YPlaneMinMaxDifference(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,
bool "interlaced"=false,float "threshold"=0.0, int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns int value luma range (maximum - minimum difference) (0 -> 255) in frame(n+delta) for area x,y,w,h.
Threshold is a percentage, stating how many percent of the pixels are allowed below minimum or above maximum (ignore extreme pixels ie noise).
Threshold is % of valid mask pixels processed if Mask supplied. The threshold is optional and defaults to 0.0.


8 bit only.

[VoodooFX]

# Version 3.1 : 2022/10/11. Fixed: "Deblend" with positive "Analyze" was unreliable on the colored logos.

Issue was gross, but not so apparent on the long videos.

Original / pre-v3.1 bug Analyze=2 / deprecated Analyze=-2 / v3.1 Analyze=2 / + TriggerDynamic=1 hack / Reel.Deel's "official":
[the differences would be more visible without Interp & dPP blur]


https://i.imgur.com/1jE7FyX.png https://i.imgur.com/D9Ltq3D.png https://i.imgur.com/PcMvXF4.png https://i.imgur.com/KYbxaZf.png https://i.imgur.com/Gof9Sdp.png https://i.ibb.co/N33vHQY/Clattanoia-Ox-T-688.png


RT_YPlaneMinMaxDifference

You thought about everything when you made RT_Stats. :)

I wanted to do a check inside the logos too, now the suitable frames search is looking only at the outsides [managed to do it with some workarounds, but it can't be done for the insides].
Some unsuitable stuff can be only inside logo and it can "crawl" into the masks.

For example, trash burned inside logo alpha:
https://i.imgur.com/fKsudYc.png

[StainlessS]

You thought about everything
I think was Martin53 request.

[kedautinh12]

This is what you said: "masking with mspaint"??
https://m.youtube.com/watch?v=OtVR9_Ua-C8

[VoodooFX]

This is what you said: "masking with mspaint"??
https://m.youtube.com/watch?v=OtVR9_Ua-C8
Yeap, basic masking, just paint full white over an image and that's called a mask, InpaintDelogo will convert non full white pixels to black automatically.
MS Paint is OK to mask some squares, other shapes are easier with more sophisticated tools like Photoshop ect..

[kedautinh12]

Photoshop very complicated for me :D

[kedautinh12]

Red Logo HD:
https://drive.google.com/file/d/19iB1ttvhIPeZ1uUIo6544n9BnEmMQbTI/view?usp=sharing

[VoodooFX]

Photoshop very complicated for me :D
Here is mask for it: https://i.imgur.com/q121iFL.png
Top logo in "Daichi Miura" video is always in a solid background, just use Inpaint on it.
How that tv channel is called with "S" logo?

[kedautinh12]

Here is mask for it: https://i.imgur.com/q121iFL.png
Top logo in "Daichi Miura" video is always in a solid background, just use Inpaint on it.
How that tv channel is called with "S" logo?

Thank, i think i can use mask from other video i created successful but thanks for your Logo

Full name is SSTV (Blue Logo) and SSTV+ (Red Logo)

Edited: how about bottom letters??

[VoodooFX]

Red Logo HD:
SSTV+.v2.1080 (https://drive.google.com/drive/folders/1QWixC2n9xy6ntt9JYr8FCsl5T3VDZuVl?usp=sharing)
If you have many same logo clips then a better deblending mask could be build if you analyze them all. [sometimes there are not so visible differences and some clip can make it worse]

how about bottom letters??
If you can't deblend it then inpaint it too.

[kedautinh12]

If Daichi Miura hard to got mask, you can try with this for (1920x1080i):
https://drive.google.com/file/d/1Dc_xJLKtAr0Oz_JWPmFJ7npt9j4874No/view?usp=sharing

[VoodooFX]

If Daichi Miura hard to got mask, you can try with this for (1920x1080i):

Added as "SSTV.v2.1080" to the logo collection folder.
Still there is no point in deblending top "Daichi Miura" logo as inpainting would be much better on this video.

Try that video with InpaintDelogo from temp branch (https://github.com/Purfview/InpaintDelogo/tree/temp), Inpaint should be very fast on it.
Use Optimize=1 to enable faster conditional inpainting, visually it shouldn't look worse than with normal inpainting. [Not tested, atm just quick theoretical implementation]

PS:
Btw, ebmp mask can be used for Inpaint mode too.

[kedautinh12]

Added as "SSTV.v2.1080" to the logo collection folder.
Still there is no point in deblending top "Daichi Miura" logo as inpainting would be much better on this video.

Try that video with InpaintDelogo from temp branch (https://github.com/Purfview/InpaintDelogo/tree/temp), Inpaint should be very fast on it.
Use Optimize=1 to enable faster conditional inpainting, visually it shouldn't look worse than with normal inpainting. [Not tested, atm just quick theoretical implementation]

PS:
Btw, ebmp mask can be used for Inpaint mode too.

How about Turbo=-2?? Result will better than -1 very much?? And optimise=1 only use for Inpainting?? How if it's use for deblending and both mode??

[VoodooFX]

How about Turbo=-2?? Result will better than -1 very much??
You tell me. "-2" I not tested at all, with "-1" I came up only from one video, with "0" I came up as generally best after a lot of SD de-logos.

And optimise=1 only use for Inpainting?? How if it's use for deblending and both mode??
Currently should work only in Inpaint mode, in the future something similar can be implemented for other modes.
Anyway, Optimize is not tested at all, so you tell me if there is increase in speed and if there is no ~visible quality loss.

EDIT:
Btw, Inpaint mode should benefit from Prefetch().

[kedautinh12]

Ok, thanks for your help, i will test more to understand your scripts :D

[VoodooFX]

How about Turbo=-2?? Result will better than -1 very much??
Inpainting is kinda fluid, you can't have strict good settings for it, only some generally ~OK settings, and it depends from how big your logo is and from the content around logo.
I think "-2" should be more suitable for UHD. On the small logos I expect it to hurt quality.

How to compare inpainting: Don't look at the single frames, you need to watch video on original speed, least noticeable delogo = better.

https://i.imgur.com/CXE1OWd.png https://i.imgur.com/O3NiNc8.png

Tested speed on "Daichi Miura" logo:
175.4 FPS with Optimize=1
25.61 FPS with Optimize=0


Mask: https://fileupload.win/hTjOZdRLAvkcO06
Forgot to tell you, that site offered me to download a Vittalia Trojan (https://howtofix.guide/trojan-vittalia/) 3 times before it gave me your file.

[kedautinh12]

Ok, thanks for your guide :D

[cagure]

Hi VoodooFX

For my convenience, I would like to remove these subtitles,

but I can't get your script to work,

I use this :

1)
LWLibavVideoSource("BL.mkv") ## Source is 3840x2160
z_Spline36Resize(dither="error_diffusion",1920,1080)
InpaintLoc(Loc="126,886,-1118,-128")
https://thumbs2.imgbox.com/d2/4d/I9zibBT2_t.png (https://imgbox.com/I9zibBT2)


2)
LWLibavVideoSource("BL.mkv")
z_Spline36Resize(dither="error_diffusion",1920,1080)
InpaintDelogo(automask=1,analyze=2, mask="E:\Gt\logomask.bmp" ,Loc="126,886,-1118,-128",Turbo=1) ### to create the 2 masks :


but the 2 masks are all 2 black https://slow.pics/c/pT50uuqb

3)
LWLibavVideoSource("BL.mkv") ## Source is 3840x2160
z_Spline36Resize(dither="error_diffusion",1920,1080)
InpaintDelogo(mask="E:\Gt\logomask.bmp",automask=0 ,Loc="126,886,-1118,-128",, Mode="Inpaint") #Deblend and Both don't works
https://thumbs2.imgbox.com/4e/48/PlFTT4nF_t.png (https://imgbox.com/PlFTT4nF)

but it practically doesn't work, surely I'm wrong something,
I got this frame, but there are other subtitles and in different x, y on the frames,
I can probably recall them with conditional filter with others masks, but in the meantime I would like to start it
i checked there are all the .dlls, i didn't install x264vfw
S.O. 7 64bit
Cheers

[VoodooFX]

but it practically doesn't work, surely I'm wrong something

Of course it doesn't work because there is nothing in your mask.
I wouldn't bother inpainting subtitles, that would look ugly in most cases, unless there are few subtitles and they are in solid or static background.
Automask is for usual static logos, if you want to use it on one subtitle then you need to limit video with Trim() only to that subtitle, or read about Analyze=-4.

If you want a dynamic mask for all subtitles then there are various DynMask methods.

[cagure]

Ok I'll try with the trim and with others DynMask methods
Thanks

[VoodooFX]

I've no idea how you suddenly came up "430,830,-1000,-180" coords.
And isn't that video HDR?

Anyway, example on your screenshot:

ImageSource("D:\PlFTT4nF_o.png")
InpaintDelogo(mask="D:\AxFwWIN.png", Loc="126,886,-1118,-128", Inflate=2, Turbo=1)
Mask: https://i.imgur.com/AxFwWIN.png

https://thumbs2.imgbox.com/26/30/iSG5DD1G_t.png (https://images2.imgbox.com/26/30/iSG5DD1G_o.png)

https://thumbs2.imgbox.com/02/a7/NtZ0nHLw_t.jpg (https://images2.imgbox.com/02/a7/NtZ0nHLw_o.jpg)

[kedautinh12]

after extracted images, how to use all images for delogo?? thanks
https://forum.doom9.org/showthread.php?p=1974638#post1974638

[VoodooFX]

after extracted images, how to use all images for delogo??
You extract images for OCR, for inpainting you just use that dynamic mask, disable Show to inpaint, use DynInflate to expand mask.

[katzenjoghurt]

Forgot to tell you, that site offered me to download a Vittalia Trojan (https://howtofix.guide/trojan-vittalia/) 3 times before it gave me your file.

Ouch. At least it was some hentai stuff for me, when I clicked on it now.
I uploaded it somewhere else and edited the posting.
[even though I doubt anyone will have interest in downloading these masks anymore]

[kedautinh12]

You extract images for OCR, for inpainting you just use that dynamic mask, disable Show to inpaint, use DynInflate to expand mask.

when i disable show=0, i got error: "source and mask clip have different dimensions"
https://drive.google.com/file/d/1OXGs7k5G-4ohnRSYmotA-NRPO8wgFgZy/view?usp=sharing

scripts:
LoadPlugin("C:\Megui\MeGUI-2924-64\tools\dgindexnv\DGDecodeNV.dll")
DGSource("C:\Users\ADMIN\Downloads\back number - Kaitou [1440x1080i MPEG2 M-ON! HD].dgi")
AnimeIVTC(mode=1)
trim(336,0).InpaintDelogo(Loc="1324,110,-60,-160", Show=0, DynLocUp=2, DynMask=2, DynTune=190, DynTEdge=1, DynTEdgeThY=30, DynTEdgeInf=4, KillNoise=5, DynInflate=4)

And i can't create mask logo at top-left on video above, can you have any suggest??

Video with same logo but can't create mask logo too:
https://drive.google.com/file/d/1nQHiXqIqQrjb-qDIooATbUo1JOSFM7mc/view?usp=sharing

[VoodooFX]

when i disable show=0, i got error: "source and mask clip have different dimensions"


Because you are doubling mask resolution with DynLocUp=2.

Video with same logo but can't create mask logo
If it's same then why you don't use same bmp?

[kedautinh12]

If it's same then why you don't use same bmp?

Can't create mask logo on two video. If i can createable, i will use same mask on video can't create mask

[VoodooFX]

Can't create mask logo on two video.
It's few clicks in PhotoShop -> https://i.imgur.com/pK3EInq.png

[kedautinh12]

It's few clicks in PhotoShop -> https://i.imgur.com/pK3EInq.png

Can you have video guide?? Few clicks but can't describe how :D

[VoodooFX]

https://www.youtube.com/watch?v=8eL5HAyNan0

[cagure]

I've no idea how you suddenly came up "430,830,-1000,-180" coords.
And isn't that video HDR?
it's my mistake, corrected in the previous post.

Excellent news, certainly by removing the subtitle I will have to compensate for the loss,but that's what I was looking for

But if I use Your mask everything works fine
https://thumbs2.imgbox.com/5b/f5/bTZ4UKlC_t.png (https://imgbox.com/bTZ4UKlC)
InpaintDelogo(mask="E:Gt\AxFwWIN.png", Loc="126,886,-1118,-128",Inflate=2, Turbo=1)

is that I can't create a mask like yours, but have you used photoshop?

Thanks.

[VoodooFX]

But if I use Your mask everything workse a mask like yours, but have you used photoshop?
Yes.
But you can use Automask on this image:

ImageSource("D:\PlFTT4nF_o.png")
InpaintDelogo(mask="D:\mask.bmp", Loc="126,886,-1118,-128", Analyze=-4, Automask=1)

[kedautinh12]

Hi, InpaintDelogo for dynamic very good but sometime it's look unnatural

1mn 42s
https://i.imgur.com/XoBnTun.png
Source: https://drive.google.com/file/d/1OXGs7k5G-4ohnRSYmotA-NRPO8wgFgZy/view?usp=sharing
Scripts:
LoadPlugin("C:\Megui\MeGUI-2924-64\tools\dgindexnv\DGDecodeNV.dll")
DGSource("C:\Users\ADMIN\Downloads\back number - Kaitou [1440x1080i MPEG2 M-ON! HD].dgi")
AnimeIVTC(mode=1)
trim(336,0).InpaintDelogo(Loc="1324,110,-60,-160", Show=0, DynLocUp=1, DynMask=2, DynTune=80, DynTEdge=1, DynTEdgeThY=30, DynTEdgeInf=4, KillNoise=5, DynInflate=4)

[kedautinh12]

Your mask only delogo complete with inpainting mod but black color arround logo will disappear
https://i.imgur.com/l04LfDk.png

Script:

LoadPlugin("C:\Megui\MeGUI-2924-64\tools\dgindexnv\DGDecodeNV.dll")
DGSource("C:\Users\ADMIN\Downloads\back number - Kaitou [1440x1080i MPEG2 M-ON! HD].dgi")
AnimeIVTC(mode=1)
trim(49,4623).InpaintDelogo(mask="C:\Left_Top_Logo.png", Automask=0, aMix=0, Loc="40,30,-1144,-940", Mode="Inpaint", Analyze=3, Turbo=-1, Optimize=1)

[VoodooFX]

Hi, InpaintDelogo for dynamic very good but sometime it's look unnatural
Looks normal, don't expect the details to be magically recreated.
Are you sure that screenshot comes from that script?

Your mask only delogo complete with inpainting mod but black color arround logo will disappear
It doesn't make any sense, can you use the google translate?
Cut off the black borders from video.

[kedautinh12]

Looks normal, don't expect the details to be magically recreated.
Are you sure that screenshot comes from that script?
Yes, i'm sure

[VoodooFX]

Your screenshots doesn't correspond to video resolution.
Use https://imgbox.com to post thumbnailed screenshots.

What function AnimeIVTC is doing here?

[kedautinh12]

Your screenshots doesn't correspond to video resolution.
Use https://imgbox.com to post thumbnailed screenshots.

What function AnimeIVTC is doing here?

Cause i cut and paste in ms paint. AnimeITVC is deinterlaced script
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/AnimeIVTC.avsi

[VoodooFX]

Cause i cut and paste in ms paint. AnimeITVC is deinterlaced script

Does this video looks like "Anime" for you?
Right click on AvSPmod preview and "Save image as...", then upload to https://imgbox.com, copy/paste BB-Code from there.

My take on this one subtitle:
LWLibAvVideoSource("D:\back number - Kaitou [1440x1080i MPEG2 M-ON! HD].ts").TFM()
InpaintDelogo(Loc="1324,110,-60,-160", DynMask=2, DynTune=190, DynTuneInf=1, DynTEdge=1, DynTEdgeThY=30, DynTEdgeInf=1, DynInflate=4)

Frame 2794:
Before:
https://thumbs2.imgbox.com/4c/d0/vdLnjcCW_t.png (https://images2.imgbox.com/4c/d0/vdLnjcCW_o.png)

After:
https://thumbs2.imgbox.com/c7/48/nb0gTPna_t.png (https://images2.imgbox.com/c7/48/nb0gTPna_o.png)

Your anime ms paint: https://i.imgur.com/XoBnTun.png

[kedautinh12]

AnimeITVC can use for real world too. Are you check all subtitles?? Cause sometime DynTune=190 can't detect all letters, that why i deceased Dyntune to 80

[Reel.Deel]

https://i.imgur.com/CXE1OWd.png https://i.imgur.com/O3NiNc8.png

Can the post-processing steps of InpaintDelogo be applied to an already delogo'd clip? I created an lgd, and while the result is not perfect I think the remaining artifacts can be mitigated.

https://i.ibb.co/JnS46bf/SSTV-1092.png https://i.ibb.co/VH1wCX3/SSTV-1115.png

The problem with TV MPEG-2 live footage is that there is always lots of artifacts in and around the logo. In some scenes the delogo'd area is better and in other the outline is clearly more visible. I also experimented with denoising the logo area before analyzing it in AviUtl but the result while better at times, it can also be inconsistent.

[kedautinh12]

AnimeITVC can you for real world too. Are you check all subtitles?? Cause sometime DynTune=190 can't detect all letters, that why i deceased Dyntune to 80

After check, i seen your result detect all letters with DunTune=190. Very good, thanks

[VoodooFX]

AnimeITVC can you for real world too. Are you check all subtitles??
You can use Invert() on any videos too.
No, I didn't check, at the bright backgrounds you would want to expand mask more to cover the shadows. Get new script from temp branch where I allowed higher DynInflate values.
And you are not leaving enough space from logo borders [shadows there are part of logo/subs].

[VoodooFX]

Can the post-processing steps of InpaintDelogo be applied to an already delogo'd clip? I created an lgd, and while the result is not perfect I think the remaining artifacts can be mitigated.

https://i.ibb.co/JnS46bf/SSTV-1092.png https://i.ibb.co/VH1wCX3/SSTV-1115.png


You are comparing apples to oranges there. Those examples there are inpainting.
Use Deblend for comparison, and Interp=0, dPP=0 to disable pp.
I think there is some avs function ported from aviutl to denoise delogo area.

First row - Interp=0, dPP=0, second - Interp=0, dPP=5:

https://i.imgur.com/pYT2Wb6.png https://i.imgur.com/apwVKlE.png

https://i.imgur.com/9BY0vzo.png https://i.imgur.com/tkwj6nu.png

Ebmp from there: SSTV.v3.1080 (https://drive.google.com/drive/folders/1QWixC2n9xy6ntt9JYr8FCsl5T3VDZuVl?usp=sharing)

EDIT:
Found it, logoNR [you added it to wiki], just FFT3DFilter there, added it on top of blur as negative dPP.
Interp=0, dPP=-5:

https://i.imgur.com/y4kGEtp.png https://i.imgur.com/exIWnEf.png

[kedautinh12]

Will SSTV.v3.1080 better than SSTV.v2.1080??

[kedautinh12]

You can use Invert() on any videos too.
No, I didn't check, at the bright backgrounds you would want to expand mask more to cover the shadows. Get new script from temp branch where I allowed higher DynInflate values.
And you are not leaving enough space from logo borders [shadows there are part of logo/subs].

I tried leaving more space from logo borders, result still same

[VoodooFX]

Will SSTV.v3.1080 better than SSTV.v2.1080??
Only you can see what better fits your videos, I'm not a fortune teller.

[Reel.Deel]

You are comparing apples to oranges there. Those examples there are inpainting.
Use Deblend for comparison, and Interp=0, dPP=0 to disable pp.
I think there is some avs function ported from aviutl to denoise delogo area.

First row - Interp=0, dPP=0, second - Interp=0, dPP=5:

<pictures>

Ebmp from there: SSTV.v3.1080 (https://drive.google.com/drive/folders/1QWixC2n9xy6ntt9JYr8FCsl5T3VDZuVl?usp=sharing)

EDIT:
Found it, logoNR [you added it to wiki], just FFT3DFilter there, added it on top of blur as negative dPP.
Interp=0, dPP=-5:

<pictures>

Those are some nice results. I had never used InpaintDelogo for logo deblending. Thanks for sharing.

Yes I added logoNR which is just a simple script that denoises the logo area and then overlays it on top of the delogo'd clip. I assume InpaintDelogo offers a more sophisticated technique for post-processing the logo area, hence my question if it can be used used for that purpose on an already delogo'd clip.

Regarding the ebmp you linked to. Are the numbers in the end supposed to be the x,y location? The name says SSTV.v3.1080_106-36 so I tried use loc="106,36,..." but it did not line up, but loc="56,36,..." did.