InpaintDelogo - advanced logo removal & hardcoded subtitles extraction

[StainlessS]

Originally Posted by manolito

Mani, where did you get above info ?

Might try doing UGlarm thing, and add to s_ExLogo2()

EDIT: I presume that 'distance' is by Pythagoras.

[manolito]

It comes from here:
https://www.sendspace.com/file/odmobw

It is for version 3.01 (the latest plugin version is 4.01), but this is the last help version I could find...

[StainlessS]

Thanks Mani, was in plugin docs eh, I was trying to find webpage.

Bout 25% done now, will initially be for SOME avs standard colorspaces only, I will though want avs+ eventually. [will also supply v2.58 plug + x64]

Doin' it summick like this,

Function Uglarm(clip c,                                                     [* Avs v2.58 & v2.60 std supports YV12/YUY2/RGB32/RGB24 : Avs+ r2915+ also YV16, YV12, YV24, Y8 *]
        \ int LogoX,int LogoY,Int LogoW,Int LogoH,                          [* ALL coords can be -ve, where RHS/BOT relative, LogoW & LogoH can additionally be 0 where RHS/BOT relative*]
        \ int "ClipX"=0,int "ClipY"=0,Int "ClipW"=0,Int "ClipH"=0,          [* ClipW, and ClipH coords can be zero or -ve, where RHS/BOT relative (but ClipX:ClipY cannot be -ve) *]
        \ float "SAR"=1.0,                                                  [* Sample (Pixel) Aspect Ratio (x/y)*]
        \ float "BlurPower"=1.0,                                            [* 1.0 -> ??? *]
        \ Int "ShowMode"=0                                                  [* 0=Overlay Logo Patch(Clipped Logo), 1=Overlay Logo Patch(Clipped) & Border, 2=DeLogo *]
        \ Bool "Blank"=false                                                [* Where ShowMode==2 and Blank==True then Blank to Black any border area (border is outside of Clip args) *]
        \ )

EDITED:

EDIT: I'm not taking any pixel Aspect Ratio into account for distance, dont think it worth the (small) effort. [EDIT: See SAR above]

EDIT: ShowMode = 1 Example showing Patch Area (after clipping, in Yellowish), Clipped out area (In Reddish), and border Area in Purplish.
Yellowish + Reddish is original non clipped Logo Coords.

Avisource(".\Parade.avi")
Crop(0,0,Width/4*4,height/4*4)      # Make Mod 4

ConvertToRGB24     # Internally Converted to RGB32 and back to RGB24 again [Layer() requirement, supports only YUY2 & RGB32].

LogoX    = -100    # Can be -ve, Width  relative
LogoY    = -100    # Can be -ve, Heigth relative
LogoW    =  -20    # Can be 0 or -ve, Width  relative
LogoH    =  -20    # Can be 0 or -ve, Height relative

                   # Clipping Coords (inside is valid graphic, outside of it is border/Letterbox)
ClipX    =    0    # Black border area, +ve ONLY
ClipY    =    0    #   ditto
ClipW    =  -50    #   Can be 0 or -ve, Width  relative
ClipH    =  -50    #   Can be 0 or -ve, Height relative

BLURPOWER=  1.0
SHOWMODE =    1
BLANK    = False

Uglarm(LogoX,LogoY,LogoW,LogoH,ClipX,ClipY,ClipW,ClipH,BlurPower,ShowMode,BLANK)

[Reel.Deel]

@VoodooFX

Thank you for this script. I tried it out and I was very pleased with the results.

[StainlessS]

Function prototype in post #43 updated.

Got the Uglam overlay stuff working fine, but the delogo part was well slow (and did not quite work on initial attempts).
Did some searching and found this source patch file for ffmpeg, cant say it made a lot of sense to me (at first, never used SubVersioning stuff before),
but managed to figure it out a bit but probably a little more time till working OK. Is a fast way to precalc tables for UGLARM interpolation,
unfortunately, they use coords for logo which include the border pixels, and I exclude them, so need a bit of a redo, but getting there.
[also seems to expect valid perimeter pixels all around the logo, whereas I allow for letterbox border overlap of logo, so more changes there too].

Here the patch source code thing that I found on UGLARM search:- https://patchwork.ffmpeg.org/patch/11585/
probably will not make much sense, presume that you are supposed to use some kind of dedicated patch code reader software to be able to understand it.
anyways, I mostly got it figured out [and dropped some dross that was either dormant or did not work].

Anyways, I then (just now) found original source of the Uglarm tables code from ffdshow:- https://sourceforge.net/p/ffdshow-tr...erLogoaway.cpp
ffdshow stuff makes a lot more sense than the ffmpeg patch file I was working with. Relevant table creation stuff is in TimgFilterLogoaway.cpp functions,

void TimgFilterLogoaway::Tplane::calcUweWeightTable(int w, int h, int power)

void TimgFilterLogoaway::Tplane::uwe(const TlogoawaySettings *cfg)

I've still gotta convert from their definition of logo coords to mine, but at least easier to read.

Leaving this here so I can more easily find it again (ffdshow link), maybe of interest to some others too [incl VDX].
[Also wanna keep record of where I got source from so that I can point to it as per GPL requirements].

Also, here on-line version of Manolito linked LogoAways docs:- http://vander.wojdon.net/virtualdub/.../logoaway.html

[VoodooFX]

Originally Posted by StainlessS

I'm not taking any pixel Aspect Ratio into account for distance, dont think it worth the (small) effort. [EDIT: See SAR above]

I don't see any practical effect of that, or even theoretical... unless to make a parameters list longer. Thats would be same as adding SAR parameter to Blur() or any other function, or I miss something here!?

[StainlessS]

I obviously wanted to make the parameter list longer, so as to seem more complicookered, me likes that.

Still not got the damn thing working properley, on last try (using more like the original patch code), it was prodcuing something quite akin
to the mandelbrot stuff, looks well weird. Maybe I see when working, if SAR stuff makes any difference, eliminate if not.
[SAR affects distance calcs for uglarm, so for chroma of eg YUY2 or YV16, should make a fair bit of difference, maybe. SAR auto modified for chroma of YUY2 and YV16].
Although, when I've ever used any delogo thing in eg Vdub, I never bothered with SAR.

EDIT:




EDIT: If there had been a logo in above images, they would certainly have been eliminated.
Had I done above patterns deliberately, I would have been quite proud if it.
I bet you mathematicians out there can tell what I'm doin' wrong just by lookin' at it.
EDIT: Gone t' pub.

[StainlessS]

Uglarm v0.0 Logo Interpolator now posted, Guinea pigs required, Here:- https://forum.doom9.org/showthread.php?t=176891

No idea how it compares to others tested in this thread.

EDIT: Seems a bit slow on YV12 UT_Video FullHD 192x192 logo, bout 11.8 FPS on my crap Core Duo Quad 2.8GHz m/c. [23% CPU usage, x64 version, bout 9FPS x86]

Mani, as you seem to like Uglarm Method, hows bout give it a whirl, with comparison times etc.
(Likely some bugs to iron out, tis still a baby).

[StainlessS]

Hi VDX, can I request a feature update.
I like to be able to use -ve width and height as in crop, but not quite so keen on being forced to only use -ve values,
so how bout making +ve width and height possible in your loc string thing ?

Below should do it in both your InpaintLoc(), and InpaintDelogo() functions.

InpaintLoc() mod in BLUE

    a           = int(value(leftstr(loc,posvirg1-1)))
    b           = int(value(leftstr(rightstr(loc,long-posvirg1),posvirg2-posvirg1-1)))
    c           = int(value(leftstr(rightstr(loc,long-posvirg2),posvirg3-posvirg2-1)))
    d           = int(value(rightstr(loc,long-posvirg3)))

    c           = (c > 0 ) ? a+c-x: c         # Added, convert +ve width  to -ve width
    d           = (d > 0 ) ? b+d-y: d         # Added, convert +ve height to -ve height

    hightlight  = blankclip(width=x-a+c,height=y-b+d,color=$96ff2d).converttoRGB32    # EDIT: This depends upon -ve crop values, may do elsewhere in script too.

InpaintDelogo() mod in BLUE

    a           = int(value(leftstr(loc,posvirg1-1)))
    b           = int(value(leftstr(rightstr(loc,long-posvirg1),posvirg2-posvirg1-1)))
    c           = int(value(leftstr(rightstr(loc,long-posvirg2),posvirg3-posvirg2-1)))
    d           = int(value(rightstr(loc,long-posvirg3)))

    c           = (c > 0 ) ? a+c-x: c         # Added, convert +ve width  to -ve width
    d           = (d > 0 ) ? b+d-y: d         # Added, convert +ve height to -ve height


    inYV12      = clp.Crop(a,b,c,d).ConvertToYV12
    inRGB24     = clp.Crop(a,b,c,d).ConvertToRGB24
    inRGB32     = clp.Crop(a,b,c,d).ConvertToRGB32

Instead of converting -ve values to +ve values, I've converted +ve values to your -ve style values, just in case your logic depends upon -ve crop values.


Thankyou in advance.

EDIT:
I now see that you updated since the version script I was using,
Here mod for InpaintLoc() my mod in BLUE, you addition in v1.12 in DarkOrchid [ do similar for InpaintDelogo() ]

    a           = int(value(leftstr(loc,posvirg1-1)))
    b           = int(value(leftstr(rightstr(loc,long-posvirg1),posvirg2-posvirg1-1)))
    c           = int(value(leftstr(rightstr(loc,long-posvirg2),posvirg3-posvirg2-1)))
    d           = int(value(rightstr(loc,long-posvirg3)))

    c           = (c > 0 ) ? a+c-x: c         # Added, convert +ve width  to -ve width
    d           = (d > 0 ) ? b+d-y: d         # Added, convert +ve height to -ve height

    Assert (c   <= 0 && d <= 0,"""Incorrect "Loc".""")
	Assert (a   >= 0 && b >= 0,"""Incorrect "Loc".""")
	Assert (a%2 == 0 && b%2 == 0 && c%2 == 0 && d%2 == 0,"""Use even numbers for "Loc".""")

    hightlight  = blankclip(width=x-a+c,height=y-b+d,color=$96ff2d).converttoRGB32

[StainlessS]

Modified InpaintDelogo_v1.13.avsi.7z @ SendSpace in sig beneath this post. [available for 30 days only, will attach when ascertained that works ok]

# Version 1.11 : 2019/08/31. Public release.
# Version 1.12 : 2019/09/06. Various tweaks. New parameters. "Inflate" and "Deep" changed.
# Version 1.13 : 2019/12/08. Allow -ve LogoX,LogoY, and +ve LogoW,LogoH in Loc string. StainlessS.

# string "Loc" : Select (crop) the part of the picture around the logo.
# Example: Loc="30,420,-540,-30".
# Example: Loc="-200,-100,-30,-20". [ssS: Added, LogoX=Width-200, LogoY=Height-100, LogoX2=Width-30, LogoY2=Height-20]

EDIT: Actually above should be Loc="-200,-100,-30,-20". [ssS: Added, LogoX=Width-200, LogoY=Height-100, LogoX2=Width-30-1, LogoY2=Height-20-1, same as -ve cropping for width and height]

EDIT: OR eg, Loc="-200,-100,30,20". [ LogoX=Width-200, LogoY=Height-100, LogoW=30, LogoH=20]

bugloc2 = (loc == "0,0,-0,-0" || loc == "0,0,0,0") ? true : false # ssS: Added:- || loc == "0,0,0,0"

# assert ( findstr(loc,"-" ) != 0 , """Incorrect "Loc".""") # ssS: Commented OUT, we now allow +ve width and height in loc

Both in InpaintDelogo() and InpaintLoc()

a = (a < 0 ) ? x+a : a # ssS: Added, convert -ve logoX to +ve LogoX
b = (b < 0 ) ? y+b : b # ssS: Added, convert -ve logoY to +ve LogoY
c = (c > 0 ) ? a+c-x: c # ssS: Added, convert +ve width to -ve width
d = (d > 0 ) ? b+d-y: d # ssS: Added, convert +ve height to -ve height

Manolito,

What magic arguments do you use for removing fully opaque logos ?

I've done a mod to VoodooFX InpaintDelogo, v1.13, but dont know how to test it as I cannot get it working.
[I am real busy and dont got no time to figure it out, so pleading to you for assist].

I'm not sure is possible without any kind of analysis, and below will fail if analysis is mandatory ['mask is full' or similar error].

I want to test with something like this.

Import("InpaintDelogo_v1.13.avsi")
WH=512
A=BlankClip(width=WH/2,height=WH/2)
B=A.Invert StackHorizontal(A,B)
StackVertical(Invert)
trim(0,-100)                # 100 frames, RGB32

LogoW    = Width/2
LogoH    = Height/2
LogoX    = (Width-LogoW)/2
LogoY    = (Height-LogoH)/2

Overlay(Last.BlankClip(Width=LogoW,height=LogoH,color=$8080C0),x=LogoX,y=LogoY)     # Make dummy logo

loc=RT_string("%d,%d,%d,%d",LogoX,LogoY,LogoW,LogoH)                                # location of logo

# Return InpaintLoc(loc)                                                            # Mark/overlay logo area with 0.5 transparent light green

#InpaintDelogo(loc=loc,mode="Inpaint",mask="d:\EXLOGO\tmp.bmp",AutoMask=1)          # NO Idea what to use here for static fully opaque logo, We do not need any kind of analyse, we know where it is.

#Uglarm(LogoX,LogoY,LogoW,LogoH)                                                    # test delogo using Uglarm

Return Last

__END__

InpaintDelogo( mask="d:\EXLOGO\tmp.bmp",
\ Automask=0, aMix=0, Loc=loc,
\ Mode="Inpaint",
\ Analyze=1, Fr1=0, Fr2=0, FrS=0, ReAnalyze=0,
\ dPP=3, dPPm=3, Interp=2, InterpM=0, Deep=2,
\ oPP=6, oPPm=3, oTune=0, KillShape=0, Inflate=1, Turbo=0,
\ Sharpness=45, PreBlur=6.5, PostBlur=4.0, Radius=8.0, prePP=0,
\ GrainPP=2,
\ Show=0, mShow=0, rSize=4)

Baseclip


Dummy Logo


InpaintLoc marked


Uglarm delogo test


EDIT: VoodooFX dont seem to have been around lately, no indication on user profile when he/she/it last visited. [no idea why last visit missing from user profile, is usually there]

[manolito]

When you have "AutoMask=1" in your InpaintDelogo call, then it will only create a mask and not remove any logo. You need to call InpaintDelogo twice, the second time with "AutoMask=0".

For creating the mask you might want to use the "Analyze=4" parameter instead of the default "Analyze=1". But for this you need to specify a frame number with the "Fr1" parameter where the logo is surrounded by black.

Another thing in your script is problematic. Your loc area is identical to the logo size, and this will give you bad results. The loc area must be at least 10 pixels larger than the logo, I use 14 to 16 pixels.

Your changes for the possible loc formats look interesting, but I will probably have no use for the new formats. To determine the loc coefficients I do not use "InpaintLoc" at all, instead I open my clip in VDub and apply the NullTransform filter. From there I can crop the image visually, and VDub shows the coefficients in the correct format which I can enter into my AVS script.


Cheers
manolito

[StainlessS]

Thanks Mani, guess that this is no good to me at all then, unless we can artificially create a mask in whatever required format.
Currently modding s_ExLogo2(), target is work from v2.58 to avs+ ALL colorspaces [EDIT: except YV411], and nice gauss blurry stuff, with soft edges.

[manono]

Originally Posted by manolito

Your changes for the possible loc formats look interesting, but I will probably have no use for the new formats.

I think you're being polite. I took one look and thought to myself, "WTF?". They're fine the way they are.

I'd like to thank the creator for this. It opens almost instantly where when using InPaintFunc as I did before, it took roughly the length of the video to open. And since I work with long movies, mostly, you can imagine the frustration. And the results are better. I've used it twice recently on opaque logos and am very happy with the results.

[StainlessS]

I also thank VoodooFX for his useful tool.

I think you're being polite. I took one look and thought to myself, "WTF?". They're fine the way they are.

-ve width and height are useful when logo is at RHS or bottom, but less so when at LHS or top,
-ve LogoX and LogoY are useful in the same way as -ve width and height.

Nothing wrong with the way they were, and also nothing wrong with having options.
I did request VoodooFX to make the mod[sept 2019], but he aint been back since then.

Also, Manolito is always polite [well sometimes he is].

EDIT:

Thanks Mani, guess that this is no good to me at all then, unless we can artificially create a mask in whatever required format.

I meant that is no use where single frame delogo required, as analysis stage would find nothing [same where static clip/static logo as in prev test script].

[manolito]

Yes, the results of the InpaintDelogo plugin blow everything I have tested and used before out of the water...

A few hints:
If you use the "Analyze=1" parameter for the AutoMask creation then 3% of the source clip will be used for analyzing. For a longer movie (duration 90 min or above) this takes a long time. You can use "Trim", but for me I get better results after specifying a lower percentage for analyzing. I edited the InpaintDelogo.avsi file and replaced "every=33" with "every=400", and this is way faster without damaging the mask.

If you use AVS+ and want to use MultiThreading, there are a few catches. As VoodoFX pointed out, the InPaint routines are not really MT friendly. Analyzing the speed with AVSMeter you will find that turning off MT will result in a higher speed. But you may have other filters in your AVS script which do profit from MT, and the overall speed will be higher when using MT.

From my InpaintDelogo tests I found out that it tends to crash quite a bit when using MT. Reducing the prefetch threads did not help, but I got it working without crashes now.

First you need to force MT_SERIALIZED mode for all Inpaint functions:

SetFilterMTMode("InpaintLogo", MT_SERIALIZED)
SetFilterMTMode("DeblendLogo", MT_SERIALIZED)
SetFilterMTMode("AnalyzeLogo", MT_SERIALIZED)
SetFilterMTMode("DistanceFunction", MT_SERIALIZED)
SetFilterMTMode("RequestLinear", MT_SERIALIZED)

And then the InpaintDelogo call must be followed by a RequestLinear call like this:

RequestLinear(rlim=50, clim=50)

This will decrease the possible MT speed gain quite a bit, but for my usual conversions I still get a moderate gain of several fps. And no more crashes for quite some time...


Cheers
manolito

[manolito]

Originally Posted by StainlessS

Also, Manolito is always polite [well sometimes he is].

Should I be offended?

[StainlessS]

You are seldom offended when I say such nice things about you [Esp as I was so staunchly defending you ].

[VoodooFX]

I just came to update op, lots of new post I see.

So what for these new Loc arguments are for? To use your new plugin with InpaintDelogo, StainlessS? Does it needs some testing?

Yesterday I delogo'ed one video:

[StainlessS]

So what for these new Loc arguments are for?

Originally Posted by StainlessS

Hi VDX, can I request a feature update.
I like to be able to use -ve width and height as in crop, but not quite so keen on being forced to only use -ve values,
so how bout making +ve width and height possible in your loc string thing ?

Not new arguments, just alternative way of specifiying existing ones. [EDIT: crop() without +ve width,height options, would be a little less perfect]
I dont mind if you remove -ve LogoX,LogoY, sometimes just a little easier to guess relative RHS/BOT, your choice. I've marked all changes in script with 'ssS'.

Does it needs some testing?

Yes please if you would be so kind.

Real good job on you new posted delogo sample, is quite nifty.

I did not know if you were gone for good, nice to see you back again

EDIT: The changes were made so as not to foul up your existing logic where code might be reliant upon -ve width,height.

EDIT: This is the only line that I can see that might require a little bit of your time [maybe delay detect until you extract a,b,c,d from loc string, and after conversion to +ve a,b, and -ve c,d].

    bugloc2     = (loc == "0,0,-0,-0" || loc == "0,0,0,0") ? true : false       # ssS: Added:- || loc == "0,0,0,0"

[VoodooFX]

Found some old testing files, just need to remember and understand all this new stuff in your plugin.

-ve +ve is confusing as you need to add at least 10px borders from logo edges in InpaintDelogo.
Real minimum I think is 7px or 8px, added some room for user error... that's why 10 in help, it will be best for quality and speed to keep at minimum 8-10px.