InpaintDelogo - advanced logo removal & hardcoded subtitles extraction

[Shinkiro]

Originally Posted by VoodooFX

Thx for the script. Don't forget StainlessS, he done heavy lifting to make it happen.

Corrected that unfortunate inaccuracy.

Originally Posted by VoodooFX

I found Subtitle Edit's nOCR feature better and faster than Tesseract.

Can it recognize italics and render it into text? because Tesseract has a problem with that.

[VoodooFX]

Originally Posted by Shinkiro

Can it recognize italics and render it into text? because Tesseract has a problem with that.

Yes, it's like a binary image compare, but different. Tesseract produces many errors and correct them takes lots of time, when after nOCR I got result with no errors.

[VoodooFX]

# Version 1.36 : 2021/10/04.  Added auto generation of the dynamic mask for subtitles with halo (aka "DynMask=4").
#                             Updated: "KillNoise", and some other tweaks.

Faster, better, simpler - "DynMask=4" method:

InpaintDelogo(Loc="0,0,-0,-0",
\ oPP=0,
\ DynMask=4, DynTune=220, DynMask4H=60, KillNoise=-5,
\ Show=4)

~30% faster than "DynMask=3" + "DynMask3H" method. Artifact at ~3073 frame is gone, one dot is still present at ~3159, so I won't be happy till it's there.

Originally Posted by Shinkiro

...converting received images into srt subtitles, using text recognition software Tesseract...

Could you share to pastebin the output from your Tesseract (OCR subs from that anime sample)?

[Shinkiro]

Originally Posted by VoodooFX

Could you share to pastebin the output from your Tesseract (OCR subs from that anime sample)?

https://pastebin.com/kWzxLUMm (model fast)
https://pastebin.com/r9ia78qy (model best) this model gave off some garbage in some places.
Tesseract adds some sort of sign at the end of each line, I manually remove it.
He's having some trouble identifying the letter I

[VoodooFX]

Originally Posted by Shinkiro

(model best) this model gave off some garbage in some places.
He's having some trouble identifying the letter I

Are these done with Tesseract 4/5 neural nets LSTM? Did you tried inverted images (SE inverts them automatically when Tesseract 5 is selected)?

Tesseract in Subtitle Edit can detect italics.
Here are my tests (I deleted all SE's dictionaries to prevent auto-correction and to see pure Tesseract):

Tesseract 3:
https://pastebin.com/7BX287Fp
InpaintDelogo(Loc="0,0,-0,-0", oPP=0, DynMask=4, DynTune=220, DynMask4H=60, KillNoise=-5, Show=4, Extract=1, ImgDir="D:\Images")

Tesseract 5 (original, no LSTM):
https://pastebin.com/BLc5b09t
InpaintDelogo(Loc="0,0,-0,-0", oPP=0, DynMask=4, DynTune=220, DynMask4H=60, KillNoise=-5, Show=4, DynMaskUp=2, Extract=1, ImgDir="D:\Images")

I think T5 likes bigger images, I used "DynMaskUp"(new parameter in v1.37) instead of "ImgSize", dunno if it helps, I think that for nOCR it should.

[VoodooFX]

# Version 1.40 : 2021/10/28.  Some speed optimizations. AvsInpaint v1.3 is required.
# Version 1.39 : 2021/10/23.  Various improvements.
# Version 1.38 : 2021/10/22.  Fixed: Bug with "SubSuspect" parameter.
# Version 1.37 : 2021/10/07.  New: "DynMaskUp" parameter.

[VoodooFX]

# Version 1.45 : 2021/11/07.  Fixed: Automask wasn't working if coords were "0,0,0,0".
# Version 1.44 : 2021/11/05.  Refactored multipass deblending code.
# Version 1.43 : 2021/11/03.  Fixed: One pass Deblend didn't see A/C masks from multi pass. Changed: "Deep".
# Version 1.42 : 2021/11/03.  Fixed: A/C masks file name if coord is "-0".
# Version 1.41 : 2021/11/03.  Fixed: "Show=1" was broken with "Deblend" since v1.18.

[VoodooFX]

@StainlessS
Maybe you could spot something why func below doesn't run on Avs 2.6 (it runs OK on Avs+)?
Posting here to not litter your thread. Relevant link: https://forum.doom9.org/showthread.p...68#post1956968

Error at clp2x2: "Crop: Top and Left must be more than 0".

function Analyze5FX(clip clp, val "mask", int "Mode", int "Analyze5Th", string "Ignore", int "Show") {
    mask       = default (mask         , "")
    Mode       = default (Mode         , 1 )
    Analyze5Th = default (Analyze5Th   , 15)
    Ignore     = default (Ignore   , "none")
    Show       = default (Show         ,  0)

    Assert (!(Mode       < 1 || Mode       >   2), """"Mode" value must be between 1 and 2.""")
    Assert (!(Analyze5Th < 1 || Analyze5Th > 255), """"Analyze5Th" value must be between 1 and 255.""")
    Assert (!(Show       < 0 || Show       >   2), """"Show" value must be between 0 and 2.""")
    Assert ((Ignore == "top" || Ignore == "bottom" || Ignore == "left" || Ignore == "right" || Ignore == "none"), """"Ignore" string must be 'top', 'bottom', 'left', 'right' or 'none'.""")

    VW   = clp.Width
    VH   = clp.Height
    Th   = Analyze5Th
    mask = (IsClip(mask)) ? mask.ConvertToY8(matrix="PC.709").MTballoon2PX(2) : nop
    clpS = clp.ScriptClip("subtitle(string(current_frame))")

    # Construct a clip to check thresholds (for deblending analysis) #
    #================================================================#

    if(Mode == 1) {
        # Get coords of the logo mask
        VFX2="""
            Status = (mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255)) ? 1 : 2
            if(Status==1) {
                RT_DBaseSet(Analyze5DB,0,Status,YIRL_X,YIRL_Y,YIRL_W,YIRL_H)
            } else {
                RT_DBaseSetField(Analyze5DB,0,0,Status)
            }
            return Last
        """
        Analyze5DB = ("~InpaintDelogo_Analyze5_"+RT_LocalTimeString+".DB").RT_GetFullPathName
        RT_DBaseAlloc(Analyze5DB,1,"iiiii")

        Parse = mask.GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true)
        Parse.RT_ForceProcess

        Assert (!(RT_DBaseGetField(Analyze5DB,0,0) == 2), "Mask image is invalid.")
        mX = RT_DBaseGetField(Analyze5DB,0,1)
        mY = RT_DBaseGetField(Analyze5DB,0,2)
        mW = RT_DBaseGetField(Analyze5DB,0,3)
        mH = RT_DBaseGetField(Analyze5DB,0,4)
        RT_FileDelete(Analyze5DB)

        # Convert X,Y,W,H coords to L,T,-R,-B coords
        mL = mX
        mT = mY
        mR = -VW + mW + mX
        mB = -VH + mY + mH
        Assert (!(mL < 8 && -1*mR < 8 || mT < 8 && -1*mB < 8), "Mask image or Loc is invalid.")

        # Coords mod2 round up
        mL = mL + (mL % 2)
        mT = mT + (mT % 2)
        mR = mR - (mR % 2)
        mB = mB - (mB % 2)

        # Crop 2x2 pxs area from above or below the logo and make a background from it
        mid = (VW-mL+mR)/2+mL
        mid = mid + (mid % 2)
        clp2x2 = (mT >= -1*mB) ? clp.crop(mid-2, mT-2, -1*(VW-mid), -VH+mT+2) : clp.crop(mid-2, VH+mB-2, -1*(VW-mid), mB+2)
        clpB = clp2x2.PointResize(VW, VH)

        Msk1 = mask.MTballoon2PX(2).mt_logic(mask, mode="xor")  # 2px wide mask around logo (+2px offset from the base mask)
        clp5 = clpB.Overlay(clp, mask=Msk1)

    } else {

        # Construct a clip to check thresholds (for Automask=1) #
        #=======================================================#

        cT = clp.crop(4, 4, -4, -1*(VH-6))
        cB = clp.crop(4, VH-6, -4, -4)
        cL = clp.crop(4, 4, -1*(VW-6), -4)
        cR = clp.crop(VW-6, 4, -4, -4)

        if(Ignore == "top") {
          cT = cB
        } else if (Ignore == "bottom") {
          cB = cT
        } else if (Ignore == "left") {
          cL = cR
        } else if (Ignore == "right") {
          cR = cL
        }

        wc = StackHorizontal(cr,cl).TurnLeft
        hc = StackVertical(ct,cb)

        if(wc.width > hc.width) {
          hc  = hc.PointResize(wc.width, hc.height)
          out = StackVertical(wc,hc)
          } else if(wc.width < hc.width) {
              wc  = wc.PointResize(hc.width, wc.height)
              out = StackVertical(wc,hc)
          } else {
              out = StackVertical(wc,hc)
          }
        clp5 = (Show == 2) ? out.PointResize(VW, VH) : out
    }

    # Delete frames above thresholds (that leaves us with the suitable frames for analysis) #
    #=======================================================================================#

    # Mod of Gavino's DeleteFrames() - https://forum.doom9.org/showthread.php?t=163107
    clp5
    fc  = FrameCount()
    res = BlankClip(clp, length=0)
    condition = "YPlaneMinMaxDifference >" +String(Th)+ " || UPlaneMinMaxDifference >" +String(Round(Th/2))+ " || VPlaneMinMaxDifference >" +String(Round(Th/2))
    current_frame = 0

    while (current_frame < fc) {
      while (Eval(condition) && current_frame < fc) {
        current_frame = current_frame+1
      }
      if (current_frame < fc) { # not at end
        start = current_frame # start of wanted section
        while (!Eval(condition) && current_frame < fc) {
          current_frame = current_frame+1
        }
        clpX = (Show == 1) ? clpS : (Show == 2) ? clp5 : clp   # 0: Original frames, 1: Original frames + frameNo, 2: Constructed clips (Mode=2 clip is upsized for show)
        res  = res + clpX.Trim(start, start-current_frame)
      }
    }
    Assert (!(res.FrameCount < 2), """No frames found to analyze, increase "Analyze5Th".""")
    return res
 }

EDIT:
Clip: https://drive.google.com/file/d/1Pdt...ew?usp=sharing
Mask: https://i.imgur.com/WieMHwJ.png
Test script:

GImport("c:\Analyze5FX.avs")
x = LSMASHVideoSource("D:\899.mp4")
x = x.crop(1124,24,-52,-636).Trim(30,100) # trimed for speed 
mask = ImageSource("D:\mask.bmp",0,0)

# Mode 1 test
m0 = x.Analyze5FX(mask=mask, mode=1, show=0)
m1 = x.Analyze5FX(mask=mask, mode=1, show=1)
m2 = x.Analyze5FX(mask=mask, mode=1, show=2)
mu = Stackhorizontal(m0.AddBorders(0,0,2,0,$000000), m0)
md = Stackhorizontal(m1.AddBorders(0,0,2,0,$000000), m2)
Stackvertical(mu.AddBorders(0,0,0,2,$000000), md)

[StainlessS]

Thanks for the source files.

Confirm, you are trying to get data for only current_frame, yes, [in Analyze5FX] ?

If so, then dont really want to use RT_ForceProcess, which scans entire clip.
Can use RT_YankChain [force single frame process].

RT_YankChain(clip,int "n"=current_frame,int "delta"=0)
 Compile/runtime clip function.
 n (default = current_frame) frame number.
 delta (default = 0), frame number offset.
 Forcibly process the filter graph chain for frame n + delta. Makes intention more explicit than eg RT_AverageLuma(n=i,w=1,h=1).
 See also RT_GraphLink().
 eg, Below, TmpA plays no part in output clip and so bitmap would not be written. Uncomment the RT_YankChain line and it will.
    a=Colorbars.ShowFrameNumber
    TmpA = a.subtitle("TestA").ImageWriter("TestA_", type="BMP")
    #RT_YankChain(TmpA,n=42)                # Uncomment to forcibly write frame 42 of TmpA to bitmap
    return MessageClip("All Done")

[VoodooFX]

Originally Posted by StainlessS

Confirm, you are trying to get data for only current_frame, yes, [in Analyze5FX] ?

If so, then dont really want to use RT_ForceProcess, which scans entire clip.
Can use RT_YankChain [force single frame process].

In if(Mode == 1) block only process that one frame (image mask), I'll change that to RT_YankChain.
I think GScriptClip there is not needed, nothing is used there from the return, is it possible to get RT_YInRangeLocate coords vars directly into function without RT_DBase thingy there?

EDIT:
Just realized Analyze5FX is using some stuff from InpaintDelogo.avsi, so you need to load that too. [I split Analyze5FX from unpublished version, but v1.45 should work]
Or just replace MTballoon2PX(2) with mt_expand().mt_expand()

[StainlessS]

is it possible

Instead of

    if(Mode == 1) {
        # Get coords of the logo mask
        VFX2="""
            Status = (mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255)) ? 1 : 2
            if(Status==1) {
                RT_DBaseSet(Analyze5DB,0,Status,YIRL_X,YIRL_Y,YIRL_W,YIRL_H)
            } else {
                RT_DBaseSetField(Analyze5DB,0,0,Status)
            }
            return Last
        """
        Analyze5DB = ("~InpaintDelogo_Analyze5_"+RT_LocalTimeString+".DB").RT_GetFullPathName
        RT_DBaseAlloc(Analyze5DB,1,"iiiii")

        Parse = mask.GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true)
        Parse.RT_ForceProcess

        Assert (!(RT_DBaseGetField(Analyze5DB,0,0) == 2), "Mask image is invalid.")
        mX = RT_DBaseGetField(Analyze5DB,0,1)
        mY = RT_DBaseGetField(Analyze5DB,0,2)
        mW = RT_DBaseGetField(Analyze5DB,0,3)
        mH = RT_DBaseGetField(Analyze5DB,0,4)
        RT_FileDelete(Analyze5DB)

This should be about right

    if(Mode == 1) {
        # Get coords of the logo mask
        if(!(mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255))) {
            Assert (FALSE, "Mask image is invalid.")
        }
        mX = YIRL_X
        mY = YIRL_Y
        mW = YIRL_W
        mH = YIRL_H

Oops, messed up, fixed.
EDIT: A bit more tinker. [Baffle might be a bit low]

EDIT: Or,

   if(Mode == 1) {
        # Get coords of the logo mask
        Assert (mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255), "Mask image is invalid.")
        mX = YIRL_X
        mY = YIRL_Y
        mW = YIRL_W
        mH = YIRL_H

EDIT: Baffle, about thickness of text + halo. [maybe slightly less]

[VoodooFX]

Originally Posted by StainlessS

This should be about right

But RT_YInRangeLocate must be in runtime script.

[StainlessS]

Yeah, I'm tryin' to figure out what you are trying to do, give me a bit of time.

[VoodooFX]

Originally Posted by StainlessS

Yeah, I'm tryin' to figure out what you are trying to do, give me a bit of time.

No rush here, I wrote it a month ago, still need to rewrite some InpaintDelogo's conditional madness to fit this func in.
It selects frames to do similar autoanalysis like AviUtl tool, just a bit better as we supply a base mask and scan area very close around logo (mode=1), mode=2 should be almost same as in AviUtl tool (except that it can deal with logos on the border with Ignore parameter. [actually I dunno exactly what AviUtl does, but it should be something similar]

[StainlessS]

OK, some backward steps here,

You seem to be wantin to handle some stuff in runtime and some other in frameserve periods, maybe you can figure out what you want from this

   if(Mode == 1) {
        VFX2="""
            if(mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255)) {
                # Convert X,Y,W,H coords to L,T,-R,-B coords
                mL =  YIRL_X
                mT =  YIRL_Y
                mR = -VW + mL + YIRL_W
                mB = -VH + mT + YIRL_H

                RT_DBaseSet(Analyze5DB,current_frame,1, mL,mT,mR,mB)            # Write VoodooFX style coords to DB

            } else {
                RT_DBaseSetField(Analyze5DB,current_frame,0,2)
                Assert (FALSE, "Mask image is invalid.")                        # Not really a good place for Assert()
            }
            return Last
        """
        Analyze5DB = ("~InpaintDelogo_Analyze5_"+RT_LocalTimeString+".DB").RT_GetFullPathName
        RT_DBaseAlloc(Analyze5DB,Mask.FrameCount,"iiiii")                       # Alloc DBase records, number of frames in mask clip
        mask.GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true)
        Parse.RT_ForceProcess                                                   # FORCE PROCESS ENTIRE CLIP

# mL, mT etc not available here in frameserver (non-runtime) section. (so below will not work]

        Assert (!(mL < 8 && -1*mR < 8 || mT < 8 && -1*mB < 8), "Mask image or Loc is invalid.")    # Not really a good place for Assert()

        # Coords mod2 round up    # Maybe move into GSCriptClip, before write to DBase, later available in DBase.
        mL = mL + (mL % 2)
        mT = mT + (mT % 2)
        mR = mR - (mR % 2)
        mB = mB - (mB % 2)

        # Crop 2x2 pxs area from above or below the logo and make a background from it
        mid = (VW-mL+mR)/2+mL
        mid = mid + (mid % 2)
        clp2x2 = (mT >= -1*mB) ? clp.crop(mid-2, mT-2, -1*(VW-mid), -VH+mT+2) : clp.crop(mid-2, VH+mB-2, -1*(VW-mid), mB+2)

        clpB = clp2x2.PointResize(VW, VH)



        Msk1 = mask.MTballoon2PX(2).mt_logic(mask, mode="xor")  # 2px wide mask around logo (+2px offset from the base mask)
        clp5 = clpB.Overlay(clp, mask=Msk1)

    } else {

The GScriptclip stuff force processes entire length of mask clip,
stuff after that seems to wanna access coords (which are only accessable in runtime script), maybe you can slot some of it into the Gscriptscript part.

Any good at all ?

EDIT: Maybe instead of Assert, write eg -1 as status to DBase, to detect error later, or something ???
EDIT: clp2x2 = (mT >= -1*mB) ? clp.crop(mid-2, mT-2, -1*(VW-mid), -VH+mT+2) : clp.crop(mid-2, VH+mB-2, -1*(VW-mid), mB+2)

Are above two parts same size, and same size as clp ? [EDIT: no not same size as clp, Scriptclip output MUST be same size as input]
If so, then I can maybe slot into GSCriptClip. [if not, then no idea].

Probably gone to bed soon. ZZZZzzzzzzz.....

[VoodooFX]

I don't see reason why we need to move all this stuff to GScriptClip, I think I would even get rid of GScriptClip...

Is it related to "Crop: Top and Left must be more than 0" error (on Avs 2.6!, func runs OK on Avs+)?

[StainlessS]

If you need coords, then you need runtime.

Maybe something like this [it dont work, but maybe roughly what is required]

   if(Mode == 1) {
        VFX2="""
            if(!mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255)) {
                RT_DBaseSetField(Analyze5DB,current_frame,0,2)
                Assert (FALSE, "Mask image is invalid.")                        # Not really a good place for Assert()
            }
            VW   = Last.Width
            VH   = Last.Height
            # Convert X,Y,W,H coords to L,T,-R,-B coords
            mL =  YIRL_X
            mT =  YIRL_Y
            mR = -VW + mL + YIRL_W
            mB = -VH + mT + YIRL_H
            Assert (!(mL < 8 && -1*mR < 8 || mT < 8 && -1*mB < 8), "Mask image or Loc is invalid.")    # Not really a good place for Assert()
            # Coords mod2 round up
            mL = mL + (mL % 2)
            mT = mT + (mT % 2)
            mR = mR - (mR % 2)
            mB = mB - (mB % 2)
            RT_DBaseSet(Analyze5DB,current_frame,1, mL,mT,mR,mB)                # Write VoodooFX style coords to DB
            # Crop 2x2 pxs area from above or below the logo and make a background from it
            mid = (VW-mL+mR)/2+mL
            mid = mid + (mid % 2)
            clp2x2 = (mT >= -1*mB) ? Last.crop(mid-2, mT-2, -1*(VW-mid), -VH+mT+2) : Last.crop(mid-2, VH+mB-2, -1*(VW-mid), mB+2)
            Return clp2x2.PointResize(VW, VH)                                   # Same size as input clip
        """
        Analyze5DB = ("~InpaintDelogo_Analyze5_"+RT_LocalTimeString+".DB").RT_GetFullPathName
        RT_DBaseAlloc(Analyze5DB,Mask.FrameCount,"iiiii")                       # Alloc DBase records, number of frames in mask clip
        clpB = clp.GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true)
        clpB.RT_ForceProcess                                                   # FORCE PROCESS ENTIRE CLIP

        Msk1 = mask.MTballoon2PX(2).mt_logic(mask, mode="xor")  # 2px wide mask around logo (+2px offset from the base mask)
        clp5 = clpB.Overlay(clp, mask=Msk1)

        # RT_FileDelete(Analyze5DB)

    } else {

ZZZZZZzzzzz......

EDIT:
Or if you dont need coords again later, then ony 1 frame at a time, no force process, no DB.

   if(Mode == 1) {
        VFX2="""
            if(!mask.RT_YInRangeLocate(Baffle=1,lo=255,hi=255)) {
                Assert (FALSE, "Mask image is invalid.")                        # Not really a good place for Assert()
            }
            VW   = Last.Width
            VH   = Last.Height
            # Convert X,Y,W,H coords to L,T,-R,-B coords
            mL =  YIRL_X
            mT =  YIRL_Y
            mR = -VW + mL + YIRL_W
            mB = -VH + mT + YIRL_H
            Assert (!(mL < 8 && -1*mR < 8 || mT < 8 && -1*mB < 8), "Mask image or Loc is invalid.")    # Not really a good place for Assert()
            # Coords mod2 round up
            mL = mL + (mL % 2)
            mT = mT + (mT % 2)
            mR = mR - (mR % 2)
            mB = mB - (mB % 2)
            # Crop 2x2 pxs area from above or below the logo and make a background from it
            mid = (VW-mL+mR)/2+mL
            mid = mid + (mid % 2)
            clp2x2 = (mT >= -1*mB) ? Last.crop(mid-2, mT-2, -1*(VW-mid), -VH+mT+2) : Last.crop(mid-2, VH+mB-2, -1*(VW-mid), mB+2)
            Return clp2x2.PointResize(VW, VH)                                   # Same size as input clip
        """
        clpB = clp.GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true)

        Msk1 = mask.MTballoon2PX(2).mt_logic(mask, mode="xor")  # 2px wide mask around logo (+2px offset from the base mask)
        clp5 = clpB.Overlay(clp, mask=Msk1)

    } else {

OOps, changed clp to Last inside GSCriptClip's. [using clp as input to GScriptClip]

EDIT: I get "No frames found to analyze, increase "Analyze5Th"." [with the top code block]

[VoodooFX]

Originally Posted by StainlessS

EDIT: I get "No frames found to analyze, increase "Analyze5Th"." [with the top code block]

Yes, same error, no frames on output for some reason.
Btw, I had to remove Analyze5DB from GScriptClip(VFX2, local=true, args="mask,Analyze5DB", after_frame=true).

EDIT:
Your edit run OK on Avs+, but it run OK there before too.
Maybe Gavino's thingy needs to be in it's own GSCriptClip too for Avs 2.6?
Or easier just to say goodbye to 2.6, I don't use it anyway.

[StainlessS]

Btw, I had to remove Analyze5DB

Yes of course, forgot that and did not try that block.
So, you dont need coords later then.
EDIT: And does the mod seem OK-ish, for what you want.

EDIT: Tomorrow is another day ... toodle pip.

[VoodooFX]

Originally Posted by StainlessS

EDIT: And does the mod seem OK-ish, for what you want.

Those asserts won't work from there I think, so only that. For me it's not a problem, but newbies do weird stuff and that would be uncatched.

From Mode=1/2 blocks I need only clip as clp5.