Making a mask to remove hardsubs

[pcroland]

Hi!

I have two video sources. The first source has good picture quality and hardcoded subs, the second source has low PQ but it does not contain hardsubs.

I want to remove the hardsubs from the video and replacing the texts with actual video.

This is my script so far:

a=FFMS2("X:\DL\hardsub.mkv")
b=FFMS2("X:\DL\nohardsub.mkv")
mask=Subtract(a,b).Levels(0, 0.16, 255, 0, 255, coring=true)
Overlay(a,b,0,0,mask)

I subtract the two image from each other and then tweaking the colors to make it kinda black and white only.

Here are some frames:
video with good PQ
video with bad PQ
difference between them
after tweaking it with levels
using the mask to Overlay the text

So my questions are:

How can I make the mask pure black and white? Is there any script that turns the image into 1 bit?
How can I expand the mask with for example 10-15 pixels? I tried to mess with mt_expand but no luck so far.

[wonkey_monkey]

There's such a microscopic difference between those two frames that I have to ask if it's really worth it - or if other scenes show a much bigger difference in quality. Presumably you're going to be re-encoding, so you may find out your re-encode looks just as "bad" as the one you've called bad, if the whole thing has such a small difference.

You'll need to do some more tweaking, anyway, since your "tweaked" difference frame doesn't include the drop shadow - maybe invert() and then use overlay with mode="lighten".

binarize in Masktools will do the black/white thing, or you can probably achieve the same thing with careful use of levels(). But you'll also need to spread out the mask a bit, otherwise you'll get some ghostly subtitle outlines.

[pcroland]

At other scenes the difference between the two sources is more visible so it's worth the effort. I'll use good settings to make it transparent to the source script.

Subtract(a,b).mt_binarize(170)

works really great Now the only thing missing is the expand

[poisondeathray]

mt_expand_multi
it's an avsi that comes with dither tools

[pcroland]

With which of mt_expand_multi's setting should I set the expand radius?

[poisondeathray]

Originally Posted by pcroland

With which of mt_expand_multi's setting should I set the expand radius?

sw, sh for width, height, respectively.

Hint : It tells you in the instructions

# Parameters:
# - sw : Growing/shrinking shape width. 0 is allowed. Default: 1
# - sh : Growing/shrinking shape height. 0 is allowed. Default: 1
# - mode : "rectangle" (default), "ellipse" or "losange". Replaces the
# mt_xxpand mode. Ellipses are actually combinations of
# rectangles and losanges and look more like octogons.
# Losanges are truncated (not scaled) when sw and sh are not
# equal.
# Other parameters are the same as mt_xxpand.

[pcroland]

Thanks Some frames are black and white and some frame has a green tint What should I do with that?

[poisondeathray]

Originally Posted by pcroland

Thanks Some frames are black and white and some frame has a green tint What should I do with that?

u=-128,v=-128

[pcroland]

Works great, thank you guys

I changed the Subtract(a,b) part to a.Levels(0, 0.01, 250, 0, 255), it seems that Level tweaking the hardsubbed video gives better results than subtracting the two videos.

[StainlessS]

This detects subs, but only if they have a halo, was never completed, op closed thread.
https://forum.doom9.org/showthread.p...36#post1782036

EDIT: Finds halo color and swells all instances several pixels, does the same with the text color,
where both swellings overlap (halo swells into text, and text swells into halo), then you have detected subtitle.

EDIT: Detector works, just never did repair script.

[pcroland]

It turned out that

Levels(0, 0.01, 250, 0, 255).Crop(200,800,-200,-50).AddBorders(200,800,200,50).mt_binarize(180).mt_expand_multi(mode="ellipse",sw=8,sh=8,u=-128,v=-128)

is not the best way to create the mask. Sometimes it picks up white parts other than the subtitles.

StainlessS, could you recommend settings to DetectSub_MI() ?

[StainlessS]

These are the important ones, but dependent upon your clip/subs.

# Subtitle coords
X        = 180
Y        = Height-140
W        = -X
H        = -44
# Text, Halo color
TEXTCOL   = $FDFDFD
TEXTVAR   = 2                             # Text RGB channels can vary by as much as TEXTVAR, eg $FEFEFE with TEXTVAR=1 ranges $FDFDFD -> $FFFFFF
HALOCOL   = $020202
HALOVAR   = 2                             # Halo RGB channels can vary by as much as HALOVAR, eg $FEFEFE with HALOVAR=1 ranges $FDFDFD -> $FFFFFF
# Text pixel width,height
TEXT_W    = 4                             # minimum text pixel 'thickness' Horizontal (horizontal thickness of vertical strokes)
TEXT_H    = 4                             # minimum text pixel 'thickness' Vertical (vertical thickness of horizontal strokes)
# Halo pixel width,height
HALO_L    = 5                             # Left Halo width in pixels
HALO_R    = 5                             # Right Halo width in pixels
HALO_T    = 5                             # Top Halo height in pixels
HALO_B    = 5                             # Bot Halo height in pixels
# Mt_Hysteresis (reduce stray pixels):- http://forum.doom9.org/showthread.php?p=1780589#post1780589
THYSCNT   = 5                             # Text Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
HHYSCNT   = 5                             # Halo Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
DHYSCNT   = 5                             # Detect clip Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0

These can (mostly) default based on above.

THRESH_W  = -(Text_w+(Halo_L+Halo_R)/2)   # -ve, specifies Abs(Pixel width)  {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
THRESH_H  = -(Text_h+(Halo_T+Halo_B)/2)   # -ve, specifies Abs(Pixel Height) {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
#
BAFFLE_W  = Text_w+Halo_L+Halo_R-2        # Width of outermost Vertical edges.
BAFFLE_H  = Text_h+Halo_T+Halo_B          # Height of outermost Horizontal edges.

Will likely not work on XP (with current masktools) due to no support for mt_Polish(InFix), in current masktools2.

Might be an amount of playing with settings to get it to work.

EDIT:

Sometimes it picks up white parts other than the subtitles.

Thats what the below swelling stuff overcomes (and why it needs halo).

Finds halo color and swells all instances several pixels, does the same with the text color,
where both swellings overlap (halo swells into text, and text swells into halo), then you have detected subtitle

[pcroland]

I tried this script:

X        = 200
Y        = 800
W        = -X
H        = -50
TEXTCOL   = $FEFEFE
TEXTVAR   = 2                             # Text RGB channels can vary by as much as TEXTVAR, eg $FEFEFE with TEXTVAR=1 ranges $FDFDFD -> $FFFFFF
HALOCOL   = $000000
HALOVAR   = 2                             # Halo RGB channels can vary by as much as HALOVAR, eg $FEFEFE with HALOVAR=1 ranges $FDFDFD -> $FFFFFF
TEXT_W    = 4                             # minimum text pixel 'thickness' Horizontal (horizontal thickness of vertical strokes)
TEXT_H    = 4                             # minimum text pixel 'thickness' Vertical (vertical thickness of horizontal strokes)
HALO_L    = 5                             # Left Halo width in pixels
HALO_R    = 5                             # Right Halo width in pixels
HALO_T    = 5                             # Top Halo height in pixels
HALO_B    = 5                             # Bot Halo height in pixels
THYSCNT   = 5                             # Text Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
HHYSCNT   = 5                             # Halo Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
DHYSCNT   = 5                             # Detect clip Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
THRESH_W  = -(Text_w+(Halo_L+Halo_R)/2)   # -ve, specifies Abs(Pixel width)  {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
THRESH_H  = -(Text_h+(Halo_T+Halo_B)/2)   # -ve, specifies Abs(Pixel Height) {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
BAFFLE_W  = Text_w+Halo_L+Halo_R-2        # Width of outermost Vertical edges.
BAFFLE_H  = Text_h+Halo_T+Halo_B          # Height of outermost Horizontal edges.
RESCAN    = true                          # Switching on might get rid of the odd stray pixel (most likely not, was already pretty good, suggest always on).
SHOW      = true                          # Show metrics.
AREACOLOR = $8040FF                       # Subtitle area marker color
SHOWAREA  = true                          # Set True to mark subtitle scan area.  (Show=False:No Effect)
STACK     = 2                             # 0=Movie, 1=Movie + detection clip, 2=Movie  detection clip + Text and Halo Mask clips. (Show=False:No Effect)
SHOWINPAND = false                        # Show Mt_Inpand masks without doing Mt_Hysteresis, ie for THysCnt, HHysCnt, DHysCnt. (Show=False:No Effect)
DB        = "MyDB.DB"                     # If Named, then DB not deleted on exit. (Maybe DBase used by other scripts later)
DEBUG     = false                         # To DebugView (Google). RT_ v2.00 beta02 extends RT_YInRangeLocate() debug info a little.
OverRide  = ""                            # ""=Not used, "*..." multiline string of ranges, "OverRide.txt"=Text file of ranges.

DetectSub_MI(Last,Override=Override,
    \ x=X,y=Y,w=W,h=H,
    \ Text_w=TEXT_W,Text_h=TEXT_H,TextCol=TEXTCOL,TextVar=TEXTVAR,
    \ Halo_L=HALO_L,Halo_T=HALO_T,Halo_R=HALO_R,Halo_B=HALO_B,HaloCol=HALOCOL,HaloVar=HALOVAR,
    \ THysCnt=THYSCNT,HHysCnt=HHYSCNT,DHysCnt=DHYSCNT,ShowInPand=SHOWINPAND,
    \ Baffle_w=BAFFLE_W,Baffle_h=BAFFLE_H,Thresh_w=THRESH_W,Thresh_h=THRESH_H,ReScan=RESCAN,
    \ Show=SHOW,stack=STACK,ShowArea=SHOWAREA,AreaColor=AREACOLOR,db=DB,Debug=DEBUG)

But I got an error:

Should I modify the mt_expand lines to mt_expand_multi or change them to ConvertYV12().mt_expand?

[pcroland]

And why do I see pale colors if I only use mt_binarize?

https://vgy.me/9gQ2Nb.png

If I use mt_expand_multi after mt_binarize it gets rid of the pale colors, but still weird.

[StainlessS]

Should I modify the mt_expand lines to mt_expand_multi or change them to ConvertYV12().mt_expand?

Ensure that source is YV12.

Post the same frame as source clip (avi saved with UT_Video codec "ULY0" FOURCC).

EDIT:

TEXTCOL   = $FEFEFE
TEXTVAR   = 2

$FE+2=$100, invalid color. (I imagine that I clipped it at $00 and $FF, but still invalid as is Halo color $00 - 2, just a be aware style warning)

EDIT: I think that the original script function was to find the rectangle where the subtitle lived, rather than a mask for just the subtitle.
Rectangle saved into the DBase for a second scan for fixing that rectangle.

[StainlessS]

This detects on your first posted image.

###
# DetectSub.MI_Client.avs"
###

Import(".\DetectSub_MI.avsi")


ImageSource("D:\jlS191.png",end=0).ConvertToYV12

X        = 640
Y        = 700
W        = 700
H        = 210

#return Overlay(Last.BlankClip(width=w,Height=h),x=x,y=y,opacity=0.3)

TEXTCOL   = $FAFAFA
TEXTVAR   = 5                             # Text RGB channels can vary by as much as TEXTVAR, eg $FEFEFE with TEXTVAR=1 ranges $FDFDFD -> $FFFFFF
HALOCOL   = $080808
HALOVAR   = 8                             # Halo RGB channels can vary by as much as HALOVAR, eg $FEFEFE with HALOVAR=1 ranges $FDFDFD -> $FFFFFF
TEXT_W    = 5                             # minimum text pixel 'thickness' Horizontal (horizontal thickness of vertical strokes)
TEXT_H    = 5                             # minimum text pixel 'thickness' Vertical (vertical thickness of horizontal strokes)
HALO_L    = 4                             # Left Halo width in pixels
HALO_R    = 4                             # Right Halo width in pixels
HALO_T    = 4                             # Top Halo height in pixels
HALO_B    = 4                             # Bot Halo height in pixels
THYSCNT   = 0                             # Text Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
HHYSCNT   = 0                             # Halo Mask Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
DHYSCNT   = 0                             # Detect clip Mt_Hysteresis inpand count (prior to mt_hysteresis), Dont Mt_Hysteresis if 0
THRESH_W  = -8                            # -ve, specifies Abs(Pixel width)  {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
THRESH_H  = -8                            # -ve, specifies Abs(Pixel Height) {Else +ve = Percent width 0.0->100.0%) {0.0=any pixel in range)
BAFFLE_W  = 10                            # Width of outermost Vertical edges.
BAFFLE_H  = 10                            # Height of outermost Horizontal edges.
RESCAN    = true                          # Switching on might get rid of the odd stray pixel (most likely not, was already pretty good, suggest always on).
SHOW      = true                          # Show metrics.
AREACOLOR = $8040FF                       # Subtitle area marker color
SHOWAREA  = true                          # Set True to mark subtitle scan area.  (Show=False:No Effect)
STACK     = 2                             # 0=Movie, 1=Movie + detection clip, 2=Movie  detection clip + Text and Halo Mask clips. (Show=False:No Effect)
SHOWINPAND =true                        # Show Mt_Inpand masks without doing Mt_Hysteresis, ie for THysCnt, HHysCnt, DHysCnt. (Show=False:No Effect)
DB        = "MyDB.DB"                     # If Named, then DB not deleted on exit. (Maybe DBase used by other scripts later)
DEBUG     = false                         # To DebugView (Google). RT_ v2.00 beta02 extends RT_YInRangeLocate() debug info a little.
OverRide  = ""                            # ""=Not used, "*..." multiline string of ranges, "OverRide.txt"=Text file of ranges.

DetectSub_MI(Last,Override=Override,
    \ x=X,y=Y,w=W,h=H,
    \ Text_w=TEXT_W,Text_h=TEXT_H,TextCol=TEXTCOL,TextVar=TEXTVAR,
    \ Halo_L=HALO_L,Halo_T=HALO_T,Halo_R=HALO_R,Halo_B=HALO_B,HaloCol=HALOCOL,HaloVar=HALOVAR,
    \ THysCnt=THYSCNT,HHysCnt=HHYSCNT,DHysCnt=DHYSCNT,ShowInPand=SHOWINPAND,
    \ Baffle_w=BAFFLE_W,Baffle_h=BAFFLE_H,Thresh_w=THRESH_W,Thresh_h=THRESH_H,ReScan=RESCAN,
    \ Show=SHOW,stack=STACK,ShowArea=SHOWAREA,AreaColor=AREACOLOR,db=DB,Debug=DEBUG)

I got your PM''ed MKV's, one is 25FPS, other 23.976, given how close they are to each other, think I would go with what DavidHorman said,

There's such a microscopic difference between those two frames that I have to ask if it's really worth it

Way too much work to be worth the bother.

EDIT: The detector works better with colored halo, which is what it was written for (I think, white and black are a little too common).
Anyway, locates the rectangle (in purple) which is what it was intended to do.

EDIT: DBase fields set for each frame/record., (purple area, X,Y,W,H), set after complete clip scan.

    /*  Fields
            0) Status,  0=Unknown (unvisited), 1=Sub, 2 Not sub, 3 User OverRide
            1) X
            2) Y
            3) W
            4) H
    */

eg

AviSource("...")
DB = "MyDB.DB"
SSS="""
    n=current_frame
    Stat=RT_DBaseGetField(DB,n,0)
    if(Stat == 1) {
        X=RT_DBaseGetField(DB,n,1)
        Y=RT_DBaseGetField(DB,n,2)
        W=RT_DBaseGetField(DB,n,3)
        H=RT_DBaseGetField(DB,n,4)
        RT_Subtitle("%d] X=%d Y=%d W=%d H=%d",n,x,y,w,h)
    }
"""
ScriptClip(SSS)

EDIT: In the 1st client script, x,y,w,h should be greatest enclosing coords of any/all subtitles.