Script to detect intertitles?

[Lyris]

Hey forum,
I have a really challenging source here. It's a long, silent film that's going on one DVD9. The length dictates an average bit rate of just about 5mbps.

Here's the problem. Being a silent film, there are frequent inter-titles. These have quite different characteristics to the rest of the film (I'm using CCE SP3 which allows you to tune Residual, Activity, and less usefully, Luminance characteristics).

There are a couple of ways to deal with this. One is to do some sort of filtering on the intertitles (clipping low-end luminance, some sort of temporal filter, perhaps) or to adjust the quantization or bitrate settings on the CCE side.

All of the solutions need some way to detect the intertitles relative to the rest of the film. Maybe it could be done based on motion analysis (if it doesn't move much, chances are it's an intertitle?) or preferrably, based on the presence of dark pixels in the frame (if it's predominantly dark, it's an intertitle).

It's a long shot, I know. But if I can just find a way of detecting the ranges of frames where the intertitles lie, I can write some sort of solution to automatically deal with them (be it feeding new settings to CCE or AviSynth or whatever). Anyone ever come across this sort of thing? Sadly I can't post a sample.

[Guest]

Not even a screenshot of a typical intertitle?

[StainlessS]

Are we talking white text on a black background between scenes ?

Perhaps RT_YInRange() could provide a means to get percentage of pixels in background range and also
percentage of pixels in text white range, if not much in between then is inter title.
(Obviously there should also be a lot less white than black).

EDIT: Also, are there also black only frames in inter title.

[johnmeyer]

Stainless' tools (which include RT_YInRange() and other functions) would definitely let you do this. The key "telltale" attribute of a silent film title is that it contains (usually) only two discrete luma values: the value for the near-white text, and the value for the near-black background. If you were to look at a luma histogram of a silent film title, you'd see two "spikes," one for the text somewhere between 200 and 235 (or 255) and the other for the background, somewhere between 0 (or 15) and 40. There might be some other luma values due to the fringing between the text and the background, and also due to noise or dust, but the predominant luma "energy" would be in just two areas. So, the approach would be to use Stainless' tools to look for frames that lack any appreciable luma values outside of the two ranges described above.

I don't know if you could write code to automatically detect bimodal luma values, or whether you'd have to find these yourself and then use fixed values for all frames. If there isn't much variation from one title to the next, the simpler approach should be sufficient.

Having just completed a project using Stainless' tools, I'm pretty certain they can do what you want.

[StainlessS]

Can you test this out to see if it is detecting.

Have included a test routine to simulate problem.


AVISource("D:\avs\test.avi")

#TestClip() # Just for testing not crash

BLKHI = 40 # hiest Black level
WHTLO = 200 # Loest White level
THRESH = 97.5 # Total WHITE + BLACK, minimum Perc to be Inter Title
HILITE = True # Switch on text location hi-liting

return ShowMetrics(blk_hi=BLKHI,Wht_Lo=WHTLO,IT_THRESH=THRESH,HiLite=HILITE)

Function TestClip(clip c) {
c
K=c.BlankClip()
KT=K.Subtitle("This is just a test text ABCDEFGHIJKLM\nAnd some more text abcdefghijklm\nAaBbCcDdeEFfGgHhIiJjKkLlMm\n", \
x=K.width/2-150,y=K.Height-100,text_color=$E0E0E0,lsp=0)
C2=0
GScript("""
for(i=0,Framecount-1,200) {
s=i
e=min(FrameCount-1,s+149)
NewC=Trim(s,e)
C2 = (C2.IsClip()) ? C2 + NewC : NewC
if(i+200 < FrameCount -1) {
C2=C2+ K.Trim(e+1,-10)
C2=C2+ KT.Trim(e+11,-30)
C2=C2+ K.Trim(e+41,-10)
}
}
""")
return C2
}

Function ShowMetrics(clip c,int "Blk_hi",int "Wht_Lo",Float "IT_Thresh",Bool "HiLite") {
c
ConvertToRGB32() # for HiLite non mod coords
Blk_hi=Default(Blk_hi,40)
Wht_lo=Default(Wht_lo,200)
IT_Thresh=Float(Default(IT_Thresh,97.5))
HiLite=Default(HiLite,False)
ScriptClip("""
KPerc = RT_YInRange(lo=0,hi=Blk_hi) * 100.0
WPerc = RT_YInRange(lo=Wht_lo,hi=255) * 100.0
itot = WPerc + KPerc # Total frame area that is Black OR White
IsTit = itot >= IT_Thresh && WPerc < KPerc
Got=(IsTit && HiLite) ? RT_YInRangeLocate(Baffle=2,lo=Wht_lo/2,hi=255,prefix="SHWM_") : False
(Got) ? OverLay(Last.BlankClip(color=$FF0000,width=SHWM_W,height=SHWM_H),x=SHWM_X,y=SHWM_Y,opacity=0.5) : NOP
SS=String(current_frame) + "] W%="+String(WPerc,"%-6.2f")+" K%="+String(KPerc,"%-6.2f")+" Tot%="+String(itot,"%-6.2f") + \
" Title="+((IsTit)?"YES":"NO")+ " : FoundText="+((Got)?"YES":"NO")
Subtitle(SS)

""",After_Frame=True,args="Blk_hi,Wht_lo,IT_Thresh,HiLite") # Needs Grunt for args
return Last
}


EDIT: Need RT_Stats, GScript, Grunt.
EDIT: Code edited. THRESH could probably be 98.0% without problems (guessing).
EDIT: Used Wht_Lo/2 in Hilite detect as subtitle creates non white fringes @ ~130.

[StainlessS]

Added HiLite arg to above post ShowMetrics .

EDIT: What are your requirements, just a list of ranges or to pull out, edit and replace Title ranges, either is not too difficult.

[kolak]

Hey forum,
I have a really challenging source here. It's a long, silent film that's going on one DVD9. The length dictates an average bit rate of just about 5mbps.

Here's the problem. Being a silent film, there are frequent inter-titles. These have quite different characteristics to the rest of the film (I'm using CCE SP3 which allows you to tune Residual, Activity, and less usefully, Luminance characteristics).

There are a couple of ways to deal with this. One is to do some sort of filtering on the intertitles (clipping low-end luminance, some sort of temporal filter, perhaps) or to adjust the quantization or bitrate settings on the CCE side.

All of the solutions need some way to detect the intertitles relative to the rest of the film. Maybe it could be done based on motion analysis (if it doesn't move much, chances are it's an intertitle?) or preferrably, based on the presence of dark pixels in the frame (if it's predominantly dark, it's an intertitle).

It's a long shot, I know. But if I can just find a way of detecting the ranges of frames where the intertitles lie, I can write some sort of solution to automatically deal with them (be it feeding new settings to CCE or AviSynth or whatever). Anyone ever come across this sort of thing? Sadly I can't post a sample.

Who pays for such a level of tweaking :) ?
Use low values for Residual and Activity and save yourself time :)

[StainlessS]

What ?, people pay you to do this, do they ?


Think maybe we should go halfer's Lyris.

EDIT: By the way Lyris, the above comment is intended to be a joke, you keep your cash, I dont mind assisting with your little earner.

[Lyris]

Who pays for such a level of tweaking ?
My sanity :)

In all seriousness, I'm always looking for ways to improve quality through automation (see CCE Assist).

StainlessS: I will try this out and will be in touch if I end up using it :)

A less elaborate solution I've found is just to set a higher minimum bitrate to stop the intertitles being overly compressed...

[StainlessS]

Lyris, would still like to know if it successfully detected your titles (even if you dont use it), also see above edit.

[Lyris]

I will try it out, and don't worry, I picked up on the humor (which is hard to do online!). At the very least I'd get you a copy of the disc :)

[StainlessS]

I'm rarely serious about anything, but dont bother about the disk, just a report on how the script fared would be sufficient thanks.

[Lyris]

StainlessS: wow, just tried it, and it works brilliantly, for the most part.

Once, I've seen it throw a false positive; it detected a dark scene as an intertitle. But that is a really impressive script, were I to use it, the occasional dark scene having non optimal compression settings applied really wouldn't be a big deal.

I don't want to waste your time if using a higher minimum bitrate will do a similar job, but out of curiosity, is there any way to have it spit ranges of frames out to a text file?

[StainlessS]

spit ranges of frames out to a text file

Yes of course, but probably a bit occupied until about Wednesday, I just wanted to know if it was in the right ball park before perhaps wasting further time on a lost cause.

What were the approx numbers on the false +ve (given metrics). Did you try changing the defaults?

Take a peek at histogram as suggested by JohnMeyer to see where the peaks live for the titles, ( and approx limits for false
+ve and also for the rest).

EDIT: Was the dark scene a complete dark scene or perhaps a fade to black.
Also, is there a black border which up's the black % in the dark + ve.
Musta been an unusual false +ve scene to have all non white/black squashed into 2.5 % of pixels.

Perhaps reducing BLKHI from 40 to maybe 32 (or below) will remove the false +ve, yep, that sounds like it should detect OK
and skip the false +ve. Of course its all guess work at this end without even a jpg to go on.

[StainlessS]

Script update, including frames command files and frame range files generation.
Demo to extract both Title and Scene as individual clips, edit, and then put them back again.


AVISource("D:\avs\test.avi")
#------------------------
TestClip() # Just for testing not crash
#------------------------
#EDIT BELOW Parameters
X=0 Y=0 W=0 H=0 # Coords, exclude borders
BLKHI = 32 # Highest Black level
WHTLO = 200 # Lowest White level
THRESH = 97.5 # Total WHITE + BLACK, minimum Percent to be Inter Title
HILITE = True # Switch on text location hi-liting in ShowMetrics, Added nicety, no particular use here, demo only.
SEP = "," # Separator for Range files in MakeFiles().
#---------------------
# Show Metrics to get correct parameters prior to MakeFiles()
return ShowMetrics(blk_hi=BLKHI,Wht_Lo=WHTLO,IT_THRESH=THRESH,HiLite=HILITE,x=X,y=Y,w=W,h=H) # Show Metrics ONLY
#---------------------
# Make command files, MUST use before ANY of remainding functions.
return MakeFiles(blk_hi=BLKHI,Wht_Lo=WHTLO,IT_THRESH=THRESH,x=X,y=Y,w=W,h=H,sep=SEP).AssumeFPS(250.0) # Commands files for both titles & scenes
#---------------------
# Just a demo which modifies BOTH Scene and Title frames, and puts them BOTH back into original clip. Requires MakeFiles generated command files.
TitleClip=SelectTitleFrames().FlipHorizontal() # Select Titles only Clip, and demo EDIT
SceneClip=SelectSceneFrames().Invert() # Select Scene only clip, and demo EDIT
return ReplaceSceneFrames(SceneClip).ReplaceTitleFrames(TitleClip) # Return demo EDITED Video, original Audio.
#---------------------

Function TestClip(clip c) {
# TestClip generator.
c
K=c.BlankClip()
KT=K.Subtitle("This is just a test text ABCDEFGHIJKLM\nAnd some more text abcdefghijklm\nAaBbCcDdeEFfGgHhIiJjKkLlMm\n", \
x=K.width/2-150,y=K.Height-100,text_color=$E0E0E0,lsp=0)
C2=0
GScript("""
for(i=0,Framecount-1,200) {
s=i
e=min(FrameCount-1,s+149)
NewC=Trim(s,e)
C2 = (C2.IsClip()) ? C2 + NewC : NewC
if(i+200 < FrameCount -1) {
C2=C2+ K.Trim(e+1,-10)
C2=C2+ KT.Trim(e+11,-30)
C2=C2+ K.Trim(e+41,-10)
}
}
""")
return C2
}

Function ShowMetrics(clip c,int "Blk_hi",int "Wht_Lo",Float "IT_Thresh",Bool "HiLite",int "X",int "Y",int "W",int "H") {
# Use for finding best parameters to MakeFiles()
c
ConvertToRGB32() # for HiLite non mod coords
Blk_hi=Default(Blk_hi,40)
Wht_lo=Default(Wht_lo,200)
IT_Thresh=Float(Default(IT_Thresh,97.5))
HiLite=Default(HiLite,False)
X=Default(X,0) Y=Default(Y,0) W=Default(W,0) H=Default(H,0)
ScriptClip("""
KPerc = RT_YInRange(lo=0,hi=Blk_hi,x=X,y=Y,w=W,h=H) * 100.0
WPerc = RT_YInRange(lo=Wht_lo,hi=255,x=X,y=Y,w=W,h=H) * 100.0
itot = WPerc + KPerc # Total frame area that is Black OR White
IsTit = itot >= IT_Thresh && WPerc < KPerc
Got=(IsTit && HiLite) ? RT_YInRangeLocate(Baffle=2,lo=Wht_lo/2,hi=255,prefix="SHWM_",x=X,y=Y,w=W,h=H) : False
(Got) ? OverLay(Last.BlankClip(color=$FF8080,width=SHWM_W,height=SHWM_H),x=SHWM_X,y=SHWM_Y,opacity=0.5) : NOP
SS=String(current_frame) + "] W%="+String(WPerc,"%-6.2f")+" K%="+String(KPerc,"%-6.2f")+" Tot%="+String(itot,"%-6.2f") + \
" Title="+((IsTit)?"YES":"NO")+ " : FoundText="+((Got)?"YES":"NO")
Subtitle(SS)
""",args="Blk_hi,Wht_lo,IT_Thresh,HiLite,X,Y,W,H") # Needs Grunt for args
return Last
}

Function MakeFiles(clip c,int "Blk_hi",int "Wht_Lo",Float "IT_Thresh",int "X",int "Y",int "W",int "H",String "Sep", \
String "TitleFrames",String "SceneFrames",String "TitleRange",String "SceneRange") {
# Make Command files for Select and Replace functions
c
Blk_hi=Default(Blk_hi,40) Wht_lo=Default(Wht_lo,200) IT_Thresh=Float(Default(IT_Thresh,97.5))
X=Default(X,0) Y=Default(Y,0) W=Default(W,0) H=Default(H,0) Sep=Default(Sep,",")
TitleFrames=Default(TitleFrames,"TitleFrames.txt")
SceneFrames=Default(SceneFrames,"SceneFrames.txt")
TitleRange=Default(TitleRange,"TitleRange.txt")
SceneRange=Default(SceneRange,"SceneRange.txt")
(Exist(TitleFrames)) ? RT_FileDelete(TitleFrames) : NOP # Delete existing
(Exist(SceneFrames)) ? RT_FileDelete(SceneFrames) : NOP
(Exist(TitleRange)) ? RT_FileDelete(TitleRange) : NOP
(Exist(SceneRange)) ? RT_FileDelete(SceneRange) : NOP
KPerc = RT_YInRange(n=0,lo=0,hi=Blk_hi,x=X,y=Y,w=W,h=H) * 100.0
WPerc = RT_YInRange(n=0,lo=Wht_lo,hi=255,x=X,y=Y,w=W,h=H) * 100.0
itot = WPerc + KPerc # Total frame area that is Black OR White
IsTit = itot >= IT_Thresh && WPerc < KPerc
Global PTit = IsTit
Global PS = 0
ScriptClip("""
n=current_frame
KPerc = RT_YInRange(n=n+1,lo=0,hi=Blk_hi,x=X,y=Y,w=W,h=H) * 100.0
WPerc = RT_YInRange(n=n+1,lo=Wht_lo,hi=255,x=X,y=Y,w=W,h=H) * 100.0
itot = WPerc + KPerc # Total frame area that is Black OR White
IsTit = itot >= IT_Thresh && WPerc < KPerc # Title status for NEXT FRAME
RT_TxtWriteFile(String(n),(PTit)?TitleFrames:SceneFrames,append=True) # Write frame commands (sampled on previous iteration)
Close = (PTit!=IsTit || n==FrameCount-1) # Next frame changed OR Last Frame
(Close) ? RT_TxtWriteFile(String(PS)+Sep+String(n),(PTit)?TitleRange:SceneRange,append=True): NOP
Global PS= (Close) ? n+1 : PS # Next frame is new range ?
Global PTit=IsTit
return Last
""",args="Blk_hi,Wht_lo,IT_Thresh,X,Y,W,H,TitleFrames,SceneFrames,TitleRange,SceneRange,Sep") # Needs Grunt for args
return Last
}

Function SelectTitleFrames(clip c,bool "Show",String "fn") {
# Select Title Frames from clip c using fn command file.
c
fn=Default(fn,"TitleFrames.txt")
# fn=Default(fn,"TitleRange.txt") # TitleRange.txt could also be used if SEP="," or " "
Show=Default(Show,False)
FrameSelect(CMD=fn,show=Show) # Returns NO audio
}

Function SelectSceneFrames(clip c,bool "Show",String "fn") {
# Select Scene Frames from clip c using fn command file.
c
fn=Default(fn,"SceneFrames.txt")
# fn=Default(fn,"SceneRange.txt") # SceneRange.txt could also be used if SEP="," or " "
Show=Default(Show,False)
FrameSelect(CMD=fn,show=Show) # Returns NO audio
}

Function ReplaceTitleFrames(clip c,clip RepC,bool "Show",String "fn") {
# Replace Titles RepC back into clip c using fn command file.
c
fn=Default(fn,"TitleFrames.txt")
# fn=Default(fn,"TitleRange.txt") # TitleRange.txt could also be used if SEP="," or " "
Show=Default(Show,False)
FrameReplace(RepC,CMD=fn,show=Show) # Returns Audio from clip c
}

Function ReplaceSceneFrames(clip c,clip RepC,bool "Show",String "fn") {
# Replace Scene RepC back into clip c using fn command file.
c
fn=Default(fn,"SceneFrames.txt")
# fn=Default(fn,"SceneRange.txt") # SceneRange.txt could also be used if SEP="," or " "
Show=Default(Show,False)
FrameReplace(RepC,CMD=fn,show=Show) # Returns Audio from clip c
}

EDITED: Added optional filenames to MakeFiles()

When using MakeFiles, must play all the way through without jumping around.
Requires RT_Stats, FrameSelect, Gscript, Grunt.