Detect And Remove Black frames

[boshreek]

hi
i have lots of clips that has black frames at the beggining and end of the clips varies between 1 to 10 seconds, i am new to avisynth and i tried to right a script to do that
the script removes the black frames at the beggining but at the end of the clip the last frame is duplicated as a still image to the end of the original clip duration (i do not now why)

ie
if the clip is 10 seconds and has 3 second of black frames
the first 3 seconds are removed but the clip duration is still 10 seconds cause the last frame was duplicated to the end of the clip

the code is as follows


loadplugin("C:\Program Files\AviSynth 2.5\plugins\GScript.dll")

global vid=DirectShowSource("C:\Documents and Settings\Administrator\Desktop\t.avi").ConvertToYV12 # DV

ScriptClip(vid,""" # this is a multiline string
n = current_frame

GScript("
i = 1
g=0
while ( i < 80 )
{
current_frame = i
lm_k = AverageLuma()
if(lm_k <17)
{
g=0
}
else
{
g=i
i=100
}
i = i+1
}
return vid.Trim(g,0)
")
""")

so is there another way to do that or is there a fix to this script

thanks

[Gavino]

The basic problem here is that ScriptClip cannot change the number of frames, so the last frame is repeated up to the length of the original clip. You can get round this by doing the processing outside ScriptClip, something like:
current_frame = 0
GScript("
while (current_frame < 80 && vid.AverageLuma() < 17) {
current_frame = current_frame + 1
}
")
vid.Trim(current_frame, 0)
This is a bit of a hack, since AverageLuma is meant to be used only inside run-time filters like ScriptClip - it works by setting current_frame directly, 'fooling' AverageLuma into working normally.

If you need to remove blank frames from the end as well as the beginning, add something similar working back from the end, or make the code into a suitable function that you apply also to vid.Reverse().

[boshreek]

very helpful solution
also your plug-in is very very nice

thank you very much

[Blatman]

The basic problem here is that ScriptClip cannot change the number of frames, so the last frame is repeated up to the length of the original clip. You can get round this by doing the processing outside ScriptClip, something like:
current_frame = 0
GScript("
while (current_frame < 80 && vid.AverageLuma() < 17) {
current_frame = current_frame + 1
}
")
vid.Trim(current_frame, 0)
This is a bit of a hack, since AverageLuma is meant to be used only inside run-time filters like ScriptClip - it works by setting current_frame directly, 'fooling' AverageLuma into working normally.

If you need to remove blank frames from the end as well as the beginning, add something similar working back from the end, or make the code into a suitable function that you apply also to vid.Reverse().

Hello Gavino ! I am trying to use the script portion to delete black frames at the end of the video I am processing. I think there is something missing in the script you shared no ? Also, I don't understand the "current_frame > 80".

In my case it seems that AVISynth only trim video from "current_frame = 0" to "0" and don't use the GScript part :(

If you can help me it could be great ! Thank you !

[StainlessS]

This should do ya.


A=ColorBars.Trim(0,-100).KillAudio # 100 frames
B=A.BlankClip # 100 frames
A++B # 200 frames, Black Starts at frame 100, till 199
ConvertToYV12
#Return ShowFrameNumber

ABOVE_BLACK = 17.0 # Float, 16.xx and less interpreted as black

current_frame=FrameCount-1
#GScript("""
while (current_frame >= 0 && AverageLuma < ABOVE_BLACK) {
current_frame = current_frame - 1
}
#""")

Return (current_frame<0)
\ ? MessageClip("Pre-Black Not found (All Black)")
\ : Trim(0,-(current_frame+1)).Subtitle("Trimmed off frame "+String(current_frame+1,"%.0f Till end of clip"))


EDIT: I assume that you are using Avs+, so GScript() call [In BLUE] commented out.

EDIT:
Also, I don't understand the "current_frame > 80"
Only chop off at most 80 frames (0-79). I have not included a limit like that (If you need, then say).
AverageLuma really only is intended to run in the Runtime Environment eg Scriptclip, but ScriptClip (and other runtime filters)
set a special variable 'current_frame' so that AvergeLuma etc knows which frame its supposed to process.
Gavino has abused the avsynth system by artifically creating a current_frame, so as to use AverageLuma outside of runitime,
he's a very abusive person, is Gavino, but we still love him despite all of his faults. :)

EDIT:
If you need to remove blank frames from the end as well as the beginning, add something similar working back from the end, or make the code into a suitable function that you apply also to vid.Reverse().


... # Insert Gavino script, chop off start, max 80 frames
vid=Reverse
... # Insert Gavino script, to just chop off from the end, max 80 frames
Reverse

or from end only

vid=vid.Reverse
... # Insert Gavino script, to just chop off from the end, max 80 frames
Reverse

[jmc718]

I came here looking for a way to remove all black frames from a video. Turns out that's not what everyone else here wanted so I wrote my own. If someone finds this in the future and wants to do the same, I used this code:

vid = LWLibavVideoSource("C:\some\path\video.mp4", format="YUV420P8")

current_frame = 0
# While we are not at the end of the video
while (current_frame <= vid.FrameCount) {
# If we are below the black threshold
if (vid.AverageLuma() < 17) {
# set the video equal to everything except the current frame
vid = vid.trim(0,current_frame-1) ++ vid.trim(current_frame+1,0)
}
# otherwise, simply advance to the next frame
else {
current_frame = current_frame + 1
}
}
# Output
vid

I realize forums don't always like it when you resurrect dead threads, but I'm hoping this will be more of a help than a hindrance.

[StainlessS]

jmc718,
Your script might get slow if many trim/splices (1000's, 2 trims and 1 splice for each killed frame),
Maybe better and easier if just create a frames file and use eg FrameSel or Prune via SelectRanges() or RejectRanges().

See here:- https://forum.doom9.org/showthread.php?p=1959013#post1959013
You could use the (above script at above link) PruneGenny.avsi to replace your script totally,
just need give a condition to output chosen frames.

Some related links here:- https://forum.doom9.org/showthread.php?p=1930974#post1930974

NOTE,
If using some frames creator script, then later use frames file to remove frames, then can combine scripts into single pass, eg

pseudo code

SourceSource("...")
Frames=".\Frames.txt"
RT_FileDelete(Frames) # kill existing

# Pass 1
MakeSomeKindOfFramesFile(Frames=Frames)

RT_ForceProcess() # Force creation of Frames file using above code, scans complete file returning only on completion where Frames file will exist.

# Pass 2
RejectRanges(cmd=Frames) # Reject frames in Frames File
#SelectRanges(cmd=Frames) # Select frames in Frames File

Return last


You can also temp extract frames in frames file for perusal, eg
FrameSel(cmd=Frames,Ordered=True,Reject=False)

Or look at frames NOT in Frames file, eg
FrameSel(cmd=Frames,Ordered=True,Reject=True)

EDIT: FrameSel/Prune could process about 40,000 individual frames in frames file in about 1 second, on an XP core Duo machine (before frameserving begins),
and would to all intents and purposes have overhead of a single trim or splice function.

EDIT:
Command Generator functions : EDIT: DONT USE THIS, SEE NEXT POST
PruneGenny.avsi

# PruneGenny.avsi
Function PruneKeepFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional KEEP frames Command File generator for Prune()
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# May require additional condition for end frames for eg YDifferenceFromPrevious and YDifferenceToNext.
# Any-DifferenceFromPrevious,
# Gives 0 for frame zero and so you may want to add to condition " || current_frame==0" to include frame 0.
# Any-DifferenceToNext,
# Gives 0 for last frame and so you may want to add to condition " || current_frame==framecount-1 " to include last frame.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
WriteFileIf(c, fileName, condition, """ "0," """, "current_frame", append=false)
}

Function PruneDeleteFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional DELETE frames Command File generator for Prune()
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# Is EXACT opposite of PruneKeepFrames() for same conditional.
# NOTE, any additional end frame condition as used in PruneKeepFrames, will also be inverted
# eg " || current_frame==0", frame 0 would be deleted.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
condition = "!(" + condition + ")"
WriteFileIf(c, fileName, condition, """ "0," """, "current_frame", append=false)
}

###########################
# Below :HINT ON USE:
/* ################
Avisource("In.avi").ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious
##### pick one of below [can replace 0.1 with whatever threshold you like]
#PruneKeepFrames(".\cmd.txt","AverageLuma>=0.1")
#PruneDeleteFrames(".\cmd.txt","AverageLuma>=0.1")
##### Or with additional conditional for end frame [YDifferenceFromPrevious ALWAYS 0.0 on frame 0, YDifferenceToNext ALWAYS 0.0 on last frame]
#PruneKeepFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always keeps first frame 0 (else delete)
#PruneDeleteFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always deletes first frame 0 (else keep)
#PruneKeepFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always keeps last frame (else delete)
#PruneDeleteFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always deletes last frame (else keep)
Return last

##### And Pass2
Avisource("In.avi") #.ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious NOT REQUIRED HERE
Return Prune(cmd=".\cmd.txt")
################ */


so can use for your case, eg

PruneDeleteFrames(Frames,"AverageLuma() < 17")

As the Pass1 MakeSomeKindOfFramesFile(Frames=Frames)

[StainlessS]

OOoops, the PruneGenny.avsi thingy was probably not best flexible for you needs, is intended only for use with Prune as writes output clip index 0 too.

Maybe better,

# MakeframesGenny.avsi # General Purpose conditional frames file writer. # https://forum.doom9.org/showthread.php?p=1980401#post1980401

Function MakeframesKeepFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional KEEP frames Command File generator
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# May require additional condition for end frames for eg YDifferenceFromPrevious and YDifferenceToNext.
# Any-DifferenceFromPrevious,
# Gives 0 for frame zero and so you may want to add to condition " || current_frame==0" to include frame 0.
# Any-DifferenceToNext,
# Gives 0 for last frame and so you may want to add to condition " || current_frame==framecount-1 " to include last frame.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
WriteFileIf(c, fileName, condition, "current_frame", append=false)
}

Function MakeframesDeleteFrames(clip c, string fileName, string condition, bool "Fast") {
# Conditional DELETE frames Command File generator
# MUST call with correct colorspace for condition eg YDifferenceFromPrevious requires a Planar colorspace.
# Is EXACT opposite of MakeframesKeepFrames() for same conditional.
# NOTE, any additional end frame condition as used in FramesKeepFrames, will also be inverted
# eg " || current_frame==0", frame 0 would be deleted.

c=(Default(Fast,true))?c.AssumeFPS(250.0):c # Fast as we can if Fast = true (Default true)
condition = "!(" + condition + ")"
WriteFileIf(c, fileName, condition, "current_frame", append=false)
}

###########################
# Below :HINT ON USE:
/* ################
Avisource("In.avi").ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious
##### pick one of below [can replace 0.1 with whatever threshold you like]
#MakeframesKeepFrames(".\cmd.txt","AverageLuma>=0.1")
#MakeframesDeleteFrames(".\cmd.txt","AverageLuma>=0.1")
##### Or with additional conditional for end frame [YDifferenceFromPrevious ALWAYS 0.0 on frame 0, YDifferenceToNext ALWAYS 0.0 on last frame]
#MakeframesKeepFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always keeps first frame 0 (else delete)
#MakeframesDeleteFrames(".\cmd.txt","YDifferenceFromPrevious>=0.1 || current_frame==0") # Always deletes first frame 0 (else keep)
#MakeframesKeepFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always keeps last frame (else delete)
#MakeframesDeleteFrames(".\cmd.txt","YDifferenceToNext>=0.1 || current_frame==framecount-1") # Always deletes last frame (else keep)
Return last

##### And Pass2
Avisource("In.avi") #.ConvertToYV12() # Planar Required for AverageLuma and YDifferenceFromPrevious NOT REQUIRED HERE
Return FrameSel(cmd=".\cmd.txt")
################ */


Test

Colorbars.ConvertToYV12.KillAudio
F = Last.BlankClip(length=1)
A = Trim(0,-100) # 100 frames
A++F++A # frame 100 is black
SSS="""
Y=AverageLuma
RT_Subtitle("%d] y=%f",current_frame,Y)
"""
#ScriptClip(SSS) return last

RT_FileDelete(".\cmd.txt")
MakeframesKeepFrames(".\cmd.txt","AverageLuma < 17.0") # for above, writes Only Frame 100 (We are Writing the FrameNo we are going to Delete). [reverse Logic, fewer frames to write]

Return last


so can use for your case, eg

MakeFramesKeepFrames(Frames,"AverageLuma() < 17")

As the Pass1 MakeSomeKindOfFramesFile(Frames=Frames)

So can use simple frame file it produces with SelectRanges(), RejectRanges(), or FrameSel().

In your case we would use RejectRanges(cmd=Frames), to delete the frames in frames file.

So, 2 pass script

SourceSource("...")
Frames=".\Frames.txt"
RT_FileDelete(Frames) # kill existing

# Pass 1
MakeFramesKeepFrames(Frames,"AverageLuma() < 17") # write framenos we will delete

RT_ForceProcess() # Force creation of Frames file using above code, scans complete file returning only on completion where Frames file will exist.

# Pass 2
RejectRanges(cmd=Frames) # Reject frames in Frames File

Return last

[goorawin]

StainlessS happy new year to you and all the others that read this.

I have a big problem with an 8mm film transfer that was done many moons ago. It was standard 8mm filmed @16fps. As can be seen it has very inconsistent frame blending which nothing that I have tried seems to deal with them properly. However manually deleting them seems to give the best results so far, but of course that is rather slow.
Would it be possible to use “MakeframesDeleteFrames” to delete bad frames based on their differences as illustrated by DA-DB in this script? See image
FFVideoSource("D:\Avisynth Testing02\Clip1.mov")
ConvertBits(16)
src = last
DA = src.DeHalo_Alpha(darkstr=1.0)
DB = src.DeHalo_Beta()
StackVertical(StackHorizontal(src.Subtitle("Src"),DA.Subtitle("DA")),StackHorizontal(Subtract(DA,DB).Subtitle("DA-DB"),DB.Subtitle("DB")))

If so how could it be done?
Or is there a better way of doing this.
It certainly works using that visual output as a reference.
Here is the link to a sample clip
https://u.pcloud.link/publink/show?code=kZYwmeVZxkBM2yrKQ94TPJU82wXuVzqq7pPk

[StainlessS]

what/where is Dehalo_Beta ?

EDIT: OK, I found it as "dehalo_beta_Package". [google did not find it without the "_package" extension]
https://forum.doom9.org/showthread.php?p=1974526

[Reel.Deel]

what/where is Dehalo_Beta ?

https://github.com/DonCanjas/Avisynth-Functions/tree/main/Dehalo_Beta_Package

[StainlessS]

Yeah I got it thanks RD.

[StainlessS]

Can somebody post avsresize somewhere, my ISP blocks Archive.org
Wiki page with link:- http://avisynth.nl/index.php/Avsresize

Req for Dehalo_Beta_Package

[Reel.Deel]

Can somebody post avsresize somewhere, my ISP blocks Archive.org
Wiki page with link:- http://avisynth.nl/index.php/Avsresize

Req for Dehalo_Beta_Package

Here: https://forum.doom9.org/showthread.php?p=1973326#post1973326 or direct link: https://files.videohelp.com/u/223002/avsresize_r19.7z

[StainlessS]

Cheers Dude !

[StainlessS]

It certainly works using that visual output as a reference
OK Goorawin, you can try this lot,

No need to worry, I've left plenty for you to do, lots of fun tweaking [there is NO attempt at tweaking here, its ALL for you to do].


FFVideoSource(".\Clip1.mov")
ConvertBits(8) # RT_Stats 8 bit only
src = last
DA = src.DeHalo_Alpha(darkstr=1.0)
DB = src.DeHalo_Beta()
Subtract(DA,DB) # Subtract Tv.Levels mid point = 126
###
Lo = 126
Hi = 126

# COMMENT OUT ONCE YOU DECIDE ON LO & HI ARGS
ScriptClip(RT_String("Show(Last,current_frame, %d,%d)", Lo,Hi)) Return StackHorizontal(Src,Last) # show combined results Full/Even/Odd



Mode = "Full" # Or "Even" or "Odd"
Th = 0.5 # WHATEVER YOU DECIDE IT TO BE
Frames = ".\Frames.txt".RT_GetFullPathName

RT_FileDelete(Frames)
Func = "Test(Last,current_frame," + String(Lo) + "," + String(Hi) + "," + Chr(34) + Mode + Chr(34) + ")"

# Choose your condition, comment out one of below
Condition = "(" + Func + " < " + String(Th) + ")" # Smaller than whatever
#Condition = "(" + Func + " > " + String(Th) + ")" # Bigger than whateverr

RT_DebugF("Condition = '%s'",Condition) # Show in DebugView :- https://learn.microsoft.com/en-us/sysinternals/downloads/debugview

MakeframesKeepFrames(Last, Frames, Condition)



Return last

Function Test(clip c,int n, Int Lo, Int Hi, String "Mode") {
c
Mode = Default(Mode,"")
Return (mode=="even") ? RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : (mode=="odd") ? RT_YInRange(n=n,x=8,y=9,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=False,lo=Lo,hi=hi)

}

Function Show(clip c, int n, Int Lo, Int hi) { # Show results for Full frame, Even and Odd scanlines
c
Full = RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=False,lo=Lo,hi=hi) # Full Frame
Even = RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi) # Even scanlines only
Odd = RT_YInRange(n=n,x=8,y=9,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi) # Odd Scanlines only
S = RT_String("%d]\\nFULL = %f\\nEVEN = %f\\nODD = %f",n,Full,Even,Odd)
Return Subtitle(S,lsp=0,font="Courier New")
}



__END__

****************************************
******* MASKED Luma Y Functions ********
****************************************

Compile time/runtime functions, Planar, YUY2, RGB24 & RGB32. (RGB internally converted to YUV-Y).

The compiletime/runtime clip functions share some common characteristics:-

The 'n' arg is an optional frame number and defaults to 'current_frame' if not specified.
The x,y,w,h, coords specify the source rectangle under scrutiny and are specified as for Crop(), the default 0,0,0,0 is full frame.
If 'interlaced' is true, then every other line is ommited from the scan, so if eg y=1, then scanlines 1,3,5,7 etc are scanned,
if eg y=4 then scanlines 4,6,8,10 etc are scanned. The 'h' coord specifies the full height scan ie same whether interlaced is true
or false, although it will not matter if the 'h' coord is specified as eg odd or even, internally the height 'h' is reduced by 1
when interlaced=true and 'h' is even.

Matrix: Conversion matrix for conversion of RGB to YUV-Y Luma. 0=REC601 : 1=REC709 : 2 = PC601 : 3 = PC709,
Default = (Width > 1100 OR Height>600) then 3(PC709) else 2(PC601). YUV not used

The optional mask clip, governs which pixels are processed by the functions. Where a luma pixel in selected area of the the mask clip is
in range "MaskMin" to "MaskMax" inclusive, then those pixels will be processed. The Mask must also be Planar but not
necessarily the same colorspace as clip c, also must be same dimensions and have at least the same number of frames as clip c.
Calling without mask clip OR with MaskMin=0,MaskMax=255 will effectively ignore the mask and scan full x,y,w,h area.

###

RT_YInRange(clip c,int "n"=current_frame,int "delta"=0,int "x"=0,int "y"=0,int "w"=0,int "h"=0,bool "interlaced"=false,
int "lo"=128,int "hi"=lo,int "Matrix"=(Width>1100||Height>600?3:2),clip "mask"=NOT_USED,int "MaskMin"=128,"MaskMax"=255)
Return int -1, if no valid pixels in Mask clip.
Returns FLOAT value (0.0 -> 1.0) being the amount of pixels in the range "lo" to "hi" (inclusive), 1.0 is equivalent to 100%.
Implemented as requested by Martin53 (thankyou), NOTE, differs from other funcs that return range 0.0 to 255.0.
NOTE, lo defaults to 128, "hi" defaults to "lo".

EDIT: Oops: Bugfix changes.

You can use the Show() func to decide on whatever you choose to use as detector,
change lo, hi, and then decide Mode and Th.

Good luck.

EDIT: The Test() function returns amount of pixels in inclusive range Lo to Hi, returns 0.0 -> 1.0 where 1.0 equivalent to 100%.
Mode will test "full" frame, "even" or "odd" scanlines. [EDIT: with an 8 pixel edge border ignored]

https://i.postimg.cc/tJXJQ4xk/Goorawin-00.jpg (https://postimg.cc/tZMpFyDn)
Above image with lo and hi args BOTH 126 [tv.Levels mid point returned by Subtract()]

EDIT: Let us know what you eventually decide upon, Thanx :)

Shows Condition in DebugView:- https://learn.microsoft.com/en-us/sysinternals/downloads/debugview
eg
00004500 18:02:16.663 RT_DebugF: Condition = '(Test(Last,current_frame,126,126,"Full") > 0.500000)'

[goorawin]

Thank you so much for your input, its been great. As usual it has been very helpful.
I have tried many different settings and compared the frames with what has been done visually. Based on that information we had to many bad frames kept while deleting to many good frames. To try and get rid of the bad frames the frame rate became far to low.
So I do not think the script will do what is required, but then maybe I am missing something that you can point out to me.
To date I have only compared the frame output of your script with the visual frame output that I did in the first instance, which by the way gave very little field blending (see script “Deleting frames.avs”). As well it gave me a frame rate of nearly 15fps which is 1fps less than it was film at, and I think I can live with that.
Based on what I see, I believe it has to be done on a frame basis using the Y level difference between good and bad frames (see images Frame01 to Frame52). Good frames have a low Y level while bad frames have a higher Y level (see script Levels.avs). So if we can select on the Y level basis we might just get close to deleting the correct frames.
I have up loaded the images and the scripts that I used, all of which I hope you can follow.
Thank you once again for all your great work.
Here is the link again:
https://u.pcloud.link/publink/show?code=kZYwmeVZxkBM2yrKQ94TPJU82wXuVzqq7pPk

[StainlessS]

Thanks, that Histogram thingy was quite useful in guessin best lo and hi,

I still have not tweaked it too much, but seems to be in ball park.

Watch the subtitled numbers when crap there.
I only checked the 1st half dozen or so numbers produced, but seems to be as your manual frames list.

FFVideoSource(".\Clip1.mov")
ConvertBits(8) # RT_Stats 8 bit only
src = last
DA = src.DeHalo_Alpha(darkstr=1.0)
DB = src.DeHalo_Beta()
Subtract(DA,DB) # Subtract Tv.Levels mid point = 126
SB=Last

### Detect Amount of pixels in range 127 -> 147 (all above tv.levels mid grey 126)
Lo = 130 # CHANGED from 127
Hi = 147 # make anything up to 255 but best localized for crap range
Frames = ".\Frames.txt".RT_GetFullPathName

# COMMENT OUT ONCE YOU DECIDE ON LO & HI ARGS
ScriptClip(RT_String("Show(Last,current_frame, %d,%d)", Lo,Hi)) Return StackHorizontal(Src,Last,SB.Histogram()) # show combined results Full/Even/Odd



Mode = "Full" # Or "Even" or "Odd"
Th = 0.07 # WHATEVER YOU DECIDE IT TO BE : CHANGED from 0.4

RT_FileDelete(Frames)
Func = "Test(Last,current_frame," + String(Lo) + "," + String(Hi) + "," + Chr(34) + Mode + Chr(34) + ")"

# Choose your condition, comment out one of below
#Condition = "(" + Func + " < " + String(Th) + ")" # Smaller than whatever
Condition = "(" + Func + " > " + String(Th) + ")" # Bigger than whateverr is crap frame

RT_DebugF("Condition = '%s'",Condition) # Show in DebugView :- https://learn.microsoft.com/en-us/sysinternals/downloads/debugview

MakeframesKeepFrames(Last, Frames, Condition)
Return last

Function Test(clip c,int n, Int Lo, Int Hi, String "Mode") {
c
Mode = Default(Mode,"")
Return (mode=="even") ? RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : (mode=="odd") ? RT_YInRange(n=n,x=8,y=9,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=False,lo=Lo,hi=hi)

}

Function Show(clip c, int n, Int Lo, Int hi) { # Show results for Full frame, Even and Odd scanlines
c
Full = RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=False,lo=Lo,hi=hi) # Full Frame
Even = RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi) # Even scanlines only
Odd = RT_YInRange(n=n,x=8,y=9,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi) # Odd Scanlines only
S = RT_String("%d]\\nFULL = %f\\nEVEN = %f\\nODD = %f",n,Full,Even,Odd)
Return Subtitle(S,lsp=0,font="Courier New")
}

EDIT: Modified in RED, Do histogram on new Subtract clip SB instead of clip with subtitles on it.

EDIT: 127 -> 147 is approx range where crap lives in histogram.
Look at Subtitled numbers frame before crap, and crap frame to set the Th and Mode.
Odd field seem to have more crap, BUT, frame before crap frame also seems to have bigger odd field numbers, so I just left at full frame.
EDIT: Frames file with given numbers. [Maybe try with lo=128 too <or even 129>, so range 128 -> 147, will also likely need Th change too]
Might also benefit from playing with hi arg too, but 147 is probably not far off.

3
6
15
21
24
27
30
33
36
39
42
45
48
51
54
57
60
63
66
69
72
75
78
81
84
87
90
93
96
99
102
105
107
108
110
111
113
116
119
125
128
131
134
137
140
143
146
149
152
155
158
161
164
167
170
173
179
182
185
188
191
194
197
200
203
206
209
212
215
218
226
229
232
235
238
241
244
247
250
253
256
259
262
265
268
271
280


Your manual edit frames list

DeleteFrames(3,6,15,21,24,27,30,33,36,39,42,45,48,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93,96,99,101,102,104,105,107,108,110,111,113,114,116,117,119,120,122,123,125,126,128,131,134,137,140,143,
\146,149,152,155,158,161,164,167,170,173,176,179,182,185,188,191,194,197,200,203,206,209,211,212,214,215,217,218,220,221,223,224,226,227,229,230,232,233,235,236,238,239,241,242,244,245,
\247,248,250,253,256,259,262,265,268,271,274,277,280)

EDIT: FrameNo 101 is first mismatch compared to your list (I think), take particular note of subtitle metrics where mismatch, and frame before mismatch, for tweaking.

EDIT: I've just tried a little bit with lo=130, hi=147, Th=0.07, and numbers look maybe better than original guess.
(typical frame before crap = 0.02, crap frame 0.2, so like 10 times bigger where crap, give it a try)

EDIT: Frames file with lo=130, hi=147, Th=0.07, Mode="Full". CHANGED SCRIPT args as per this line.

3
6
15
21
24
27
30
33
36
39
42
45
48
51
54
57
60
63
66
69
72
75
78
81
84
87
90
93
95
96
98
99
101
102
104
105
107
108
110
111
113
114
116
119
122
125
128
131
134
137
140
143
146
149
152
155
158
161
164
167
170
173
176
179
182
185
188
191
194
197
200
203
206
209
211
212
214
215
217
218
220
221
223
224
226
227
229
230
232
233
235
236
238
241
244
247
250
253
256
259
262
265
268
271
274
277
280

Well thats enough tweaking from me, now your turn :)

[goorawin]

Thanks that is great, and has allowed the fine tuning to be much easier.
Now that I have the selected the frames to keep, how do I use that saved txt file or the existing script to actually keep the good frames and delete the bad frames from the base clip. I thought I could just use MakeframesKeepFrames to do that job, but seems not as nothing I have tried seem to work. I must have lost the plot somewhere along the line. Could you please head me in the right direction to do that final step.
Thanks

[goorawin]

I found a possible solution that may work.
FrameSel(last,"","G:\Family Movies\December 2022\Images15\Frames.txt")

[StainlessS]

Try the RejectRanges() function, its either in Prune or FrameSel zip,
or FrameSel with Reject=True.
(RejectRanges requires both Prune and FrameSel).
Prune supports audio cutting and prevents 'cracks' in audio where cut/spliced, but I guess you dont need that.

I did knock up a script to iterate through "full", "Even", Odd",
lo = 127 -> 150, and hi = Lo -> 180,
and no variation of those found a combination that complies with your provided list of required frames (to drop).
The most noisy good frame was always worse than the least noisy bad frame, in all cases.
Tweaking may not be at all easy [ie is impossible]. :(

The iterating script took about 2.5 -> 3 hours to produce an "Impossible" type answer
[i7-8700, iterating script single core only. EDIT: whilst watching movie on machine, and debug scrolling up on 2nd monitor].
(It did not look for most compliant result, that would take much longer probably EDIT: Well longer anyways)

I did try a couple of comb detectors too, but results similar to our script. (I presumed that you had already tried those. EDIT: without the DeHalo Subtract stuff).

EDIT: I think this is "Impossible" script as I used it

FFVideoSource(".\Clip1.mov")
ConvertBits(8) # RT_Stats 8 bit only
src = last
DA = src.DeHalo_Alpha(darkstr=1.0)
DB = src.DeHalo_Beta()
Subtract(DA,DB) # Subtract Tv.Levels mid point = 126
SB=Last

DB = ".\MyDB.DB"
RT_DBaseAlloc(DB,FrameCount,"b")
# Goorawin list of Bad frames
BADFRAMES = """
3 6 15 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87 90 93 96 99 101 102 104 105 107 108 110 111 113 114 116 117 119 120 122 123 125 126 128 131 134 137 140 143
146 149 152 155 158 161 164 167 170 173 176 179 182 185 188 191 194 197 200 203 206 209 211 212 214 215 217 218 220 221 223 224 226 227 229 230 232 233 235 236 238 239 241 242 244 245
247 248 250 253 256 259 262 265 268 271 274 277 280
"""

# Scan bad frames into DBase
While(!BadFrames.ChrIsNul) {
BadFrames = BadFrames.ChrEatWhite
If(!BadFrames.ChrIsNul) {
Assert(BadFrames.ChrIsDigit,"Parse Error")
f = BadFrames.RT_NumberValue
BadFrames = BadFrames.ChrEatDigits
RT_DBaseSetField(DB,f,0,True) # Bad frame
# RT_DebugF("%d] BAD",f,name="ScanFrames: ")
}
}


BestFact = 0.0
BestMd = 0
BestTh = 0.0
BestLo = 255
BestHi = 0
BestLoFrm=-1
BestHiFrm=-1
BestLowestBad=1.0
BestHighestGood=0.0
for(Md=0,0) { # 0=full, 1=even, 2=Odd : Pick Start and End
Mode = Select(Md,"Full","Even","Odd")
for (lo=127,127) { # Lo scan : Pick Start and End
for(hi=lo,lo) { # Hi scan : Pick Start and End (must be >= lo)
LowestBad = 1.0
LowestFrm = -1
HighestGood = 0.0
HighestFrm = -1
RT_DebugF("SCAN: Lo=%d Hi=%d Mode='%s'",Lo,Hi,Mode,Name="ThSCan: ")
for(frm=0,SB.FrameCount-1) { # Scan all frames
m = Test(SB,frm,Lo,Hi,Mode) # Metric
IsBad = RT_DBaseGetField(DB,frm,0) # Bad Frame ?
if(IsBad) {
if(m < LowestBad) {
LowestBad = m
LowestFrm = Frm
}
} Else {
if(m > HighestGood) {
HighestGood = m
HighestFrm = Frm
}
}
}
if(HighestGood > LowestBad) {
RT_DebugF("BAD_BADFRAMES_LIST: For Lo=%d Hi=%d Mode='%s' [%d]HighestGood{%f} > [%d]LowestBad{%f}",lo,hi,Mode,HighestFrm,HighestGood,LowestFrm,LowestBad,name="BADLIST: ")
} Else {
Fact = LowestBad / Max(HighestGood,0.0001)
if(Fact > BestFact) {
BestFact = Fact
BestMd = md
BestTh = (HighestGood + LowestBad) / 2.0
BestLo = Lo
BestHi = Hi
BestLoFrm=LowestFrm
BestHiFrm=HighestFrm
BestLowestBad=LowestBad
BestHighestGood=HighestGood
}
}
(BestFact>1.0) ? RT_DebugF("BESTSOFAR: BestFact=%f BestLowestBad=%f[%d] BestHighestGood=%f[%d] BestLo=%d BestHi=%d BestMode='%s' BestTh=%f",BestFact,BestLowestBad,BestLoFrm,BestHighestGood,BestHiFrm,BestLo,BestHi,Select(BestMd,"Full","Even","Odd"),BestTh,Name="ThSCan: ") : NOP
}
}
}

Mode = Select(BestMd,"Full","Even","Odd")
S1 = RT_String("BEST: Lo=%d Hi=%d Th=%f Mode='%s'",BestLo,BestHi,BestTh,Mode)
S2 = RT_String("BEST NOT FOUND, BAD Data list")

Return MessageClip(BestFact>1.0?S1:S2)

Function Test(clip c,int n, Int Lo, Int Hi, String Mode) {
c
Return (mode=="even") ? RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : (mode=="odd") ? RT_YInRange(n=n,x=8,y=9,w=-8,h=-8,interlaced=True,lo=Lo,hi=hi)
\ : RT_YInRange(n=n,x=8,y=8,w=-8,h=-8,interlaced=False,lo=Lo,hi=hi)

}

Function ChrIsNul(String S) {return RT_Ord(S)== 0} # End of String
Function ChrIsDigit(String S) {C=RT_Ord(S) return C>=48&&C<=57}
Function ChrEatWhite(String S) {i=1 C=RT_Ord(S,i) While(C==32||C>=8&&C<=13) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}
Function ChrEatDigits(String S) {i=1 C=RT_Ord(S,i) While(C>=48&&C<=57) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}

EDIT: NO, I must have edited the md, Lo, and Hi start and end of iteration in the for() loops,
set md to for(md=0,2), lo to for(lo=127,150), hi to for(hi=lo,180).