Gavino's: Delete frames satisfying user-defined condition

[Floatingshed]

Hello,

http://forum.doom9.org/showthread.php?t=163107

This works so very well but I would like it to delete the original as well as the duplicate (don't ask!). Is that possible?
Thanks.

[StainlessS]

Can you be more specific in requirement,
you want to delete START (where START is frame before 1st dupe) and 1st dupe only, or START + ALL following dupes ?
EDIT: Ignore Above, its rubbish

EDIT: NOTE, potential problem in linked script function is that it only compares dupes with the previous frame,
and not with the START frame, so could delete low motion sequences.

You could try this, (untested), deletes START + all dupes not just first.

DeleteFrames("YDifferenceToNext < 0.1 && current_frame < fc-1")


EDIT: Above does not remove final dupe :(

[StainlessS]

I guess above question was a bit stupid :(, you would end up deleting multiple pairs of dupes if not deleting all dupes in run.

How bout this


Function DeleteCompleteDupeRun(clip c,float "th",Bool "Debug") {
# Delete dupe START frame plus all following dupes.
# Dupes compared with run START frame rather than DupeFrame-1
# Req RT_Stats + GSCript.
myName="DeleteCompleteDupeRun: "
c
th=Float(Default(th,0.1)) Debug=Default(Debug,False)
res = BlankClip(c, length=0)
fc = FrameCount() fmx=fc-1
GSCript("""
n = 0
while(n < fc) {
DelS=n
while(n<fmx && RT_LumaDifference(Last,Last,n=DelS,n2=n+1) < th) { # YDifference To run START frame
n = n + 1
}
if(n>DelS) {
(Debug)?RT_DebugF("Deleting %d to %d",DelS,n,name=myName):NOP
n = n + 1 # Point @ start keep frame
}
if(n < fc) { # not at end
KeepS = n # start of wanted section
while((n<fmx && RT_YDifference(Last,n=n,delta=1) >= th) || n==fmx) { # YDifferenceToNext
n = n + 1
}
(Debug)?RT_DebugF("Keeping %d to %d",KeepS,n-1,name=myName):NOP
res = res + Trim(KeepS, KeepS-n)
}
}
""")
return res
}

ColorBars.ConvertToYV12.Killaudio

A=Trim(0,-1)
B=A.FlipHorizontal
C=A.Flipvertical
A=A.Subtitle("A")
B=B.Subtitle("B")
C=C.Subtitle("C")

#A+A # Del 0-1 == No Frames
#A+B # Keep 0-1 == AB
#A+B+C+ A+A+A +B+C+A # keep 0-2, Del 3-5, keep 6-8 == ABCBCA
#A+B+C+ A+A+A +B+C+C # keep 0-2, Del 3-5, keep 6, Del 7-8 == ABCB
A+B+C+ A+A+A +B # keep 0-2, Del 3-5, keep 6 == ABCB

DeleteCompleteDupeRun(0.1,Debug=True)


Only tested with above demo tests

[Floatingshed]

Works perfectly on my test clip. Many, many thanks.

[Gavino]

Hello,

http://forum.doom9.org/showthread.php?t=163107

This works so very well but I would like it to delete the original as well as the duplicate (don't ask!). Is that possible?
Thanks.
StainlessS's dedicated solution looks right (though I haven't tried it in practice).
I think it can also be done using my original generic DeleteFrames function by extending the supplied condition to be:
DeleteFrames("(YDifferenceFromPrevious < 0.1 && current_frame > 0)
\ || (YDifferenceToNext < 0.1 && current_frame < fc-1)")

[Floatingshed]

Sorry, don't understand the: ¦¦
Neither does the filter!
Thanks.

[Gavino]

Sorry, don't understand the: ¦¦
Neither does the filter!
Thanks.
Sorry, I meant to use || not ¦¦.
Edited the earlier post to fix.

[Floatingshed]

Thanks for the update but I still get syntax error. AVSP highlights the final ")" but that can be misleading...

[raffriff42]

DeleteFrames("(YDifferenceFromPrevious < 0.1 && current_frame > 0)
\ || (YDifferenceToNext < 0.1 && current_frame < fc-1)")

[Gavino]

DeleteFrames("(YDifferenceFromPrevious < 0.1 && current_frame > 0)
\ || (YDifferenceToNext < 0.1 && current_frame < fc-1)")
Thanks raffriff.
Should have tested it - I must be getting rusty. :)
Original post edited (again!)

[StainlessS]

I think it can also be done using my original generic DeleteFrames function by extending the supplied condition to be:
DeleteFrames("(YDifferenceFromPrevious < 0.1 && current_frame > 0)
\ || (YDifferenceToNext < 0.1 && current_frame < fc-1)")

Now that would have saved me some work, had I thought of it. :confused:

EDIT: With long run of (EDIT: Localized) low motion, my version might output eg
ddkddkddkddkddk instead of (d=del, k=keep)
ddddddddddddddd

due to comparisons with START frame rather than DupeFrame-1.

[Floatingshed]

Now I'm finding that on regular occasions I need to delete yet another frame.

My clip has some nasty blending (Kinescope with phase errors) at scene changes: A, A/B, B/A, damaged B, B.
If I run John Meyers' scene change detection script I get: A, A, A, damaged B, B.
Then after Stainless/Gavino's script: A, damaged B, B.
So, does anyone have any idea for removing "damaged B"? TBH its all a bit above me, I don't know how Gavino's script does what it does. Wish I had a brain!
Thanks for all the help.

[raffriff42]

You need to define 'damaged' in Avisynth terms...

Does 'Kinescope with phase errors' mean the colors are wonky? Then try
DeleteFrames("(YDifferenceFromPrevious < 0.1 && current_frame > 0)
\ || (YDifferenceToNext < 0.1 && current_frame < fc-1)"
\ || (UDifferenceToNext > 42 && current_frame < fc-1)")

http://avisynth.nl/index.php/UDifferenceToNext#Runtime_functions
- YDifferenceToNext tests luminance (brightness)
- UDifferenceToNext tests color on the blue-yellow axis
- VDifferenceToNext tests color on the red-green axis

Threshold level "42" was chosen at random. Try different values.

EDIT thx StainlessS, please remove the red quote.

[StainlessS]

RaffRiff, you got too many quotes in there ? (middle line)

Not related to previous Floatingshed post, here func for deleting dupes not-including START of run (just duplicates),
Dupes compared with inital START of run frame rather than DupeFrame-1.


Function DeleteDupRun(clip c,float "th",Bool "Debug") {
# Delete all dupes (leaving initial START frame [of which dupes are a copy]).
# Dupes compared with run START frame rather than DupeFrame-1
# Req RT_Stats + GSCript.
myName="DeleteDupRun: "
c
th=Float(Default(th,0.1)) Debug=Default(Debug,False)
res = BlankClip(c, length=0)
fc = FrameCount() fmx=fc-1
GSCript("""
n = 0
while(n < fc) {
DupS=n # n - 1 is START (if n != 0)
while(n<fc && n>0 && RT_LumaDifference(Last,Last,n=DupS-1,n2=n) < th) {
n = n + 1
}
if(DupS<n) { # DupS < n if we are deleting some
(Debug)?RT_DebugF("Deleting %d to %d",DupS,n-1,name=myName):NOP
}
if(n < fc) { # not at end
KeepS = n # start of wanted section
n = n + 1
while(n<fc && RT_YDifference(Last,n=n,delta= -1) >= th) { # YDifference From Previous
n = n + 1
}
# Exit here with n == Dupe Seq START frame + 1 (or fc)
(Debug)?RT_DebugF("Keeping %d to %d",KeepS,n-1,name=myName):NOP
res = res + Trim(KeepS, KeepS-n)
}
}
""")
return res
}

ColorBars.ConvertToYV12.Killaudio

A=Trim(0,-1)
B=A.FlipHorizontal
C=A.Flipvertical
A=A.Subtitle("A")
B=B.Subtitle("B")
C=C.Subtitle("C")

#A+A # Keep 0-0, Del 1-1 == A
#A+B # Keep 0-1 == AB
#A+B+C+ A+A+A +B+C+A # keep 0-3, Del 4-5, keep 6-8 == ABCABCA
#A+B+C+ A+A+A +B+C+C # keep 0-3, Del 4-5, keep 6-7, Del 8 == ABCABC
A+B+C+ A+A+A +B # keep 0-3, Del 4-5, keep 6 == ABCAB

DeleteDupRun(0.1,Debug=True)


Again, only tested with included DEMO tests.

EDIT:
Threshold level "42" was chosen at random. Try different values.
Nah, I'm guessin that its because that number is "The meaning of, Life, the Universe, and Everything", and
also because its part of RaffRiff42's moniker. (random my foot).

EDIT:
From Below post, Nothing is random. Nothing.
I'll take that as a confession of guilt :)

EDIT:
42, Life, the Universe, and Everything:- http://www.independent.co.uk/life-style/history/42-the-answer-to-life-the-universe-and-everything-2205734.html
https://www.youtube.com/watch?v=aboZctrHfK8
http://www.theguardian.com/books/2011/feb/03/douglas-adams-42-hitchhiker

EDIT: The Movie, (The Hitch Hiker's Guide To The Galaxy) is quite naff compared to original TV series or even the World Service
radio broadcasts. (I've got both TV series and Movie on DVD, only had the stomach to watch the movie about twice,
like many/most US backed re-makes of British classics, result is garbage, US producers seem to presume that their audience is stupid)
EDIT: The YouTube clip linked above, is the naff Movie version.

[raffriff42]

Nothing is random. Nothing.


EDIT I'm not sure what that statement means. I just typed it "at random." :)

[Floatingshed]

'Kinescope with phase errors'.. is not a colour problem, the footage is B&W.
Phase errors in this case refers to the alignment of the tv fields with the film frames. A situation where a single frame of film can contain field B of a tv frame blended with field A of the next.

Here is an example, not one of the worst by any means.

https://www.mediafire.com/?p85f5sar45autrw

[StainlessS]

FloatingShed, cant you just eg convert toYV24/YUY2, AssumeTFF/AssumeBFF, SeparateFields, delete first field, weave, to match up like fields.

...

Having looked at Exotic Interlacing by ScharfisBrain see below (See Mediafire in my sig, DATA folder for Exotic Interlacing in English),

Imitation in AviSynth: (creating problem)

AVISource("24fpsFilm.avi")
AssumeFPS(25, true).DoubleWeave().SelectOdd()


This kind of the Telecine is called phase shifted or sometimes also as Perverse Telecine and can be repaired by AVISynth quite simply, by sorting the fields again:

Recovery in AviSynth:


DoubleWeave().SelectOdd()


EDIT: I think at least one of the field matching plugins should be able to fix Phase Shifted fields that do not stay in sequence (TFM or whatever).

[Floatingshed]

No, you misunderstand. This is a kinescope/Telerecording/Film-recording. Made, in essence, by pointing a film camera at a tv monitor.
The field/frame matching errors are burned into the film frames.

[StainlessS]

FS, you got a bit bigger sample, ~30MB please (audio not necessary).

EDIT: With a few of scene cuts.

[Floatingshed]

Ok. Will sort tomorrow. Thanks.

[StainlessS]

Here a couple of simple detect/fix scripts

Detector, inspired by John Meyer's scene change detector script (works on sample, may fail on other bad frames)

# FS_Detect.avs

Avisource("blend.mpg.avi")

scenethreshold=15.0

DELETE="DELETE.txt" # BAD frame to delete
AFTER ="AFTER.txt" # Frame after BAD frame (maybe replace with frame after that)

RT_FileDelete(DELETE)
RT_FileDelete(AFTER)

SSS=("""
n=current_frame
df = RT_YDifference(Last,n=n,delta=1) # YDifferenceToNext
pdf = RT_YDifference(Last,n=n,delta=-1) # YDifferenceToPrev
DEL=(df>scenethreshold)
CPY=(pdf>scenethreshold)
(DEL)?RT_WriteFile(DELETE,"%d",n,Append=True):NOP # RejectRanges Command file, does not require clip index
# (DEL)?RT_WriteFile(DELETE,"%d",n+1,Append=True):NOP # Delete AFTER frames instead of BELOW replace AFTER frames.
(DEL)?RT_WriteFile(AFTER,"1 %d",n+1,Append=True):NOP # Clipclop Command File, ie uses replacement Clip Index 1
RT_Subtitle("%d] %.3f %s",n,df,(DEL)?"DELETE":(CPY)?"FIX":"")
""")

ScriptClip(SSS)

#Play all way Through

#Or Use ForceProcessAVI() from TwriteAVI v2.0 to force write frames files
#ForceProcessAVI()

Return Last


and fix script

# FS_Fix.avs

c=Avisource("blend.mpg.avi")
c

# clips slid over 1, same length as original
PREV = c.DeleteFrame(c.Framecount()-1).DuplicateFrame(0) # Not used here
NEXT = c.DuplicateFrame(c.Framecount()-1).DeleteFrame(0)

DELETE="DELETE.txt" # BAD frames to delete
AFTER ="AFTER.txt" # Frame after BAD frame (maybe replace with frame after that)

ClipClop(Last,NEXT,Cmd=AFTER) # Fix frames AFTER real bad frame (COMMENT out to not fix AFTER frames)
RejectRanges(Last,Cmd=DELETE) # Delete real bad frames


return Last

# From FrameSel or Prune (Requires BOTH)
Function RejectRanges(clip c,String "SCmd",String "Cmd",Bool "TrimAudio",Float "FadeMS") {
# RejectRanges() by StainlessS. Required:- FrameSel, Prune, RT_Stats
# Wrapper to delete frames/ranges along with audio, can supply frames/ranges in SCmd string And/Or Cmd file.
# The wrapper makes for easier usage of Prune() which supports up to 256 input clips, but requires a clip index,
# eg '3, 100,200' would specify clip 3, range 100 to 200. The wrapper does away with the necessity for the clip index as we
# are only using a single clip here. Prune also does not have a 'reject' arg to delete specified frames rather than select them,
# this wrapper also converts a list of frames to delete into a list of frames to select so that we can use Prune and its audio
# capability.
#
# SCmd: Frames/Ranges specified in String (Frames/Ranges either Chr(10) or ';' separated, infix ',' specifies range, eg 'start,end').
# Cmd: Frames/Ranges specified in file (one frame/range per line, comments also allowed, see FrameSel for Further info).
# TrimAudio:
# True(default), deletes audio belonging to deleted frames
# False, returns original audio, probably out of sync.
# FadeMS: (default 1.0 millisec). Linear Audio Fade duration at splices when TrimAudio==true, 0 = dont fade (might result in audio 'clicks/cracks').
c
TrimAudio=Default(TrimAudio,True) # default true trims audio, false returns original audio (audiodubbed, as Framesel returns no audio)
FadeMS=Float(Default(FadeMS,1.0)) # 1 millisecond linear fadeout/fadein at splices
PruneCmd = (TrimAudio) ? "~Prune_"+RT_LocalTimeString+".txt" : ""
(!TrimAudio)
\ ? FrameSel(scmd=SCmd,cmd=Cmd,reject=true)
\ : FrameSel_CmdReWrite(PruneCmd,scmd=SCmd,cmd=Cmd,reject=true,Prune=True,range=true)
(TrimAudio) ? Prune(Cmd=PruneCmd,FadeIn=True,FadeSplice=True,FadeOut=True,Fade=FadeMS) : NOP
# If TrimAudio==true then delete Prune temp file, Else restore original Audio to the now audio-less clip
(TrimAudio)
\ ? RT_FileDelete(PruneCmd)
\ : (c.HasAudio) ? AudioDub(c) : NOP
Return Last
}

Scripts EDITED:

Requires RT_Stats, FrameSel, Prune, ClipClop.

Try without ClipClop() fix, it might be acceptable with only a single blended frame after the real bad one. (Comment out that line)

EDIT: If you wanted to delete AFTER frames too, then in detect script, write AFTER frames to DELETE file instead of AFTER file,
and comment out the ClipClop line. [Shown in BLUE in Detect script, where comment out write AFTER frames line].

[Floatingshed]

Wow! You have been busy whilst I was snoring! Thanks.
If you still want a longer clip here it is:

https://www.mediafire.com/?xgd4qhvdndxz2q8

I'll now have a play with your latest script... thanks again.


EDIT: I've had a play, this is astonishing work. Perfect. Thanks very much.

[StainlessS]

I've had a play, this is astonishing work. Perfect. Thanks very much.

Methinks you only tried with the short sample (or you are really easily pleased :) )

The problem with previously given detector is that with YDifferenceToNext, you have two frames, each of which is equally distant from the other. A high YDifference does not tell which of the two frames is the bad one.
I was hoping that JM knew something more than I when I based the detector on previously given scene change detector, but guessed that it would not work 'as advertised'.
Anyway that detector was not a complete waste of time if I can get below stuff to select the naff frame.

I have an idea that seems like it might be able to distinguish which of the two frames is bad, but I am only guessing, perhaps someone with a better grasp on mathematics could give some insight as to whether or not there may be any merit whatsoever in my guess.

Given two frames, each with some value of Stdev, if those two frames are blended (50-50), then Stdev of resultant frame should be less than for either source frame,

Somebody give me a yah, or a nah please. :thanks:

EDIT: Scratch above gibberish, it dont work in a test.
Although for some reason it does seems to be the case with second sample supplied by FloatingShed, think I'll pursue it a little longer.

[StainlessS]

Concerning previous post, I looked at above gibberish a little more and ran this:-


Avisource("N:\V\XMen2.avi")
Trim(10000,0) # Dump CGI

FC=FrameCount
A=Trim(0,FC/2)
B=Trim(FC/2+1,0)
C=Merge(A,B)

FC=C.FrameCount
C
ABOVE=0
BELOW=0
MID=0
SSS=("""
n=current_frame
S1=RT_YPlaneStdev(A,n=n)
S2=RT_YPlaneStdev(B,n=n)
S3=RT_YPlaneStdev(C,n=n)
ABOVE= (S3>S1 && S3 > S2) ? ABOVE + 1 : ABOVE
BELOW= (S3<S1 && S3 < S2) ? BELOW + 1 : BELOW
MID = (n+1) - ABOVE - BELOW
ZA=ABOVE/(n+1.0)
ZB=BELOW/(n+1.0)
ZM=MID/(n+1.0)
RT_Subtitle("%d ] %.3f %.3f %.3f %d %d %d %.3f %.3f %.3f",n,S1,S2,S3,ABOVE,BELOW,MID,ZA,ZB,ZM)
""")

Scriptclip(SSS)


After running for 30 minutes (25FPS PAL), I got these results


BLENDED_STD_ABOVE_BOTH_SOURCE = 00.00% <<<------- PERHAPS Useful for something
BLENDED_STD_BELOW_BOTH_SOURCE = 62.70%
BLENDED_STD_BETWEEN_BOTH_SOURCE = 37.30%


Although at one point I think below both source went down to about 33%
Get better both below, when both source indoors, ie probably a little darker.

Just in case anyone is interested.
EDIT: Initial test (Incl CGI in one source) gave worst results for blend stdev below both source.

[StainlessS]

OK FloatingShed, methinks we got this beat.

Looks like the Stdev stuff is down to combing, detector does seem to work very well, I have high hopes :).
EDIT: The bad frame blending, subdues the combing and so lower stdev (I think, maybe, perhaps).
EDIT: OK, not combing, but some kind of visible CRT raster lines, dont know what to call them, or how caused. (EDIT: Perhaps related to CRT persistence)

Here Detector mark 2, MUST play straight though, no jumping about.


# FS_Detect_2.avs

#Avisource("blend.mpg.avi")
Avisource("Blends.demuxed.m2v.AVI")

#### CONFIG ####
scenethreshold=15.0
# Set up to TWO of below to TRUE (DELETE takes precedence, ie REPLACED may still be DELETED)
DELETE_BAD = True
DELETE_AFTER = False
REPLACE_BAD = False
REPLACE_AFTER = True
#####################
#####################
#####################
DELETE = "DELETE.txt" # Frames to delete
REPLACE= "REPLACE.txt" # Frames to replace
DB = "FS_DET.DB"

FC=FrameCount

RT_FileDelete(DELETE)
RT_FileDelete(REPLACE)

# DBase Fields, 0=Bool Done, 1=Int DetectType, 2=Float YDif2Nxt, 3=Float YDif2Prv, 3=Float YStdev
RT_DBaseAlloc(DB,FC,"bifff")

SSS=("""
n=current_frame
det = FSDet(Last,DB,n,scenethreshold) # 0 = OK : 1 = Bad Frame : 2 = AFTER frame

# RejectRanges (Delete) file
(DELETE_BAD&&det==1)?RT_WriteFile(DELETE,"%d",n,Append=True):NOP # Delete BAD.
(DELETE_AFTER&&det==1)?RT_WriteFile(DELETE,"%d",n+1,Append=True):NOP # Delete AFTER

# Clipclop Command File
(REPLACE_BAD&&det==1)?RT_WriteFile(REPLACE,"1 %d",n,Append=True):NOP # Replace BAD with Prev
(REPLACE_AFTER&&det==1)?RT_WriteFile(REPLACE,"2 %d",n+1,Append=True):NOP # Replace AFTER frame with Next

df2nxt=RT_DBaseGetField(DB,n,2)
stdev =RT_DBaseGetField(DB,n,3)
RT_Subtitle("%d] %.3f %s",n,df2nxt,(det==1)?"DELETE":(det==2)?"FIX":"")
RT_Subtitle("Stdev=%.3f",stdev,align=1)
""")

ScriptClip(SSS)
#Play all way Through
#Or Use ForceProcessAVI() from TwriteAVI v2.0 to force write frames files
#ForceProcessAVI()
Return Last

###############

Function FSDet(clip c,String DB,int n,Float Thresh) {
GSCript("""
c FC = FrameCount
Done = RT_DBaseGetField(DB,n,0) # Detected Already ?
if(!Done) {
DetT= 0 # Detect type not yet ascertained (not bad frame)
ndf = RT_YDifference(Last,n=n,delta= 1) # YDifferenceToNext
pdf = RT_YDifference(Last,n=n,delta=-1) # YDifferenceToPrev
std = RT_YPlaneStdev(n=n)
if(ndf > Thresh) { # Initial detection ?
nstd=RT_YPlaneStdev(n=n+1)
if(std < nstd) { # Stdev of current is less than that of next frame
DetT= 1 # We are the badly blended frame
}
if(n<FC-1 && !RT_DBaseGetField(DB,n+1,0)) { # Next frame is not yet set
# Set Next frame to DetT + 1. ie bad frame or AFTER frame
RT_DBaseSet(
\ DB,
\ n+1,
\ True,
\ DetT + 1,
\ RT_YDifference(Last,n=n+1,delta= 1),
\ ndf,
\ nstd
\ )
}
if(DetT==0) { # n + 1 is bad frame, so n + 2 is AFTER frame
if(n<FC-2 && !RT_DBaseGetField(DB,n+2,0)) {
# Set n + 2 to 2, ie AFTER frame
RT_DBaseSet(
\ DB,
\ n+2,
\ True,
\ 2,
\ RT_YDifference(Last,n=n+2,delta= 1),
\ RT_YDifference(Last,n=n+2,delta=-1),
\ RT_YPlaneStdev(n=n+2)
\ )
}
}
}
RT_DBaseSet(DB,n,True,DetT,ndf,pdf,std) # Set Current frame detection + stats
} else {
DetT=RT_DBaseGetField(DB,n,1) # Get Previously Detected Type
}
Return DetT # 0 = OK : 1 = Bad Frame : 2 = AFTER frame
""")
}


FS_Fix_2.avs

# FS_Fix_2.avs

#c=Avisource("blend.mpg.avi")
c=Avisource("Blends.demuxed.m2v.AVI")

# clips slid over 1, same length as original
PREV = c.DeleteFrame(c.Framecount()-1).DuplicateFrame(0)
NEXT = c.DuplicateFrame(c.Framecount()-1).DeleteFrame(0)

c

DELETE ="DELETE.txt" # Frames to delete
REPLACE ="REPLACE.txt" # Frames to Replace

# No edit of below necessary, decision done in detector script
(Exist(REPLACE)) ? ClipClop(Last,PREV,NEXT,Cmd=REPLACE) : NOP
(Exist(DELETE)) ? RejectRanges(Last,Cmd=DELETE) : NOP

return Last

# From FrameSel or Prune (Requires BOTH)
Function RejectRanges(clip c,String "SCmd",String "Cmd",Bool "TrimAudio",Float "FadeMS") {
# RejectRanges() by StainlessS. Required:- FrameSel, Prune, RT_Stats
# Wrapper to delete frames/ranges along with audio, can supply frames/ranges in SCmd string And/Or Cmd file.
# The wrapper makes for easier usage of Prune() which supports up to 256 input clips, but requires a clip index,
# eg '3, 100,200' would specify clip 3, range 100 to 200. The wrapper does away with the necessity for the clip index as we
# are only using a single clip here. Prune also does not have a 'reject' arg to delete specified frames rather than select them,
# this wrapper also converts a list of frames to delete into a list of frames to select so that we can use Prune and its audio
# capability.
#
# SCmd: Frames/Ranges specified in String (Frames/Ranges either Chr(10) or ';' separated, infix ',' specifies range, eg 'start,end').
# Cmd: Frames/Ranges specified in file (one frame/range per line, comments also allowed, see FrameSel for Further info).
# TrimAudio:
# True(default), deletes audio belonging to deleted frames
# False, returns original audio, probably out of sync.
# FadeMS: (default 1.0 millisec). Linear Audio Fade duration at splices when TrimAudio==true, 0 = dont fade (might result in audio 'clicks/cracks').
c
TrimAudio=Default(TrimAudio,True) # default true trims audio, false returns original audio (audiodubbed, as Framesel returns no audio)
FadeMS=Float(Default(FadeMS,1.0)) # 1 millisecond linear fadeout/fadein at splices
PruneCmd = (TrimAudio) ? "~Prune_"+RT_LocalTimeString+".txt" : ""
(!TrimAudio)
\ ? FrameSel(scmd=SCmd,cmd=Cmd,reject=true)
\ : FrameSel_CmdReWrite(PruneCmd,scmd=SCmd,cmd=Cmd,reject=true,Prune=True,range=true)
(TrimAudio) ? Prune(Cmd=PruneCmd,FadeIn=True,FadeSplice=True,FadeOut=True,Fade=FadeMS) : NOP
# If TrimAudio==true then delete Prune temp file, Else restore original Audio to the now audio-less clip
(TrimAudio)
\ ? RT_FileDelete(PruneCmd)
\ : (c.HasAudio) ? AudioDub(c) : NOP
Return Last
}


Scripts EDITED:

As configured, deletes BAD frame and replaces AFTER frame.
Can Delete one or both, OR, Replace one or both, or Delete one and replace other.
All config done in Detection script.

EDIT: In FloatingShed 2nd sample, original FS_Det script detected sometimes bad frame, sometimes frame before bad frame.

[johnmeyer]

I'm probably too late to the party with this, but here's some code I've used many times that will find ALL duplicates in a run of duplicates (using the first WriteFile statement). I'm posting it because the second WriteFile statement uses the compound logic to only delete the first in a series.

This particular code creates a file with dupe frame numbers which you can then feed to another decimation script (TDecimate), which accepts a decimation list.

StainlessS' one-pass solution is most certainly what you want to use, but I thought it would be useful to confirm Gavino's compound test to ensure that a range gets deleted. His uses OR instead of AND which means his logic gets them all, whereas mine keeps all, except the first one.

The script, as written here, is set up to just put out the YDiff numbers. You have to comment out the Scriptclip line and then enable whichever WriteIf line you want to use, in order to get the script to actually do anything useful.

So, this script is a little tangential to what you want to do, but may have a few code fragments that may help.

BTW, I added the global variable, just to make Gavino happy. :)

#This script finds exact duplicate frames and outputs the frame numbers

global blankthreshold=3

filename = "e:\output_duplicate_frames.txt"
AVISource("e:\fs.avi").killaudio()
i=AssumeBFF.ConvertToYV12

#This line below will output EVERY frame that is below threshold, which results in LOTS of frames
#Normally you don't do this, but it is included for those who want this capability.
#WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)<=blankthreshold)", "current_frame+1", append = false)

#The line below writes the FIRST frame that falls below the threshold
#WriteFileIf(last, filename, "(YDifferenceFromPrevious(i)>blankthreshold)&&YDifferenceToNext(i)<=blankthreshold", "current_frame", append = false)

#Use this instead of WriteFile in order to determine blankthreshold
ScriptClip("Subtitle(String(YDifferenceFromPrevious(i)))")

[Floatingshed]

Methinks you only tried with the short sample (or you are really easily pleased :) )

I am not easily pleased! It worked very well on the samples but when I ran it on a 25 minute clip it was a disaster, weirdly deleting runs of frames where there were no scene changes.

I put it to one side for a couple of days, not wanting to complain after all the time you had put in. Now I see that you have been busy again, can't wait to play. thanks ;)

[StainlessS]

but when I ran it on a 25 minute clip it was a disaster, weirdly deleting runs of frames where there were no scene changes.

You talking about the FS_Detect_2 script, I expected the first version to fail ? (I dont see how FS_Detect_2 could delete 'runs of frames').

If so, then can you perhaps PM a link to complete m2v.

EDIT:
Now I see that you have been busy again

Perhaps that indicates that you were talking about the previous version, please clarify.

If it does detect more scene changes than it should, then up scenethreshold a bit (after viewing metrics at false +ve detects).
If still a problem, then post a few of the false +ve detects.
EDIT: I'll see if I can make initial detector a bit better.

[StainlessS]

Here, FS_Detect_3, better initial detector.

FS_Detect_3.avs

# FS_Detect_3.avs

#Avisource("blend.mpg.avi")
#Avisource("Blends.demuxed.m2v.AVI")
Avisource("test.demuxed.m2v.AVI")


#### CONFIG #### FloatingShed responsible for CONFIG settings :)
# Initial Detection:- (dFact*PrevToCurrDiff + dMinim) < CurrToNextDiff
dFact =1.25
dMinim=7.0
# Below, set up to TWO of below to TRUE (DELETE takes precedence, ie REPLACED may still be DELETED)
DELETE_BAD = True
DELETE_AFTER = False
REPLACE_BAD = False
REPLACE_AFTER = True
#####################
#####################
#####################
DELETE = "DELETE.txt" # Frames to delete
REPLACE= "REPLACE.txt" # Frames to replace
DB = "FS_DET.DB"

FC=FrameCount

RT_FileDelete(DELETE)
RT_FileDelete(REPLACE)

RT_DBaseAlloc(DB,FC,"bifffb")

SSS=("""
n=current_frame
det = FSDet(Last,DB,n,dFact,dMinim) # 0 = OK : 1 = Bad Frame : 2 = AFTER frame

# RejectRanges (Delete) file
(DELETE_BAD&&det==1)?RT_WriteFile(DELETE,"%d",n,Append=True):NOP # Delete BAD.
(DELETE_AFTER&&det==1)?RT_WriteFile(DELETE,"%d",n+1,Append=True):NOP # Delete AFTER

# Clipclop Command File
(REPLACE_BAD&&det==1) ?RT_WriteFile(REPLACE,"1 %d",n,Append=True) :NOP # Replace BAD with Prev
(REPLACE_AFTER&&det==1)?RT_WriteFile(REPLACE,"2 %d",n+1,Append=True):NOP # Replace AFTER frame with Next

df2prv = RT_DBaseGetField(DB,n,2)
df2nxt = RT_DBaseGetField(DB,n,3)
stdev = RT_DBaseGetField(DB,n,4)
T1 = RT_DBaseGetField(DB,n,5)
RT_Subtitle("%6d] Prev->Cur : Cur->Next\n %6.2f(%6.2f) : %6.2f %s %s\n%s",
\ n,df2prv,(dFact*df2prv+dMinim),df2nxt,(T1)?"Initial Detection":"",(det==1)?"BAD":(det==2)?"AFTER":"",
\ (det==1)?(DELETE_BAD ? "ACTION: DELETE" : (REPLACE_BAD) ? "ACTION: REPLACE" : "ACTION: NO ACTION")
\ : (det==2)?(DELETE_AFTER ? "ACTION: DELETE" : (REPLACE_AFTER) ? "ACTION: REPLACE" : "ACTION: NO ACTION")
\ : "")
RT_Subtitle("Stdev=%.3f",stdev,align=1)
""")

ScriptClip(SSS)
#Play all way Through
#Or Use ForceProcessAVI() from TwriteAVI v2.0 to force write frames files
#ForceProcessAVI()
Return Last

###############

Function FSDet(clip c,String DB,int n,Float dFact,Float dMinim) {
GSCript("""
c FC = FrameCount
Done = RT_DBaseGetField(DB,n,0) # Detected Already ?
if(!Done) {
DetT= 0 # Detect type not yet ascertained (not bad frame)
dfA = RT_YDifference(Last,n=n-1,delta = 1)
dfB = RT_YDifference(Last,n=n+0,delta = 1)
T1 = (dFact*dfA+dminim) < dfB # sufficiently bigger than 'Prev->Curr' ?
std = RT_YPlaneStdev(n=n)
if(T1) { # Initial detection ?
dfC = RT_YDifference(Last,n=n+1,delta = 1)
nstd= RT_YPlaneStdev(n=n+1)
if(std < nstd) { # Stdev of current is less than that of next frame
DetT= 1 # We are the badly blended frame
if(n<FC-1 && !RT_DBaseGetField(DB,n+1,0)) { # Next frame is not yet set
# Set Next frame to 2 ie AFTER frame
RT_DBaseSet(DB,n+1,True,2,dfB,dfC,nstd,False)
}
} else {
if(n<FC-1 && !RT_DBaseGetField(DB,n+1,0)) { # Next frame is not yet set
# Set Next frame to 1. ie bad frame
RT_DBaseSet(DB,n+1,True,1,dfB,dfC,nstd,False)
if(n<FC-2 && !RT_DBaseGetField(DB,n+2,0)) {
dfD = RT_YDifference(Last,n=n+2,delta = 1)
n2std= RT_YPlaneStdev(n=n+2)
# Set n + 2 to 2, ie AFTER frame
RT_DBaseSet(DB,n+2,True,2,dfC,dfD,n2std,False)
}
}
}
}
RT_DBaseSet(DB,n,True,DetT,dfA,dfB,std,T1) # Set Current frame detection + stats
} else {
DetT=RT_DBaseGetField(DB,n,1) # Get Previously Detected Type
}
Return DetT # 0 = OK : 1 = Bad Frame : 2 = AFTER frame
""")
}


FS_Fix_3.avs

# FS_Fix_3.avs

#c=Avisource("blend.mpg.avi")
c=Avisource("Blends.demuxed.m2v.AVI")

# clips slid over 1, same length as original
PREV = c.DeleteFrame(c.Framecount()-1).DuplicateFrame(0)
NEXT = c.DuplicateFrame(c.Framecount()-1).DeleteFrame(0)

c
DELETE ="DELETE.txt" # Frames to delete
REPLACE ="REPLACE.txt" # Frames to Replace

# No edit of below necessary, decision done in detector script
(Exist(REPLACE)) ? ClipClop(Last,PREV,NEXT,Cmd=REPLACE) : NOP
(Exist(DELETE)) ? RejectRanges(Last,Cmd=DELETE) : NOP
return Last

# From FrameSel or Prune (Requires BOTH)
Function RejectRanges(clip c,String "SCmd",String "Cmd",Bool "TrimAudio",Float "FadeMS") {
# RejectRanges() by StainlessS. Required:- FrameSel, Prune, RT_Stats
# Wrapper to delete frames/ranges along with audio, can supply frames/ranges in SCmd string And/Or Cmd file.
# The wrapper makes for easier usage of Prune() which supports up to 256 input clips, but requires a clip index,
# eg '3, 100,200' would specify clip 3, range 100 to 200. The wrapper does away with the necessity for the clip index as we
# are only using a single clip here. Prune also does not have a 'reject' arg to delete specified frames rather than select them,
# this wrapper also converts a list of frames to delete into a list of frames to select so that we can use Prune and its audio
# capability.
#
# SCmd: Frames/Ranges specified in String (Frames/Ranges either Chr(10) or ';' separated, infix ',' specifies range, eg 'start,end').
# Cmd: Frames/Ranges specified in file (one frame/range per line, comments also allowed, see FrameSel for Further info).
# TrimAudio:
# True(default), deletes audio belonging to deleted frames
# False, returns original audio, probably out of sync.
# FadeMS: (default 1.0 millisec). Linear Audio Fade duration at splices when TrimAudio==true, 0 = dont fade (might result in audio 'clicks/cracks').
c
TrimAudio=Default(TrimAudio,True) # default true trims audio, false returns original audio (audiodubbed, as Framesel returns no audio)
FadeMS=Float(Default(FadeMS,1.0)) # 1 millisecond linear fadeout/fadein at splices
PruneCmd = (TrimAudio) ? "~Prune_"+RT_LocalTimeString+".txt" : ""
(!TrimAudio)
\ ? FrameSel(scmd=SCmd,cmd=Cmd,reject=true)
\ : FrameSel_CmdReWrite(PruneCmd,scmd=SCmd,cmd=Cmd,reject=true,Prune=True,range=true)
(TrimAudio) ? Prune(Cmd=PruneCmd,FadeIn=True,FadeSplice=True,FadeOut=True,Fade=FadeMS) : NOP
# If TrimAudio==true then delete Prune temp file, Else restore original Audio to the now audio-less clip
(TrimAudio)
\ ? RT_FileDelete(PruneCmd)
\ : (c.HasAudio) ? AudioDub(c) : NOP
Return Last
}


If it fails on only a few frames, might be easiest to just edit frames files.

[Floatingshed]

Hello, yes I was referring to version one.
We are a bit out of sync! Version two performed remarkably well on a 25 minute clip. I had to increase the threshold to 25 to prevent lots of false positives, but then it worked extremely well.

I will do a similar test with FS3, thanks so much for all your time and effort. I think I may be a little sad, but I find this stuff exciting!


EDIT: I see you have changed the 'scenethreshold' into two settings. What are their individual functions and range of settings? Thanks.

[Floatingshed]

Sorry to be a pain... here's another test clip that exhibits lots of false detections. Setting scenethreshold to 15 on FS_Detect_2 overcomes this, I've played with FS_3 but can't get it right... But I'm stumbling about in the dark!

https://www.mediafire.com/?tb54dbiddjjbrg6

Any thoughts? Thanks again.

[StainlessS]

OK, I've made a small alteration to FS_Detect_3, shows (dFact*PrevToCurrDiff + dMinim) in parenthesis '()' after PrevToCurrDiff, just to save you
calculating this yourself. This number should be < CurrToNextDiff (shown on right) to be an initial detection.

I have not tried with FS_Detect_2, it is now considered dead.

I've also changed to dFact =1.25, dMinim=7.0, due to last sample, where 2nd from last frame 859 had PrevToCurrDiff of 0.0(dupe), and CurrToNextDiff of 6.41, so
(dFact*PrevToCurrDiff + dMinim) would be 4.0 (when dMinim is 4.0) and fires a false detection (4.0 < 6.41).
With new dFact=1.25 and dMinim=7.0, the dMinim prevents false detection on the dupe.
Give it a try.

EDIT:
NOTE, The Initial Detection should fire on either the BAD frame or the frame before the BAD frame, the Stdev stuff
decides which one the BAD frame is.

EDIT:
EDIT: I see you have changed the 'scenethreshold' into two settings. What are their individual functions and range of settings? Thanks.
Depends upon source.
dMinim has to be marginally higher than the highest difference [EDIT: that should NOT be an initial detection] following a dupe, dFact has to be low enough to detect correctly (BAD frame or frame before BAD frame, the one with the biggish value).

# Initial Detection:- (dFact*PrevToCurrDiff + dMinim) < CurrToNextDif


EDIT: If you prefer the 'scenethreshold' version of initial detection, just set dFact to 0.0, and dMinim to your chosen scenethreshold.

[Floatingshed]

Sorry for the sudden silence, had a family emergency!

Back to the testing!

Thanks Stainless...

[Floatingshed]

Hi StainlessS, I'd like to play around with this a little more if possible. I am finding that some delete/replace combinations work perfectly, whilst on others the bad frame is labelled initial detection and the repair process is effectively 1 frame late. This leaves the bad frame untouched, deletes the less affected frame and replaces a good frame. But as I say, it is unpredictable...
Any thoughts? Thanks.

[StainlessS]

Post a little more of your clip and I'll try to find time to look at it again (I still have original clips [2 I think]).

[Floatingshed]

I have posted another sample. It is rather large I'm afraid, to get lots of scene changes...

https://www.mediafire.com/?66ymmtm2n4gmdmj

Please have a look at the following frames: 24?, 313 ok, 877 ok, 1132?, 1266 ok, 1498 ok, 1724 ok, 3926 fail, 4110 ?
I am using version 3 of the detect script.
Thanks.

[StainlessS]

OK Floaty, I'll DL your clip tomorrow.

OT stuff removed. (to RT_Stats thread)

[Floatingshed]

I haven't the slightest idea what you just said! :-)

[StainlessS]

OK floaty, got your sample.
I'll move previous ot edit to rt_stats thread when back on desktop m/c (currently mobile).
EDIT: I got a bit excitable, due to quite extraordinary results (es tut mir leid).

[StainlessS]

Floaty,
Your last sample has in some places up to 4 bad frames (well 3 bad and one with a little image persistence of bright objects).
Given that all four are basically blends, and of clean fields that do not exist in stream, I dont really think it is easily possible
to create an automatic solution that is reliable. Perhaps the best option is a manual one, The VDM_EdKey (VdubMod) &
VD_EdKey (VDub standard), utilities allow you to manually hit CTRL/INSERT [in Vdub] to mark a frame for removal. I could knock up
a script to do the removal (well all you really need is RejectRanges() from Prune or FrameSel plugs). After removal of bad frames,
then could do a second pass with BadFramesRemoved Clip in VD/Mod EdKey to mark frames to replace with previous or following
frame, using exact same method, I could easily knock up a script to do that.
Would above be acceptable to you ? (If so then get one of the Edkey utils and start removing the bad blends whenever you want).

http://forum.doom9.org/showthread.php?t=173010

EDIT: Also, seems a little silly to spend perhaps days trying to create an automatic solution (which may not be successful), when manual solution would take little longer than to watch the clip.
The frames with slightly persisting bright areas probably dont need replacing unless one is a lil bit anal about things,
me thinks that a man on a galloping horse would not be able to see them. :)

[StainlessS]

EDIT: Post #1 of 2.

Floaty, perhaps you'ed like to try this:- EDIT: LINK REMOVED,
see here for new SawBones v1.0 thread: http://forum.doom9.org/showthread.php?t=173158


SawBones v0.0.

SawBones, is a compiled AutoIt script utility, intended to assist in creating Avisynth command file for FrameSurgeon.avs script.

SawBones is used together with VirtualDub and NotePad. You must run the app, with BOTH VirtualDub and the
NotePad Editor VISIBLE, you can scroll through a video clip, and press CTRL/DELETE to insert a FrameSurgeon DELETE command for the
current frame into the NotePad Editor. EDIT: You press the keys with VDub active, not NotePad

All commands:-

CTRL/DELETE Delete current frame
CTRL/F1 CopyFromPrevious frame to current frame (ie replace current frame n with frame n - 1.
CTRL/F2 CopyFromNext frame to current frame (ie replace current frame n with frame n + 1.
CTRL/1 Interpolate current frame n using n-1 and n+1 as source frames.
CTRL/2 Interpolate 2 frames starting at current frame n, using n-1 and n+2 as source frames.
CTRL/3 Interpolate 3 frames starting at current frame n, using n-1 and n+3 as source frames.
CTRL/4 Interpolate 4 frames starting at current frame n, using n-1 and n+4 as source frames.
CTRL/5 Interpolate 5 frames starting at current frame n, using n-1 and n+5 as source frames.
CTRL/6 Interpolate 6 frames starting at current frame n, using n-1 and n+6 as source frames.
CTRL/7 Interpolate 7 frames starting at current frame n, using n-1 and n+7 as source frames.
CTRL/8 Interpolate 8 frames starting at current frame n, using n-1 and n+8 as source frames.
CTRL/9 Interpolate 9 frames starting at current frame n, using n-1 and n+9 as source frames.

After creating Command file in NotePad, Save as eg Command.txt and provide it to FrameSurgeon in Cmd arg.
FrameSurgeon.avs requires GScript, RT_Stats, ClipClop, FrameSel and Prune Plugins.

AutoiIt compiled executable with source provided, just click Menu Tools/build to create executable (In Scite4AutoIt3 editor).
Requires AutoIt3 and Scite4AutoIt3 editor to re-build executable.

ssS


Zip contains AutoIt source and Executable, and FrameSurgeon.avs script to edit clip using created command file.
(script not shown, would probably break 16KB D9 post limit).

[StainlessS]

Post #2 of 2,

Scipt does not break D9 limit, but is too big for inclusion with previous post. Here tis.

Function FrameSurgeon(clip c, String "Scmd",String "Cmd",bool "Show",int "dv",
\ int "pel",int "sharp",int "rfilter",Float "ml") {
# FrameSurgeon() by StainlessS.
# Replace damaged frames using commands in a either command string or file.
/*
Commands:-
Eg, "CP 100" CopyFromPrevious ie replace frame 100 with frame 100 - 1.
Eg, "CN 200" CopyFromNext ie replace frame 200 with frame 200 + 1.
Eg, "I2 300" Interpolate 2 frames at frames 300 and 301 (using 299 and 302).
Eg, "I9 400" Interpolate 9 frames at frames 400 to 408 (using 399 and 409).
eg "DEL 100" Delete frame 100
eg "-100" Delete frame 100
Interpolate I1 to I9 supported.
*/
#########
Scmd = Default(Scmd,"") # User supplied list of newlineEDIT: DO NOT USE Semicolon Separators or semicolon separated commands.
Cmd = Default(Cmd,"") # User supplied Filename containing newline separated commands.
Show = Default(Show,false) # Show Info on frame
dv = Default(dv,0) # ClipClop DebugView level (0 - 4, Need DebugView utility)
# EDIT: IGNORE LINE: When Ext22 is True, will likely need eg SetMemoryMax(256) to avoid 'Out of Memory' Errors.
pel = Default(pel,2) # Default as for MSuper(), ie 2, range=1 or 2 or 4
sharp = Default(sharp,2) # Default as for MSuper(), ie 2, range=0 -> 2
rfilter= Default(rfilter,2) # Default as for MSuper(), ie 2, range=0 -> 4
ml = Float(default(ml,100.0)) # Default as for MFlowInter(), ie 100.0, range=greater than 0.0.

ClopCmdFN = "~ClipClop_" + RT_LocalTimeString(file=True)+".txt"
DelCmdFN = "~Delete_" + RT_LocalTimeString(file=True)+".txt"

S = (Cmd!="") ? RT_ReadTxtFromFile(Cmd) : ""
(S!="") ? SplitCommands(S,ClopCmdFN,DelCmdFN) : NOP
(SCmd!="") ? SplitCommands(SCmd,ClopCmdFN,DelCmdFN) : NOP

###
CP = c.DeleteFrame(c.Framecount()-1).DuplicateFrame(0)
CN = c.DuplicateFrame(c.Framecount()-1).DeleteFrame(0)
###
I = CP
sup = I.MSuper(pel=pel, sharp=sharp, rfilter=rfilter)
###
BAD = 1
I01_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I01_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I01_01 = I.MFlowInter(sup,I01_bv,I01_fv, time=100.0 * 1 / (BAD+1) , ml=ml)
BAD = 2
I02_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I02_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I02_01 = I.MFlowInter(sup,I02_bv,I02_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I02_02 = I.MFlowInter(sup,I02_bv,I02_fv, time=100.0 * 2 / (BAD+1), ml=ml)
BAD = 3
I03_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I03_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I03_01 = I.MFlowInter(sup,I03_bv,I03_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I03_02 = I.MFlowInter(sup,I03_bv,I03_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I03_03 = I.MFlowInter(sup,I03_bv,I03_fv, time=100.0 * 3 / (BAD+1), ml=ml)
BAD = 4
I04_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I04_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I04_01 = I.MFlowInter(sup,I04_bv,I04_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I04_02 = I.MFlowInter(sup,I04_bv,I04_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I04_03 = I.MFlowInter(sup,I04_bv,I04_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I04_04 = I.MFlowInter(sup,I04_bv,I04_fv, time=100.0 * 4 / (BAD+1), ml=ml)
BAD = 5
I05_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I05_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I05_01 = I.MFlowInter(sup,I05_bv,I05_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I05_02 = I.MFlowInter(sup,I05_bv,I05_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I05_03 = I.MFlowInter(sup,I05_bv,I05_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I05_04 = I.MFlowInter(sup,I05_bv,I05_fv, time=100.0 * 4 / (BAD+1), ml=ml)
I05_05 = I.MFlowInter(sup,I05_bv,I05_fv, time=100.0 * 5 / (BAD+1), ml=ml)
BAD = 6
I06_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I06_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I06_01 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I06_02 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I06_03 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I06_04 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 4 / (BAD+1), ml=ml)
I06_05 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 5 / (BAD+1), ml=ml)
I06_06 = I.MFlowInter(sup,I06_bv,I06_fv, time=100.0 * 6 / (BAD+1), ml=ml)
BAD = 7
I07_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I07_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I07_01 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I07_02 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I07_03 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I07_04 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 4 / (BAD+1), ml=ml)
I07_05 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 5 / (BAD+1), ml=ml)
I07_06 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 6 / (BAD+1), ml=ml)
I07_07 = I.MFlowInter(sup,I07_bv,I07_fv, time=100.0 * 7 / (BAD+1), ml=ml)
BAD = 8
I08_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I08_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I08_01 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I08_02 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I08_03 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I08_04 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 4 / (BAD+1), ml=ml)
I08_05 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 5 / (BAD+1), ml=ml)
I08_06 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 6 / (BAD+1), ml=ml)
I08_07 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 7 / (BAD+1), ml=ml)
I08_08 = I.MFlowInter(sup,I08_bv,I08_fv, time=100.0 * 8 / (BAD+1), ml=ml)
BAD = 9
I09_bv = sup.MAnalyse(isb=true, delta=BAD+1)
I09_fv = sup.MAnalyse(isb=false, delta=BAD+1)
I09_01 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 1 / (BAD+1), ml=ml)
I09_02 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 2 / (BAD+1), ml=ml)
I09_03 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 3 / (BAD+1), ml=ml)
I09_04 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 4 / (BAD+1), ml=ml)
I09_05 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 5 / (BAD+1), ml=ml)
I09_06 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 6 / (BAD+1), ml=ml)
I09_07 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 7 / (BAD+1), ml=ml)
I09_08 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 8 / (BAD+1), ml=ml)
I09_09 = I.MFlowInter(sup,I09_bv,I09_fv, time=100.0 * 9 / (BAD+1), ml=ml)
NICKNAME=""" # Define Command mnemonics allowed in command string and file.
CP =1 # CopyFromPrevious (frame n replaced with frame n-1)
CN =2 # CopyFromNext (frame n replaced with frame n+1)
I1 =3:1 # Interpolate 1
I2 =4:2 # Interpolate 2
I3 =6:3 # Interpolate 3
I4 =9:4 # Etc
I5 =13:5 #
I6 =18:6 #
I7 =24:7 #
I8 =31:8 #
I9 =39:9 #
"""
Clopped = (Exist(ClopCmdFN)) ? c.ClipClop(
\ CP,
\ CN,
\ I01_01,
\ I02_01,I02_02,
\ I03_01,I03_02,I03_02,
\ I04_01,I04_02,I04_03,I04_04,
\ I05_01,I05_02,I05_03,I05_04,I05_05,
\ I06_01,I06_02,I06_03,I06_04,I06_05,I06_06,
\ I07_01,I07_02,I07_03,I07_04,I07_05,I07_06,I07_07,
\ I08_01,I08_02,I08_03,I08_04,I08_05,I08_06,I08_07,I08_08,
\ I09_01,I09_02,I09_03,I09_04,I09_05,I09_06,I09_07,I09_08,I09_09,
\ cmd=ClopCmdFN,nickname=NICKNAME,show=SHOW,dv=DV)
\ : c
Result = (Exist(DelCmdFN)) ? RejectRanges(clopped,Cmd=DelCmdFN) : Clopped
RT_FileDelete(ClopCmdFN) RT_FileDelete(DelCmdFN)
Return Result
}

Function SplitCommands(String CmdS,String ClopCmdFN,String DelCmdFN) {
# Separate commands in CmdS into ClipClop and FrameSel Reject frame commands
myName="SplitCommands: "
#RT_DebugF("%s=CmdS",CmdS) EDIT: Delete this line
GSCript("""
Lines = RT_TxtQueryLines(CmdS)
For(i=0,Lines-1) {
SSS=RT_TxtGetLine(CmdS,i)
S=EatWhite(SSS)
if(S!="" && !IsHash(S)) {
if(RT_FindStr(S,"DEL",Sig=False)==1) {
S=MidStr(S,4).EatWhite
Assert(IsDigit(S),RT_String("%sExpecting frame number after DEL %s",myName,SSS))
frm = RT_NumberValue(S) S=EatDigits(S)
RT_WriteFile(DelCmdFN,"%d",frm,Append=True)
} else if(RT_FindStr(S,"-",Sig=False)==1) {
S=MidStr(S,2).EatWhite
Assert(IsDigit(S),RT_String("%sExpecting frame number after '-' %s",myName,SSS))
frm = RT_NumberValue(S) S=EatDigits(S)
RT_WriteFile(DelCmdFN,"%d",frm,Append=True)
} else {
RT_WriteFile(ClopCmdFN,"%s",S,Append=True) # Copy ClipClop Commands Verbatim
S = "" # All Done
}
S=EatWhite(S).EatHash
Assert(S=="",RT_String("%sNon-PARSE [S=%s]",myName,SSS))
}
}
""")
}

# From FrameSel or Prune (Requires BOTH)
Function RejectRanges(clip c,String "SCmd",String "Cmd",Bool "TrimAudio",Float "FadeMS") {
# RejectRanges() by StainlessS. Required:- FrameSel, Prune, RT_Stats
# Wrapper to delete frames/ranges along with audio, can supply frames/ranges in SCmd string And/Or Cmd file.
# The wrapper makes for easier usage of Prune() which supports up to 256 input clips, but requires a clip index,
# eg '3, 100,200' would specify clip 3, range 100 to 200. The wrapper does away with the necessity for the clip index as we
# are only using a single clip here. Prune also does not have a 'reject' arg to delete specified frames rather than select them,
# this wrapper also converts a list of frames to delete into a list of frames to select so that we can use Prune and its audio
# capability.
#
# SCmd: Frames/Ranges specified in String (Frames/Ranges either Chr(10) or ';' separated, infix ',' specifies range, eg 'start,end').
# Cmd: Frames/Ranges specified in file (one frame/range per line, comments also allowed, see FrameSel for Further info).
# TrimAudio:
# True(default), deletes audio belonging to deleted frames
# False, returns original audio, probably out of sync.
# FadeMS: (default 1.0 millisec). Linear Audio Fade duration at splices when TrimAudio==true, 0 = dont fade (might result in audio 'clicks/cracks').
c
TrimAudio=Default(TrimAudio,True) # default true trims audio, false returns original audio (audiodubbed, as Framesel returns no audio)
FadeMS=Float(Default(FadeMS,1.0)) # 1 millisecond linear fadeout/fadein at splices
PruneCmd = (TrimAudio) ? "~Prune_"+RT_LocalTimeString+".txt" : ""
(!TrimAudio)
\ ? FrameSel(scmd=SCmd,cmd=Cmd,reject=true)
\ : FrameSel_CmdReWrite(PruneCmd,scmd=SCmd,cmd=Cmd,reject=true,Prune=True,range=true)
(TrimAudio) ? Prune(Cmd=PruneCmd,FadeIn=True,FadeSplice=True,FadeOut=True,Fade=FadeMS) : NOP
# If TrimAudio==true then delete Prune temp file, Else restore original Audio to the now audio-less clip
(TrimAudio)
\ ? RT_FileDelete(PruneCmd)
\ : (c.HasAudio) ? AudioDub(c) : NOP
Return Last
}

Function IsDigit(String S) {return (RT_Ord(S)>=48 && RT_Ord(S)<=57)}
Function IsHash(String S) {return (RT_Ord(S)==RT_Ord("#"))}
Function EatWhite(String S) {return (RT_Ord(S)==32 || RT_Ord(S)==9) ?EatWhite(MidStr(S,2)):S}
Function EatDigits(String S) {return (IsDigit(S)) ?EatDigits(MidStr(S,2)): S}
Function EatHash(String S) {return (IsHash(S)) ?"":S}


and client scipt

Import("FrameSurgeon.avs")

AviSource("test3.avi")

SHOW=False
COMMAND = "Command.TXT"

Return FrameSurgeon(Cmd=COMMAND,Show=SHOW)


EDIT: A Sawbones produced command file might look something like this (no Interpolate commands in this one, eg "I9 666")

DEL 26
DEL 27
CN 28
DEL 315
DEL 316
CN 317
DEL 879
DEL 880
DEL 1134
DEL 1135
DEL 1268
DEL 1269
DEL 1500
DEL 1501
CN 1502
DEL 1726
DEL 1727
DEL 2204
DEL 2205
DEL 2419
DEL 2420
DEL 2906
DEL 2907
DEL 3166
DEL 3167
CN 3168
DEL 3760
DEL 3761
DEL 3928
DEL 3929
DEL 4112
DEL 4113
DEL 4166


EDIT: ALL frame deletes are done AFTER frame replacements.

EDIT: Floaty, I hope that you find above an acceptable if not automatic solution,
NOTE, FrameSurgeon will excise audio along with deleted video frames if provided with sound,
also applies a 1 millisecond linear fade out/in where frames removed to avoid 'cracks' and 'clicks' in the audio.

[Floatingshed]

Crikey, that's a lot to think about. Thanks, will study....

[StainlessS]

Crikey, that's a lot to think about. Thanks, will study....

Not really, in a nutshell,

Sawbones commands

All commands:-

CTRL/DELETE Delete current frame
CTRL/F1 CopyFromPrevious frame to current frame (ie replace current frame n with frame n - 1.
CTRL/F2 CopyFromNext frame to current frame (ie replace current frame n with frame n + 1.
CTRL/1 Interpolate current frame n using n-1 and n+1 as source frames.
CTRL/2 Interpolate 2 frames starting at current frame n, using n-1 and n+2 as source frames.
CTRL/3 Interpolate 3 frames starting at current frame n, using n-1 and n+3 as source frames.
CTRL/4 Interpolate 4 frames starting at current frame n, using n-1 and n+4 as source frames.
CTRL/5 Interpolate 5 frames starting at current frame n, using n-1 and n+5 as source frames.
CTRL/6 Interpolate 6 frames starting at current frame n, using n-1 and n+6 as source frames.
CTRL/7 Interpolate 7 frames starting at current frame n, using n-1 and n+7 as source frames.
CTRL/8 Interpolate 8 frames starting at current frame n, using n-1 and n+8 as source frames.
CTRL/9 Interpolate 9 frames starting at current frame n, using n-1 and n+9 as source frames.


Client Avisynth script

Import("FrameSurgeon.avs")

AviSource("test3.avi")

SHOW=False
COMMAND = "Command.TXT" # EDIT: Created by SawBones.Exe within VDub

Return FrameSurgeon(Cmd=COMMAND,Show=SHOW)


I'll Add Delete Range and User chosen 'FXn' (where 'n' is digit 1 to 9) which is a replacement clip, eg User supplies FX1 clip to FrameSurgeon and FXn commands in SawBones will replace ranges in source clip with those in FXn clip, eg FX1=Light Denoise, FX2=Med Denoise etc.

[Floatingshed]

It certainly does what it should! Very easy to use and the results are excellent. The question is: "Will I live long enough to process all 66 episodes manually?"

It would seem sensible to run the automatic script first and then mop up manually. Do you see any problems?

[StainlessS]

Problem is getting a reliable automatic script, I dont really know how to cope with that KineScope thing, too variable.
You would still need to check everything for errors anyway, and a lot less simple to undo mistakes than do em right in the first place.

I'll try again but dont hold your breath nor be too expectant.

EDIT: Updated SawBones v1.00 here:- http://forum.doom9.org/showthread.php?t=173158

[StainlessS]

Floaty, here you go:-


# FloatyFixKinescope.avs # http://forum.doom9.org/showthread.php?p=1756588#post1756588

Avisource("test.AVI")

Global SEQ = 0.85 # corr >= is good sequence
Global SC = 0.3 # corr below is scene change
Global SC2 = 16.0 # LumaDiff, Above is scene change (additional check to above SC)
Global dMul = 4.0 # Detect rapid decrease in corr @ c3 (wrt c2)

Global FN="FrameSurgeonCommand.txt"
Global DB="Floaty_TEMP.DB"

Global REPLACE_DROPC=False # Frame after blend pair (lil bit of persistence of bright objects, unlikely to be very evident during play)

RT_FileDelete(FN)
RT_DBaseAlloc(DB,FrameCount,"bi") # Fields: Bool, Int

GSCript("""
Function Floaty(clip c,int n) {
c
c1 = RT_LumaCorrelation(Last, Last, n=n, delta= -3, n2=n, delta2=-2)
c2 = RT_LumaCorrelation(Last, Last, n=n, delta= -2, n2=n, delta2=-1)
c3 = RT_LumaCorrelation(Last, Last, n=n, delta= -1, n2=n, delta2= 0)
c4 = RT_LumaCorrelation(Last, Last, n=n, delta= 0, n2=n, delta2= 1)
c5 = RT_LumaCorrelation(Last, Last, n=n, delta= 1, n2=n, delta2= 2)
c6 = RT_LumaCorrelation(Last, Last, n=n, delta= 2, n2=n, delta2= 3)
c7 = RT_LumaCorrelation(Last, Last, n=n, delta= 3, n2=n, delta2= 4)
scd3 = RT_LumaCorrelation(Last, Last, n=n, delta= -1, n2=n, delta2= 3) # SC across blends
scYDf= RT_LumaDifference( Last, Last, n=n, delta= -1, n2=n, delta2= 3) # SC across blends
Done = RT_DBaseGetField(DB,n,0)
Stat = RT_DBaseGetField(DB,n,1)
nStat=0
if(!Done) {
if(c3<SC) {
Stat=2 # n-1 -> n < SC : Prev was EOS, So we are SOS
} else if(c3==1.0 && n>0){
Stat=3 # n = DUPE
} else if(c4<SC) {
Stat = 1 # n -> n+1 < SC : EOS
nStat= 2
} else if(c4==1.0){ # n+1 is Dupe, so we must be norm
Stat=0
} else if(c5<SC) {
Stat = 0 # n+1 -> n+2 is EOS (We must be norm)
} else if(c5==1.0){
Stat=0 # n+2 is Dupe, so we must be norm
} else if(scd3<SC || scYDf>SC2) { # Seems like might be a scene change somewhere close ahead
if(C1>=SEQ && C2>=SEQ && C6>=SEQ) {
T1 = c3<SEQ
T2 = c4<SEQ
T3 = c3 < c2-(dMul*(Abs(c1-c2)+0.001))
RT_Subtitle("%s || (%s && %s)",T1,T2,T3,align=1)
if(T1 || (T2 && T3)) {
Stat = 4
nStat = 5
RT_DBaseSet(DB,n,True,Stat)
(n < FrameCount-2) ? RT_DBaseSet(DB,n+2,True,6) : NOP
RT_WriteFile(FN,"DEL %d,%d",n,n+1,Append=True)
(REPLACE_DROPC) ? RT_WriteFile(FN,"CN %d",n+2,Append=True) : NOP
}
}
}
RT_DBaseSet(DB,n,True,Stat)
(nStat !=0 && n < FrameCount-1) ? RT_DBaseSet(DB,n+1,True,nStat) : NOP
}
RT_Subtitle("%d] %f %f %f %f %f %f %f\n%f\n%f",n,c1,c2,c3,c4,c5,c6,c7,scd3,scYDf)
If(Stat!=0) {Subtitle(Select(Stat,"Dummy","EOS","SOS","DUPE","DROP_A","DROP_B","DROP_C"),Align=5)}
Return Last
}
""")

SSS="Floaty(Last,current_frame)"

Scriptclip(SSS,after_frame=true)

Return Last


Updated a little bit

Floater_Fix.avs

Import("FrameSurgeon.avs")
Avisource("test.AVI")
CMD="FrameSurgeonCommand.txt"
FrameSurgeon(cmd=Cmd)


Nearly gave up more than once, but as given, copes with every blend in all 4 of your samples, including the breaking frame (switch mid frame).
It may or may not work well with all of the remaining 66 episodes, good luck with that (I dont really wanna know if it dont, I'm kinda sick of it) :)
You need FrameSurgeon script.

Replacing frame after blend pair is probably overkill, but given you the option, hope you have a nice one.

EDIT: FrameSurgeon.avs comes with Sawbones.

Below is result of edited (cut out dead black frames) samples glued together (in order given I think), with REPLACE_DROPC enabled.


DEL 9,10
CN 11
DEL 81,82
CN 83
DEL 340,341
CN 342
DEL 384,385
CN 386
DEL 825,826
CN 827
DEL 1279,1280
CN 1281
DEL 2186,2187
CN 2188
DEL 2475,2476
CN 2477
DEL 3039,3040
CN 3041
DEL 3294,3295
CN 3296
DEL 3428,3429
CN 3430
DEL 3660,3661
CN 3662
DEL 3886,3887
CN 3888
DEL 4364,4365
CN 4366
DEL 4579,4580
CN 4581
DEL 5066,5067
CN 5068
DEL 5326,5327
CN 5328
DEL 5920,5921
CN 5922
DEL 6088,6089
CN 6090
DEL 6272,6273
CN 6274
DEL 6326,6327
CN 6328


EDIT: Note, if fix script (2nd one) provided with audio, will cut audio too and do linear 1ms fadeout/in at cuts (avoid 'cracks' in audio).

[Floatingshed]

Apologies, only just seen this! Family matters have kept me away and I thought you had given up.
I will try it out, thanks for all your help, much appreciated.

[StainlessS]

No probs Floaty, assumed that you had put it on the back burner.

Out of interest, what was the name of the series (seeing Windsor Davies in it), I tried to locate it (via IMDB) but not sure which it was.
There was a couple (I think) that had 'about' 66 episodes, but a number of missing/lost ones.

EDIT: Whoever would have thought that Windsor would have had a number 1 chart hit quite a few years later :)
EDIT: https://www.google.co.uk/?gfe_rd=cr&ei=KRd1VKG6N5HCVND3gGA&gws_rd=ssl#q=don+eselle+whispering+grass

[Floatingshed]

The series is "Sergeant Cork", actually a very good watch as long as you remember to wear your archive hat!
My old mate Mike Sammes of The Mike Sammes Singers described Windsor as "the only tone-deaf Welshman in captivity"! Although Windsor did the speaking parts on Whispering grass the more musical parts (bah, bah, boom!) were done by Mike!

BTW, I find that the latest script gives excellent results visually but a lot of the cuts are mid-sentence and the audio trims are noticable (not clicks, just bits missing).
Is it possible with this new script to make all repairs dupes instead of drops, the coding is a bit beyond me!

Cheers.