Deleting every frame between equal frames in video

[eduardobedoya]

Hi great community, Im looking for your help to create an avisynth script that is similar to the popular avisynth "duplicated frame function script".

I have lots of videos (1920 x 1080, tsc2 compression, avi format, about 45min long each one) that I need to trim automatically.

In these videos the movie goes forward normally but then returns exactly to a previous frame (like 3 segs of rewind),
then the movie resumes and goes forward again, but then it returns to the same previous frame again, and then resumes again, until finally the movie pass this segment and goes on, but in other segment of the movie the same thing happens until it pass that segment, and so on. In the hole movie this "rewind to especific frame, forward, rewind, forward" happens almost 30 times.

For example if each letter is a different frame, the movie goes like this:

AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ

Note that once in a while the movie goes back to exactly the same frame.
So, the lowercase letters are frames that got rewound, frames that I would like to discard,
and the capital letters are frames that I want to keep
So in the example above I would like to transform the video into this:

ABCDEFGHIJKLMNOPQ

I guess the average way to achieve this, would be detecting all 100% equal frames in the video and deleting everyframe between two equal frames, perhaps also use a threshold about 10 minutes, to delete only the frames that are between two equal frames that are not separated longer than 10 minutes.
Please note that these are HD videos and the difference between each frame in the video is very subtle.
Also if you recomend me some links to learn to use avisynth with several videos (kinda batch mode). I use premiere.

Thank you very much for your support doom9, regards.

[Guest]

Welcome to the forum. Sorry but I have to ask, where did you get these files?

[StainlessS]

This serves as a reasonable example of how you can use RT_Stats Dbase funcs,
Used your example string to produce a test clip.


##############################
# Requires GScript(), RT_Stats(), and FrameSel() Plugins
##############################

# Choose one of below lines (comment out the other)
#TestSource(Upper=true) # Test Source clip, We do tests on Upper case frames, but later select from upper/lower case mix frames
AviSource("...whatever...")

######
ScanAheadSeconds= 10 * 60
TEST_THRESH = 1.0
DIF_THRESH = 0.1
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB="MyDbase.DB"
FN="Frames.txt"
DEBUG=True # Info to DebugView window (Google)
DELETEDBASE=True # Delete DBase file
##############################
GSCript("""
RT_FileDelete(FN) # Delete any existing frames file
RT_DBaseAlloc(DB,FrameCount,"f") # Allocate a single field database of type Float
LastFrame=FrameCount-1
(DEBUG) ? RT_DebugF("Filling DBase with AverageLuma data") : NOP
For(i=0,LastFrame) { # Set AveLuma for each and every frame
RT_DBaseSet(DB,i,RT_AverageLuma(Last,i))
}
(DEBUG) ? RT_DebugF("DBase Filled, Scanning frames.") : NOP
For(i=0,LastFrame) {
LumaI = RT_DBaseGetField(DB,i,0)
EndLimit= Min(i + ScanAheadFrames,LastFrame)
OutFrame = i # Init frame to output
For(j=EndLimit,i + 1,-1) {
LumaJ = RT_DBaseGetField(DB,j,0)
if(Abs(LumaI - LumaJ) <= TEST_THRESH) { # about same ave luma, test it
if(RT_LumaDifference(Last,Last,n=i,n2=j) <= DIF_THRESH) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True)
if(i != OutFrame) {
(DEBUG) ? RT_DebugF("%d ] (Skipped=%d)",OutFrame,OutFrame-i) : NOP # Shows in DebugView (Google)
i = OutFrame # Skip all frames before Outframe
} Else {
(DEBUG) ? RT_DebugF("%d ]",OutFrame) : NOP
}
}
(DELETEDBASE) ? RT_FileDelete(DB) : NOP
""")

# If using TestSource(), comment out below line
Return FrameSel(Last,cmd=FN) # This will return frames using FN frames file

# Here Just testing using synthesized test clip, we do NOT upper case strings this time, to see if it worked OK
TestSource(Upper=False)
Return FrameSel(Last,cmd=FN)
#####################

# Only of use in testing, (If you dont have a real problem clip)
Function TestSource(Bool "Upper") {
Upper=Default(Upper,False)
Test = "AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ"
Test = (Upper) ? Ucase(TesT) : Test
CC = ColorBars().Trim(0,-1).KillAudio # Single frame
C=CC.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
s = s + s + s + s + s + s + s + s + s + s
C = C + CC.Subtitle(S,size=24,align=5)
}
""")
return C
}


Script to load only good frames, once main script has created FN ("Frame.txt")

FN="Frames.txt"
AviSource("...whatever...")
Return FrameSel(Last,cmd=FN)


EDIT: It will NOT be super fast !!! (main script)
EDIT: Any colorspace OK.
EDIT: Clip should be seekable, eg NOT DivX with single keyframe (way too slow).

EDIT: DebugView output (1st frame 12 is 12th letter in your example string ie 'A', 2nd is 13 ie 'B' etc)

00000010 5.15711737 [1824] RT_DebugF: Filling DBase with AverageLuma data
00000011 5.96010494 [1824] RT_DebugF: DBase Filled, Scanning frames.
00000012 6.72434425 [1824] RT_DebugF: 12 ] (Skipped=12)
00000013 6.91525364 [1824] RT_DebugF: 13 ]
00000014 7.10134315 [1824] RT_DebugF: 14 ]
00000015 7.22855568 [1824] RT_DebugF: 29 ] (Skipped=14)
00000016 7.34694576 [1824] RT_DebugF: 30 ]
00000017 7.46073008 [1824] RT_DebugF: 31 ]
00000018 7.57038546 [1824] RT_DebugF: 32 ]
00000019 7.65050507 [1824] RT_DebugF: 40 ] (Skipped=7)
00000020 7.72205544 [1824] RT_DebugF: 41 ]
00000021 7.78988838 [1824] RT_DebugF: 42 ]
00000022 7.85289049 [1824] RT_DebugF: 43 ]
00000023 7.91187668 [1824] RT_DebugF: 44 ]
00000024 7.93679857 [1824] RT_DebugF: 53 ] (Skipped=8)
00000025 7.95319843 [1824] RT_DebugF: 54 ]
00000026 7.96542358 [1824] RT_DebugF: 55 ]
00000027 7.97333288 [1824] RT_DebugF: 56 ]
00000028 7.97700548 [1824] RT_DebugF: 57 ]


EDIT:
Tested on a SD PAL 30 mins, 39 secs clip without any repeats, with 10 mins scan ahead and it takes about 12 seconds to output each frame,
would be faster if repeats were present and did not have to scan ahead 10 * 60 * 25 frames, for each and every output frame.
Above on Core Duo Dual Core @ 2.13GHz, Sata 2.

[eduardobedoya]

Thanks you StainlessS

yes 10 min is the average time it would have to look ahead, looking for repeats, the source is very clean, meaning that are HD digitally recorded videos, good quality, no blur, no death pixels.


Please StainlessS could you give me some links, where could I study, (figure out) how to apply avisynth script inside an windows 8 laptop?
wich software do I have to install? wich plugins?
Thanks you very much man, u have save me hours of editing, THANKS!

[eduardobedoya]

Welcome to the forum. Sorry but I have to ask, where did you get these files?

the source is very clean, meaning that are HD digitally recorded videos, good quality, no blur, no death pixels.


Please neuron2 could you give me some links, where could I study, (figure out) how to apply avisynth script inside an windows 8 laptop?
wich software do I have to install? wich plugins? in orther to run Stainless script??
Thank you advanced.

[StainlessS]

eduardobedoya,

Neuron2 wantz to know how you obtained the samples, ie " Where did you get these files ? ".

For a Laptop, suggest you get the FILTERSDK-HELP (compressed CHM help file version of docs installed with Avisynth v2.6,
suggest you get this even if using v2.58), available via my sig ie StainlessS@MediaFire below this post.
You can put it on Hot-Key for easy acces and it is easily searchable.

When on-line, Avisynth Wiki (has list of many plugins) here http://avisynth.nl/index.php/Main_Page

2 Lists of plugins at top of Usage forum 1st page in stickies.

Most of my plugins are not not in any of the lists (one or maybe two only EDIT: they may be in the New plugins sticky),
see Mediafire in sig for mine.

FrameSel thread here, http://forum.doom9.org/showthread.php?t=167971

RT_Stats here http://forum.doom9.org/showthread.php?t=165479&highlight=rt_stats

GScript here, http://forum.doom9.org/showthread.php?t=147846&highlight=gscript

There is a search option at top of all forum pages where you could have found the plugins yourself.

EDIT: And DebugView from SysInternals (Now MicroSoft) Here http://technet.microsoft.com/en-gb/sysinternals/bb896647.aspx

and Welcome to the Forum.

EDIT: I may have a go it increasing speed of given script, perhaps speed increase of 50X or 100X possible
(but maybe not possible, I'll try to give it a go though).

[StainlessS]

Post #1 of 3

QwikAveLumaScan.Avs

# QwikAveLumaScan.Avs

Function QwikAveLumaScanCreateDB(clip c, String DB, String "PrevDB", String "NextDB", Bool "Debug") {
# QwikAveLumaScanCreateDB: By StainlessS
# Requires GScript(), RT_Stats() plugins
#
# Creates a DB DataBase file of c.FrameCount records with 1 Float field per record, set to AverageLuma of each frame.
# Usage:- eg AveLuma = RT_DBaseGetField(DB,FrameNo,0)
#
# Optional creation of PrevDB and NextDB used by QwikAveLumaScanGetNear() function.
# PrevDB is a database of c.FrameCount records, and 256 Int fields per record, where content of fields point to the
# nearest frame BEFORE the record number (frame) which has the same Int(AverageLuma) value as the field index. If there is no
# frame prior to the record number (frame) that has an Int(AverageLuma) the same as the field index, then the field will hold Int -1.
# As the above is about as clear as mud, I shall try to explain with an example.
# If we want to know which frame is the nearest frame prior to frame 1000 which has an AverageLuma greater or equal to 128.0
# and less than 129.0 then PrevDB(1000,128) will hold that frame number, and the RT_Stats call to get it is
# prevfrm = RT_DBaseGetField(PrevDB,1000,128)
# NextDB is the same as PrevDB but the fields point to the nearest frame AFTER the record number (frame).
#
# c, Clip. Planar, YUY2, RGB24, RGB32.
# DB, String. Filename of created DB file.
# PrevDB, String. Filename of optional PrevDB file name.
# NextDB, String. Filename of optional NextDB file name.
# Debug, Bool, default false. True outputs info (eg progress) to DebugView window (Google).
#
# This function is really quite slow and took about 30 mins to create all three DataBases on a 10 Mins 13 secs PAL SD clip.
# Core Duo Duel Core 2.13Ghz and Sata 2 Drives. The AverageLuma of each frame is only gotten once, and the optional DataBases
# are created using the DBase data rather than scanning the frames again. The Optional DBases could be quite big with
# (Framecount * 256 * 4) + $4000 bytes each, but should support up to about 1.9 million frames (where it would break the
# 2 Gig +ve Int maximum, ie go -ve)
# It is recommended to create the databases only once and to re-use them where possible rather than re-creating time and again.
# Once created, the databases make it possible to fairly quickly find similar frames and could perhaps greatly speed up
# scripts to find where cuts have been made between 2 clips or some kind of frame matching scripts.
#
GSCript("""
c
myName="QwikAveLumaScanCreateDB: "
VER = 0.0
Debug = Default(Debug, False)
Start = RT_Timer()
PrevDB=Default(PrevDB,"")
NextDB=Default(NextDB,"")
(DEBUG) ? RT_DebugF("\nQwikAveLumaScanCreateDB() v%.2f by StainlessS\n",VER,name=myName) : NOP
(DEBUG) ? RT_DebugF("Filling DBase with AverageLuma data (Will take some time)\n",name=myName) : NOP
LastFrame=FrameCount-1
RT_DBaseAlloc(DB,FrameCount,"f") # FrameCount records with 1 field each, Float
if(PrevDB=="") {
(DEBUG) ? RT_DebugF("Creating DB",name=myName) : NOP
For(frame=0,LastFrame) {
L = RT_AverageLuma(Last,frame) # Ave luma of frame
RT_DBaseSet(DB,frame,L) # Set DBase Aveluma
(DEBUG && frame % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",frame,(frame+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
if(PrevDB!="" || NextDB!="") {
DBT=DB+"_"+RT_LocalTimeString(File=True) # Temporary DBase filename
RT_DBaseAlloc(DBT,256,"i") # Create temp DBT
DBS = RT_StrPad("",256,"i") # Type String of 256 'i' characters, ie 256 int fields
if(PrevDB!="") {
(DEBUG) ? RT_DebugF("Initializing TEMP DB",name=myName) : NOP
For(i=0,255) { # Init DBT field 0 to not valid
RT_DBaseSet(DBT,i, -1)
}
(DEBUG) ? RT_DebugF("Creating PrevDB",name=myName) : NOP
RT_DBaseAlloc(PrevDB,FrameCount,DBS)
For(frame=0,LastFrame) {
L = RT_AverageLuma(Last,frame) # Ave luma of frame
RT_DBaseSet(DB,frame,L) # Set Aveluma
for(i=0,255) {
prev = RT_DBaseGetField(DBT,i,0) # Get latest frame number with approx luma = i
RT_DBaseSetField(PrevDB,frame,i,prev)
}
Li = Int(L) # AveLuma of frame as index into temp DBase DBT
RT_DBaseSet(DBT,Li,frame) # Update temp DB with current frame as previous for following frames
(DEBUG && frame % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",frame,(frame+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
if(NextDB!="") {
(DEBUG) ? RT_DebugF("Initializing TEMP DB",name=myName) : NOP
For(i=0,255) { # Init DBT field 0 to not valid
RT_DBaseSet(DBT,i, -1)
}
(DEBUG) ? RT_DebugF("Creating NextDB",name=myName) : NOP
RT_DBaseAlloc(NextDB,FrameCount,DBS)
For(frame=LastFrame, 0, -1) {
L = RT_DBaseGetField(DB,frame,0) # Aveluma
for(i=0,255) {
next = RT_DBaseGetField(DBT,i,0)
RT_DBaseSetField(NextDB,frame,i,next)
}
Li = Int(L) # AveLuma of frame as index into temp DBase DBT
RT_DBaseSet(DBT,Li,frame) # Update temp DB with current frame as next for prev frames
Ftmp=LastFrame - Frame
(DEBUG && Ftmp % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",Ftmp,(Ftmp+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
RT_FileDelete(DBT) # Delete Temp Dbase
}
End = RT_Timer()
(DEBUG) ? RT_DebugF("\nTotal Time = %.2f Seconds (%.2f Mins)\n",End-Start,(End-Start)/60.0,name=myName) : NOP
""")
}

Function QwikAveLumaScanGetNear(String DB, String PNDB,int Frame,Float "AveLuma", Float "AveLumaDiff",int "MaxDistance") {
# Function to find the frame nearest (in frame number distance) to the target Frame, whose AverageLuma is within
# +- AveLumaDiff (inclusive) of AveLuma.
#
# DB, String. Filename of DB file as created by QwikAveLumaScanCreateDB()
# PNDB, String, Filename of PrevDB OR NextDB file as created by QwikAveLumaScanCreateDB()
# Frame, Int. Frame number nearest to which, you want to find a NEAR frame.
# AveLuma, Float. Optional value of AverageLuma, defaults to the AverageLuma of arg Frame.
# AveLumaDiff, Float. Optional (default 1.0). Difference Threshold acceptable as similar to AveLuma. AveLumaDiff is inclusive.
# MaxDistance, Int. Optinal maximum number of frames to search relative to Frame arg. Default number of records in DB-1, ie FrameCount - 1.
#
# The PNDB arg controls whether we are looking for frames BEFORE or AFTER the target Frame, use the PrevDB to find BEFORE
# frames or NextDB to find AFTER frames (as described for QwikAveLumaScanCreateDB().
# To find the nearest frame BEFORE frame 1000 with a Aveluma difference within 1.5 of AverageLuma of frame 1000, use eg
# PrvFrm = QwikAveLumaScanGetNear(DB,PrevDB,1000,AveLumaDiff=1.5)
# To find the nearest frame AFTER frame 1000 with an aveluma difference within 1.5 of AverageLuma = 128.0, use eg
# NxtFrm = QwikAveLumaScanGetNear(DB,NextDB,1000,AveLuma=128.0,AveLumaDiff=1.5)
# In this case the AverageLuma of the returned frame number would be greater or equal to (128.0 - 1.5) and less or
# equal to (128.0 + 1.5). This allows you to use an AveLumaDiff of 0.0 to find an EXACT MATCH.
# To Find exact match eg (needs clip and prior DB setup),
# q = RT_AverageLuma(n=1000) # Get AverageLuma of frame 1000
# z = QwikAveLumaScanGetNear(DB, NextDB, 0, AveLuma=q, AveLumaDiff=0.0) # Get next frame after frame 0 exactly same as frame 1000 AverageLuma.
# Subtitle(String(q)+" " + String(z))
# Above will show AverageLuma of frame 1000 and likely show '1000' as next nearest frame to 0 with that exact same AverageLuma.
#
# The Function returns -1 if there is no frame that satisfies conditions.
#
AveLumaDiff=Float(Default(AveLumaDiff,1.0))
L = Defined(AveLuma) ? Float(AveLuma) : RT_DBaseGetField(DB,Frame,0) # Use AveLuma if given, else get from DB
MaxDistance = Default(MaxDistance,RT_DBaseRecords(DB) - 1) # Defaults to FrameCount - 1
Li = Int(L) # Approx ave luma as Int, index into PNDB
tmn = L - AveLumaDiff ## Valid luma range +- AveLumaDiff
tmx = L + AveLumaDiff ##
mni = Max(0, Int(tmn)) # Start PNDB Index to search
mxi = Min(255, Int(tmx)) # End PNDB Index to search
NearLumaFrm = -1 # Not Found
Distance = MaxDistance + 1 # Has to be closer than this
GSCript("""
# Search center interval 1st as likely nearest to target frame (assuming any given AveLuma is that of target Frame)
# Intent to cut down on search distance in mni->mxi loop
frm=RT_DBaseGetField(PNDB,Frame,Li) # Get nearest before/after frame @ approx Li ave luma
if(frm != -1) { # Are there any frames in that luma interval ?
Break=False # Wish there was a Break (sigh)
While(!Break && Abs(Frame-frm) < Distance) {
v = RT_DBaseGetField(DB,frm,0) # AveLuma for roving frm
if(v >= tmn && v <= tmx) { # Is it within FLOAT Thresh range of L ?
NearLumaFrm = frm # We got a nearer one, HeHe
Distance = Abs(Frame-frm)
Break=True # We got less distant, break out of While Loop
} Else {
frm = RT_DBaseGetField(PNDB,frm,Li) # Scan more distant
if(frm== -1) {
Break = True # No More
}
}
}
}
if(Distance > 1) { # If Distance == 1, then we cannot find closer frame
for(i=mni,mxi) { # Int intervals where AveLuma +- AveLumaDiff is valid
if(i != Li) { # We already did Li
frm=RT_DBaseGetField(PNDB,Frame,i) # Get nearest before/after frame @ approx i ave luma
if(frm != -1) { # Are there any frames in that luma interval ?
Break=False # Wish there was a Break (sigh)
While(!Break && Abs(Frame-frm) < Distance) {
v = RT_DBaseGetField(DB,frm,0) # AveLuma for roving frm
if(v >= tmn && v <= tmx) { # Is it within FLOAT Thresh range of L ?
NearLumaFrm = frm # We got a nearer one, HeHe
Distance = Abs(Frame-frm)
Break=True # We got less distant, break out of While Loop
} Else {
frm = RT_DBaseGetField(PNDB,frm,i) # Scan more distant
if(frm== -1) {
Break = True # No More
}
}
}
if(Distance==1) {
i = mxi # Found adjacent frame, No more search needed, Early Break out of For Loop
}
}
}
}
}
""")
return NearLumaFrm
}

[StainlessS]

Post #2 of 3.

MakePrevDB.avs, Creates Database's

# MakePrevDB.avs

Import("QwikAveLumaScan.avs")

AviSource("Test.avi") # Or Whatever

DB ="MyDB.DB"
PrevDB ="MyDB_Prev.DB"

QwikAveLumaScanCreateDB(Last, DB, PrevDB, debug=True)

return MessageClip("DB PREV set Created")


Eduardobedoya.avs, needs pre-created database's

# Eduardobedoya.avs

# MUST create DataBases before using this script using MakePrevDB.avs
# This script creates a Frames.txt file, can then use SelectFrames.avs to get frames instantly.

Import("QwikAveLumaScan.avs")

AviSource("Test.avi") # Or Whatever

# Alter below to suit
ScanAheadSeconds = 10 * 60 # Search range in seconds
AVELUMA_DIFF = 0.1 # Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
MAX_LDIFF = 0.001 # LumaDifference between i frame and candidate duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB ="MyDB.DB"
PrevDB ="MyDB_Prev.DB"
FN="Frames.txt" # Output frame command file for FrameSel()
DEBUG=True # Info to DebugView window (Google)
####
GSCript("""
Start = RT_Timer()
RT_FileDelete(FN) # Delete any existing frames file
LastFrame=FrameCount-1
OCnt = 0 # Frames output count
For(i=0,LastFrame) {
OutFrame = i # Init frame to output
LumaI = RT_DBaseGetField(DB,i,0) # AverageLuma of i frame from DB
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit - 1 to i+1 (downwards)
Fails = 0 # Count false candidates, too many may indicate AVELUMA_DIFF too big
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to j but higher than i that has AverageLuma difference with i, less or equal to AVELUMA_DIFF
j = QwikAveLumaScanGetNear(DB,PrevDB,j,AveLuma=LumaI,AveLumaDiff=AVELUMA_DIFF,maxdistance=j-i-1)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= MAX_LDIFF) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
} Else {
Fails = Fails + 1 # Count False candidate frames for Debug
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True) # For Prune instead of FrameSel (supports audio), use "0,"+String(OutFrame)
OCnt = OCnt + 1
if(i != OutFrame) {
If(DEBUG) {
E=RT_Timer
RT_DebugF("%d ] Failed_LDifs= %-4d LDif=%f ALDif=%f Skipped=%d InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f",
\ OutFrame,Fails,dif,Abs(LumaI-RT_DBaseGetField(DB,OutFrame,0)),OutFrame-i,i+1,OCnt,OCnt/(E-Start),(OutFrame+1)/(E-Start))
}
i = OutFrame # Skip all frames before Outframe
} Else {
If(DEBUG) {
E=RT_Timer
RT_DebugF("%d ] Failed_LDifs= %-4d InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f",
\ OutFrame,Fails,i+1,OCnt,OCnt/(E-Start),(OutFrame+1)/(E-Start))
}
}
}
E = RT_Timer()
S=RT_String("\nTotal Time = %.2f Seconds (%.2f Mins) InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f\n",
\ E-Start,(E-Start)/60.0,LastFrame+1,OCnt,OCnt/(E-Start),(LastFrame+1)/(E-Start))
RT_DebugF(S)
RT_TxtWriteFile(S,"Eduardobedoya.Log",Append=False)
""")

Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug, or for audio support see a few lines above.


SelectFrames.avs, instantly get frames afterdatabases created and frames file created via Eduardobedoya.avs

# SelectFrames.avs

AviSource("Test.avi") # Or Whatever
FN="Frames.txt"
Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug.


Somewhat faster than previous script, but still not startling. The QwikAveLumaScan.Avs funcs may well benefit by converting to plugins.

[StainlessS]

Post #3 of 3.

DummyTest.avs, if you dont have suitable test clip

# DummyTest.avs

Import("QwikAveLumaScan.avs")

TestClip(True) # Get Testclip all UpperCase
#Return Last

# Below are just for show for TestClip, both difs can be 0.0 as we will find exact matches
ScanAheadSeconds = 10 * 60 # Search range in seconds
AVELUMA_DIFF = 0.1 # Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
MAX_LDIFF = 0.001 # LumaDifference between i frame and candidate duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB ="Dummy.DB"
PrevDB ="Dummy_Prev.DB"
FN ="DummyFrames.txt" # Output frame command file for FrameSel()
DEBUG=True # Info to DebugView window (Google)
DELETEDBASE=TRUE # Delete DBase files
QwikAveLumaScanCreateDB(Last, DB, PrevDB,debug=true) # Create DataBases (would normally do as separate job, SLOW)
####
GSCript("""
Start = RT_Timer()
RT_FileDelete(FN) # Delete any existing frames file
LastFrame=FrameCount-1
For(i=0,LastFrame) {
OutFrame = i # Init frame to output
LumaI = RT_DBaseGetField(DB,i,0) # AverageLuma of i frame from DB
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit - 1 to i+1 (downwards)
Fails = 0 # Count false candidates, too many may indicate AVELUMA_DIFF too big
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to j but higher than i that has AverageLuma difference with i, less or equal to AVELUMA_DIFF
j = QwikAveLumaScanGetNear(DB,PrevDB,j,AveLuma=LumaI,AveLumaDiff=AVELUMA_DIFF,maxdistance=j-i-1)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= MAX_LDIFF) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
} Else {
Fails = Fails + 1 # Count False candidate frames for Debug
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True)
if(i != OutFrame) {
(DEBUG)
\ ? RT_DebugF("%d ] Failed_LDifs= %-4d LDif=%f ALDif=%f Skipped=%d",
\ OutFrame,Fails,dif,Abs(LumaI-RT_DBaseGetField(DB,OutFrame,0)),OutFrame-i)
\ : NOP
i = OutFrame # Skip all frames before Outframe
} Else {
(DEBUG) ? RT_DebugF("%d ] Failed_LDifs= %-4d",OutFrame,Fails) : NOP
}
}
if(DELETEDBASE) {
RT_FileDelete(DB)
RT_FileDelete(PrevDB)
}
End = RT_Timer()
(DEBUG) ? RT_DebugF("\nTotal Time = %.2f Seconds (%.2f Mins)\n",End-Start,(End-Start)/60.0) : NOP
""")

Testclip(False) # Now change to upper/lower case mixed to see if it worked OK
Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug.

# Only of use in testing, (If you dont have a real problem clip)
Function TestClip(Bool "Upper") {
Upper=Default(Upper,False)
Test = "AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ"
Test = (Upper) ? Ucase(Test) : Test
CC = ColorBars().Trim(0,-1).KillAudio # Single frame
C=CC.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
s = RT_StrPad("",20,s)
s=RT_String("%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n",
\ s,s,s,s,s,s,s,s,s,s,s,Esc=0)
C = C + CC.Subtitle(s,lsp=0,size=40,align=8,y=20)

}
""")
return C
}

[eduardobedoya]

Thanks Stainless, man, thank you very much for your support, you just freak me out with all that huge amount of scripts, I will read and try everything tomorrow. Please man, just give me the minimal ammount of links necesary to achieve this specific task, the easier way possible, sorry I dident post my laptop specs, it is a laptop with 16gbram, nvidia GTX 680M, multicore 2.9ghz. Please Stainless since I need to apply this script in many videos (batch operation) I would like you suggest me the fastest application possible, if it is possible the easier app also, sorry I am not a programmer. Thanks Advanced.

PD: I get the videos from camtasia recording the laptop screen.
Regards.

[StainlessS]

Have already given a few links in previous post,
Would also suggest VirtualDub, see main forum page.

Have only just now converted the QwikAveLumaScanCreateDB func to a plugin (in coming version of RT_Stats),
previous script created all 3 databases (for a test avi) in about 45 mins, currently got it running
as plugin doing same in 41 seconds, so there is some improvement.

I shall now have a go at the QwikAveLumaScanGetNear function, hoping for a reasonable improvement there also,
I shall not though be touching the Eduardobedoya.avs script, its too much a one-off to bother with,
but it should be pretty nippy I think when other plugs done.

I'll see what I can do to make a simple idiot proof script to auto create Databases and FrameSelect command files,
and also create scripts to load the resultant fixed files, all you will have to do is run VirtualDub, select compression and
then Queue them for Batch Saving (file menu).

maybe a day or two.

[eduardobedoya]

Man from 45 min to 41 seconds, thats optimization!
thanks you stainless
I have read your scripts so I kinda understand that...

First I run the QwikAveLumaScan.avs (the plugin version) inside Virtualdub in orther to create a database (but I dont understand, which application plugin? avisynth? Virtualdub? do I need to install this plugin??)
Then I run the MakePrevDB.avs also inside Virtualdub in orther to create a database
Then I run the Eduardobedoya.avs also inside Virtualdub in orther to create a frames.txt file
and finally use the SelectFrames.avs to get the frames inside Virtualdub, right? inside Virtualdub timeline? or I will get avi files directly?

I have 20 videos (45 minutes footage each one), Do I need to proceed with these steps above for each one of these videos??? or, Can I put in the avisource 4 videos, in orther to create the database of 4 videos in one shot????

I have read this
http://neuron2.net/LVG/avisynth.html
So I beleive that I will have to install avisynth and virtualdub, then create the avs scriptfiles that you provided changing the sorcefiles and paths, then open the avs scripts inside virtualdub in the order posted above??? That would be all right? Do I missing something?? those are all the steps right?

I will wait for you last instruction, to perform a test. thanks again StainlessS.

[StainlessS]

Dont bother with those scripts just now, There are a few little problems, especially around the
end of clip (last frame), so I'm still tinkering a little bit. Shall leave those scripts (when updated)
in place for example usage.

I'll make a script, that allows avi group selection via a file selector (GUI),
does ALL of the dbase creation, processes the avi's using the DB's and output
frames files, and also auto create avs scripts, which should be loaded later into vdub and queued for
compressed output. You can execute the creation script by simply loading into VDub or
playing in Windows Media Player (but will seem to hang until all done).

[eduardobedoya]

pls man, let me know when you will release it, I kinda look around this page everyday xD, Thanks, thanks advanced.

[StainlessS]

Sorry, eduardobedoya, had to go away for a few days but have been working on it.
Have had to increase maximum number of fields in RT_Stats DBase's from 256 to 1024
(same for Attributes) and made a few other small changes. Have nearly implemented
plugin functions for fast finding of frames using a 'fingerprint' of the frame to find.
Fingerprint uses (RT_Stats equivalents to) YPlaneMedian, AveraLuma, YPlaneStdev,
and YInRange (last two RT_Stats only). With a tolerance of about +- 1.0 on fingerprint
primitives (eg difference in AvergeLuma between find and found frames) I'm getting
something like 20 FPS output (to frames text file) on a clip that has no duplicates (ie there is no
frame to find so does a full search) where test clip has 10,000 SD PAL frames searching from
current frame to last frame for every output frame. If tolerances are 0.0, ie exact matches
only, then am getting about 170 to 180 FPS in same case as above. (Core 2 Duo 2.13 Ghz, Sata 2).

Have been reluctant to release RT_Stats update until the new funcs are implemented as flexibly as possible,
will be developer ONLY functions, and not for the casual scriptor. Will I think be released as experimental, as
could still change args, but mostly already there.

Question on your particular problem, are there any unique sequences within duplicate sequence range (10 mins you say) ?,
as currently implemented it searches for LAST frame that matches, so it can skip large numbers of frames all in a single
jump. However, if there are isolated unique sequences beteen Find and Found frames, you would lose the unique sequence.
I could implement modified script to drop the current frame instead of all frames between current Find frame and Found frame,
but it would be significantly slower dropping 1 frame at a time instead of vast swathes of frames.

Please be patient, I am still working on it.

EDIT: Also, do we need to cope with Audio ?


# Below XYX isolated unique sequence, Keep uppercase.
"abcdabcdXYZdefdefgABCdefdefgDEFghijGHIJKLmnopmnopMNOPQRSTUVW"

[eduardobedoya]

Thanks Stainless - audio is not important, in fact, the files are avi. without audio.

Thanks advanced, will wait till you release it.

[eduardobedoya]

Thanks Stainless - No, there are not any unique sequence between two duplicated frames, it would be very strange to have something usefull between two duplicated frames, basically I need to delete ALL frames between to equal frames. Not only delete the duplicated frames, but delete everything between two duplicated frames. (inside a range of 10 minutes).

The footage is 30fps Techsmith codec avi - no audio.

Thanks Advanced, I will wait till your release

[StainlessS]

Post #1 of 2

Not releasing RT_Stats v1.31 with docs just yet, but here it is without new docs or source.
Also includes Prune for Audio. (copy plugs to Plugins dir).

Have implemented for Audio anyways, and two scripts, one to keep unique frames, and one that dont.

See beginning of avs for config.

Here are dll's (still need Gscript and FrameSel v1.12) : LINK REMOVED

Keep Unique script:

# ############################
# Fsel_EduardobedoyaKeepUnique_Batch.avs, by StainlessS
# ############################

# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

####
# Below settings, as close to zero as possible (faster but might miss duplicates), used by QWIK scan routines
LODIFFMED = 1 # INT, Max diff (-ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFMED = 1 # INT, Max diff of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFAL = 0.01 # Float, Max diff (-ve tolerance) of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFAL = 0.01 # Float, Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFSTD = 0.01 # Float, Max diff (-ve tolerance) of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFSTD = 0.01 # Float, Max diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFINR = 0.01 # Float, Max diff (-ve tolerance) of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFINR = 0.01 # Float, Max diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good.
MAX_LDIFF = 0.01 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####

####
AUDIO = True # False, no audio. True Supports audio, Audio requires Prune Plugin
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI, files Will Batch create FrameSel command files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_EduardobedoyaKeepUnique_Batch: "
LOG="Fsel_EduardobedoyaKeepUnique_Batch.Log"
RT_TxtWriteFile(LOG,LOG,Append=False)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
DropDupSequencesKeepUnique(FN,ScanAheadSecs,
\ ldm=LODIFFMED, hdm=HIDIFFMED,
\ ldL=LODIFFAL, hdL=HIDIFFAL,
\ lds=LODIFFSTD, hds=HIDIFFSTD,
\ ldr=LODIFFINR, hdr=HIDIFFINR,
\ LDThresh=MAX_LDIFF,audio=AUDIO,log=LOG)
T = RT_Timer - START
S=RT_String(" %s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_TxtWriteFile(S,LOG,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropDupSequencesKeepUnique(String "AviName",int "ScanAheadSecs",
\ int "ldM",int "hdM",Float "ldL",Float "hdL",Float "ldS",Float "hdS",Float "ldR",Float "hdR",Float "LDThresh",Bool "Audio",String "Log") {
myName="DropDupSequencesKeepUnique: "
ScanAheadSecs=Default(ScanAheadSecs,10*60)
ldM = Default(ldM,1) hdM = Default(hdM,1)
ldL = Float(Default(ldL,0.01)) hdL = Float(Default(hdL,0.01))
ldS = Float(Default(ldS,0.01)) hdS = Float(Default(hdS,0.01))
ldR = Float(Default(ldR,0.01)) hdR = Float(Default(hdR,0.01))
LDThresh = Float(Default(LDThresh,0.01))
LOG = Default(LOG,"DropDupSequencesKeepUnique.LOG")
Assert(Exist(AviName),myName+AviName+" Does Not Exist")
Avisource(AviName)
Audio=Default(Audio,False) # Needs Prune plugin if Audio
Audio=(!HasAudio) ? False : Audio
Assert(ScanAheadSecs>=0,myName+"ScanAheadSecs Must be greater than zero")
Assert(ldM>=0 && hdM >0,myName+"ldM and hdM Must be greater than zero")
Assert(ldL>=0.0 && hdL >=0.0,myName+"ldL and hdL Must be greater or equal to zero")
Assert(ldS>=0.0 && hdS >=0.0,myName+"ldS and hdS Must be greater or equal to zero")
Assert(ldR>=0.0 && hdR >=0.0,myName+"ldR and hdR Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_KUnique_Frames.txt"
ScriptFile=PathAndNode+"_KUnique_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
NextDB=PathAndNode+"_Next.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
FrameSel_Select="""
Avisource("%s")
CmdFrames="%s"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=True) : NOP
Return Last
"""
Prune_Select="""
Avisource("%s")
CmdFrames="%s"
PruneFrames=CmdFrames+"_Prune.txt"
Ex=(Exist(CmdFrames))
(Ex) ? FrameSel_CmdReWrite(PruneFrames,Cmd=CmdFrames,reject=True,Range=True,Prune=True) : NOP
(Ex) ? Prune(cmd=PruneFrames,Fade=10,FadeIn=True,FadeSplice=True,FadeOut=True) : NOP
Return Last
"""

Select_S = (Audio) ? Prune_Select : FrameSel_Select # Select FrameSel or Prune extraction
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,CmdFrames) # Insert filenames

START = RT_Timer
RT_QwikAveLumaScanCreateDB(DB,prevdb="",nextdb=NextDB,debug=true) # Forward scanning
T= RT_Timer - START
S = RT_String(" QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_TxtWriteFile(S,LOG,Append=True)
GSCript("""
START = RT_Timer
Dropped = 0
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)
For(i=0,LastFrame) {
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching i+1 to Endlimit inclusive
# QWIK Find a series of likely matching frames
For(j = i + 1,EndLimit) {
j = RT_QwikAveLumaScanGetNear(DB,NextDB,j,
\ findframe=i,
\ LoDiffMed =ldM, HiDiffMEd =hdM,
\ LoDiffAl =ldL, HiDiffAL =hdL,
\ LoDiffStd =ldS, HiDiffStd =hdS,
\ LoDiffInr =ldR, HiDiffInr =hdR,
\ maxdistance=EndLimit-j, Inclusive=True)
if(j < 0) { # j Will be -1, ie Not Found
j = EndLimit # Early break from j, DONT delete frame
} Else { # We found a candidate frame, j will be greater than i
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Average pixel diff between i frame and candidate j
if(dif <= LDThresh) { # is candidate frame a good match ?
Dropped = Dropped + 1
RT_TxtWriteFile(String(i),
\ CMDFrames,Append=True) # Frame to DELETE, Keep later frame, delete earlier
j = EndLimit # Early break, move on to next i frame
} # Otherwise continue QWIK scan for next candidate in series
}
}
}
RT_TxtWriteFile(Select_S,ScriptFile,Append=False)
RT_FileDelete(DB)
RT_FileDelete(NextDB)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String(" Dropping %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]\n QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", \
Dropped, FrameCount,Width,Height,FT,FT/60.0,FrameRate,T,T/60.0,FrameCount/T,(FrameCount-Dropped)/T)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
""")
Return 0
}

EDITED:

and log


Fsel_EduardobedoyaKeepUnique_Batch.Log


1/4 ] Processing D:\AVS\AVI\IN\FLASHTEST.avi
QWIK Scan DBase creation = 15.75 Secs (0.26 Mins)
Dropping 0 of 7342 frames [640x400 293.68 secs (4.89 Mins) @ 25.00 FPS]
QWIK SCAN 5.42 Secs (0.09 Mins) InFPS=1354.36 OutFPS=1354.36
D:\AVS\AVI\IN\FLASHTEST.avi Tot File Time = 21.25 Seconds (0.35 Mins)


2/4 ] Processing D:\AVS\AVI\IN\Scrambled.avi
QWIK Scan DBase creation = 244.28 Secs (4.07 Mins)
Dropping 25500 of 45000 frames [720x576 1800.00 secs (30.00 Mins) @ 25.00 FPS]
QWIK SCAN 524.80 Secs (8.75 Mins) InFPS=85.75 OutFPS=37.16
D:\AVS\AVI\IN\Scrambled.avi Tot File Time = 769.27 Seconds (12.82 Mins)


3/4 ] Processing D:\AVS\AVI\IN\TEST2.avi
QWIK Scan DBase creation = 46.72 Secs (0.78 Mins)
Dropping 0 of 15319 frames [720x576 612.76 secs (10.21 Mins) @ 25.00 FPS]
QWIK SCAN 20.47 Secs (0.34 Mins) InFPS=748.40 OutFPS=748.40
D:\AVS\AVI\IN\TEST2.avi Tot File Time = 67.47 Seconds (1.12 Mins)


4/4 ] Processing D:\AVS\AVI\IN\TEST.avi
QWIK Scan DBase creation = 57.47 Secs (0.96 Mins)
Dropping 0 of 13861 frames [720x576 554.44 secs (9.24 Mins) @ 25.00 FPS]
QWIK SCAN 13.17 Secs (0.22 Mins) InFPS=1052.31 OutFPS=1052.31
D:\AVS\AVI\IN\TEST.avi Tot File Time = 70.84 Seconds (1.18 Mins)


DONE

TOTAL Time = 928.83 Seconds (15.48 Mins)



Only Scrambled.avi is scrambled, others just to test that it dont delete frames where not required.

Here MakeScrambled.avs, test clip

###################
# MakeScrambled.avs
###################
# Only of use in testing, (If you dont have a real problem clip)

Function ScrambleClip(clip c) { # intended for PAL source [numbers may not work for NTSC, have not tried, use AssumeFPS(25.0) beforehand]
c
fc=FrameCount
Need = 27 * 750
Assert(fc >= Need,"ScrambleClip:, need at least "+String(Need)+" Frame clip as source(got "+String(fc)+")")

# Below XYZ isolated sequence. Will keep all upper case letters (including XYZ in Keep Unique script).
# Keep unique script will keep LAST instance of ALL frames within Scanahead.
# Non Keep Unique avs will jump past XYZ frames and they will not be kept, but faster. (jumps from 1st lowercase 'a' to
# last 'A', and omits everything between.

Test = Ucase("abcdabcdXYZdefdefgABCdefdefgDEFghijGHIJKLmnopmnopMNOPQRSTUVW")
CC=C.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
si = RT_ORD(s) - RT_Ord("A") # Subtract 65, make 'A'=0, 'Z' = 25
CT=C.Trim(si*750,-750).Subtitle(S,Size=20)
CT = (s>="X" && s<="Z") ? CT.Subtitle("UNIQUE",Align=5,size=60) : CT
CC = CC ++ CT
}
""")
return CC
}

avisource("JurassicPark.avi")
Trim(16854,182049) # Chop off intro
ScrambleClip()

return last

[StainlessS]

Post #2 of 2

No Keep unique frames

# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# ############################

# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

####
# Below settings, as close to zero as possible (faster but might miss duplicates), used by QWIK scan routines
LODIFFMED = 1 # INT, Max diff (-ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFMED = 1 # INT, Max diff of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFAL = 0.01 # Float, Max (-ve tolerance) diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFAL = 0.01 # Float, Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFSTD = 0.01 # Float, Max (-ve tolerance) diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFSTD = 0.01 # Float, Max diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFINR = 0.01 # Float, Max (-ve tolerance) diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFINR = 0.01 # Float, Max diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good.
MAX_LDIFF = 0.01 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####

####
AUDIO = True # False, no audio. True Supports audio, Audio requires Prune Plugin
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI, files Will Batch create FrameSel command files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_TxtWriteFile(LOG,LOG,Append=False)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
DropDupSequences(FN,ScanAheadSecs,
\ ldm=LODIFFMED, hdm=HIDIFFMED,
\ ldL=LODIFFAL, hdL=HIDIFFAL,
\ lds=LODIFFSTD, hds=HIDIFFSTD,
\ ldr=LODIFFINR, hdr=HIDIFFINR,
\ LDThresh=MAX_LDIFF,audio=AUDIO,log=LOG)
T = RT_Timer - START
S=RT_String(" %s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_TxtWriteFile(S,LOG,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropDupSequences(String "AviName",int "ScanAheadSecs",
\ int "ldM",int "hdM",Float "ldL",Float "hdL",Float "ldS",Float "hdS",Float "ldR",Float "hdR",Float "LDThresh",Bool "Audio",String "Log") {
myName="DropDupSequences: "
ScanAheadSecs=Default(ScanAheadSecs,10*60)
ldM = Default(ldM,1) hdM = Default(hdM,1)
ldL = Float(Default(ldL,0.01)) hdL = Float(Default(hdL,0.01))
ldS = Float(Default(ldS,0.01)) hdS = Float(Default(hdS,0.01))
ldR = Float(Default(ldR,0.01)) hdR = Float(Default(hdR,0.01))
LDThresh = Float(Default(LDThresh,0.01))
LOG = Default(LOG,"DropDupSequences.LOG")
Assert(Exist(AviName),myName+AviName+" Does Not Exist")
Avisource(AviName)
Audio=Default(Audio,False) # Needs Prune plugin if Audio
Audio=(!HasAudio) ? False : Audio
Assert(ScanAheadSecs>0,myName+"ScanAheadSecs Must be greater than zero")
Assert(ldM>=0 && hdM >=0,myName+"ldM and hdM Must be greater or equal to zero")
Assert(ldL>=0.0 && hdL >=0.0,myName+"ldL and hdL Must be greater or equal to zero")
Assert(ldS>=0.0 && hdS >=0.0,myName+"ldS and hdS Must be greater or equal to zero")
Assert(ldR>=0.0 && hdR >=0.0,myName+"ldR and hdR Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.txt"
ScriptFile=PathAndNode+"_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
PrevDB=PathAndNode+"_Prev.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
FrameSel_Select="""
Avisource("%s")
CmdFrames="%s"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last
"""
Prune_Select="""
Avisource("%s")
CmdFrames="%s"
PruneFrames=CmdFrames+"_Prune.txt"
Ex=(Exist(CmdFrames))
(Ex) ? FrameSel_CmdReWrite(PruneFrames,Cmd=CmdFrames,reject=False,Range=True,Prune=True) : NOP
(Ex) ? Prune(cmd=PruneFrames,Fade=10,FadeIn=True,FadeSplice=True,FadeOut=True) : NOP
Return Last
"""

Select_S = (Audio) ? Prune_Select : FrameSel_Select # Select FrameSel or Prune extraction
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,CmdFrames) # Insert filenames

START = RT_Timer
RT_QwikAveLumaScanCreateDB(DB,prevdb=PrevDB,nextdb="",debug=true) # Backwards scanning
T= RT_Timer - START
S = RT_String(" QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_TxtWriteFile(S,LOG,Append=True)
GSCript("""
START = RT_Timer
Dropped = 0
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)
For(i=0,LastFrame) {
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit to i+1 inclusive (downwards)
# QWIK Find a series of likely matching frames
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to EndLimit but higher than i
j = RT_QwikAveLumaScanGetNear(DB,PrevDB,j,
\ findframe=i,
\ LoDiffMed =ldM, HiDiffMEd =hdM,
\ LoDiffAl =ldL, HiDiffAL =hdL,
\ LoDiffStd =ldS, HiDiffStd =hdS,
\ LoDiffInr =ldR, HiDiffInr =hdR,
\ maxdistance=j-(i+1), Inclusive=True)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= LDThresh) {
Dropped = Dropped + (j-i)
i = j # Last frame within ScanAheadFrames that is similar to i
j = 0 # Early break
}
} # Otherwise continue QWIK scan for next candidate in series
}
RT_TxtWriteFile(String(i),CMDFrames,Append=True)
}
RT_TxtWriteFile(Select_S,ScriptFile,Append=False)
RT_FileDelete(DB)
RT_FileDelete(PrevDB)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String(" Dropping %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]\n QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", \
Dropped, FrameCount,Width,Height,FT,FT/60.0,FrameRate,T,T/60.0,FrameCount/T,(FrameCount-Dropped)/T)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
""")
Return 0
}

EDITED:

and log

Fsel_Eduardobedoya_Batch.Log

1/4 ] Processing D:\AVS\AVI\IN\FLASHTEST.avi
QWIK Scan DBase creation = 16.30 Secs (0.27 Mins)
Dropping 0 of 7342 frames [640x400 293.68 secs (4.89 Mins) @ 25.00 FPS]
QWIK SCAN 8.44 Secs (0.14 Mins) InFPS=870.11 OutFPS=870.11
D:\AVS\AVI\IN\FLASHTEST.avi Tot File Time = 24.83 Seconds (0.41 Mins)


2/4 ] Processing D:\AVS\AVI\IN\Scrambled.avi
QWIK Scan DBase creation = 236.22 Secs (3.94 Mins)
Dropping 27750 of 45000 frames [720x576 1800.00 secs (30.00 Mins) @ 25.00 FPS]
QWIK SCAN 23.17 Secs (0.39 Mins) InFPS=1942.00 OutFPS=744.43
D:\AVS\AVI\IN\Scrambled.avi Tot File Time = 259.59 Seconds (4.33 Mins)


3/4 ] Processing D:\AVS\AVI\IN\TEST2.avi
QWIK Scan DBase creation = 53.75 Secs (0.90 Mins)
Dropping 0 of 15319 frames [720x576 612.76 secs (10.21 Mins) @ 25.00 FPS]
QWIK SCAN 27.56 Secs (0.46 Mins) InFPS=555.80 OutFPS=555.80
D:\AVS\AVI\IN\TEST2.avi Tot File Time = 81.61 Seconds (1.36 Mins)


4/4 ] Processing D:\AVS\AVI\IN\TEST.avi
QWIK Scan DBase creation = 69.00 Secs (1.15 Mins)
Dropping 0 of 13861 frames [720x576 554.44 secs (9.24 Mins) @ 25.00 FPS]
QWIK SCAN 18.75 Secs (0.31 Mins) InFPS=739.25 OutFPS=739.25
D:\AVS\AVI\IN\TEST.avi Tot File Time = 87.91 Seconds (1.47 Mins)


DONE



TOTAL Time = 453.94 Seconds (7.57 Mins)


Is somewhat better than 1st script in thread where we were getting about 1 frame every 12 seconds.

EDIT: Has GUI File Selector to group select AVI files, auto creates DataBases, Frames command files and
avs files to instantly select edited clips, and deletes Dbases files.

EDIT: Just spotted potential (but unlikely) problem, if mutiple consecutive SPACES in AVI filenames, will
convert to single space and NOT find them. (RT_StrReplace line).

EDIT: Scripts edited for minor problems or typo's.
The Prune extractor removes possible audio glitches ('cracks') at splices when audio present.

PS, were the scripts idiot proof ? :)

[eduardobedoya]

Thanks Stainlesss, I read your notes and tried your script

First I installed the 32bit version of Virtualdub (I dont know why windows media player fail to open Fsel_Eduardobedoya_Batch.avs)
Is it the same if I have the 32 or 64 bits version of Virtualdub?

Then I installed Avisynth_258 and the RT Stats plugin that you provided
I couldn't find FrameSel v1.12 so I guessed v2.12 could do the trick, so I installed from here
http://forum.doom9.org/showthread.php?p=1566581#post1566581
Then I installed GScript_11 from here
http://forum.doom9.org/showthread.php?t=147846
(I copy all the plugins' dlls inside the Avisynth's plugin directory)
Finally I installed debug view from here
http://technet.microsoft.com/en-gb/sysinternals/bb896647

I run VirtualDub and open Fsel_Eduardobedoya_Batch.avs
the open windows showed, then I pick up some 1920x1080 20min video
here is the LOG:

00000001 0.00000000 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000002 15.38796234 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000003 15.40174389 [7920] Fsel_Eduardobedoya_Batch:
00000004 15.40177345 [7920] Fsel_Eduardobedoya_Batch:
00000005 15.40181732 [7920] Fsel_Eduardobedoya_Batch: 1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\cap19 -4.avi
00000006 15.41219711 [7920] TSC2: Instantiating codec with Q=2/6, force KFs=800, force KFbytes=5000000
00000007 15.43127728 [7920]
00000008 15.43127728 [7920] RT_QwikAveLumaScanCreateDB: RT_QwikAveLumaScanCreateDB() by StainlessS
00000009 15.43131065 [7920] RT_QwikAveLumaScanCreateDB: Filling DBase with YStats data (Will take some time)
00000010 15.43134689 [7920] RT_QwikAveLumaScanCreateDB: Creating DB
00000011 23.23528481 [7920] RT_QwikAveLumaScanCreateDB: record(1024) 5.5%
00000012 31.00768471 [7920] RT_QwikAveLumaScanCreateDB: record(2048) 10.9%
00000013 35.70775223 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000014 35.70792007 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000015 38.72195816 [7920] RT_QwikAveLumaScanCreateDB: record(3072) 16.4%
00000016 46.43200684 [7920] RT_QwikAveLumaScanCreateDB: record(4096) 21.8%
00000017 54.15245438 [7920] RT_QwikAveLumaScanCreateDB: record(5120) 27.3%
00000018 61.86000061 [7920] RT_QwikAveLumaScanCreateDB: record(6144) 32.7%
00000019 69.56040955 [7920] RT_QwikAveLumaScanCreateDB: record(7168) 38.2%
00000020 77.31515503 [7920] RT_QwikAveLumaScanCreateDB: record(8192) 43.6%
00000021 78.57512665 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000022 80.63726044 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000023 82.39134979 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000024 85.02700043 [7920] RT_QwikAveLumaScanCreateDB: record(9216) 49.1%
00000025 88.59996033 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 2
00000026 92.72863007 [7920] RT_QwikAveLumaScanCreateDB: record(10240) 54.5%
00000027 100.43165588 [7920] RT_QwikAveLumaScanCreateDB: record(11264) 60.0%
00000028 108.12608337 [7920] RT_QwikAveLumaScanCreateDB: record(12288) 65.4%
00000029 115.83764648 [7920] RT_QwikAveLumaScanCreateDB: record(13312) 70.9%
00000030 123.53649902 [7920] RT_QwikAveLumaScanCreateDB: record(14336) 76.3%
00000031 131.22393799 [7920] RT_QwikAveLumaScanCreateDB: record(15360) 81.8%
00000032 138.88714600 [7920] RT_QwikAveLumaScanCreateDB: record(16384) 87.2%
00000033 146.58955383 [7920] RT_QwikAveLumaScanCreateDB: record(17408) 92.7%
00000034 154.30934143 [7920] RT_QwikAveLumaScanCreateDB: record(18432) 98.1%
00000035 156.98666382 [7920] RT_QwikAveLumaScanCreateDB: record(18785) 100.0%
00000036 156.98908997 [7920] RT_QwikAveLumaScanCreateDB: Creating PrevDB
00000037 157.05241394 [7920] RT_QwikAveLumaScanCreateDB: record(1024) 5.5%
00000038 157.05616760 [7920] RT_QwikAveLumaScanCreateDB: record(2048) 10.9%
00000039 157.05995178 [7920] RT_QwikAveLumaScanCreateDB: record(3072) 16.4%
00000040 157.06365967 [7920] RT_QwikAveLumaScanCreateDB: record(4096) 21.8%
00000041 157.06732178 [7920] RT_QwikAveLumaScanCreateDB: record(5120) 27.3%
00000042 157.07098389 [7920] RT_QwikAveLumaScanCreateDB: record(6144) 32.7%
00000043 157.07464600 [7920] RT_QwikAveLumaScanCreateDB: record(7168) 38.2%
00000044 157.07839966 [7920] RT_QwikAveLumaScanCreateDB: record(8192) 43.6%
00000045 157.08206177 [7920] RT_QwikAveLumaScanCreateDB: record(9216) 49.1%
00000046 157.08575439 [7920] RT_QwikAveLumaScanCreateDB: record(10240) 54.5%
00000047 157.08943176 [7920] RT_QwikAveLumaScanCreateDB: record(11264) 60.0%
00000048 157.09320068 [7920] RT_QwikAveLumaScanCreateDB: record(12288) 65.4%
00000049 157.09689331 [7920] RT_QwikAveLumaScanCreateDB: record(13312) 70.9%
00000050 157.10060120 [7920] RT_QwikAveLumaScanCreateDB: record(14336) 76.3%
00000051 157.10430908 [7920] RT_QwikAveLumaScanCreateDB: record(15360) 81.8%
00000052 157.10803223 [7920] RT_QwikAveLumaScanCreateDB: record(16384) 87.2%
00000053 157.11178589 [7920] RT_QwikAveLumaScanCreateDB: record(17408) 92.7%
00000054 157.11546326 [7920] RT_QwikAveLumaScanCreateDB: record(18432) 98.1%
00000055 157.11676025 [7920] RT_QwikAveLumaScanCreateDB: record(18785) 100.0%
00000056 157.11859131 [7920] RT_QwikAveLumaScanCreateDB: total time = 141.69 seconds (2.36 mins)
00000057 157.12307739 [7920] DropDupSequences: QWIK Scanning file ... Please Wait
00000058 178.10548401 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 2
00000059 217.35881042 [7920] DropDupSequences: Dropping 18526 of 18785 frames [1920x1080 1252.33 secs (20.87 Mins) @ 15.00 FPS]
00000060 217.35885620 [7920] DropDupSequences: QWIK SCAN 60.23 Secs (1.00 Mins) InFPS=311.88 OutFPS=4.30
00000061 217.36831665 [7920] Fsel_Eduardobedoya_Batch: C:\Users\Eduardo\Desktop\FOR AVISYNTH\cap19 -4.avi Tot File Time = 201.97 Seconds (3.37 Mins)
00000062 217.37132263 [7920] Fsel_Eduardobedoya_Batch:
00000063 217.37135315 [7920] Fsel_Eduardobedoya_Batch:
00000064 217.37139893 [7920] Fsel_Eduardobedoya_Batch: TOTAL Time = 201.97 Seconds (3.37 Mins)
00000065 217.37141418 [7920] Fsel_Eduardobedoya_Batch:
00000066 217.37489319 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73647561(auds), 0)
00000067 217.37492371 [7920] 05533D48->CAVIStreamSynth(audio)
00000068 217.37496948 [7920] 05533D48->CAVIStreamSynth::AddRef() (audio) gRefCnt=3, m_refs=1
00000069 217.37498474 [7920] 003C8740->CAVIFileSynth::AddRef() gRefCnt=4, m_refs=3
00000070 217.37503052 [7920] 05533D48->CAVIStreamSynth::Info(0018F948, 204) (audio)
00000071 217.37504578 [7920] 05533D48->CAVIStreamSynth::ReadFormat() (audio)
00000072 217.37536621 [7920] 05533D48->CAVIStreamSynth::ReadFormat() (audio)
00000073 217.37565613 [7920] 05533D48->CAVIStreamSynth::Info(0018F9E0, 204) (audio)
00000074 217.37568665 [7920] 05533D48->CAVIStreamSynth::Info(0018F9D8, 204) (audio)
00000075 217.37573242 [7920] 05533D48->CAVIStreamSynth::Info(0018F9D8, 204) (audio)
00000076 217.37577820 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73647561(auds), 1)
00000077 217.37579346 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73766169(iavs), 1)
00000078 217.37583923 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73646976(vids), 0)
00000079 217.37586975 [7920] 05533C28->CAVIStreamSynth(video)
00000080 217.37591553 [7920] 05533C28->CAVIStreamSynth::AddRef() (video) gRefCnt=5, m_refs=1
00000081 217.37593079 [7920] 003C8740->CAVIFileSynth::AddRef() gRefCnt=6, m_refs=4
00000082 217.37596130 [7920] 05533C28->CAVIStreamSynth::Info(0018F978, 204) (video)
00000083 217.37600708 [7920] 05533C28->CAVIStreamSynth::ReadFormat() (video)
00000084 217.37629700 [7920] 05533C28->CAVIStreamSynth::ReadFormat() (video)
00000085 217.37661743 [7920] 05533C28->CAVIStreamSynth::Info(0018FA0C, 204) (video)
00000086 217.37663269 [7920] 05533C28->CAVIStreamSynth::Info(0018FA04, 204) (video)
00000087 217.37666321 [7920] 05533C28->CAVIStreamSynth::Info(0018FA04, 204) (video)
00000088 217.37803650 [7920] 05533C28->CAVIStreamSynth::QueryInterface() (video) {00020022-0000-0000-c000-000000000046} (IAVIStreaming)
00000089 217.37808228 [7920] 05533C28->CAVIStreamSynth::AddRef() (video) gRefCnt=7, m_refs=2
00000090 217.37809753 [7920] 05533C28->CAVIStreamSynth::Begin(0, 2073600, 2000) (video)
00000091 217.37814331 [7920] 05533C28->CAVIStreamSynth::Release() (video) gRefCnt=6, m_refs=1
00000092 217.43962097 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 6

the process finished and an black screen appeared with yellow letters indicating some time
also created the "cap19 -4_Frames.txt" and "cap19 -4_SelectFrames.AVS" files inside the source directory
but I dont know how to export the video, If I put File>save as avi, it export a black screen video
If I try to open the created "cap19 -4_SelectFrames.AVS" it will pop a dialog box saying...
Avisynth open failure:
Script error: there is no function named "FrameSel"

Do I need to install some plugin in VirtualDub?
Is there something I am doing wrong?
Is there a way to import the result frames directly inside premiere? or another editor that could render faster than VirtualDub?
I found that VirtualDub render to slow, I need to render almost 20 hours with x264 compression, pls Stainlesss what do you suggest?

[StainlessS]

Not sure, I think you need 32 bit VDub (I still use XP32, and will continue to after APRIL 8 [when
dropped from Ms support, I think])

I see no reason why windows Media player should not open the script, although it would just seem to hang
until script comleted.

Should be able to open avs script containing only eg:

return Colorbars()



Yes, it looks like you have Avisynth v2.58 installed, I think many of the DebugView messages
are coming from v2.58 eg "CAVIStreamSynth::QueryInterface()" messages, v2.58 was I think complied
with debug logging enabled.
I use v2.6, might be beter choice than v2.58, despite it being alpha, is more stable than v2.58, IMHO.
If you set DebugView filter (EDIT menu), to only display lines with a colon (':') then it would only
capture relevant messages (mostly), I nearly always use a colon in debug messages.

yep, you were correct, FrameSel v2.12, not v1.12 (my mistake).
However, it may be that you installed v2.6 plugin (for Avisynth v2.6) and so Avisynth cannot find it.
(Avisynth v2.6 can load v2.5 plugins but Avisynth v2.58 cannot load v2.6 plugins).
In the folder named 'Avisynth25' not 'Avisynth26' unless you change to Avisynth v2.6Alpha5.

Looks like we are dropping way too many frames

DropDupSequences: Dropping 18526 of 18785 frames [1920x1080 1252.33 secs (20.87 Mins) @ 15.00 FPS]

As you say you are using screencaps (TechSmith codec), I assumed duplicate sequences would be pretty
identical, looks like (by dropped frame numbers) that this is not the case.
Is there something other than purely desktop involved (do the captures contain windows displaying
streamed video or something else from lossy compressed source). (I'm under impression that TechSmith
is lossless but intended for non complex image capture eg DeskTop with a limited number of colors,
and little change from frame to frame).

Once we get it working OK, then you can use Premiere (I think) or MeGUI to output x264/mp4.

For just now, suggest install correct FrameSel plug, and change config to something like

LODIFFMED = 1
HIDIFFMED = 1
LODIFFAL = 1.0
HIDIFFAL = 1.0
LODIFFSTD = 1.0
HIDIFFSTD = 1.0
LODIFFINR = 1.0
HIDIFFINR = 1.0
MAX_LDIFF = 20.0

and see what happens

Is it possible that you could use VirtualDub to chop out two sets of duplicate frames (about half a dozen of each)
and save them with "Direct Stream Copy" on Video Menu. I dont want to be downloading gigabytes of stuff on my Mobile BroadBand,
ideally no more than about 40-50MB. You can send me PM with link if you prefer.
EDIT: Upload to Mediaifre or SendSpace(no account needed for SendSpace), and maybe 100MB max, I can use broadband
instead.

I'm still playing with this, mainly the KeepUnique version as it seems more useful, also, if you are not using purely desktop
caps then the non keep unique version is probably not the way to go for you either.

[StainlessS]

currently mobile (in pub), now I see the sample, I better know what we are dealing with, quite some problem. as most frames identical and with very small differences between edits and undos, suggest all config stuff be set to 0 or 0.0.
had also come to conclusion that no keep unique of little use, however it seems that it is exactly what we need but with all zero tolerances.
if it does work, resultant clip will be only a few frames and may need for each frame to be duplicated. maybe 60 times each, or setting Framerate to 0.5 fps.
I shall have a play and get back to you.
nice art work by the way, wish I could paint.

[eduardobedoya]

Thanks man,
programmers are digital architects, digital engineers
all digital painting n 3d modeling is based on programming, so
Thanks man again, wish I could understand more of programming, with all that planning and design.

[StainlessS]

Seem to have hit a problem, I was getting weird numbers from the script, and it seems that Avisynth
AviSource video source filter, aint working terribly well with TechSmith codec. We gets lots of
black flashing frames, which presumably mean something like 'Copy the previous frame exactly",
ie in the video stream there are probably some kind of flags that tell the app to duplicate previous frames,
and for those frames TechSmith delivers only black frames or frame 'differences'.
I did try cutting out just frames 14 and 16 from your sample and saving with "Direct Stream Copy",
to another AVI, and it actually saved 17 frames (0-16). DivX and XDiv only allow you to cut video at
KEY frames, I just checked in Vdub, first KEY frame = 0, next is 807, then 836, 848 , 1657, 1807, 1815, and then
no more.

EDIT: From post #3, Clip should be seekable, eg NOT DivX with single keyframe (way too slow)

So TechSmith is NOT cutt-able at each frame, and to properley extract eg frame 806, it may need to seek
all the way back to frame 0 to decode that frame (and decode all frames between 0 and 806).
DirectShowSource() seems to work OK, but Directshow is notorious for delivering the wrong frames (guesses),
I'll have a go at getting the script working using DirectShowSource, but cannot guarentee the results as we
do really need frame accurate seeking. You may have to re-encode to uncompressed RGB (or better, UT_Video
codec as RGB24 or 32) before trying out the script.

There is a thread on-site mentioning the exact same problem here:- http://forum.doom9.org/showthread.php?p=1062041&highlight=techsmith+avisource#post1062041

and there seemed to have been no resolution for that problem.

Would you mind if I gave a link to your sample, it may be that developers are not aware that AviSource does not play nice with video samples
containing TechSmith 'Repeat frame' flags (or whatever they might be called) ?

below just so you can see what I mean.

# Loads using FOURCC 'tsc2', TechSmith Screen Codec 2 (v1.0.6 April 23 2013)

AVISource("painting test for avisynth.avi") # Junk frames where duplicate frame flags
#AVIFileSource("painting test for avisynth.avi") # Junk frames where duplicate frame flags
#OpenDMLSource("painting test for avisynth.avi") # Junk frames where duplicate frame flags

# Info() # This actually forces AviSource to work as it modifies the input frame
# BUT NOT with below Trim()

#DirectShowSource("painting test for avisynth.avi") # Seems to work (but lots of seeking probably)

#Trim(806,-1) # Frame before 2nd KEY frame
return last

EDIT:
From Avisynth v2.6 Docs: built-in OpenDML code (taken from VirtualDub)
Stange that Vdub current and even VDubMod load/play clip OK, whereas Avisynth dont, looks like an AviSource bug to me,
have not checked against Avisynth v2.58, dont seem any point as above unresolved problem link was back in 2007.

EDIT: And just so I dont forget where I got it, here the decoder only tsc2 codec from TechSmith: https://support.techsmith.com/entries/22849238-Camtasia-Studio-Download-the-TSC2-standalone-codec

EDIT:
Looks like it may be connected to 'dropped_frame's, perhaps somewhere near here

PVideoFrame AVISource::GetFrame(int n, IScriptEnvironment* env)

[eduardobedoya]

Thanks StainlessS,
Yes, Techsmith is a very compressed video format, but it remains lossless, I guess it uses that kind of artifacts when recording, because the filesize is very small when most of the screen remain the same while recording, whereas when the screen is zoomed and panned very often while recording the file size is quite large (both files with the same duration).

Sure man, use the link as you want, pls let me know if there is some improvement, I saw the other post with the same problem, but that guy never upload any sample right? I hope you could deliver this sample to good hands.

On the other hand, Its possible that I reencode all the videos in order to avisynth to read them? what encoder do you suggest? a MPEG kind? there is some lossless alternatives? sometime ago I tried lots of codecs to decide whichone use to record the speedpaintings, Techsmith was the best.

PD: what is the tsc2 decoder link for?

[StainlessS]

what is the tsc2 decoder link for?
so I know where I got it, and for anyone else eg developer wanting to test out problem.

Find here, new version RT_Stats (without FrameSel) and update scripts.

Tried Fsel_Eduardobedoya_Batch.avs using DirectShowSource on TSCC, works but very slow (instantiates codec
on every frame and sends bucket loads of identical text to DebugView). Unfortunately, it delivers
wrong frames (ie guesses, as I said earlier), so of no use.

If you convert TSCC clips to either RGB compressed or YV12, maybe HuffYUV or UT_Video codecs, will work fine.
Here is update script and dll, works great. LINK REMOVED

Also included a script to remove duplicate sequences from a clip.
Tested using 3 trailer type clips with some differing sequences in each, joined them together and it removes the duplicate seqs.

You could try this to convert to UT_Video, named 'Whatever.bat'

setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get input file, No terminating Backslash, "." = current directory
set INDIR="."


REM Where to place output file, No terminating Backslash.
set OUTDIR="D:\AVS\AVI"


FOR %%A IN (*.avi) DO (
REM %FFMPEG% -i "%INDIR%\%%A" huffyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
%FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo "%OUTDIR%\%%~nxA.AVI"

)

Pause

EDITED: REM'ed out statement converts to HUFFYUV and PCM audio (if present)

You can get ffmpeg from eg MeGUI or google it.

EDIT: I've posted your sample TSCC clip in AviSynth 2.6.0 Alpha5 thread as a bug report sample. here:http://forum.doom9.org/showthread.php?p=1675858#post1675858

[eduardobedoya]

Thanks StainlessS
I upgrade the RT_Stats and the avs script.

I really get used to record using camtasia tsc2 codec, do you think it may be possible to solve that avisynth issue with tsc2 in the near future?

I have started to look for another lossless 24bit color screen capture codec (in order to avoid having to convert all tsc2 videos before using avisynth), do you have a screen capture codec to suggest?

About the convertion, I was thinking about a batch converter
what do you think about this app? it has CUDA acceleration.
http://www.mediacoderhq.com/
Is there another way to convert all 20 videos at once?
I am really concern about lossing quality in the convertion, do you think ffmpeg is my best choice, to which codec should I convert?

Thanks Advanced.

[StainlessS]

TSCC is not an editable format, meant for capture or archival purposes, you want to edit it, and so need in editable format.
Whether AviSource is fixed or not, it will be very slow (as it aint an editable format) and as my script does a lots of seeking
(as it has to, to do the job), even if Avisource is fixed, it may still suffer from seek problems (that may actually be fault of the
codec). Techsmith is probably as good as any other screen cap codec, but they will all suffer from same problem, ie not an
editable format. (EDIT: at least lossless ones will eg TechSmith, MJpeg, Motion Jpeg used in eg Camera or TV caps is not
lossless, and to some extent is editable, but as you want a lossless real time screen capture codec, they will likely not be editable).

The Batch .bat script provided will convert all AVI files in the same dir as batch script, at once. I edited the bat file with the REM'ed out
statement to do the same using HuffYUV codec instead (and with PCM audio if present).

You could even use ffmpeg to batch convert all you TSCC clips at once, directly to x264-mp4, I dont off hand know the command
for that but I'm sure someone else could offer that info. However, it does not solve your problem of wanting to remove the EDIT/UNDO's
from the clip.
I'm not really sure what is inside the TSCC clips, think that it's RGB but MediaInfo gives little info, and one app I tried mentioned YV24
although did not seem to be sure.
The UNDO removal avs script converts to REC601 YV12, this is so that whatever is input (YV12, RGB or YUY2 or whatever), uses the same settings
and dont need to keep changing with different clips, there is also a config option to extract edited clip with a ConvertToYV12 (for conversion to x264/mp4) if required.
Suggest convert to compressed RGB (will be without conversion, if it is indeed RGB to begin with), use edit/undo script
(which will only convert to YV12 during detection and extraction script creation) and then use resultant scripts as input to whatever you choose to render
eg x264/mp4.

EDIT:
I used to use MediaCoder (not the hq version) some time ago, but got annoyed with it for some reason and no longer use it,
I'm sure it will be just fine if you abandon the idea of editing out the UNDO's (or decide to do it by hand in MediaCoder, likely a lot more
work than already suggested options).

[StainlessS]

Seems that content of TSCC2 is YV24 (rec601 I think, 16 -> 235)

Tried conversion of TSCC to YV24 uncompressed, managed to do it but U and V were crossed over,
seems to be some confusion about which way around they should be.
Suggest convert to RGB, almost no difference in colorspace conversion and a heluva lot less bother than YV24,
results of a little experimentation below.





setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR="."


REM Where to place output file, No terminating Backslash.
set OUTDIR="D:\AVS\AVI"


REM Converts all below input formats in INDIR to avi files in OUTDIR directory (could add to below formats).
FOR %%A IN (*.wmv *.asf *.mpg *.mpeg *.avi *.flv *.mov *.mp4 *.m4v *.RAM *.RM *.mkv *.TS) DO (



REM Uncomment ie REMOVE 'REM' from beginning of one of below lines (And REM out the currently Un-commented line).



REM Convert Video to forced RGB24 (ULRG) UT_Video Codec, audio PCM 16 Bit (if Present)
%FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -pix_fmt rgb24 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to UT_Video Codec (YV24 and YV12 to ULY0(YV12), YUY2 to ULY2, RGB to ULRG), audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to forced RGB24 HuffYUV Codec, audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec huffyuv -pix_fmt rgb24 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to HuffYUV Codec (if YUV then YUY2, Else RGB), audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec huffyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"



)


Pause


By the way, a UT_Video compressed YV12 version of your 27MB TechSmith clip is 2.57GB in size, and in RGB is 4.53GB,
but they are editable.

EDIT: Seems Virtualdub has a screen capture mode, might be worth a look, see the help, Video Capture.

[StainlessS]

Not sure if this will work for you, as I have read somewhere that Windows 8 64 bit, broke Video For Windows interface
(used by VDub).

If you find that whatever program you want to use to encode (eg Premiere, MeGUI), cannot accept avs files,
then you could batch process them all to edited AVI files.

See Here: http://forum.doom9.org/showthread.php?p=1628221&highlight=script#post1628221

Forgot all about doing that.

After tscc2 conversion to AVI using eg ffmpeg .bat,

you could then process resultant clips using my avs batch script for the EDIT/UNDO removal

then, copy the resultant avs files to another empty directory (just the avs files),
so that they are NOT in same place as original AVI files (reason hilited at end of last
post in link above).

then process the copied AVS using another batch script (as in above link) with the
directories edited to suit.
this would enable you to do an entire job lot in 3 batch processes.

Batch file from above link repeated here
VdBatch.bat

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


You need to set up settings eg codec, save settings then execute the batch file, and it will
do the lot, pulling in the avs files (results of EDIT/UNDO script), and then create AVI's in the output dir.

[eduardobedoya]

I test some codecs to convert the tsc2 avis, the
UTVideo in ULRG mode is the only one that has good color but it is 5.2gb, Lagarith codec is also RGB and it is 4.2 gb

I regret myself had to install Mediacoder, I dont know why but after installed it avisynth dont work anymore
when dropping the avs into virtualdub it show a message "AVI Import Filter error: (Unknown) 80040154"
I google it and in this thread
http://forum.videohelp.com/threads/302249-VirtualDub-AVI-Import-Filter-error-(Unknown)-80040154
people suggest run ccleaner (I would not like to do that) I upgraded the K-lite codec, but the message ramins, what can I do?

EDIT: I unistalled avisynth n virtualdub (delete its folder) then installed ccleaner run it, then reinstall avisynth (and its plugins) n virtualdub, but I still get that message when dropping the avs into virtual dub "AVI Import Filter error: (Unknown) 80040154"


Thanks advanced.

[eduardobedoya]

Since I cant get virtualdub working with the batch avs that you provided, I tried dropping the batch avs into Media Player Classic, it worked, here are the results:


Fsel_Eduardobedoya_Batch.Log
CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000


1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi
QWIK Scan DBase creation = 227.69 Secs (3.79 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 562.68 Secs (9.38 Mins) InFPS=4.09 OutFPS=2.62
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi Tot File Time = 790.98 Seconds (13.18 Mins)


DONE



TOTAL Time = 790.98 Seconds (13.18 Mins)







painting test for avisynth Lagarith_SelectFrames.AVS

Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last




What can I do from here since I can not use virtualdub??? is it normal that Media Player Classic take 13min to process a 2.5 min file? or virtualdub could do it faster?

Thanks advanced

PD:
There is a line in your Batch.avs
####
CONVYV12 = True # True ConvertTo YV12 in created script, Else False leaves as is
####
I changed it to "False" I would like to keep the color in RGB.

Thanks


Edit: I tried changing the virtualdub version (I was using the 64amd version), and using the 32 bit version it worked ok, (its strange that I was able to work with the 64bit amd version before and now I cant) anyway, I processed the avs again...

Fsel_Eduardobedoya_Batch.Log

CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000

1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi
QWIK Scan DBase creation = 182.26 Secs (3.04 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 447.19 Secs (7.45 Mins) InFPS=5.15 OutFPS=3.30
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi Tot File Time = 629.93 Seconds (10.50 Mins)


DONE



TOTAL Time = 629.93 Seconds (10.50 Mins)





painting test for avisynth Lagarith_SelectFrames.AVS

Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last



then what...? I read what you post above, but it looks kinda strange to me. I just cant understand if I got to put the "=" sings, Sorry if I am not kinda smart enough understanding codes, I tried this:

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory ####what is this??????
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI ####I guess this line is replaced by the next line, right???
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=C:\Program Files (x86)\VirtualDub-1.10.4\VDub.exe"


REM Where to Find VDUB Settings #####I have no clue where are the *.vcf file placed
set VDS=C:\Program Files (x86)\VirtualDub-1.10.4\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=F:\2 AVISYNTH\INPUT


REM Where to place output file, No terminating Backslash.
set OUTDIR=F:\2 AVISYNTH\OUTPUT

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit ##is it ok?
%VD% /s %VDS% /c /b F:\2 AVISYNTH\INPUT F:\2 AVISYNTH\OUTPUT /r /x

Pause



I save this file as a VdBatch.bat and place it inside F:\2 AVISYNTH\INPUT with the "batch avs" file and execute it via cmd?


PD: Pls StainlessS, do you have any clue why I cant use VirtualDub amd64 anymore? there is a way that I could get it working again? because if the 32bit version render in 10min perhaps the 64bit version could render in 6minutes??? This is very important since I have about 20 hours of video to process. I would like install avisynth+ with VDub amd64bit:
http://forum.doom9.org/showthread.php?t=168856
what do you suggest??

PD2: It is ok that I had converted the video using Lagarith? perhaps there is another RGB lossless codec that could be processed much faster with avisynth??? at this time I would prefer the performance factor than file size.

Thanks advanced.

[StainlessS]

Glad you got it working, as I said, W8 64bit is supposed to be broken for Video For Windows
so I'm surprised it worked before MediaCoder install.

What can I do from here since I can not use virtualdub??? is it normal that Media Player Classic
take 13min to process a 2.5 min file? or virtualdub could do it faster?

It is Avisynth that does the processing, as soon as Media Player OR Vdub displays first frame, then it is already complete.

From this
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480

Perhaps below would work if differences do not vastly change between your TSSC clips.

ldM=0 hdM=0 ldL=0.005000 hdL=0.005000 ldS=0.004000 hdS=0.004000 ldR=0.003000 hdR=0.003000 MAX_LDIFF=0.004000

If misses some UNDO's in a clip, then redo with original values.

10:50 secs processing for your 3min clip is not so bad considering that you scan up to 2304 HD frames for each output frame,
but above mods to thresholds should increase speed considerably (but you need to check that the UNDO's were correctly detected/removed).
EDIT: The 1st script version output 1 frame about every 12 seconds, so with 1474 output frames it would perhaps have
taken about 5 hours.

As you have chosen to use RGB, I may have a go at using an RGB fingerprint, should I think be a little faster,
but for movie clips, probably a lot faster (there are so many possible matches in your clips as many frames are
identical/almost identical).

do you have any clue why I cant use VirtualDub amd64 anymore?

No idea, I use XP32bit with no intention of changing to W7, W8 (most likely change, Linux).

REM VirtualDub Directory, No terminating Backslash, "." = current directory ####what is this??????
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"

Just means that set VDDIR="." # would set VDDIR to the current directory, dont worry about it.
'.' just means current directory, '..' means parent of current directory, in command line processor, eg
dir .
dir ..

shows you files in current directory, and the directory above current directory respectively.

The only paths you might want to change hilited in blue, those in Magenta are currently set RELATIVE to the current directory,
ie relative to the directory that the bat file is in.


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


NOTE, UT_Video is I think faster than Lagarith, especially for decode and as you will be doing LOTS
of seeking and decoding during frame scanning, it is problably the better choice, check it out for yourself
how long everything takes.

[eduardobedoya]

Hi StainlessS I tried to get what you posted, this is what I did:

I opened virtualdub, then.. (file>save processing settings)
and save the file to same directory as CURRENT version of VD (C:\Program Files (x86)\VirtualDub-1.10.4) as VD.vcf. (the result file was VD.vcf.vdscript)

Then I placed this VdBatch.bat inside F:\2 AVISYNTH


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR=C:\Program Files (x86)\VirtualDub-1.10.4

REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"

REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=F:\2 AVISYNTH\INPUT


REM Where to place output file, No terminating Backslash.
set OUTDIR=F:\2 AVISYNTH\OUTPUT

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause




finally a run the VdBatch.bat by double click it, it sent what it looked to be the command promp with the script text and at the end a "puase" and a text saying "press any key to continue"
I press space and the black screen closed, then nothing happens in about half hour.

PD: I tried replacing the VD.vcf.vdscript file with a VD.vcf file, and the same result

What I am doing wrong??? please could you post just exactly how you did it in your particular case, in your PC??? Thanks Advanced

PD: It is important to configure something inside Virtualdub before running the bat file??? perhaps there are some important thread that I should read to learn how to edit settings inside Virtualdub. Thanks advanced.

[StainlessS]

You lost the double quotes in a number of places.


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR="F:\2 AVISYNTH\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR="F:\2 AVISYNTH\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause



try this

EDIT: Settings should be saved as VD.vcf not VD.vcf.vdscript, dont know if VirtualDub added the 'vdscript' bit. Either save as
VD.vcf or edit bat file

set VDS=%VDDIR%"\VD.vcf.vdscript"


Also, if bat file is 1 directory above both INPUT and OUTPUT folders, then could have left original bat script INPUT and OUTPUT
lines alone eg


set INDIR=".\INPUT"



set OUTDIR=".\OUTPUT"


OK, I see that I have to keep adding EDITs to answer your edits :)

You need to set up eg CODEC at least, on video Menu, they (settings) will be saved in the settings file, VD.vcf

[StainlessS]

OK, no more edits.

Load a clip into Vdub, set codec, save VD.vcf into you VirtualDub folder.
run this bat

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


Only the path to virtualdub folder has changed since original posted bat file.
The bat file needs to be in your "F:\2 AVISYNTH" folder.

EDIT: You can move/rename the "2 AVISYNTH" folder anyhere you like and it will still work, so long as it still contains the bat file and both
INPUT and OUTPUT directories.

[eduardobedoya]

The same result I did everything as you said, but I still got the same result, when I double clik VdBatch.bat I get the exact same result, it sent what it looked to be the command promp with the script text and at the end a "puase" and a text saying "press any key to continue"
I press space and then the black screen closed, then nothing happens in about half hour.
I have tried to launch the bat file via Win Run>cmd, and the same result.

I modified the Bat file as you said,

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause

What does REM mean??

I also opened a video inside VirtualDub and clicked in Video>Compression, and choose Lagarith, then I saved the processing settings as VD.vcf (the previous time it was Virtualdub who was triying to save the processing settings as *.vdscript file) inside the VirtualDub directory.

And I placed the VdBatch.bat file inside F:\2 AVISYNTH

But the same result, what I am doing wrong??? THanks advanced.

PD: I am placing the resulted avs files of the second UNDObatch process inside the "F:\2 AVISYNTH\INPUT" folder, do I have to place the frames.txt files also???

By the way, I did convert the tsc2 avis into a Lagarith avi to make this try, all that I have posted have been using Lagarith avi. You said that if I try UTvideo ULRG it could be faster than Lagarith, so I convert the tsc2 file into UTvideo ULRG and try to process it with avisynth by running the Eduardobedoya_Batch inside virtual dub, but when the open windows appears and I pick up the UTvideo ULRG it shows this message:

VirtualDub Error
Avisynth open failure:
AVISource: couldn't locate a decompressor for fourcc ULRG
(C:\Users\Eduardo\Desktop\Avisynth\Eduardobedoya_Batch_3\Fsel_Eduardobedoya_Batch.avs, line 103)
([GScript], line 16)
([GScript], line 22)
(C:\Users\Eduardo\Desktop\Avisynth\Eduardobedoya_Batch_3\Fsel_Eduardobedoya_Batch.avs, line 86)

Pls help.

[StainlessS]

REM is short for REMark, ie comment follows it.

ffmpeg converts it to UT_Video, however you need a UT_Video codec to decode for Video For Windows.
http://www.videohelp.com/tools/Ut-Video-Codec-Suite

What you have done according to previous post looks correct to me, should not need to move frames files to the INPUT directory.
Try load any avs file from INPUT directory into VDUB directly, if there is some problem it should present an Avisynth error message in VDub.
EDIT: The missing UT_Video codec is problably the vdub batch problem if that was what the TSCC files were converted to.
EDIT: It is the ffmpeg conversion that best output UT_Video for the UNDO removal, can use whatever codec you like for the
VD batch output.

.

[eduardobedoya]

I tried to load the avs file from INPUT directory into VDUB directly, it shows this message

Avisynth open failure:
Script error: there is no function named "FrameSel"
(F:\2 AVISYNTH\INPUT\painting test for avisynth Lagarith_SelectFrames.avs, line4)

I tried to load it from the first folder where the videos where placed, it shows the same message

Avisynth open failure:
Script error: there is no function named "FrameSel"
(C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_SelectFrames.AVS, line4)




Note that I am just triying everything with a single video, (it has been converted to Lagarith),I recently tried to test with more videos, using other codecs, I installed UTvideo and convert the tsc2 into UTvideoULRG then process it with the UNDO batch, it do the task in 3min aprox (good improve) but in the last batch when I try to run the VdBatch with the new UTvideo avs file in the INPUT folder, I got the same result.

Why this final step does not work???

I converted that tsc2 video to Lagarith and UTVideoULRG using camtasia studio.
PD: I just have ffdshow Video Codec installed, is that ffmpeg???

[StainlessS]

You need FrameSel plugin for avisynth, I think I included it in 1st zip I posted, not included since then.
Get here and put in your plugins dir. http://forum.doom9.org/showthread.php?t=167971

If you are converting with camstudio, you dont need ffmpeg.

[eduardobedoya]

I do have FrameSel v2.6 dll in my avisynth plugin folder

I have avisynth 26 installed as you suggested.

[StainlessS]

Then you have some kind of Avisynth problem, re-install.
Copy only required plugins to plugins dir, and establish that avisynth is working, no
point in trying to get the rest working without working avisynth.
You only put 1 of each of the plugs in plugins dir, ie only v.26 version not v2.5.

Before install try this


Version()


It should show Avisynth version. v2.5 cannot load v2.6 plugins.
EDIT: Only thing I can think of is that you actually have Avisynth v2.58 installed.

[eduardobedoya]

I tried the version () code (creating a avs with that code and open it in virtualdub), here is the result


AviSynth 2.60, build:Sep 18 2013 [17:36:36]
C 2000-2013 Ben Rudiak-Gould, et al.
http://www.avisynth.org


What I have noticed is that even when I installing Avisynth 2.6 it gets installed inside a folder name Avisynth 2.5, I guess it does not matter

The files inside my current avisynth installation directory (C:\Program Files (x86)\AviSynth 2.5\plugins) are:

by default:
colors_rgb.avsi
DirectShowSource.dll
TCPDeliver.dll

Installed by me
FrameSelect26.dll
Gscript.dll
RT_Stats26.dll


I desinstall and reinstall Avisynth 2.6
I get the defult plugins:
colors_rgb.avsi
DirectShowSource.dll
TCPDeliver.dll

and I added only the FrameSelect26.dll (84kb)


Then I lunched the avs file but get the same error message
Avisynth open failure:
Script error: there is no function named "FrameSel"
(C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_SelectFrames.AVS, line4)

I dont know what is wrong with my system, should I try to install AviSynthPlus-r1576.exe?? or AviSynth 2.5.8????
http://avisynth.nl/index.php/Main_Page

Thanks Advaced StainlessS

[StainlessS]

Oops, sorry.
Seems I may have supplied you with FrameSelect plugin rather than FrameSel plugin.
FrameSelect is older version, needed to change args in non compatible way, and so I renamed the plugin
to avoid compatibility problems. Delete the older FrameSelect, you need FrameSel plugin, v2.6.
See MediaFire below in sig.

[eduardobedoya]

Thanks StainlessS
I was already thinking about installing a virtual machine


Thanks man, it worked, I did a TEST>>>


First Here is the UNDO Batch Log:

Fsel_Eduardobedoya_Batch.Log
CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000


1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi
QWIK Scan DBase creation = 49.06 Secs (0.82 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 127.67 Secs (2.13 Mins) InFPS=18.05 OutFPS=11.55
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi Tot File Time = 177.32 Seconds (2.96 Mins)

DONE


TOTAL Time = 177.33 Seconds (2.96 Mins)


Here is the resulted AVS file:


Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last



Then here is the message that appeared in the command promp when launching VdBatch.bat

F:\2 AVISYNTH>REM Where to get input file, No terminating Backslash, "." = current directory

F:\2 AVISYNTH>set INDIR=".\INPUT"

F:\2 AVISYNTH>REM Where to get input files, No terminating Backslash.

F:\2 AVISYNTH>set INDIR=".\OUTPUT"

F:\2 AVISYNTH>REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit

F:\2 AVISYNTH>"C:\Program FIles <x86>\VirtualDub-1.10.4""/VDub.exe" \s "C:Program Files <x86>\VirtualDub-1.10.4""\VD.vcf" /c /b ".\INPUT" ".\OUTPUT" /r /x VirtualDub CLI Video Processor Version 1.10.4 <build 35491/release> for 80x86 Copyright <C> Avery Lee 1998-2009. Licensed under GNU General Public License

AVI: Opening file "F:\2 AVISYNTH\INPUT\painting test for avisynth UTVideoULRG_SelectFrames.AVS"
AVI: Avisynth detected. Extended error handling enabled
Beginning dub operation
Dub: Input <decompression> format is : XRGB8888.
Dub: Output <compression> format is: RGB888.

[eduardobedoya]

Thanks a lot StainlessS, it worked!

I used UTVideoULRG because it keep the color very similar to the original in RGB
Also I changed this line, of the Undo BATCH.AVS that you provided:
####
CONVYV12 = False # True ConvertTo YV12 in created script, Else False leaves as is
####in orther to keep the video in RGB

But when I run the VdBatch.bat the color of the OUTPUT video drastically changed
I guessed it depends on the video>compression properties that you choose on Virtualdub (VD.vcf)
So I changed Virtualdub compression properties to tscc codec, and then it rendered the video in a good RGB color,

Thanks a lot StainlessS, I just have some doubts:

Is there a big difference between converting the original tsc2 video to UTVideoULRG or UTVideoULY0 codecs for running the UNDOBatch.avs??
I mean, cuz, the sample tsc2 26mb video converted into UTVideoULRG is 5.2gb and into UTVideoULY0 is 2.7gb,
So is there any problem if I convert the original tsc2 into UTVideoULY0 video file in order to run the UNDObatch in YUV and then re-convert it with the VDBatch.bat into RGB again? Do I loss some color information in the final OUTPUT RGB video by doing this way??? Will the batch process be run in YUV colors so it could loss some accuracy in color detection (similar color painting strokes)??

I run the test and it really cut all the UNDOS, but, is there any way to merge all equal frames to a single frame?
I mean, cuz, the result video got in fact all the painting strokes without UNDOS, but there is so much space(time)(equal frames) between each different stroke, is there a way to merge all that spaces,
so the time between each different painting stroke could be just one frame?
For example:
the original video was
AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ
the applying your avs it turns into
AABCDDDDEFGHHIKJLMMMNOPQ
is there a way to turn it into (perhpas using the same batch.avs)
ABCDEFGHIJKLMNOPQ

Thanks a lot StainlessS!!!

[StainlessS]

But when I run the VdBatch.bat the color of the OUTPUT video drastically changed
I guessed it depends on the video>compression properties that you choose on Virtualdub (VD.vcf)
So I changed Virtualdub compression properties to tscc codec, and then it rendered the video in a good RGB color,

I guess that has something to do with color matrix used/assumed, see Vdub menu "Video/ColorDepth", perhaps playing around with
'decompression format' and 'output format to compressor/display' is required, I would be reluctant to go back to TSCC codec, we already
established it to be unreliable for eg seeking, suggest find alternative. I dont usually touch HD, I dont have BD player or other source
of it, perhaps others could advise on best settings for VD 'decompression format' and 'output format to compressor/display', to
pass through RGB untouched to UT_Video ULRG. I dont really see why pulling in RGB and compressing to UT_Video RGB should give
a problem. What are you using to judge the 'drastically changed output', did you look at the input to UNDO script too ?

UTVideo ULY0 is rec601 YV12, ie half resolution chroma, that is why is is smaller compressed file, half chroma detail gone.

but, is there any way to merge all equal frames to a single frame?

I initially did that but assumed that you really would not want it, ie results will be a clip of maybe 3 or 4 seconds out of your 3 min clip.
OK, I'll put it back like that.

EDIT: From your last post

Dub: Input <decompression> format is : XRGB8888.
Dub: Output <compression> format is: RGB888.

Looks like straight through RGB in -> out (RGB32 to RGB24), why drastically changed, no idea.

[eduardobedoya]

UTVideo ULY0 is rec601 YV12, ie half resolution chroma, that is why is is smaller compressed file, half chroma detail gone.

So it means that runing the UNDOBatch.avs with a UTVideoULRG video and not converting to YV12 could be more accurate in detected diferent color strokes?? (subtle diferent color strokes)
compared to ULY0????

[StainlessS]

Yep..

[StainlessS]

OK, I've added framecount limiting for static sequences, with KEEPMAX==1 (default) will produce output clip with
148 frames (on your sample, I think) with no pauses between edits. setting to eg 15 will give max 1 second pause between edits
when 15FPS clip.

Added a ranges.txt file, created during resultant avs load, informational only.
Removed ConvertToYV12 functionality.
Reduced number of tolerance settings (+ve and -ve tolerances set same).

here: LINK REMOVED

script here:

# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# Remove Undos from painting captures
# ############################



# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

#### BELOW SETTINGS WORK FINE WITH SUPPLIED SAMPLE WITH A LITTLE LEEWAY.
# Below settings, as close to zero as possible (faster but might miss matches), used by QWIK scan routines to identify possible match frames.
dM = 1 # INT, Max diff (+ve and -ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dL = 0.005 # Float, Max diff (+ve and -ve tolerance) of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dS = 0.005 # Float, Max diff (+ve and -ve tolerance) of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dR = 0.005 # Float, Max diff (+ve and -ve tolerance) of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good match.
MAX_LDIFF = 0.004 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####
# Maximum number of frames to keep per static sequence (0 = no limit)
KEEPMAX=1
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_WriteFileF(LOG,"%s\n",LOG,Append=False)
S=RT_String("dM=%d dL=%f dS=%f dR=%f MAX_LDIFF=%f",dM,dL,dS,dR,MAX_LDiff)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_String("KeepMax=%d",KeepMax)
RT_DebugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s\n",S,Append=True)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)

DropUndoSequences(FN,ScanAheadSecs,dm=dM, dL=dL, ds=dS, dr=dR,LDThresh=MAX_LDIFF, log=LOG, keepmax=KEEPMAX)

T = RT_Timer - START
S=RT_String("%s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropUndoSequences(String "AviName",Float "ScanAheadSecs",int "dM",Float "dL",Float "dS",Float "dR",Float "LDThresh",String "Log",int "KeepMax") {
myName="DropUndoSequences: "
ScanAheadSecs=Float(Default(ScanAheadSecs,10.0*60.0))
dM = Default(dM,1)
dL = Float(Default(dL,0.005))
dS = Float(Default(dS,0.005))
dR = Float(Default(dR,0.005))
LDThresh = Float(Default(LDThresh,0.004))
LOG = Default(LOG,"DropUndoSequences.LOG")
KeepMax=Default(KeepMax,0) # 0 = no limit
Assert(Exist(AviName),myName+AviName+" Does Not Exist")

Avisource(AviName).ConvertToYV12() # So we use the same settings whether orig RGB or YV12

Assert(ScanAheadSecs>0.0,myName+"ScanAheadSecs Must be greater than zero")
Assert(dM>=0, myName+"ldM Must be greater than or equal to zero")
Assert(dL>=0.0,myName+"ldL Must be greater or equal to zero")
Assert(dS>=0.0,myName+"ldS Must be greater or equal to zero")
Assert(dR>=0.0,myName+"ldR Must be greater or equal to zero")
Assert(KeepMax>=0,myName+"KeepMax Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
ScriptFile=PathAndNode+"_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
PNDB=PathAndNode + "_Prev.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
RT_FileDelete(Ranges) # Delete any existing Ranges file
### TEMPLATE script

FrameSel_Select="""
fn="%s"
Avisource(fn)
PathAndNode="%s"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last
"""

###
Select_S = FrameSel_Select
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,PathAndNode) # Insert filenames and ConvertToYV12

START = RT_Timer
RT_QwikLumaScanCreateDB(DB,prevdb=PNDB,nextdb="",debug=true)
T= RT_Timer - START
S = RT_String("QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_WriteFileF(LOG,"%s",S,Append=True)
OCNT=0
GSCript("""
START = RT_Timer
LastFrame=FrameCount-1
FivePercFrames = Round(0.05 * FrameCount)
RT_DebugF("QWIK Scanning file ... Please Wait",name=myName)
MDeltaMax = 0 LDeltaMax = 0.0 SDeltaMax = 0.0 RDeltaMax = 0.0
Kept = 0

For(i=0,LastFrame) {
WrStart = i WrEnd = i # Init to keep 1 frame (if not found)
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit to i+1 inclusive (downwards)
# Scan for static sequence
FindStart = i + 1
for(k=FindStart,EndLimit) {
dif=RT_LumaDifference(Last,Last,n=i,n2=k)
if(dif <= LDThresh) {
WrEnd = k
} Else {
FindStart = k
k = EndLimit # break
}
}
# Here:- WStart to WrEnd are similar, search from EndLimit downward to FindStart for later dupe sequence
For(j=EndLimit,FindStart,-1) {
# Search for i frame dupe nearest to EndLimit but higher than FindStart
j = RT_QwikLumaScanGetNear(DB,PNDB,j, \
ldm = dM, ldl = dL, lds = dS, ldr = dR, \
maxdistance=j-FindStart, Inclusive=True, \
findframe=i)
if(j > FindStart) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= LDThresh) {
# We found a single duplicate frame and will drop earlier sequence, look for start of similar frames in this sequence
DupeEnd = j
DupeStart = j
for(k=j-1,FindStart+1,-1) {
tdif=RT_LumaDifference(Last,Last,n=i,n2=k)
if(tdif <= LDThresh) {
DupeStart = k
} Else {
k = FindStart # break
}
}
RT_DebugF("%d,%d --> %d,%d Matched LumaDif=%f",WrStart,WrEnd,DupeStart,DupeEnd,dif,name=myName)
S=RT_String("%d,%d Skipping earlier duplicate sequence and UNDOS",WrStart,DupeStart-1)
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
RT_Ystats(Last,i,flgs=$78,lo=64,hi=192,Prefix="I_") # Default DBase create, YInRange lo and hi
RT_Ystats(Last,j,flgs=$78,lo=64,hi=192,Prefix="J_")
I_yInRng = I_yInRng * 255.0 J_yInRng = J_yInRng * 255.0 # Make range same as other YStats (YInrange is 0.0->1.0)
MDelta=Abs(I_yMed-J_yMed) LDelta=Abs(I_yAve-J_yAve)
SDelta=Abs(I_yStdev-J_yStdev) RDelta=Abs(I_yInRng-J_yInRng)
MDeltaMax=Max(MDeltaMax,MDelta) LDeltaMax=Max(LDeltaMax,LDelta)
SDeltaMax=Max(SDeltaMax,SDelta) RDeltaMax=Max(RDeltaMax,RDelta)
RT_DebugF("MDelta=%d LDelta=%f SDelta=%f RDelta=%f",MDelta,LDelta,SDelta,RDelta,name=myName)
WrStart = DupeStart
WrEnd = DupeEnd
j = 0 # Early break
}
}
}
Keep = (WrEnd-WrStart+1)
if(KeepMax>0 && Keep>KeepMax) {
# Limit frames to keep
RT_DebugF("KeepMax Limiting: %d frames limited to %d (%d,%d -> %d,%d)",Keep,KeepMax,WrStart,WrEnd,WrEnd-KeepMax+1,WrEnd,name=myName)
Keep = KeepMax
WrStart=WrEnd-Keep+1
}
Kept = Kept + Keep
if(Keep == 1) {
RT_WriteFileF(CMDFrames,"%d",WrStart,Append=True)
} else {
RT_WriteFileF(CMDFrames,"%d,%d",WrStart,WrEnd,Append=True)
}
i = WrEnd
If(i > (OCNT+FivePercFrames)) {
T= RT_Timer - START
RT_DebugF("%d ] Progress = %.2f%% InFrames=%d OutFrames=%d InFPS=%.2fFPS OutFPS=%.2f",
\ i,(i+1)*100.0/(LastFrame+1),i+1,Kept,(i+1)/T,Kept/T,name=myName)
OCNT=i
}
}
RT_WriteFileF(ScriptFile,"%s",Select_S,Append=False)
RT_FileDelete(DB)
RT_FileDelete(PNDB)
S=RT_String("MDeltaMax=%d LDeltaMax=%f SDeltaMax=%f RDeltaMax=%f",MDeltaMax,LDeltaMax,SDeltaMax,RDeltaMax)
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
S="(Above, maximum values of dM, dL, dS, dR, that would have worked, but may have been faster)"
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String("Kept %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]", Kept, FrameCount,Width,Height,FT,FT/60.0,FrameRate)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_String("QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", T,T/60.0,FrameCount/T,Kept/T)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
""")
Return 0
}