Deleting every frame between equal frames in video

[eduardobedoya]

Hi great community, Im looking for your help to create an avisynth script that is similar to the popular avisynth "duplicated frame function script".

I have lots of videos (1920 x 1080, tsc2 compression, avi format, about 45min long each one) that I need to trim automatically.

In these videos the movie goes forward normally but then returns exactly to a previous frame (like 3 segs of rewind),
then the movie resumes and goes forward again, but then it returns to the same previous frame again, and then resumes again, until finally the movie pass this segment and goes on, but in other segment of the movie the same thing happens until it pass that segment, and so on. In the hole movie this "rewind to especific frame, forward, rewind, forward" happens almost 30 times.

For example if each letter is a different frame, the movie goes like this:

AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ

Note that once in a while the movie goes back to exactly the same frame.
So, the lowercase letters are frames that got rewound, frames that I would like to discard,
and the capital letters are frames that I want to keep
So in the example above I would like to transform the video into this:

ABCDEFGHIJKLMNOPQ

I guess the average way to achieve this, would be detecting all 100% equal frames in the video and deleting everyframe between two equal frames, perhaps also use a threshold about 10 minutes, to delete only the frames that are between two equal frames that are not separated longer than 10 minutes.
Please note that these are HD videos and the difference between each frame in the video is very subtle.
Also if you recomend me some links to learn to use avisynth with several videos (kinda batch mode). I use premiere.

Thank you very much for your support doom9, regards.

[Guest]

Welcome to the forum. Sorry but I have to ask, where did you get these files?

[StainlessS]

This serves as a reasonable example of how you can use RT_Stats Dbase funcs,
Used your example string to produce a test clip.


##############################
# Requires GScript(), RT_Stats(), and FrameSel() Plugins
##############################

# Choose one of below lines (comment out the other)
#TestSource(Upper=true) # Test Source clip, We do tests on Upper case frames, but later select from upper/lower case mix frames
AviSource("...whatever...")

######
ScanAheadSeconds= 10 * 60
TEST_THRESH = 1.0
DIF_THRESH = 0.1
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB="MyDbase.DB"
FN="Frames.txt"
DEBUG=True # Info to DebugView window (Google)
DELETEDBASE=True # Delete DBase file
##############################
GSCript("""
RT_FileDelete(FN) # Delete any existing frames file
RT_DBaseAlloc(DB,FrameCount,"f") # Allocate a single field database of type Float
LastFrame=FrameCount-1
(DEBUG) ? RT_DebugF("Filling DBase with AverageLuma data") : NOP
For(i=0,LastFrame) { # Set AveLuma for each and every frame
RT_DBaseSet(DB,i,RT_AverageLuma(Last,i))
}
(DEBUG) ? RT_DebugF("DBase Filled, Scanning frames.") : NOP
For(i=0,LastFrame) {
LumaI = RT_DBaseGetField(DB,i,0)
EndLimit= Min(i + ScanAheadFrames,LastFrame)
OutFrame = i # Init frame to output
For(j=EndLimit,i + 1,-1) {
LumaJ = RT_DBaseGetField(DB,j,0)
if(Abs(LumaI - LumaJ) <= TEST_THRESH) { # about same ave luma, test it
if(RT_LumaDifference(Last,Last,n=i,n2=j) <= DIF_THRESH) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True)
if(i != OutFrame) {
(DEBUG) ? RT_DebugF("%d ] (Skipped=%d)",OutFrame,OutFrame-i) : NOP # Shows in DebugView (Google)
i = OutFrame # Skip all frames before Outframe
} Else {
(DEBUG) ? RT_DebugF("%d ]",OutFrame) : NOP
}
}
(DELETEDBASE) ? RT_FileDelete(DB) : NOP
""")

# If using TestSource(), comment out below line
Return FrameSel(Last,cmd=FN) # This will return frames using FN frames file

# Here Just testing using synthesized test clip, we do NOT upper case strings this time, to see if it worked OK
TestSource(Upper=False)
Return FrameSel(Last,cmd=FN)
#####################

# Only of use in testing, (If you dont have a real problem clip)
Function TestSource(Bool "Upper") {
Upper=Default(Upper,False)
Test = "AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ"
Test = (Upper) ? Ucase(TesT) : Test
CC = ColorBars().Trim(0,-1).KillAudio # Single frame
C=CC.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
s = s + s + s + s + s + s + s + s + s + s
C = C + CC.Subtitle(S,size=24,align=5)
}
""")
return C
}


Script to load only good frames, once main script has created FN ("Frame.txt")

FN="Frames.txt"
AviSource("...whatever...")
Return FrameSel(Last,cmd=FN)


EDIT: It will NOT be super fast !!! (main script)
EDIT: Any colorspace OK.
EDIT: Clip should be seekable, eg NOT DivX with single keyframe (way too slow).

EDIT: DebugView output (1st frame 12 is 12th letter in your example string ie 'A', 2nd is 13 ie 'B' etc)

00000010 5.15711737 [1824] RT_DebugF: Filling DBase with AverageLuma data
00000011 5.96010494 [1824] RT_DebugF: DBase Filled, Scanning frames.
00000012 6.72434425 [1824] RT_DebugF: 12 ] (Skipped=12)
00000013 6.91525364 [1824] RT_DebugF: 13 ]
00000014 7.10134315 [1824] RT_DebugF: 14 ]
00000015 7.22855568 [1824] RT_DebugF: 29 ] (Skipped=14)
00000016 7.34694576 [1824] RT_DebugF: 30 ]
00000017 7.46073008 [1824] RT_DebugF: 31 ]
00000018 7.57038546 [1824] RT_DebugF: 32 ]
00000019 7.65050507 [1824] RT_DebugF: 40 ] (Skipped=7)
00000020 7.72205544 [1824] RT_DebugF: 41 ]
00000021 7.78988838 [1824] RT_DebugF: 42 ]
00000022 7.85289049 [1824] RT_DebugF: 43 ]
00000023 7.91187668 [1824] RT_DebugF: 44 ]
00000024 7.93679857 [1824] RT_DebugF: 53 ] (Skipped=8)
00000025 7.95319843 [1824] RT_DebugF: 54 ]
00000026 7.96542358 [1824] RT_DebugF: 55 ]
00000027 7.97333288 [1824] RT_DebugF: 56 ]
00000028 7.97700548 [1824] RT_DebugF: 57 ]


EDIT:
Tested on a SD PAL 30 mins, 39 secs clip without any repeats, with 10 mins scan ahead and it takes about 12 seconds to output each frame,
would be faster if repeats were present and did not have to scan ahead 10 * 60 * 25 frames, for each and every output frame.
Above on Core Duo Dual Core @ 2.13GHz, Sata 2.

[eduardobedoya]

Thanks you StainlessS

yes 10 min is the average time it would have to look ahead, looking for repeats, the source is very clean, meaning that are HD digitally recorded videos, good quality, no blur, no death pixels.


Please StainlessS could you give me some links, where could I study, (figure out) how to apply avisynth script inside an windows 8 laptop?
wich software do I have to install? wich plugins?
Thanks you very much man, u have save me hours of editing, THANKS!

[eduardobedoya]

Welcome to the forum. Sorry but I have to ask, where did you get these files?

the source is very clean, meaning that are HD digitally recorded videos, good quality, no blur, no death pixels.


Please neuron2 could you give me some links, where could I study, (figure out) how to apply avisynth script inside an windows 8 laptop?
wich software do I have to install? wich plugins? in orther to run Stainless script??
Thank you advanced.

[StainlessS]

eduardobedoya,

Neuron2 wantz to know how you obtained the samples, ie " Where did you get these files ? ".

For a Laptop, suggest you get the FILTERSDK-HELP (compressed CHM help file version of docs installed with Avisynth v2.6,
suggest you get this even if using v2.58), available via my sig ie StainlessS@MediaFire below this post.
You can put it on Hot-Key for easy acces and it is easily searchable.

When on-line, Avisynth Wiki (has list of many plugins) here http://avisynth.nl/index.php/Main_Page

2 Lists of plugins at top of Usage forum 1st page in stickies.

Most of my plugins are not not in any of the lists (one or maybe two only EDIT: they may be in the New plugins sticky),
see Mediafire in sig for mine.

FrameSel thread here, http://forum.doom9.org/showthread.php?t=167971

RT_Stats here http://forum.doom9.org/showthread.php?t=165479&highlight=rt_stats

GScript here, http://forum.doom9.org/showthread.php?t=147846&highlight=gscript

There is a search option at top of all forum pages where you could have found the plugins yourself.

EDIT: And DebugView from SysInternals (Now MicroSoft) Here http://technet.microsoft.com/en-gb/sysinternals/bb896647.aspx

and Welcome to the Forum.

EDIT: I may have a go it increasing speed of given script, perhaps speed increase of 50X or 100X possible
(but maybe not possible, I'll try to give it a go though).

[StainlessS]

Post #1 of 3

QwikAveLumaScan.Avs

# QwikAveLumaScan.Avs

Function QwikAveLumaScanCreateDB(clip c, String DB, String "PrevDB", String "NextDB", Bool "Debug") {
# QwikAveLumaScanCreateDB: By StainlessS
# Requires GScript(), RT_Stats() plugins
#
# Creates a DB DataBase file of c.FrameCount records with 1 Float field per record, set to AverageLuma of each frame.
# Usage:- eg AveLuma = RT_DBaseGetField(DB,FrameNo,0)
#
# Optional creation of PrevDB and NextDB used by QwikAveLumaScanGetNear() function.
# PrevDB is a database of c.FrameCount records, and 256 Int fields per record, where content of fields point to the
# nearest frame BEFORE the record number (frame) which has the same Int(AverageLuma) value as the field index. If there is no
# frame prior to the record number (frame) that has an Int(AverageLuma) the same as the field index, then the field will hold Int -1.
# As the above is about as clear as mud, I shall try to explain with an example.
# If we want to know which frame is the nearest frame prior to frame 1000 which has an AverageLuma greater or equal to 128.0
# and less than 129.0 then PrevDB(1000,128) will hold that frame number, and the RT_Stats call to get it is
# prevfrm = RT_DBaseGetField(PrevDB,1000,128)
# NextDB is the same as PrevDB but the fields point to the nearest frame AFTER the record number (frame).
#
# c, Clip. Planar, YUY2, RGB24, RGB32.
# DB, String. Filename of created DB file.
# PrevDB, String. Filename of optional PrevDB file name.
# NextDB, String. Filename of optional NextDB file name.
# Debug, Bool, default false. True outputs info (eg progress) to DebugView window (Google).
#
# This function is really quite slow and took about 30 mins to create all three DataBases on a 10 Mins 13 secs PAL SD clip.
# Core Duo Duel Core 2.13Ghz and Sata 2 Drives. The AverageLuma of each frame is only gotten once, and the optional DataBases
# are created using the DBase data rather than scanning the frames again. The Optional DBases could be quite big with
# (Framecount * 256 * 4) + $4000 bytes each, but should support up to about 1.9 million frames (where it would break the
# 2 Gig +ve Int maximum, ie go -ve)
# It is recommended to create the databases only once and to re-use them where possible rather than re-creating time and again.
# Once created, the databases make it possible to fairly quickly find similar frames and could perhaps greatly speed up
# scripts to find where cuts have been made between 2 clips or some kind of frame matching scripts.
#
GSCript("""
c
myName="QwikAveLumaScanCreateDB: "
VER = 0.0
Debug = Default(Debug, False)
Start = RT_Timer()
PrevDB=Default(PrevDB,"")
NextDB=Default(NextDB,"")
(DEBUG) ? RT_DebugF("\nQwikAveLumaScanCreateDB() v%.2f by StainlessS\n",VER,name=myName) : NOP
(DEBUG) ? RT_DebugF("Filling DBase with AverageLuma data (Will take some time)\n",name=myName) : NOP
LastFrame=FrameCount-1
RT_DBaseAlloc(DB,FrameCount,"f") # FrameCount records with 1 field each, Float
if(PrevDB=="") {
(DEBUG) ? RT_DebugF("Creating DB",name=myName) : NOP
For(frame=0,LastFrame) {
L = RT_AverageLuma(Last,frame) # Ave luma of frame
RT_DBaseSet(DB,frame,L) # Set DBase Aveluma
(DEBUG && frame % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",frame,(frame+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
if(PrevDB!="" || NextDB!="") {
DBT=DB+"_"+RT_LocalTimeString(File=True) # Temporary DBase filename
RT_DBaseAlloc(DBT,256,"i") # Create temp DBT
DBS = RT_StrPad("",256,"i") # Type String of 256 'i' characters, ie 256 int fields
if(PrevDB!="") {
(DEBUG) ? RT_DebugF("Initializing TEMP DB",name=myName) : NOP
For(i=0,255) { # Init DBT field 0 to not valid
RT_DBaseSet(DBT,i, -1)
}
(DEBUG) ? RT_DebugF("Creating PrevDB",name=myName) : NOP
RT_DBaseAlloc(PrevDB,FrameCount,DBS)
For(frame=0,LastFrame) {
L = RT_AverageLuma(Last,frame) # Ave luma of frame
RT_DBaseSet(DB,frame,L) # Set Aveluma
for(i=0,255) {
prev = RT_DBaseGetField(DBT,i,0) # Get latest frame number with approx luma = i
RT_DBaseSetField(PrevDB,frame,i,prev)
}
Li = Int(L) # AveLuma of frame as index into temp DBase DBT
RT_DBaseSet(DBT,Li,frame) # Update temp DB with current frame as previous for following frames
(DEBUG && frame % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",frame,(frame+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
if(NextDB!="") {
(DEBUG) ? RT_DebugF("Initializing TEMP DB",name=myName) : NOP
For(i=0,255) { # Init DBT field 0 to not valid
RT_DBaseSet(DBT,i, -1)
}
(DEBUG) ? RT_DebugF("Creating NextDB",name=myName) : NOP
RT_DBaseAlloc(NextDB,FrameCount,DBS)
For(frame=LastFrame, 0, -1) {
L = RT_DBaseGetField(DB,frame,0) # Aveluma
for(i=0,255) {
next = RT_DBaseGetField(DBT,i,0)
RT_DBaseSetField(NextDB,frame,i,next)
}
Li = Int(L) # AveLuma of frame as index into temp DBase DBT
RT_DBaseSet(DBT,Li,frame) # Update temp DB with current frame as next for prev frames
Ftmp=LastFrame - Frame
(DEBUG && Ftmp % 1000 == 0) ? RT_DebugF("Record(%d) %.1f%%",Ftmp,(Ftmp+1)*100.0/(LastFrame+1),name=myName) : NOP
}
}
RT_FileDelete(DBT) # Delete Temp Dbase
}
End = RT_Timer()
(DEBUG) ? RT_DebugF("\nTotal Time = %.2f Seconds (%.2f Mins)\n",End-Start,(End-Start)/60.0,name=myName) : NOP
""")
}

Function QwikAveLumaScanGetNear(String DB, String PNDB,int Frame,Float "AveLuma", Float "AveLumaDiff",int "MaxDistance") {
# Function to find the frame nearest (in frame number distance) to the target Frame, whose AverageLuma is within
# +- AveLumaDiff (inclusive) of AveLuma.
#
# DB, String. Filename of DB file as created by QwikAveLumaScanCreateDB()
# PNDB, String, Filename of PrevDB OR NextDB file as created by QwikAveLumaScanCreateDB()
# Frame, Int. Frame number nearest to which, you want to find a NEAR frame.
# AveLuma, Float. Optional value of AverageLuma, defaults to the AverageLuma of arg Frame.
# AveLumaDiff, Float. Optional (default 1.0). Difference Threshold acceptable as similar to AveLuma. AveLumaDiff is inclusive.
# MaxDistance, Int. Optinal maximum number of frames to search relative to Frame arg. Default number of records in DB-1, ie FrameCount - 1.
#
# The PNDB arg controls whether we are looking for frames BEFORE or AFTER the target Frame, use the PrevDB to find BEFORE
# frames or NextDB to find AFTER frames (as described for QwikAveLumaScanCreateDB().
# To find the nearest frame BEFORE frame 1000 with a Aveluma difference within 1.5 of AverageLuma of frame 1000, use eg
# PrvFrm = QwikAveLumaScanGetNear(DB,PrevDB,1000,AveLumaDiff=1.5)
# To find the nearest frame AFTER frame 1000 with an aveluma difference within 1.5 of AverageLuma = 128.0, use eg
# NxtFrm = QwikAveLumaScanGetNear(DB,NextDB,1000,AveLuma=128.0,AveLumaDiff=1.5)
# In this case the AverageLuma of the returned frame number would be greater or equal to (128.0 - 1.5) and less or
# equal to (128.0 + 1.5). This allows you to use an AveLumaDiff of 0.0 to find an EXACT MATCH.
# To Find exact match eg (needs clip and prior DB setup),
# q = RT_AverageLuma(n=1000) # Get AverageLuma of frame 1000
# z = QwikAveLumaScanGetNear(DB, NextDB, 0, AveLuma=q, AveLumaDiff=0.0) # Get next frame after frame 0 exactly same as frame 1000 AverageLuma.
# Subtitle(String(q)+" " + String(z))
# Above will show AverageLuma of frame 1000 and likely show '1000' as next nearest frame to 0 with that exact same AverageLuma.
#
# The Function returns -1 if there is no frame that satisfies conditions.
#
AveLumaDiff=Float(Default(AveLumaDiff,1.0))
L = Defined(AveLuma) ? Float(AveLuma) : RT_DBaseGetField(DB,Frame,0) # Use AveLuma if given, else get from DB
MaxDistance = Default(MaxDistance,RT_DBaseRecords(DB) - 1) # Defaults to FrameCount - 1
Li = Int(L) # Approx ave luma as Int, index into PNDB
tmn = L - AveLumaDiff ## Valid luma range +- AveLumaDiff
tmx = L + AveLumaDiff ##
mni = Max(0, Int(tmn)) # Start PNDB Index to search
mxi = Min(255, Int(tmx)) # End PNDB Index to search
NearLumaFrm = -1 # Not Found
Distance = MaxDistance + 1 # Has to be closer than this
GSCript("""
# Search center interval 1st as likely nearest to target frame (assuming any given AveLuma is that of target Frame)
# Intent to cut down on search distance in mni->mxi loop
frm=RT_DBaseGetField(PNDB,Frame,Li) # Get nearest before/after frame @ approx Li ave luma
if(frm != -1) { # Are there any frames in that luma interval ?
Break=False # Wish there was a Break (sigh)
While(!Break && Abs(Frame-frm) < Distance) {
v = RT_DBaseGetField(DB,frm,0) # AveLuma for roving frm
if(v >= tmn && v <= tmx) { # Is it within FLOAT Thresh range of L ?
NearLumaFrm = frm # We got a nearer one, HeHe
Distance = Abs(Frame-frm)
Break=True # We got less distant, break out of While Loop
} Else {
frm = RT_DBaseGetField(PNDB,frm,Li) # Scan more distant
if(frm== -1) {
Break = True # No More
}
}
}
}
if(Distance > 1) { # If Distance == 1, then we cannot find closer frame
for(i=mni,mxi) { # Int intervals where AveLuma +- AveLumaDiff is valid
if(i != Li) { # We already did Li
frm=RT_DBaseGetField(PNDB,Frame,i) # Get nearest before/after frame @ approx i ave luma
if(frm != -1) { # Are there any frames in that luma interval ?
Break=False # Wish there was a Break (sigh)
While(!Break && Abs(Frame-frm) < Distance) {
v = RT_DBaseGetField(DB,frm,0) # AveLuma for roving frm
if(v >= tmn && v <= tmx) { # Is it within FLOAT Thresh range of L ?
NearLumaFrm = frm # We got a nearer one, HeHe
Distance = Abs(Frame-frm)
Break=True # We got less distant, break out of While Loop
} Else {
frm = RT_DBaseGetField(PNDB,frm,i) # Scan more distant
if(frm== -1) {
Break = True # No More
}
}
}
if(Distance==1) {
i = mxi # Found adjacent frame, No more search needed, Early Break out of For Loop
}
}
}
}
}
""")
return NearLumaFrm
}

[StainlessS]

Post #2 of 3.

MakePrevDB.avs, Creates Database's

# MakePrevDB.avs

Import("QwikAveLumaScan.avs")

AviSource("Test.avi") # Or Whatever

DB ="MyDB.DB"
PrevDB ="MyDB_Prev.DB"

QwikAveLumaScanCreateDB(Last, DB, PrevDB, debug=True)

return MessageClip("DB PREV set Created")


Eduardobedoya.avs, needs pre-created database's

# Eduardobedoya.avs

# MUST create DataBases before using this script using MakePrevDB.avs
# This script creates a Frames.txt file, can then use SelectFrames.avs to get frames instantly.

Import("QwikAveLumaScan.avs")

AviSource("Test.avi") # Or Whatever

# Alter below to suit
ScanAheadSeconds = 10 * 60 # Search range in seconds
AVELUMA_DIFF = 0.1 # Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
MAX_LDIFF = 0.001 # LumaDifference between i frame and candidate duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB ="MyDB.DB"
PrevDB ="MyDB_Prev.DB"
FN="Frames.txt" # Output frame command file for FrameSel()
DEBUG=True # Info to DebugView window (Google)
####
GSCript("""
Start = RT_Timer()
RT_FileDelete(FN) # Delete any existing frames file
LastFrame=FrameCount-1
OCnt = 0 # Frames output count
For(i=0,LastFrame) {
OutFrame = i # Init frame to output
LumaI = RT_DBaseGetField(DB,i,0) # AverageLuma of i frame from DB
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit - 1 to i+1 (downwards)
Fails = 0 # Count false candidates, too many may indicate AVELUMA_DIFF too big
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to j but higher than i that has AverageLuma difference with i, less or equal to AVELUMA_DIFF
j = QwikAveLumaScanGetNear(DB,PrevDB,j,AveLuma=LumaI,AveLumaDiff=AVELUMA_DIFF,maxdistance=j-i-1)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= MAX_LDIFF) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
} Else {
Fails = Fails + 1 # Count False candidate frames for Debug
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True) # For Prune instead of FrameSel (supports audio), use "0,"+String(OutFrame)
OCnt = OCnt + 1
if(i != OutFrame) {
If(DEBUG) {
E=RT_Timer
RT_DebugF("%d ] Failed_LDifs= %-4d LDif=%f ALDif=%f Skipped=%d InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f",
\ OutFrame,Fails,dif,Abs(LumaI-RT_DBaseGetField(DB,OutFrame,0)),OutFrame-i,i+1,OCnt,OCnt/(E-Start),(OutFrame+1)/(E-Start))
}
i = OutFrame # Skip all frames before Outframe
} Else {
If(DEBUG) {
E=RT_Timer
RT_DebugF("%d ] Failed_LDifs= %-4d InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f",
\ OutFrame,Fails,i+1,OCnt,OCnt/(E-Start),(OutFrame+1)/(E-Start))
}
}
}
E = RT_Timer()
S=RT_String("\nTotal Time = %.2f Seconds (%.2f Mins) InCnt=%d OutCnt=%d InFPS=%.3f OutFPS=%.3f\n",
\ E-Start,(E-Start)/60.0,LastFrame+1,OCnt,OCnt/(E-Start),(LastFrame+1)/(E-Start))
RT_DebugF(S)
RT_TxtWriteFile(S,"Eduardobedoya.Log",Append=False)
""")

Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug, or for audio support see a few lines above.


SelectFrames.avs, instantly get frames afterdatabases created and frames file created via Eduardobedoya.avs

# SelectFrames.avs

AviSource("Test.avi") # Or Whatever
FN="Frames.txt"
Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug.


Somewhat faster than previous script, but still not startling. The QwikAveLumaScan.Avs funcs may well benefit by converting to plugins.

[StainlessS]

Post #3 of 3.

DummyTest.avs, if you dont have suitable test clip

# DummyTest.avs

Import("QwikAveLumaScan.avs")

TestClip(True) # Get Testclip all UpperCase
#Return Last

# Below are just for show for TestClip, both difs can be 0.0 as we will find exact matches
ScanAheadSeconds = 10 * 60 # Search range in seconds
AVELUMA_DIFF = 0.1 # Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
MAX_LDIFF = 0.001 # LumaDifference between i frame and candidate duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
###
ScanAheadFrames = Int(ScanAheadSeconds*FrameRate)
DB ="Dummy.DB"
PrevDB ="Dummy_Prev.DB"
FN ="DummyFrames.txt" # Output frame command file for FrameSel()
DEBUG=True # Info to DebugView window (Google)
DELETEDBASE=TRUE # Delete DBase files
QwikAveLumaScanCreateDB(Last, DB, PrevDB,debug=true) # Create DataBases (would normally do as separate job, SLOW)
####
GSCript("""
Start = RT_Timer()
RT_FileDelete(FN) # Delete any existing frames file
LastFrame=FrameCount-1
For(i=0,LastFrame) {
OutFrame = i # Init frame to output
LumaI = RT_DBaseGetField(DB,i,0) # AverageLuma of i frame from DB
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit - 1 to i+1 (downwards)
Fails = 0 # Count false candidates, too many may indicate AVELUMA_DIFF too big
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to j but higher than i that has AverageLuma difference with i, less or equal to AVELUMA_DIFF
j = QwikAveLumaScanGetNear(DB,PrevDB,j,AveLuma=LumaI,AveLumaDiff=AVELUMA_DIFF,maxdistance=j-i-1)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= MAX_LDIFF) {
OutFrame = j # Last frame within ScanAheadFrames that is similar to i
j = i # Early break
} Else {
Fails = Fails + 1 # Count False candidate frames for Debug
}
}
}
RT_TxtWriteFile(String(OutFrame),FN,Append=True)
if(i != OutFrame) {
(DEBUG)
\ ? RT_DebugF("%d ] Failed_LDifs= %-4d LDif=%f ALDif=%f Skipped=%d",
\ OutFrame,Fails,dif,Abs(LumaI-RT_DBaseGetField(DB,OutFrame,0)),OutFrame-i)
\ : NOP
i = OutFrame # Skip all frames before Outframe
} Else {
(DEBUG) ? RT_DebugF("%d ] Failed_LDifs= %-4d",OutFrame,Fails) : NOP
}
}
if(DELETEDBASE) {
RT_FileDelete(DB)
RT_FileDelete(PrevDB)
}
End = RT_Timer()
(DEBUG) ? RT_DebugF("\nTotal Time = %.2f Seconds (%.2f Mins)\n",End-Start,(End-Start)/60.0) : NOP
""")

Testclip(False) # Now change to upper/lower case mixed to see if it worked OK
Return FrameSel(Last,cmd=FN) # Select frames via FrameSel plug.

# Only of use in testing, (If you dont have a real problem clip)
Function TestClip(Bool "Upper") {
Upper=Default(Upper,False)
Test = "AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ"
Test = (Upper) ? Ucase(Test) : Test
CC = ColorBars().Trim(0,-1).KillAudio # Single frame
C=CC.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
s = RT_StrPad("",20,s)
s=RT_String("%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n%s\n",
\ s,s,s,s,s,s,s,s,s,s,s,Esc=0)
C = C + CC.Subtitle(s,lsp=0,size=40,align=8,y=20)

}
""")
return C
}

[eduardobedoya]

Thanks Stainless, man, thank you very much for your support, you just freak me out with all that huge amount of scripts, I will read and try everything tomorrow. Please man, just give me the minimal ammount of links necesary to achieve this specific task, the easier way possible, sorry I dident post my laptop specs, it is a laptop with 16gbram, nvidia GTX 680M, multicore 2.9ghz. Please Stainless since I need to apply this script in many videos (batch operation) I would like you suggest me the fastest application possible, if it is possible the easier app also, sorry I am not a programmer. Thanks Advanced.

PD: I get the videos from camtasia recording the laptop screen.
Regards.

[StainlessS]

Have already given a few links in previous post,
Would also suggest VirtualDub, see main forum page.

Have only just now converted the QwikAveLumaScanCreateDB func to a plugin (in coming version of RT_Stats),
previous script created all 3 databases (for a test avi) in about 45 mins, currently got it running
as plugin doing same in 41 seconds, so there is some improvement.

I shall now have a go at the QwikAveLumaScanGetNear function, hoping for a reasonable improvement there also,
I shall not though be touching the Eduardobedoya.avs script, its too much a one-off to bother with,
but it should be pretty nippy I think when other plugs done.

I'll see what I can do to make a simple idiot proof script to auto create Databases and FrameSelect command files,
and also create scripts to load the resultant fixed files, all you will have to do is run VirtualDub, select compression and
then Queue them for Batch Saving (file menu).

maybe a day or two.

[eduardobedoya]

Man from 45 min to 41 seconds, thats optimization!
thanks you stainless
I have read your scripts so I kinda understand that...

First I run the QwikAveLumaScan.avs (the plugin version) inside Virtualdub in orther to create a database (but I dont understand, which application plugin? avisynth? Virtualdub? do I need to install this plugin??)
Then I run the MakePrevDB.avs also inside Virtualdub in orther to create a database
Then I run the Eduardobedoya.avs also inside Virtualdub in orther to create a frames.txt file
and finally use the SelectFrames.avs to get the frames inside Virtualdub, right? inside Virtualdub timeline? or I will get avi files directly?

I have 20 videos (45 minutes footage each one), Do I need to proceed with these steps above for each one of these videos??? or, Can I put in the avisource 4 videos, in orther to create the database of 4 videos in one shot????

I have read this
http://neuron2.net/LVG/avisynth.html
So I beleive that I will have to install avisynth and virtualdub, then create the avs scriptfiles that you provided changing the sorcefiles and paths, then open the avs scripts inside virtualdub in the order posted above??? That would be all right? Do I missing something?? those are all the steps right?

I will wait for you last instruction, to perform a test. thanks again StainlessS.

[StainlessS]

Dont bother with those scripts just now, There are a few little problems, especially around the
end of clip (last frame), so I'm still tinkering a little bit. Shall leave those scripts (when updated)
in place for example usage.

I'll make a script, that allows avi group selection via a file selector (GUI),
does ALL of the dbase creation, processes the avi's using the DB's and output
frames files, and also auto create avs scripts, which should be loaded later into vdub and queued for
compressed output. You can execute the creation script by simply loading into VDub or
playing in Windows Media Player (but will seem to hang until all done).

[eduardobedoya]

pls man, let me know when you will release it, I kinda look around this page everyday xD, Thanks, thanks advanced.

[StainlessS]

Sorry, eduardobedoya, had to go away for a few days but have been working on it.
Have had to increase maximum number of fields in RT_Stats DBase's from 256 to 1024
(same for Attributes) and made a few other small changes. Have nearly implemented
plugin functions for fast finding of frames using a 'fingerprint' of the frame to find.
Fingerprint uses (RT_Stats equivalents to) YPlaneMedian, AveraLuma, YPlaneStdev,
and YInRange (last two RT_Stats only). With a tolerance of about +- 1.0 on fingerprint
primitives (eg difference in AvergeLuma between find and found frames) I'm getting
something like 20 FPS output (to frames text file) on a clip that has no duplicates (ie there is no
frame to find so does a full search) where test clip has 10,000 SD PAL frames searching from
current frame to last frame for every output frame. If tolerances are 0.0, ie exact matches
only, then am getting about 170 to 180 FPS in same case as above. (Core 2 Duo 2.13 Ghz, Sata 2).

Have been reluctant to release RT_Stats update until the new funcs are implemented as flexibly as possible,
will be developer ONLY functions, and not for the casual scriptor. Will I think be released as experimental, as
could still change args, but mostly already there.

Question on your particular problem, are there any unique sequences within duplicate sequence range (10 mins you say) ?,
as currently implemented it searches for LAST frame that matches, so it can skip large numbers of frames all in a single
jump. However, if there are isolated unique sequences beteen Find and Found frames, you would lose the unique sequence.
I could implement modified script to drop the current frame instead of all frames between current Find frame and Found frame,
but it would be significantly slower dropping 1 frame at a time instead of vast swathes of frames.

Please be patient, I am still working on it.

EDIT: Also, do we need to cope with Audio ?


# Below XYX isolated unique sequence, Keep uppercase.
"abcdabcdXYZdefdefgABCdefdefgDEFghijGHIJKLmnopmnopMNOPQRSTUVW"

[eduardobedoya]

Thanks Stainless - audio is not important, in fact, the files are avi. without audio.

Thanks advanced, will wait till you release it.

[eduardobedoya]

Thanks Stainless - No, there are not any unique sequence between two duplicated frames, it would be very strange to have something usefull between two duplicated frames, basically I need to delete ALL frames between to equal frames. Not only delete the duplicated frames, but delete everything between two duplicated frames. (inside a range of 10 minutes).

The footage is 30fps Techsmith codec avi - no audio.

Thanks Advanced, I will wait till your release

[StainlessS]

Post #1 of 2

Not releasing RT_Stats v1.31 with docs just yet, but here it is without new docs or source.
Also includes Prune for Audio. (copy plugs to Plugins dir).

Have implemented for Audio anyways, and two scripts, one to keep unique frames, and one that dont.

See beginning of avs for config.

Here are dll's (still need Gscript and FrameSel v1.12) : LINK REMOVED

Keep Unique script:

# ############################
# Fsel_EduardobedoyaKeepUnique_Batch.avs, by StainlessS
# ############################

# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

####
# Below settings, as close to zero as possible (faster but might miss duplicates), used by QWIK scan routines
LODIFFMED = 1 # INT, Max diff (-ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFMED = 1 # INT, Max diff of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFAL = 0.01 # Float, Max diff (-ve tolerance) of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFAL = 0.01 # Float, Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFSTD = 0.01 # Float, Max diff (-ve tolerance) of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFSTD = 0.01 # Float, Max diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFINR = 0.01 # Float, Max diff (-ve tolerance) of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFINR = 0.01 # Float, Max diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good.
MAX_LDIFF = 0.01 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####

####
AUDIO = True # False, no audio. True Supports audio, Audio requires Prune Plugin
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI, files Will Batch create FrameSel command files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_EduardobedoyaKeepUnique_Batch: "
LOG="Fsel_EduardobedoyaKeepUnique_Batch.Log"
RT_TxtWriteFile(LOG,LOG,Append=False)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
DropDupSequencesKeepUnique(FN,ScanAheadSecs,
\ ldm=LODIFFMED, hdm=HIDIFFMED,
\ ldL=LODIFFAL, hdL=HIDIFFAL,
\ lds=LODIFFSTD, hds=HIDIFFSTD,
\ ldr=LODIFFINR, hdr=HIDIFFINR,
\ LDThresh=MAX_LDIFF,audio=AUDIO,log=LOG)
T = RT_Timer - START
S=RT_String(" %s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_TxtWriteFile(S,LOG,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropDupSequencesKeepUnique(String "AviName",int "ScanAheadSecs",
\ int "ldM",int "hdM",Float "ldL",Float "hdL",Float "ldS",Float "hdS",Float "ldR",Float "hdR",Float "LDThresh",Bool "Audio",String "Log") {
myName="DropDupSequencesKeepUnique: "
ScanAheadSecs=Default(ScanAheadSecs,10*60)
ldM = Default(ldM,1) hdM = Default(hdM,1)
ldL = Float(Default(ldL,0.01)) hdL = Float(Default(hdL,0.01))
ldS = Float(Default(ldS,0.01)) hdS = Float(Default(hdS,0.01))
ldR = Float(Default(ldR,0.01)) hdR = Float(Default(hdR,0.01))
LDThresh = Float(Default(LDThresh,0.01))
LOG = Default(LOG,"DropDupSequencesKeepUnique.LOG")
Assert(Exist(AviName),myName+AviName+" Does Not Exist")
Avisource(AviName)
Audio=Default(Audio,False) # Needs Prune plugin if Audio
Audio=(!HasAudio) ? False : Audio
Assert(ScanAheadSecs>=0,myName+"ScanAheadSecs Must be greater than zero")
Assert(ldM>=0 && hdM >0,myName+"ldM and hdM Must be greater than zero")
Assert(ldL>=0.0 && hdL >=0.0,myName+"ldL and hdL Must be greater or equal to zero")
Assert(ldS>=0.0 && hdS >=0.0,myName+"ldS and hdS Must be greater or equal to zero")
Assert(ldR>=0.0 && hdR >=0.0,myName+"ldR and hdR Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_KUnique_Frames.txt"
ScriptFile=PathAndNode+"_KUnique_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
NextDB=PathAndNode+"_Next.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
FrameSel_Select="""
Avisource("%s")
CmdFrames="%s"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=True) : NOP
Return Last
"""
Prune_Select="""
Avisource("%s")
CmdFrames="%s"
PruneFrames=CmdFrames+"_Prune.txt"
Ex=(Exist(CmdFrames))
(Ex) ? FrameSel_CmdReWrite(PruneFrames,Cmd=CmdFrames,reject=True,Range=True,Prune=True) : NOP
(Ex) ? Prune(cmd=PruneFrames,Fade=10,FadeIn=True,FadeSplice=True,FadeOut=True) : NOP
Return Last
"""

Select_S = (Audio) ? Prune_Select : FrameSel_Select # Select FrameSel or Prune extraction
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,CmdFrames) # Insert filenames

START = RT_Timer
RT_QwikAveLumaScanCreateDB(DB,prevdb="",nextdb=NextDB,debug=true) # Forward scanning
T= RT_Timer - START
S = RT_String(" QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_TxtWriteFile(S,LOG,Append=True)
GSCript("""
START = RT_Timer
Dropped = 0
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)
For(i=0,LastFrame) {
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching i+1 to Endlimit inclusive
# QWIK Find a series of likely matching frames
For(j = i + 1,EndLimit) {
j = RT_QwikAveLumaScanGetNear(DB,NextDB,j,
\ findframe=i,
\ LoDiffMed =ldM, HiDiffMEd =hdM,
\ LoDiffAl =ldL, HiDiffAL =hdL,
\ LoDiffStd =ldS, HiDiffStd =hdS,
\ LoDiffInr =ldR, HiDiffInr =hdR,
\ maxdistance=EndLimit-j, Inclusive=True)
if(j < 0) { # j Will be -1, ie Not Found
j = EndLimit # Early break from j, DONT delete frame
} Else { # We found a candidate frame, j will be greater than i
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Average pixel diff between i frame and candidate j
if(dif <= LDThresh) { # is candidate frame a good match ?
Dropped = Dropped + 1
RT_TxtWriteFile(String(i),
\ CMDFrames,Append=True) # Frame to DELETE, Keep later frame, delete earlier
j = EndLimit # Early break, move on to next i frame
} # Otherwise continue QWIK scan for next candidate in series
}
}
}
RT_TxtWriteFile(Select_S,ScriptFile,Append=False)
RT_FileDelete(DB)
RT_FileDelete(NextDB)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String(" Dropping %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]\n QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", \
Dropped, FrameCount,Width,Height,FT,FT/60.0,FrameRate,T,T/60.0,FrameCount/T,(FrameCount-Dropped)/T)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
""")
Return 0
}

EDITED:

and log


Fsel_EduardobedoyaKeepUnique_Batch.Log


1/4 ] Processing D:\AVS\AVI\IN\FLASHTEST.avi
QWIK Scan DBase creation = 15.75 Secs (0.26 Mins)
Dropping 0 of 7342 frames [640x400 293.68 secs (4.89 Mins) @ 25.00 FPS]
QWIK SCAN 5.42 Secs (0.09 Mins) InFPS=1354.36 OutFPS=1354.36
D:\AVS\AVI\IN\FLASHTEST.avi Tot File Time = 21.25 Seconds (0.35 Mins)


2/4 ] Processing D:\AVS\AVI\IN\Scrambled.avi
QWIK Scan DBase creation = 244.28 Secs (4.07 Mins)
Dropping 25500 of 45000 frames [720x576 1800.00 secs (30.00 Mins) @ 25.00 FPS]
QWIK SCAN 524.80 Secs (8.75 Mins) InFPS=85.75 OutFPS=37.16
D:\AVS\AVI\IN\Scrambled.avi Tot File Time = 769.27 Seconds (12.82 Mins)


3/4 ] Processing D:\AVS\AVI\IN\TEST2.avi
QWIK Scan DBase creation = 46.72 Secs (0.78 Mins)
Dropping 0 of 15319 frames [720x576 612.76 secs (10.21 Mins) @ 25.00 FPS]
QWIK SCAN 20.47 Secs (0.34 Mins) InFPS=748.40 OutFPS=748.40
D:\AVS\AVI\IN\TEST2.avi Tot File Time = 67.47 Seconds (1.12 Mins)


4/4 ] Processing D:\AVS\AVI\IN\TEST.avi
QWIK Scan DBase creation = 57.47 Secs (0.96 Mins)
Dropping 0 of 13861 frames [720x576 554.44 secs (9.24 Mins) @ 25.00 FPS]
QWIK SCAN 13.17 Secs (0.22 Mins) InFPS=1052.31 OutFPS=1052.31
D:\AVS\AVI\IN\TEST.avi Tot File Time = 70.84 Seconds (1.18 Mins)


DONE

TOTAL Time = 928.83 Seconds (15.48 Mins)



Only Scrambled.avi is scrambled, others just to test that it dont delete frames where not required.

Here MakeScrambled.avs, test clip

###################
# MakeScrambled.avs
###################
# Only of use in testing, (If you dont have a real problem clip)

Function ScrambleClip(clip c) { # intended for PAL source [numbers may not work for NTSC, have not tried, use AssumeFPS(25.0) beforehand]
c
fc=FrameCount
Need = 27 * 750
Assert(fc >= Need,"ScrambleClip:, need at least "+String(Need)+" Frame clip as source(got "+String(fc)+")")

# Below XYZ isolated sequence. Will keep all upper case letters (including XYZ in Keep Unique script).
# Keep unique script will keep LAST instance of ALL frames within Scanahead.
# Non Keep Unique avs will jump past XYZ frames and they will not be kept, but faster. (jumps from 1st lowercase 'a' to
# last 'A', and omits everything between.

Test = Ucase("abcdabcdXYZdefdefgABCdefdefgDEFghijGHIJKLmnopmnopMNOPQRSTUVW")
CC=C.BlankClip(length=0) # zero length clip
GScript("""
For(i=1,StrLen(Test)) {
s = MidStr(Test,i,1)
si = RT_ORD(s) - RT_Ord("A") # Subtract 65, make 'A'=0, 'Z' = 25
CT=C.Trim(si*750,-750).Subtitle(S,Size=20)
CT = (s>="X" && s<="Z") ? CT.Subtitle("UNIQUE",Align=5,size=60) : CT
CC = CC ++ CT
}
""")
return CC
}

avisource("JurassicPark.avi")
Trim(16854,182049) # Chop off intro
ScrambleClip()

return last

[StainlessS]

Post #2 of 2

No Keep unique frames

# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# ############################

# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

####
# Below settings, as close to zero as possible (faster but might miss duplicates), used by QWIK scan routines
LODIFFMED = 1 # INT, Max diff (-ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFMED = 1 # INT, Max diff of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFAL = 0.01 # Float, Max (-ve tolerance) diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFAL = 0.01 # Float, Max diff of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFSTD = 0.01 # Float, Max (-ve tolerance) diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFSTD = 0.01 # Float, Max diff of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
LODIFFINR = 0.01 # Float, Max (-ve tolerance) diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
HIDIFFINR = 0.01 # Float, Max diff of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good.
MAX_LDIFF = 0.01 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####

####
AUDIO = True # False, no audio. True Supports audio, Audio requires Prune Plugin
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI, files Will Batch create FrameSel command files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_TxtWriteFile(LOG,LOG,Append=False)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
DropDupSequences(FN,ScanAheadSecs,
\ ldm=LODIFFMED, hdm=HIDIFFMED,
\ ldL=LODIFFAL, hdL=HIDIFFAL,
\ lds=LODIFFSTD, hds=HIDIFFSTD,
\ ldr=LODIFFINR, hdr=HIDIFFINR,
\ LDThresh=MAX_LDIFF,audio=AUDIO,log=LOG)
T = RT_Timer - START
S=RT_String(" %s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_TxtWriteFile(S,LOG,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropDupSequences(String "AviName",int "ScanAheadSecs",
\ int "ldM",int "hdM",Float "ldL",Float "hdL",Float "ldS",Float "hdS",Float "ldR",Float "hdR",Float "LDThresh",Bool "Audio",String "Log") {
myName="DropDupSequences: "
ScanAheadSecs=Default(ScanAheadSecs,10*60)
ldM = Default(ldM,1) hdM = Default(hdM,1)
ldL = Float(Default(ldL,0.01)) hdL = Float(Default(hdL,0.01))
ldS = Float(Default(ldS,0.01)) hdS = Float(Default(hdS,0.01))
ldR = Float(Default(ldR,0.01)) hdR = Float(Default(hdR,0.01))
LDThresh = Float(Default(LDThresh,0.01))
LOG = Default(LOG,"DropDupSequences.LOG")
Assert(Exist(AviName),myName+AviName+" Does Not Exist")
Avisource(AviName)
Audio=Default(Audio,False) # Needs Prune plugin if Audio
Audio=(!HasAudio) ? False : Audio
Assert(ScanAheadSecs>0,myName+"ScanAheadSecs Must be greater than zero")
Assert(ldM>=0 && hdM >=0,myName+"ldM and hdM Must be greater or equal to zero")
Assert(ldL>=0.0 && hdL >=0.0,myName+"ldL and hdL Must be greater or equal to zero")
Assert(ldS>=0.0 && hdS >=0.0,myName+"ldS and hdS Must be greater or equal to zero")
Assert(ldR>=0.0 && hdR >=0.0,myName+"ldR and hdR Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.txt"
ScriptFile=PathAndNode+"_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
PrevDB=PathAndNode+"_Prev.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
FrameSel_Select="""
Avisource("%s")
CmdFrames="%s"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last
"""
Prune_Select="""
Avisource("%s")
CmdFrames="%s"
PruneFrames=CmdFrames+"_Prune.txt"
Ex=(Exist(CmdFrames))
(Ex) ? FrameSel_CmdReWrite(PruneFrames,Cmd=CmdFrames,reject=False,Range=True,Prune=True) : NOP
(Ex) ? Prune(cmd=PruneFrames,Fade=10,FadeIn=True,FadeSplice=True,FadeOut=True) : NOP
Return Last
"""

Select_S = (Audio) ? Prune_Select : FrameSel_Select # Select FrameSel or Prune extraction
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,CmdFrames) # Insert filenames

START = RT_Timer
RT_QwikAveLumaScanCreateDB(DB,prevdb=PrevDB,nextdb="",debug=true) # Backwards scanning
T= RT_Timer - START
S = RT_String(" QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_TxtWriteFile(S,LOG,Append=True)
GSCript("""
START = RT_Timer
Dropped = 0
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)
For(i=0,LastFrame) {
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit to i+1 inclusive (downwards)
# QWIK Find a series of likely matching frames
For(j=EndLimit,i + 1,-1) {
# Search for frame nearest to EndLimit but higher than i
j = RT_QwikAveLumaScanGetNear(DB,PrevDB,j,
\ findframe=i,
\ LoDiffMed =ldM, HiDiffMEd =hdM,
\ LoDiffAl =ldL, HiDiffAL =hdL,
\ LoDiffStd =ldS, HiDiffStd =hdS,
\ LoDiffInr =ldR, HiDiffInr =hdR,
\ maxdistance=j-(i+1), Inclusive=True)
if(j > i) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= LDThresh) {
Dropped = Dropped + (j-i)
i = j # Last frame within ScanAheadFrames that is similar to i
j = 0 # Early break
}
} # Otherwise continue QWIK scan for next candidate in series
}
RT_TxtWriteFile(String(i),CMDFrames,Append=True)
}
RT_TxtWriteFile(Select_S,ScriptFile,Append=False)
RT_FileDelete(DB)
RT_FileDelete(PrevDB)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String(" Dropping %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]\n QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", \
Dropped, FrameCount,Width,Height,FT,FT/60.0,FrameRate,T,T/60.0,FrameCount/T,(FrameCount-Dropped)/T)
RT_DebugF(S,name=myName)
RT_TxtWriteFile(S,LOG,Append=True)
""")
Return 0
}

EDITED:

and log

Fsel_Eduardobedoya_Batch.Log

1/4 ] Processing D:\AVS\AVI\IN\FLASHTEST.avi
QWIK Scan DBase creation = 16.30 Secs (0.27 Mins)
Dropping 0 of 7342 frames [640x400 293.68 secs (4.89 Mins) @ 25.00 FPS]
QWIK SCAN 8.44 Secs (0.14 Mins) InFPS=870.11 OutFPS=870.11
D:\AVS\AVI\IN\FLASHTEST.avi Tot File Time = 24.83 Seconds (0.41 Mins)


2/4 ] Processing D:\AVS\AVI\IN\Scrambled.avi
QWIK Scan DBase creation = 236.22 Secs (3.94 Mins)
Dropping 27750 of 45000 frames [720x576 1800.00 secs (30.00 Mins) @ 25.00 FPS]
QWIK SCAN 23.17 Secs (0.39 Mins) InFPS=1942.00 OutFPS=744.43
D:\AVS\AVI\IN\Scrambled.avi Tot File Time = 259.59 Seconds (4.33 Mins)


3/4 ] Processing D:\AVS\AVI\IN\TEST2.avi
QWIK Scan DBase creation = 53.75 Secs (0.90 Mins)
Dropping 0 of 15319 frames [720x576 612.76 secs (10.21 Mins) @ 25.00 FPS]
QWIK SCAN 27.56 Secs (0.46 Mins) InFPS=555.80 OutFPS=555.80
D:\AVS\AVI\IN\TEST2.avi Tot File Time = 81.61 Seconds (1.36 Mins)


4/4 ] Processing D:\AVS\AVI\IN\TEST.avi
QWIK Scan DBase creation = 69.00 Secs (1.15 Mins)
Dropping 0 of 13861 frames [720x576 554.44 secs (9.24 Mins) @ 25.00 FPS]
QWIK SCAN 18.75 Secs (0.31 Mins) InFPS=739.25 OutFPS=739.25
D:\AVS\AVI\IN\TEST.avi Tot File Time = 87.91 Seconds (1.47 Mins)


DONE



TOTAL Time = 453.94 Seconds (7.57 Mins)


Is somewhat better than 1st script in thread where we were getting about 1 frame every 12 seconds.

EDIT: Has GUI File Selector to group select AVI files, auto creates DataBases, Frames command files and
avs files to instantly select edited clips, and deletes Dbases files.

EDIT: Just spotted potential (but unlikely) problem, if mutiple consecutive SPACES in AVI filenames, will
convert to single space and NOT find them. (RT_StrReplace line).

EDIT: Scripts edited for minor problems or typo's.
The Prune extractor removes possible audio glitches ('cracks') at splices when audio present.

PS, were the scripts idiot proof ? :)

[eduardobedoya]

Thanks Stainlesss, I read your notes and tried your script

First I installed the 32bit version of Virtualdub (I dont know why windows media player fail to open Fsel_Eduardobedoya_Batch.avs)
Is it the same if I have the 32 or 64 bits version of Virtualdub?

Then I installed Avisynth_258 and the RT Stats plugin that you provided
I couldn't find FrameSel v1.12 so I guessed v2.12 could do the trick, so I installed from here
http://forum.doom9.org/showthread.php?p=1566581#post1566581
Then I installed GScript_11 from here
http://forum.doom9.org/showthread.php?t=147846
(I copy all the plugins' dlls inside the Avisynth's plugin directory)
Finally I installed debug view from here
http://technet.microsoft.com/en-gb/sysinternals/bb896647

I run VirtualDub and open Fsel_Eduardobedoya_Batch.avs
the open windows showed, then I pick up some 1920x1080 20min video
here is the LOG:

00000001 0.00000000 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000002 15.38796234 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000003 15.40174389 [7920] Fsel_Eduardobedoya_Batch:
00000004 15.40177345 [7920] Fsel_Eduardobedoya_Batch:
00000005 15.40181732 [7920] Fsel_Eduardobedoya_Batch: 1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\cap19 -4.avi
00000006 15.41219711 [7920] TSC2: Instantiating codec with Q=2/6, force KFs=800, force KFbytes=5000000
00000007 15.43127728 [7920]
00000008 15.43127728 [7920] RT_QwikAveLumaScanCreateDB: RT_QwikAveLumaScanCreateDB() by StainlessS
00000009 15.43131065 [7920] RT_QwikAveLumaScanCreateDB: Filling DBase with YStats data (Will take some time)
00000010 15.43134689 [7920] RT_QwikAveLumaScanCreateDB: Creating DB
00000011 23.23528481 [7920] RT_QwikAveLumaScanCreateDB: record(1024) 5.5%
00000012 31.00768471 [7920] RT_QwikAveLumaScanCreateDB: record(2048) 10.9%
00000013 35.70775223 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000014 35.70792007 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000015 38.72195816 [7920] RT_QwikAveLumaScanCreateDB: record(3072) 16.4%
00000016 46.43200684 [7920] RT_QwikAveLumaScanCreateDB: record(4096) 21.8%
00000017 54.15245438 [7920] RT_QwikAveLumaScanCreateDB: record(5120) 27.3%
00000018 61.86000061 [7920] RT_QwikAveLumaScanCreateDB: record(6144) 32.7%
00000019 69.56040955 [7920] RT_QwikAveLumaScanCreateDB: record(7168) 38.2%
00000020 77.31515503 [7920] RT_QwikAveLumaScanCreateDB: record(8192) 43.6%
00000021 78.57512665 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000022 80.63726044 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000023 82.39134979 [7920] DllMain: hModule=0x10000000, ulReason=3, lpReserved=0x00000000, gRefCnt = 2
00000024 85.02700043 [7920] RT_QwikAveLumaScanCreateDB: record(9216) 49.1%
00000025 88.59996033 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 2
00000026 92.72863007 [7920] RT_QwikAveLumaScanCreateDB: record(10240) 54.5%
00000027 100.43165588 [7920] RT_QwikAveLumaScanCreateDB: record(11264) 60.0%
00000028 108.12608337 [7920] RT_QwikAveLumaScanCreateDB: record(12288) 65.4%
00000029 115.83764648 [7920] RT_QwikAveLumaScanCreateDB: record(13312) 70.9%
00000030 123.53649902 [7920] RT_QwikAveLumaScanCreateDB: record(14336) 76.3%
00000031 131.22393799 [7920] RT_QwikAveLumaScanCreateDB: record(15360) 81.8%
00000032 138.88714600 [7920] RT_QwikAveLumaScanCreateDB: record(16384) 87.2%
00000033 146.58955383 [7920] RT_QwikAveLumaScanCreateDB: record(17408) 92.7%
00000034 154.30934143 [7920] RT_QwikAveLumaScanCreateDB: record(18432) 98.1%
00000035 156.98666382 [7920] RT_QwikAveLumaScanCreateDB: record(18785) 100.0%
00000036 156.98908997 [7920] RT_QwikAveLumaScanCreateDB: Creating PrevDB
00000037 157.05241394 [7920] RT_QwikAveLumaScanCreateDB: record(1024) 5.5%
00000038 157.05616760 [7920] RT_QwikAveLumaScanCreateDB: record(2048) 10.9%
00000039 157.05995178 [7920] RT_QwikAveLumaScanCreateDB: record(3072) 16.4%
00000040 157.06365967 [7920] RT_QwikAveLumaScanCreateDB: record(4096) 21.8%
00000041 157.06732178 [7920] RT_QwikAveLumaScanCreateDB: record(5120) 27.3%
00000042 157.07098389 [7920] RT_QwikAveLumaScanCreateDB: record(6144) 32.7%
00000043 157.07464600 [7920] RT_QwikAveLumaScanCreateDB: record(7168) 38.2%
00000044 157.07839966 [7920] RT_QwikAveLumaScanCreateDB: record(8192) 43.6%
00000045 157.08206177 [7920] RT_QwikAveLumaScanCreateDB: record(9216) 49.1%
00000046 157.08575439 [7920] RT_QwikAveLumaScanCreateDB: record(10240) 54.5%
00000047 157.08943176 [7920] RT_QwikAveLumaScanCreateDB: record(11264) 60.0%
00000048 157.09320068 [7920] RT_QwikAveLumaScanCreateDB: record(12288) 65.4%
00000049 157.09689331 [7920] RT_QwikAveLumaScanCreateDB: record(13312) 70.9%
00000050 157.10060120 [7920] RT_QwikAveLumaScanCreateDB: record(14336) 76.3%
00000051 157.10430908 [7920] RT_QwikAveLumaScanCreateDB: record(15360) 81.8%
00000052 157.10803223 [7920] RT_QwikAveLumaScanCreateDB: record(16384) 87.2%
00000053 157.11178589 [7920] RT_QwikAveLumaScanCreateDB: record(17408) 92.7%
00000054 157.11546326 [7920] RT_QwikAveLumaScanCreateDB: record(18432) 98.1%
00000055 157.11676025 [7920] RT_QwikAveLumaScanCreateDB: record(18785) 100.0%
00000056 157.11859131 [7920] RT_QwikAveLumaScanCreateDB: total time = 141.69 seconds (2.36 mins)
00000057 157.12307739 [7920] DropDupSequences: QWIK Scanning file ... Please Wait
00000058 178.10548401 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 2
00000059 217.35881042 [7920] DropDupSequences: Dropping 18526 of 18785 frames [1920x1080 1252.33 secs (20.87 Mins) @ 15.00 FPS]
00000060 217.35885620 [7920] DropDupSequences: QWIK SCAN 60.23 Secs (1.00 Mins) InFPS=311.88 OutFPS=4.30
00000061 217.36831665 [7920] Fsel_Eduardobedoya_Batch: C:\Users\Eduardo\Desktop\FOR AVISYNTH\cap19 -4.avi Tot File Time = 201.97 Seconds (3.37 Mins)
00000062 217.37132263 [7920] Fsel_Eduardobedoya_Batch:
00000063 217.37135315 [7920] Fsel_Eduardobedoya_Batch:
00000064 217.37139893 [7920] Fsel_Eduardobedoya_Batch: TOTAL Time = 201.97 Seconds (3.37 Mins)
00000065 217.37141418 [7920] Fsel_Eduardobedoya_Batch:
00000066 217.37489319 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73647561(auds), 0)
00000067 217.37492371 [7920] 05533D48->CAVIStreamSynth(audio)
00000068 217.37496948 [7920] 05533D48->CAVIStreamSynth::AddRef() (audio) gRefCnt=3, m_refs=1
00000069 217.37498474 [7920] 003C8740->CAVIFileSynth::AddRef() gRefCnt=4, m_refs=3
00000070 217.37503052 [7920] 05533D48->CAVIStreamSynth::Info(0018F948, 204) (audio)
00000071 217.37504578 [7920] 05533D48->CAVIStreamSynth::ReadFormat() (audio)
00000072 217.37536621 [7920] 05533D48->CAVIStreamSynth::ReadFormat() (audio)
00000073 217.37565613 [7920] 05533D48->CAVIStreamSynth::Info(0018F9E0, 204) (audio)
00000074 217.37568665 [7920] 05533D48->CAVIStreamSynth::Info(0018F9D8, 204) (audio)
00000075 217.37573242 [7920] 05533D48->CAVIStreamSynth::Info(0018F9D8, 204) (audio)
00000076 217.37577820 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73647561(auds), 1)
00000077 217.37579346 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73766169(iavs), 1)
00000078 217.37583923 [7920] 003C8740->CAVIFileSynth::GetStream(*, 73646976(vids), 0)
00000079 217.37586975 [7920] 05533C28->CAVIStreamSynth(video)
00000080 217.37591553 [7920] 05533C28->CAVIStreamSynth::AddRef() (video) gRefCnt=5, m_refs=1
00000081 217.37593079 [7920] 003C8740->CAVIFileSynth::AddRef() gRefCnt=6, m_refs=4
00000082 217.37596130 [7920] 05533C28->CAVIStreamSynth::Info(0018F978, 204) (video)
00000083 217.37600708 [7920] 05533C28->CAVIStreamSynth::ReadFormat() (video)
00000084 217.37629700 [7920] 05533C28->CAVIStreamSynth::ReadFormat() (video)
00000085 217.37661743 [7920] 05533C28->CAVIStreamSynth::Info(0018FA0C, 204) (video)
00000086 217.37663269 [7920] 05533C28->CAVIStreamSynth::Info(0018FA04, 204) (video)
00000087 217.37666321 [7920] 05533C28->CAVIStreamSynth::Info(0018FA04, 204) (video)
00000088 217.37803650 [7920] 05533C28->CAVIStreamSynth::QueryInterface() (video) {00020022-0000-0000-c000-000000000046} (IAVIStreaming)
00000089 217.37808228 [7920] 05533C28->CAVIStreamSynth::AddRef() (video) gRefCnt=7, m_refs=2
00000090 217.37809753 [7920] 05533C28->CAVIStreamSynth::Begin(0, 2073600, 2000) (video)
00000091 217.37814331 [7920] 05533C28->CAVIStreamSynth::Release() (video) gRefCnt=6, m_refs=1
00000092 217.43962097 [7920] DllMain: hModule=0x10000000, ulReason=2, lpReserved=0x00000000, gRefCnt = 6

the process finished and an black screen appeared with yellow letters indicating some time
also created the "cap19 -4_Frames.txt" and "cap19 -4_SelectFrames.AVS" files inside the source directory
but I dont know how to export the video, If I put File>save as avi, it export a black screen video
If I try to open the created "cap19 -4_SelectFrames.AVS" it will pop a dialog box saying...
Avisynth open failure:
Script error: there is no function named "FrameSel"

Do I need to install some plugin in VirtualDub?
Is there something I am doing wrong?
Is there a way to import the result frames directly inside premiere? or another editor that could render faster than VirtualDub?
I found that VirtualDub render to slow, I need to render almost 20 hours with x264 compression, pls Stainlesss what do you suggest?

[StainlessS]

Not sure, I think you need 32 bit VDub (I still use XP32, and will continue to after APRIL 8 [when
dropped from Ms support, I think])

I see no reason why windows Media player should not open the script, although it would just seem to hang
until script comleted.

Should be able to open avs script containing only eg:

return Colorbars()



Yes, it looks like you have Avisynth v2.58 installed, I think many of the DebugView messages
are coming from v2.58 eg "CAVIStreamSynth::QueryInterface()" messages, v2.58 was I think complied
with debug logging enabled.
I use v2.6, might be beter choice than v2.58, despite it being alpha, is more stable than v2.58, IMHO.
If you set DebugView filter (EDIT menu), to only display lines with a colon (':') then it would only
capture relevant messages (mostly), I nearly always use a colon in debug messages.

yep, you were correct, FrameSel v2.12, not v1.12 (my mistake).
However, it may be that you installed v2.6 plugin (for Avisynth v2.6) and so Avisynth cannot find it.
(Avisynth v2.6 can load v2.5 plugins but Avisynth v2.58 cannot load v2.6 plugins).
In the folder named 'Avisynth25' not 'Avisynth26' unless you change to Avisynth v2.6Alpha5.

Looks like we are dropping way too many frames

DropDupSequences: Dropping 18526 of 18785 frames [1920x1080 1252.33 secs (20.87 Mins) @ 15.00 FPS]

As you say you are using screencaps (TechSmith codec), I assumed duplicate sequences would be pretty
identical, looks like (by dropped frame numbers) that this is not the case.
Is there something other than purely desktop involved (do the captures contain windows displaying
streamed video or something else from lossy compressed source). (I'm under impression that TechSmith
is lossless but intended for non complex image capture eg DeskTop with a limited number of colors,
and little change from frame to frame).

Once we get it working OK, then you can use Premiere (I think) or MeGUI to output x264/mp4.

For just now, suggest install correct FrameSel plug, and change config to something like

LODIFFMED = 1
HIDIFFMED = 1
LODIFFAL = 1.0
HIDIFFAL = 1.0
LODIFFSTD = 1.0
HIDIFFSTD = 1.0
LODIFFINR = 1.0
HIDIFFINR = 1.0
MAX_LDIFF = 20.0

and see what happens

Is it possible that you could use VirtualDub to chop out two sets of duplicate frames (about half a dozen of each)
and save them with "Direct Stream Copy" on Video Menu. I dont want to be downloading gigabytes of stuff on my Mobile BroadBand,
ideally no more than about 40-50MB. You can send me PM with link if you prefer.
EDIT: Upload to Mediaifre or SendSpace(no account needed for SendSpace), and maybe 100MB max, I can use broadband
instead.

I'm still playing with this, mainly the KeepUnique version as it seems more useful, also, if you are not using purely desktop
caps then the non keep unique version is probably not the way to go for you either.

[StainlessS]

currently mobile (in pub), now I see the sample, I better know what we are dealing with, quite some problem. as most frames identical and with very small differences between edits and undos, suggest all config stuff be set to 0 or 0.0.
had also come to conclusion that no keep unique of little use, however it seems that it is exactly what we need but with all zero tolerances.
if it does work, resultant clip will be only a few frames and may need for each frame to be duplicated. maybe 60 times each, or setting Framerate to 0.5 fps.
I shall have a play and get back to you.
nice art work by the way, wish I could paint.

[eduardobedoya]

Thanks man,
programmers are digital architects, digital engineers
all digital painting n 3d modeling is based on programming, so
Thanks man again, wish I could understand more of programming, with all that planning and design.

[StainlessS]

Seem to have hit a problem, I was getting weird numbers from the script, and it seems that Avisynth
AviSource video source filter, aint working terribly well with TechSmith codec. We gets lots of
black flashing frames, which presumably mean something like 'Copy the previous frame exactly",
ie in the video stream there are probably some kind of flags that tell the app to duplicate previous frames,
and for those frames TechSmith delivers only black frames or frame 'differences'.
I did try cutting out just frames 14 and 16 from your sample and saving with "Direct Stream Copy",
to another AVI, and it actually saved 17 frames (0-16). DivX and XDiv only allow you to cut video at
KEY frames, I just checked in Vdub, first KEY frame = 0, next is 807, then 836, 848 , 1657, 1807, 1815, and then
no more.

EDIT: From post #3, Clip should be seekable, eg NOT DivX with single keyframe (way too slow)

So TechSmith is NOT cutt-able at each frame, and to properley extract eg frame 806, it may need to seek
all the way back to frame 0 to decode that frame (and decode all frames between 0 and 806).
DirectShowSource() seems to work OK, but Directshow is notorious for delivering the wrong frames (guesses),
I'll have a go at getting the script working using DirectShowSource, but cannot guarentee the results as we
do really need frame accurate seeking. You may have to re-encode to uncompressed RGB (or better, UT_Video
codec as RGB24 or 32) before trying out the script.

There is a thread on-site mentioning the exact same problem here:- http://forum.doom9.org/showthread.php?p=1062041&highlight=techsmith+avisource#post1062041

and there seemed to have been no resolution for that problem.

Would you mind if I gave a link to your sample, it may be that developers are not aware that AviSource does not play nice with video samples
containing TechSmith 'Repeat frame' flags (or whatever they might be called) ?

below just so you can see what I mean.

# Loads using FOURCC 'tsc2', TechSmith Screen Codec 2 (v1.0.6 April 23 2013)

AVISource("painting test for avisynth.avi") # Junk frames where duplicate frame flags
#AVIFileSource("painting test for avisynth.avi") # Junk frames where duplicate frame flags
#OpenDMLSource("painting test for avisynth.avi") # Junk frames where duplicate frame flags

# Info() # This actually forces AviSource to work as it modifies the input frame
# BUT NOT with below Trim()

#DirectShowSource("painting test for avisynth.avi") # Seems to work (but lots of seeking probably)

#Trim(806,-1) # Frame before 2nd KEY frame
return last

EDIT:
From Avisynth v2.6 Docs: built-in OpenDML code (taken from VirtualDub)
Stange that Vdub current and even VDubMod load/play clip OK, whereas Avisynth dont, looks like an AviSource bug to me,
have not checked against Avisynth v2.58, dont seem any point as above unresolved problem link was back in 2007.

EDIT: And just so I dont forget where I got it, here the decoder only tsc2 codec from TechSmith: https://support.techsmith.com/entries/22849238-Camtasia-Studio-Download-the-TSC2-standalone-codec

EDIT:
Looks like it may be connected to 'dropped_frame's, perhaps somewhere near here

PVideoFrame AVISource::GetFrame(int n, IScriptEnvironment* env)

[eduardobedoya]

Thanks StainlessS,
Yes, Techsmith is a very compressed video format, but it remains lossless, I guess it uses that kind of artifacts when recording, because the filesize is very small when most of the screen remain the same while recording, whereas when the screen is zoomed and panned very often while recording the file size is quite large (both files with the same duration).

Sure man, use the link as you want, pls let me know if there is some improvement, I saw the other post with the same problem, but that guy never upload any sample right? I hope you could deliver this sample to good hands.

On the other hand, Its possible that I reencode all the videos in order to avisynth to read them? what encoder do you suggest? a MPEG kind? there is some lossless alternatives? sometime ago I tried lots of codecs to decide whichone use to record the speedpaintings, Techsmith was the best.

PD: what is the tsc2 decoder link for?

[StainlessS]

what is the tsc2 decoder link for?
so I know where I got it, and for anyone else eg developer wanting to test out problem.

Find here, new version RT_Stats (without FrameSel) and update scripts.

Tried Fsel_Eduardobedoya_Batch.avs using DirectShowSource on TSCC, works but very slow (instantiates codec
on every frame and sends bucket loads of identical text to DebugView). Unfortunately, it delivers
wrong frames (ie guesses, as I said earlier), so of no use.

If you convert TSCC clips to either RGB compressed or YV12, maybe HuffYUV or UT_Video codecs, will work fine.
Here is update script and dll, works great. LINK REMOVED

Also included a script to remove duplicate sequences from a clip.
Tested using 3 trailer type clips with some differing sequences in each, joined them together and it removes the duplicate seqs.

You could try this to convert to UT_Video, named 'Whatever.bat'

setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get input file, No terminating Backslash, "." = current directory
set INDIR="."


REM Where to place output file, No terminating Backslash.
set OUTDIR="D:\AVS\AVI"


FOR %%A IN (*.avi) DO (
REM %FFMPEG% -i "%INDIR%\%%A" huffyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"
%FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo "%OUTDIR%\%%~nxA.AVI"

)

Pause

EDITED: REM'ed out statement converts to HUFFYUV and PCM audio (if present)

You can get ffmpeg from eg MeGUI or google it.

EDIT: I've posted your sample TSCC clip in AviSynth 2.6.0 Alpha5 thread as a bug report sample. here:http://forum.doom9.org/showthread.php?p=1675858#post1675858

[eduardobedoya]

Thanks StainlessS
I upgrade the RT_Stats and the avs script.

I really get used to record using camtasia tsc2 codec, do you think it may be possible to solve that avisynth issue with tsc2 in the near future?

I have started to look for another lossless 24bit color screen capture codec (in order to avoid having to convert all tsc2 videos before using avisynth), do you have a screen capture codec to suggest?

About the convertion, I was thinking about a batch converter
what do you think about this app? it has CUDA acceleration.
http://www.mediacoderhq.com/
Is there another way to convert all 20 videos at once?
I am really concern about lossing quality in the convertion, do you think ffmpeg is my best choice, to which codec should I convert?

Thanks Advanced.

[StainlessS]

TSCC is not an editable format, meant for capture or archival purposes, you want to edit it, and so need in editable format.
Whether AviSource is fixed or not, it will be very slow (as it aint an editable format) and as my script does a lots of seeking
(as it has to, to do the job), even if Avisource is fixed, it may still suffer from seek problems (that may actually be fault of the
codec). Techsmith is probably as good as any other screen cap codec, but they will all suffer from same problem, ie not an
editable format. (EDIT: at least lossless ones will eg TechSmith, MJpeg, Motion Jpeg used in eg Camera or TV caps is not
lossless, and to some extent is editable, but as you want a lossless real time screen capture codec, they will likely not be editable).

The Batch .bat script provided will convert all AVI files in the same dir as batch script, at once. I edited the bat file with the REM'ed out
statement to do the same using HuffYUV codec instead (and with PCM audio if present).

You could even use ffmpeg to batch convert all you TSCC clips at once, directly to x264-mp4, I dont off hand know the command
for that but I'm sure someone else could offer that info. However, it does not solve your problem of wanting to remove the EDIT/UNDO's
from the clip.
I'm not really sure what is inside the TSCC clips, think that it's RGB but MediaInfo gives little info, and one app I tried mentioned YV24
although did not seem to be sure.
The UNDO removal avs script converts to REC601 YV12, this is so that whatever is input (YV12, RGB or YUY2 or whatever), uses the same settings
and dont need to keep changing with different clips, there is also a config option to extract edited clip with a ConvertToYV12 (for conversion to x264/mp4) if required.
Suggest convert to compressed RGB (will be without conversion, if it is indeed RGB to begin with), use edit/undo script
(which will only convert to YV12 during detection and extraction script creation) and then use resultant scripts as input to whatever you choose to render
eg x264/mp4.

EDIT:
I used to use MediaCoder (not the hq version) some time ago, but got annoyed with it for some reason and no longer use it,
I'm sure it will be just fine if you abandon the idea of editing out the UNDO's (or decide to do it by hand in MediaCoder, likely a lot more
work than already suggested options).

[StainlessS]

Seems that content of TSCC2 is YV24 (rec601 I think, 16 -> 235)

Tried conversion of TSCC to YV24 uncompressed, managed to do it but U and V were crossed over,
seems to be some confusion about which way around they should be.
Suggest convert to RGB, almost no difference in colorspace conversion and a heluva lot less bother than YV24,
results of a little experimentation below.





setlocal

REM Where to Find ffmpeg
set FFMPEG="C:\BIN\ffmpeg.exe"

REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR="."


REM Where to place output file, No terminating Backslash.
set OUTDIR="D:\AVS\AVI"


REM Converts all below input formats in INDIR to avi files in OUTDIR directory (could add to below formats).
FOR %%A IN (*.wmv *.asf *.mpg *.mpeg *.avi *.flv *.mov *.mp4 *.m4v *.RAM *.RM *.mkv *.TS) DO (



REM Uncomment ie REMOVE 'REM' from beginning of one of below lines (And REM out the currently Un-commented line).



REM Convert Video to forced RGB24 (ULRG) UT_Video Codec, audio PCM 16 Bit (if Present)
%FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -pix_fmt rgb24 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to UT_Video Codec (YV24 and YV12 to ULY0(YV12), YUY2 to ULY2, RGB to ULRG), audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec utvideo -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to forced RGB24 HuffYUV Codec, audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec huffyuv -pix_fmt rgb24 -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"


REM Convert Video to HuffYUV Codec (if YUV then YUY2, Else RGB), audio PCM 16 Bit (if Present)
REM %FFMPEG% -i "%INDIR%\%%A" -vcodec huffyuv -acodec pcm_s16le "%OUTDIR%\%%~nxA.AVI"



)


Pause


By the way, a UT_Video compressed YV12 version of your 27MB TechSmith clip is 2.57GB in size, and in RGB is 4.53GB,
but they are editable.

EDIT: Seems Virtualdub has a screen capture mode, might be worth a look, see the help, Video Capture.

[StainlessS]

Not sure if this will work for you, as I have read somewhere that Windows 8 64 bit, broke Video For Windows interface
(used by VDub).

If you find that whatever program you want to use to encode (eg Premiere, MeGUI), cannot accept avs files,
then you could batch process them all to edited AVI files.

See Here: http://forum.doom9.org/showthread.php?p=1628221&highlight=script#post1628221

Forgot all about doing that.

After tscc2 conversion to AVI using eg ffmpeg .bat,

you could then process resultant clips using my avs batch script for the EDIT/UNDO removal

then, copy the resultant avs files to another empty directory (just the avs files),
so that they are NOT in same place as original AVI files (reason hilited at end of last
post in link above).

then process the copied AVS using another batch script (as in above link) with the
directories edited to suit.
this would enable you to do an entire job lot in 3 batch processes.

Batch file from above link repeated here
VdBatch.bat

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


You need to set up settings eg codec, save settings then execute the batch file, and it will
do the lot, pulling in the avs files (results of EDIT/UNDO script), and then create AVI's in the output dir.

[eduardobedoya]

I test some codecs to convert the tsc2 avis, the
UTVideo in ULRG mode is the only one that has good color but it is 5.2gb, Lagarith codec is also RGB and it is 4.2 gb

I regret myself had to install Mediacoder, I dont know why but after installed it avisynth dont work anymore
when dropping the avs into virtualdub it show a message "AVI Import Filter error: (Unknown) 80040154"
I google it and in this thread
http://forum.videohelp.com/threads/302249-VirtualDub-AVI-Import-Filter-error-(Unknown)-80040154
people suggest run ccleaner (I would not like to do that) I upgraded the K-lite codec, but the message ramins, what can I do?

EDIT: I unistalled avisynth n virtualdub (delete its folder) then installed ccleaner run it, then reinstall avisynth (and its plugins) n virtualdub, but I still get that message when dropping the avs into virtual dub "AVI Import Filter error: (Unknown) 80040154"


Thanks advanced.

[eduardobedoya]

Since I cant get virtualdub working with the batch avs that you provided, I tried dropping the batch avs into Media Player Classic, it worked, here are the results:


Fsel_Eduardobedoya_Batch.Log
CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000


1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi
QWIK Scan DBase creation = 227.69 Secs (3.79 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 562.68 Secs (9.38 Mins) InFPS=4.09 OutFPS=2.62
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi Tot File Time = 790.98 Seconds (13.18 Mins)


DONE



TOTAL Time = 790.98 Seconds (13.18 Mins)







painting test for avisynth Lagarith_SelectFrames.AVS

Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last




What can I do from here since I can not use virtualdub??? is it normal that Media Player Classic take 13min to process a 2.5 min file? or virtualdub could do it faster?

Thanks advanced

PD:
There is a line in your Batch.avs
####
CONVYV12 = True # True ConvertTo YV12 in created script, Else False leaves as is
####
I changed it to "False" I would like to keep the color in RGB.

Thanks


Edit: I tried changing the virtualdub version (I was using the 64amd version), and using the 32 bit version it worked ok, (its strange that I was able to work with the 64bit amd version before and now I cant) anyway, I processed the avs again...

Fsel_Eduardobedoya_Batch.Log

CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000

1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi
QWIK Scan DBase creation = 182.26 Secs (3.04 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 447.19 Secs (7.45 Mins) InFPS=5.15 OutFPS=3.30
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi Tot File Time = 629.93 Seconds (10.50 Mins)


DONE



TOTAL Time = 629.93 Seconds (10.50 Mins)





painting test for avisynth Lagarith_SelectFrames.AVS

Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last



then what...? I read what you post above, but it looks kinda strange to me. I just cant understand if I got to put the "=" sings, Sorry if I am not kinda smart enough understanding codes, I tried this:

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory ####what is this??????
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI ####I guess this line is replaced by the next line, right???
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=C:\Program Files (x86)\VirtualDub-1.10.4\VDub.exe"


REM Where to Find VDUB Settings #####I have no clue where are the *.vcf file placed
set VDS=C:\Program Files (x86)\VirtualDub-1.10.4\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=F:\2 AVISYNTH\INPUT


REM Where to place output file, No terminating Backslash.
set OUTDIR=F:\2 AVISYNTH\OUTPUT

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit ##is it ok?
%VD% /s %VDS% /c /b F:\2 AVISYNTH\INPUT F:\2 AVISYNTH\OUTPUT /r /x

Pause



I save this file as a VdBatch.bat and place it inside F:\2 AVISYNTH\INPUT with the "batch avs" file and execute it via cmd?


PD: Pls StainlessS, do you have any clue why I cant use VirtualDub amd64 anymore? there is a way that I could get it working again? because if the 32bit version render in 10min perhaps the 64bit version could render in 6minutes??? This is very important since I have about 20 hours of video to process. I would like install avisynth+ with VDub amd64bit:
http://forum.doom9.org/showthread.php?t=168856
what do you suggest??

PD2: It is ok that I had converted the video using Lagarith? perhaps there is another RGB lossless codec that could be processed much faster with avisynth??? at this time I would prefer the performance factor than file size.

Thanks advanced.

[StainlessS]

Glad you got it working, as I said, W8 64bit is supposed to be broken for Video For Windows
so I'm surprised it worked before MediaCoder install.

What can I do from here since I can not use virtualdub??? is it normal that Media Player Classic
take 13min to process a 2.5 min file? or virtualdub could do it faster?

It is Avisynth that does the processing, as soon as Media Player OR Vdub displays first frame, then it is already complete.

From this
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480

Perhaps below would work if differences do not vastly change between your TSSC clips.

ldM=0 hdM=0 ldL=0.005000 hdL=0.005000 ldS=0.004000 hdS=0.004000 ldR=0.003000 hdR=0.003000 MAX_LDIFF=0.004000

If misses some UNDO's in a clip, then redo with original values.

10:50 secs processing for your 3min clip is not so bad considering that you scan up to 2304 HD frames for each output frame,
but above mods to thresholds should increase speed considerably (but you need to check that the UNDO's were correctly detected/removed).
EDIT: The 1st script version output 1 frame about every 12 seconds, so with 1474 output frames it would perhaps have
taken about 5 hours.

As you have chosen to use RGB, I may have a go at using an RGB fingerprint, should I think be a little faster,
but for movie clips, probably a lot faster (there are so many possible matches in your clips as many frames are
identical/almost identical).

do you have any clue why I cant use VirtualDub amd64 anymore?

No idea, I use XP32bit with no intention of changing to W7, W8 (most likely change, Linux).

REM VirtualDub Directory, No terminating Backslash, "." = current directory ####what is this??????
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"

Just means that set VDDIR="." # would set VDDIR to the current directory, dont worry about it.
'.' just means current directory, '..' means parent of current directory, in command line processor, eg
dir .
dir ..

shows you files in current directory, and the directory above current directory respectively.

The only paths you might want to change hilited in blue, those in Magenta are currently set RELATIVE to the current directory,
ie relative to the directory that the bat file is in.


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\NON-INSTALL\VDUB\VDUB_PLAIN"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


NOTE, UT_Video is I think faster than Lagarith, especially for decode and as you will be doing LOTS
of seeking and decoding during frame scanning, it is problably the better choice, check it out for yourself
how long everything takes.

[eduardobedoya]

Hi StainlessS I tried to get what you posted, this is what I did:

I opened virtualdub, then.. (file>save processing settings)
and save the file to same directory as CURRENT version of VD (C:\Program Files (x86)\VirtualDub-1.10.4) as VD.vcf. (the result file was VD.vcf.vdscript)

Then I placed this VdBatch.bat inside F:\2 AVISYNTH


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR=C:\Program Files (x86)\VirtualDub-1.10.4

REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"

REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=F:\2 AVISYNTH\INPUT


REM Where to place output file, No terminating Backslash.
set OUTDIR=F:\2 AVISYNTH\OUTPUT

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause




finally a run the VdBatch.bat by double click it, it sent what it looked to be the command promp with the script text and at the end a "puase" and a text saying "press any key to continue"
I press space and the black screen closed, then nothing happens in about half hour.

PD: I tried replacing the VD.vcf.vdscript file with a VD.vcf file, and the same result

What I am doing wrong??? please could you post just exactly how you did it in your particular case, in your PC??? Thanks Advanced

PD: It is important to configure something inside Virtualdub before running the bat file??? perhaps there are some important thread that I should read to learn how to edit settings inside Virtualdub. Thanks advanced.

[StainlessS]

You lost the double quotes in a number of places.


setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR="F:\2 AVISYNTH\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR="F:\2 AVISYNTH\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause



try this

EDIT: Settings should be saved as VD.vcf not VD.vcf.vdscript, dont know if VirtualDub added the 'vdscript' bit. Either save as
VD.vcf or edit bat file

set VDS=%VDDIR%"\VD.vcf.vdscript"


Also, if bat file is 1 directory above both INPUT and OUTPUT folders, then could have left original bat script INPUT and OUTPUT
lines alone eg


set INDIR=".\INPUT"



set OUTDIR=".\OUTPUT"


OK, I see that I have to keep adding EDITs to answer your edits :)

You need to set up eg CODEC at least, on video Menu, they (settings) will be saved in the settings file, VD.vcf

[StainlessS]

OK, no more edits.

Load a clip into Vdub, set codec, save VD.vcf into you VirtualDub folder.
run this bat

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause


Only the path to virtualdub folder has changed since original posted bat file.
The bat file needs to be in your "F:\2 AVISYNTH" folder.

EDIT: You can move/rename the "2 AVISYNTH" folder anyhere you like and it will still work, so long as it still contains the bat file and both
INPUT and OUTPUT directories.

[eduardobedoya]

The same result I did everything as you said, but I still got the same result, when I double clik VdBatch.bat I get the exact same result, it sent what it looked to be the command promp with the script text and at the end a "puase" and a text saying "press any key to continue"
I press space and then the black screen closed, then nothing happens in about half hour.
I have tried to launch the bat file via Win Run>cmd, and the same result.

I modified the Bat file as you said,

setlocal

REM VirtualDub Directory, No terminating Backslash, "." = current directory
set VDDIR="C:\Program Files (x86)\VirtualDub-1.10.4"


REM Where to Find VDUB with GUI
REM set VD=%VDDIR%"\VirtualDub.exe"

REM Where to Find VDUB command line.
set VD=%VDDIR%"\VDub.exe"


REM Where to Find VDUB Settings
set VDS=%VDDIR%"\VD.vcf"


REM Where to get input files, No terminating Backslash, "." = current directory
set INDIR=".\INPUT"


REM Where to place output file, No terminating Backslash.
set OUTDIR=".\OUTPUT"

REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit
%VD% /s %VDS% /c /b %INDIR% %OUTDIR% /r /x

Pause

What does REM mean??

I also opened a video inside VirtualDub and clicked in Video>Compression, and choose Lagarith, then I saved the processing settings as VD.vcf (the previous time it was Virtualdub who was triying to save the processing settings as *.vdscript file) inside the VirtualDub directory.

And I placed the VdBatch.bat file inside F:\2 AVISYNTH

But the same result, what I am doing wrong??? THanks advanced.

PD: I am placing the resulted avs files of the second UNDObatch process inside the "F:\2 AVISYNTH\INPUT" folder, do I have to place the frames.txt files also???

By the way, I did convert the tsc2 avis into a Lagarith avi to make this try, all that I have posted have been using Lagarith avi. You said that if I try UTvideo ULRG it could be faster than Lagarith, so I convert the tsc2 file into UTvideo ULRG and try to process it with avisynth by running the Eduardobedoya_Batch inside virtual dub, but when the open windows appears and I pick up the UTvideo ULRG it shows this message:

VirtualDub Error
Avisynth open failure:
AVISource: couldn't locate a decompressor for fourcc ULRG
(C:\Users\Eduardo\Desktop\Avisynth\Eduardobedoya_Batch_3\Fsel_Eduardobedoya_Batch.avs, line 103)
([GScript], line 16)
([GScript], line 22)
(C:\Users\Eduardo\Desktop\Avisynth\Eduardobedoya_Batch_3\Fsel_Eduardobedoya_Batch.avs, line 86)

Pls help.

[StainlessS]

REM is short for REMark, ie comment follows it.

ffmpeg converts it to UT_Video, however you need a UT_Video codec to decode for Video For Windows.
http://www.videohelp.com/tools/Ut-Video-Codec-Suite

What you have done according to previous post looks correct to me, should not need to move frames files to the INPUT directory.
Try load any avs file from INPUT directory into VDUB directly, if there is some problem it should present an Avisynth error message in VDub.
EDIT: The missing UT_Video codec is problably the vdub batch problem if that was what the TSCC files were converted to.
EDIT: It is the ffmpeg conversion that best output UT_Video for the UNDO removal, can use whatever codec you like for the
VD batch output.

.

[eduardobedoya]

I tried to load the avs file from INPUT directory into VDUB directly, it shows this message

Avisynth open failure:
Script error: there is no function named "FrameSel"
(F:\2 AVISYNTH\INPUT\painting test for avisynth Lagarith_SelectFrames.avs, line4)

I tried to load it from the first folder where the videos where placed, it shows the same message

Avisynth open failure:
Script error: there is no function named "FrameSel"
(C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_SelectFrames.AVS, line4)




Note that I am just triying everything with a single video, (it has been converted to Lagarith),I recently tried to test with more videos, using other codecs, I installed UTvideo and convert the tsc2 into UTvideoULRG then process it with the UNDO batch, it do the task in 3min aprox (good improve) but in the last batch when I try to run the VdBatch with the new UTvideo avs file in the INPUT folder, I got the same result.

Why this final step does not work???

I converted that tsc2 video to Lagarith and UTVideoULRG using camtasia studio.
PD: I just have ffdshow Video Codec installed, is that ffmpeg???

[StainlessS]

You need FrameSel plugin for avisynth, I think I included it in 1st zip I posted, not included since then.
Get here and put in your plugins dir. http://forum.doom9.org/showthread.php?t=167971

If you are converting with camstudio, you dont need ffmpeg.

[eduardobedoya]

I do have FrameSel v2.6 dll in my avisynth plugin folder

I have avisynth 26 installed as you suggested.

[StainlessS]

Then you have some kind of Avisynth problem, re-install.
Copy only required plugins to plugins dir, and establish that avisynth is working, no
point in trying to get the rest working without working avisynth.
You only put 1 of each of the plugs in plugins dir, ie only v.26 version not v2.5.

Before install try this


Version()


It should show Avisynth version. v2.5 cannot load v2.6 plugins.
EDIT: Only thing I can think of is that you actually have Avisynth v2.58 installed.

[eduardobedoya]

I tried the version () code (creating a avs with that code and open it in virtualdub), here is the result


AviSynth 2.60, build:Sep 18 2013 [17:36:36]
C 2000-2013 Ben Rudiak-Gould, et al.
http://www.avisynth.org


What I have noticed is that even when I installing Avisynth 2.6 it gets installed inside a folder name Avisynth 2.5, I guess it does not matter

The files inside my current avisynth installation directory (C:\Program Files (x86)\AviSynth 2.5\plugins) are:

by default:
colors_rgb.avsi
DirectShowSource.dll
TCPDeliver.dll

Installed by me
FrameSelect26.dll
Gscript.dll
RT_Stats26.dll


I desinstall and reinstall Avisynth 2.6
I get the defult plugins:
colors_rgb.avsi
DirectShowSource.dll
TCPDeliver.dll

and I added only the FrameSelect26.dll (84kb)


Then I lunched the avs file but get the same error message
Avisynth open failure:
Script error: there is no function named "FrameSel"
(C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth Lagarith_SelectFrames.AVS, line4)

I dont know what is wrong with my system, should I try to install AviSynthPlus-r1576.exe?? or AviSynth 2.5.8????
http://avisynth.nl/index.php/Main_Page

Thanks Advaced StainlessS

[StainlessS]

Oops, sorry.
Seems I may have supplied you with FrameSelect plugin rather than FrameSel plugin.
FrameSelect is older version, needed to change args in non compatible way, and so I renamed the plugin
to avoid compatibility problems. Delete the older FrameSelect, you need FrameSel plugin, v2.6.
See MediaFire below in sig.

[eduardobedoya]

Thanks StainlessS
I was already thinking about installing a virtual machine


Thanks man, it worked, I did a TEST>>>


First Here is the UNDO Batch Log:

Fsel_Eduardobedoya_Batch.Log
CONVYV12=False
ldM=1 hdM=1 ldL=0.005000 hdL=0.005000 ldS=0.005000 hdS=0.005000 ldR=0.005000 hdR=0.005000 MAX_LDIFF=0.004000


1/1 ] Processing C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi
QWIK Scan DBase creation = 49.06 Secs (0.82 Mins)
184,195 Skipping earlier duplicate sequence and UNDOS
368,536 Skipping earlier duplicate sequence and UNDOS
612,706 Skipping earlier duplicate sequence and UNDOS
847,886 Skipping earlier duplicate sequence and UNDOS
910,961 Skipping earlier duplicate sequence and UNDOS
1284,1345 Skipping earlier duplicate sequence and UNDOS
1378,1438 Skipping earlier duplicate sequence and UNDOS
1506,1535 Skipping earlier duplicate sequence and UNDOS
1544,1546 Skipping earlier duplicate sequence and UNDOS
1702,1713 Skipping earlier duplicate sequence and UNDOS
1716,1797 Skipping earlier duplicate sequence and UNDOS
1983,2049 Skipping earlier duplicate sequence and UNDOS
2085,2126 Skipping earlier duplicate sequence and UNDOS
2183,2285 Skipping earlier duplicate sequence and UNDOS
MDeltaMax=0 LDeltaMax=0.002777 SDeltaMax=0.001762 RDeltaMax=0.001480
(Above, maximum values of ldM, hdM, ldL, hdL, ldS, hdS, ldR, hdR, that would have worked, but may have been faster)
Kept 1474 of 2304 frames [1920x1080 153.60 secs (2.56 Mins) @ 15.00 FPS]
QWIK SCAN 127.67 Secs (2.13 Mins) InFPS=18.05 OutFPS=11.55
C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi Tot File Time = 177.32 Seconds (2.96 Mins)

DONE


TOTAL Time = 177.33 Seconds (2.96 Mins)


Here is the resulted AVS file:


Avisource("C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG.avi")
CmdFrames="C:\Users\Eduardo\Desktop\FOR AVISYNTH\painting test for avisynth UTVideoULRG_Frames.txt"
(Exist(CmdFrames)) ? FrameSel(cmd=CmdFrames,reject=False) : NOP

Return Last



Then here is the message that appeared in the command promp when launching VdBatch.bat

F:\2 AVISYNTH>REM Where to get input file, No terminating Backslash, "." = current directory

F:\2 AVISYNTH>set INDIR=".\INPUT"

F:\2 AVISYNTH>REM Where to get input files, No terminating Backslash.

F:\2 AVISYNTH>set INDIR=".\OUTPUT"

F:\2 AVISYNTH>REM Execute Vdub using script, clear command queue, batch mode, source dir, dest dir, process queue, exit

F:\2 AVISYNTH>"C:\Program FIles <x86>\VirtualDub-1.10.4""/VDub.exe" \s "C:Program Files <x86>\VirtualDub-1.10.4""\VD.vcf" /c /b ".\INPUT" ".\OUTPUT" /r /x VirtualDub CLI Video Processor Version 1.10.4 <build 35491/release> for 80x86 Copyright <C> Avery Lee 1998-2009. Licensed under GNU General Public License

AVI: Opening file "F:\2 AVISYNTH\INPUT\painting test for avisynth UTVideoULRG_SelectFrames.AVS"
AVI: Avisynth detected. Extended error handling enabled
Beginning dub operation
Dub: Input <decompression> format is : XRGB8888.
Dub: Output <compression> format is: RGB888.

[eduardobedoya]

Thanks a lot StainlessS, it worked!

I used UTVideoULRG because it keep the color very similar to the original in RGB
Also I changed this line, of the Undo BATCH.AVS that you provided:
####
CONVYV12 = False # True ConvertTo YV12 in created script, Else False leaves as is
####in orther to keep the video in RGB

But when I run the VdBatch.bat the color of the OUTPUT video drastically changed
I guessed it depends on the video>compression properties that you choose on Virtualdub (VD.vcf)
So I changed Virtualdub compression properties to tscc codec, and then it rendered the video in a good RGB color,

Thanks a lot StainlessS, I just have some doubts:

Is there a big difference between converting the original tsc2 video to UTVideoULRG or UTVideoULY0 codecs for running the UNDOBatch.avs??
I mean, cuz, the sample tsc2 26mb video converted into UTVideoULRG is 5.2gb and into UTVideoULY0 is 2.7gb,
So is there any problem if I convert the original tsc2 into UTVideoULY0 video file in order to run the UNDObatch in YUV and then re-convert it with the VDBatch.bat into RGB again? Do I loss some color information in the final OUTPUT RGB video by doing this way??? Will the batch process be run in YUV colors so it could loss some accuracy in color detection (similar color painting strokes)??

I run the test and it really cut all the UNDOS, but, is there any way to merge all equal frames to a single frame?
I mean, cuz, the result video got in fact all the painting strokes without UNDOS, but there is so much space(time)(equal frames) between each different stroke, is there a way to merge all that spaces,
so the time between each different painting stroke could be just one frame?
For example:
the original video was
AbcdAbcAbcdeABCDefDefgDefDefgDEFGHijHijkHIJKLMnopMnopMNOPQ
the applying your avs it turns into
AABCDDDDEFGHHIKJLMMMNOPQ
is there a way to turn it into (perhpas using the same batch.avs)
ABCDEFGHIJKLMNOPQ

Thanks a lot StainlessS!!!

[StainlessS]

But when I run the VdBatch.bat the color of the OUTPUT video drastically changed
I guessed it depends on the video>compression properties that you choose on Virtualdub (VD.vcf)
So I changed Virtualdub compression properties to tscc codec, and then it rendered the video in a good RGB color,

I guess that has something to do with color matrix used/assumed, see Vdub menu "Video/ColorDepth", perhaps playing around with
'decompression format' and 'output format to compressor/display' is required, I would be reluctant to go back to TSCC codec, we already
established it to be unreliable for eg seeking, suggest find alternative. I dont usually touch HD, I dont have BD player or other source
of it, perhaps others could advise on best settings for VD 'decompression format' and 'output format to compressor/display', to
pass through RGB untouched to UT_Video ULRG. I dont really see why pulling in RGB and compressing to UT_Video RGB should give
a problem. What are you using to judge the 'drastically changed output', did you look at the input to UNDO script too ?

UTVideo ULY0 is rec601 YV12, ie half resolution chroma, that is why is is smaller compressed file, half chroma detail gone.

but, is there any way to merge all equal frames to a single frame?

I initially did that but assumed that you really would not want it, ie results will be a clip of maybe 3 or 4 seconds out of your 3 min clip.
OK, I'll put it back like that.

EDIT: From your last post

Dub: Input <decompression> format is : XRGB8888.
Dub: Output <compression> format is: RGB888.

Looks like straight through RGB in -> out (RGB32 to RGB24), why drastically changed, no idea.

[eduardobedoya]

UTVideo ULY0 is rec601 YV12, ie half resolution chroma, that is why is is smaller compressed file, half chroma detail gone.

So it means that runing the UNDOBatch.avs with a UTVideoULRG video and not converting to YV12 could be more accurate in detected diferent color strokes?? (subtle diferent color strokes)
compared to ULY0????

[StainlessS]

Yep..

[StainlessS]

OK, I've added framecount limiting for static sequences, with KEEPMAX==1 (default) will produce output clip with
148 frames (on your sample, I think) with no pauses between edits. setting to eg 15 will give max 1 second pause between edits
when 15FPS clip.

Added a ranges.txt file, created during resultant avs load, informational only.
Removed ConvertToYV12 functionality.
Reduced number of tolerance settings (+ve and -ve tolerances set same).

here: LINK REMOVED

script here:

# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# Remove Undos from painting captures
# ############################



# Alter below to Config
########################################################################
########################################################################
########################################################################

ScanAheadSecs = 10 * 60 # Search ahead range in seconds

#### BELOW SETTINGS WORK FINE WITH SUPPLIED SAMPLE WITH A LITTLE LEEWAY.
# Below settings, as close to zero as possible (faster but might miss matches), used by QWIK scan routines to identify possible match frames.
dM = 1 # INT, Max diff (+ve and -ve tolerance) of YMedian between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dL = 0.005 # Float, Max diff (+ve and -ve tolerance) of AverageLuma between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dS = 0.005 # Float, Max diff (+ve and -ve tolerance) of YPlaneStdev between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
#
dR = 0.005 # Float, Max diff (+ve and -ve tolerance) of YInRange between i frame and possible duplicate (may vary with clip, 0.0 for EXACT match)
####

####
# Below used to identify if candidate frames found by QWIK SCAN routines are good match.
MAX_LDIFF = 0.004 # Float, LumaDifference between candidate frame and duplicate (average pixel diff rather than frame diff, 0.0 Exact match)
####
# Maximum number of frames to keep per static sequence (0 = no limit)
KEEPMAX=1
####

########################################################################
########################################################################
########################################################################

FSEL_TITLE="Select AVI files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))

NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_WriteFileF(LOG,"%s\n",LOG,Append=False)
S=RT_String("dM=%d dL=%f dS=%f dR=%f MAX_LDIFF=%f",dM,dL,dS,dR,MAX_LDiff)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_String("KeepMax=%d",KeepMax)
RT_DebugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s\n",S,Append=True)
GSCript("""
TOTSTART = RT_Timer
For(i=0,NFILES-1) {
START = RT_Timer
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n%d/%d ] Processing %s",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)

DropUndoSequences(FN,ScanAheadSecs,dm=dM, dL=dL, ds=dS, dr=dR,LDThresh=MAX_LDIFF, log=LOG, keepmax=KEEPMAX)

T = RT_Timer - START
S=RT_String("%s Tot File Time = %.2f Seconds (%.2f Mins)",FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
}
T = RT_Timer - TOTSTART
S=RT_String("\n\nTOTAL Time = %.2f Seconds (%.2f Mins)\n",T,T/60.0)
RT_DebugF(S,name=myName)
S=RT_String("\n\nDONE\n\n%s\n",S)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_StrReplace(S,Chr(10),"\n")
""")
Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)


Function DropUndoSequences(String "AviName",Float "ScanAheadSecs",int "dM",Float "dL",Float "dS",Float "dR",Float "LDThresh",String "Log",int "KeepMax") {
myName="DropUndoSequences: "
ScanAheadSecs=Float(Default(ScanAheadSecs,10.0*60.0))
dM = Default(dM,1)
dL = Float(Default(dL,0.005))
dS = Float(Default(dS,0.005))
dR = Float(Default(dR,0.005))
LDThresh = Float(Default(LDThresh,0.004))
LOG = Default(LOG,"DropUndoSequences.LOG")
KeepMax=Default(KeepMax,0) # 0 = no limit
Assert(Exist(AviName),myName+AviName+" Does Not Exist")

Avisource(AviName).ConvertToYV12() # So we use the same settings whether orig RGB or YV12

Assert(ScanAheadSecs>0.0,myName+"ScanAheadSecs Must be greater than zero")
Assert(dM>=0, myName+"ldM Must be greater than or equal to zero")
Assert(dL>=0.0,myName+"ldL Must be greater or equal to zero")
Assert(dS>=0.0,myName+"ldS Must be greater or equal to zero")
Assert(dR>=0.0,myName+"ldR Must be greater or equal to zero")
Assert(KeepMax>=0,myName+"KeepMax Must be greater or equal to zero")

PathAndNode = RT_FilenameSplit(AviName,7) # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
ScriptFile=PathAndNode+"_SelectFrames.AVS"
ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
PNDB=PathAndNode + "_Prev.DB"
RT_FileDelete(CMDFrames) # Delete any existing frames file
RT_FileDelete(Ranges) # Delete any existing Ranges file
### TEMPLATE script

FrameSel_Select="""
fn="%s"
Avisource(fn)
PathAndNode="%s"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last
"""

###
Select_S = FrameSel_Select
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,PathAndNode) # Insert filenames and ConvertToYV12

START = RT_Timer
RT_QwikLumaScanCreateDB(DB,prevdb=PNDB,nextdb="",debug=true)
T= RT_Timer - START
S = RT_String("QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_WriteFileF(LOG,"%s",S,Append=True)
OCNT=0
GSCript("""
START = RT_Timer
LastFrame=FrameCount-1
FivePercFrames = Round(0.05 * FrameCount)
RT_DebugF("QWIK Scanning file ... Please Wait",name=myName)
MDeltaMax = 0 LDeltaMax = 0.0 SDeltaMax = 0.0 RDeltaMax = 0.0
Kept = 0

For(i=0,LastFrame) {
WrStart = i WrEnd = i # Init to keep 1 frame (if not found)
EndLimit = Min(i + ScanAheadFrames,LastFrame) # Searching Endlimit to i+1 inclusive (downwards)
# Scan for static sequence
FindStart = i + 1
for(k=FindStart,EndLimit) {
dif=RT_LumaDifference(Last,Last,n=i,n2=k)
if(dif <= LDThresh) {
WrEnd = k
} Else {
FindStart = k
k = EndLimit # break
}
}
# Here:- WStart to WrEnd are similar, search from EndLimit downward to FindStart for later dupe sequence
For(j=EndLimit,FindStart,-1) {
# Search for i frame dupe nearest to EndLimit but higher than FindStart
j = RT_QwikLumaScanGetNear(DB,PNDB,j, \
ldm = dM, ldl = dL, lds = dS, ldr = dR, \
maxdistance=j-FindStart, Inclusive=True, \
findframe=i)
if(j > FindStart) { # We found a candidate frame
dif=RT_LumaDifference(Last,Last,n=i,n2=j) # Ave pixel diff between i frame and candidate
if(dif <= LDThresh) {
# We found a single duplicate frame and will drop earlier sequence, look for start of similar frames in this sequence
DupeEnd = j
DupeStart = j
for(k=j-1,FindStart+1,-1) {
tdif=RT_LumaDifference(Last,Last,n=i,n2=k)
if(tdif <= LDThresh) {
DupeStart = k
} Else {
k = FindStart # break
}
}
RT_DebugF("%d,%d --> %d,%d Matched LumaDif=%f",WrStart,WrEnd,DupeStart,DupeEnd,dif,name=myName)
S=RT_String("%d,%d Skipping earlier duplicate sequence and UNDOS",WrStart,DupeStart-1)
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
RT_Ystats(Last,i,flgs=$78,lo=64,hi=192,Prefix="I_") # Default DBase create, YInRange lo and hi
RT_Ystats(Last,j,flgs=$78,lo=64,hi=192,Prefix="J_")
I_yInRng = I_yInRng * 255.0 J_yInRng = J_yInRng * 255.0 # Make range same as other YStats (YInrange is 0.0->1.0)
MDelta=Abs(I_yMed-J_yMed) LDelta=Abs(I_yAve-J_yAve)
SDelta=Abs(I_yStdev-J_yStdev) RDelta=Abs(I_yInRng-J_yInRng)
MDeltaMax=Max(MDeltaMax,MDelta) LDeltaMax=Max(LDeltaMax,LDelta)
SDeltaMax=Max(SDeltaMax,SDelta) RDeltaMax=Max(RDeltaMax,RDelta)
RT_DebugF("MDelta=%d LDelta=%f SDelta=%f RDelta=%f",MDelta,LDelta,SDelta,RDelta,name=myName)
WrStart = DupeStart
WrEnd = DupeEnd
j = 0 # Early break
}
}
}
Keep = (WrEnd-WrStart+1)
if(KeepMax>0 && Keep>KeepMax) {
# Limit frames to keep
RT_DebugF("KeepMax Limiting: %d frames limited to %d (%d,%d -> %d,%d)",Keep,KeepMax,WrStart,WrEnd,WrEnd-KeepMax+1,WrEnd,name=myName)
Keep = KeepMax
WrStart=WrEnd-Keep+1
}
Kept = Kept + Keep
if(Keep == 1) {
RT_WriteFileF(CMDFrames,"%d",WrStart,Append=True)
} else {
RT_WriteFileF(CMDFrames,"%d,%d",WrStart,WrEnd,Append=True)
}
i = WrEnd
If(i > (OCNT+FivePercFrames)) {
T= RT_Timer - START
RT_DebugF("%d ] Progress = %.2f%% InFrames=%d OutFrames=%d InFPS=%.2fFPS OutFPS=%.2f",
\ i,(i+1)*100.0/(LastFrame+1),i+1,Kept,(i+1)/T,Kept/T,name=myName)
OCNT=i
}
}
RT_WriteFileF(ScriptFile,"%s",Select_S,Append=False)
RT_FileDelete(DB)
RT_FileDelete(PNDB)
S=RT_String("MDeltaMax=%d LDeltaMax=%f SDeltaMax=%f RDeltaMax=%f",MDeltaMax,LDeltaMax,SDeltaMax,RDeltaMax)
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
S="(Above, maximum values of dM, dL, dS, dR, that would have worked, but may have been faster)"
RT_debugF("%s",S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
T = RT_Timer - START
FT=FrameCount / FrameRate
S=RT_String("Kept %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]", Kept, FrameCount,Width,Height,FT,FT/60.0,FrameRate)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
S=RT_String("QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f", T,T/60.0,FrameCount/T,Kept/T)
RT_DebugF(S,name=myName)
RT_WriteFileF(LOG,"%s",S,Append=True)
""")
Return 0
}

[eduardobedoya]

Thanks StainlessS I will try this last script with my 30min videos.
Did you see my last PM? I did a summary of all most subtle strokes in a 46mb video, I was wondering which strokes are posible to process, and which are not.

Thanks advanced.

[StainlessS]

Yes I DL your clip from last PM, thank you.
Am in process of converting to 3 channel fingerprint rather than Luma only.
Not working as good as Luma only as yet.

EDIT:
I was wondering which strokes are posible to process, and which are not.

You want to catch subtle differences, the main problem is that damn cursor that keeps jumping around the frame,
and its usually quite different to whatever its overlaying.

[eduardobedoya]

yes man, the cursor its a pain, there is no way to remove it, I did my best to kinda hide it, so I set the app to display only four little white dots (kinda pixel size) in crossshape that always appear on the screen.
Some times it changes to a circular or eliptical white shapes, but those can be removed by me, I mean I can set the app to avoid display those circular and eliptical cursors, so you can deal only with the crossshape one.

I will see if I can figure out a way to get rid of the crossshape cursor, because if I delete it, then I could not see where I am actually positioning the brush in the screen.

Thanks advanced.

[StainlessS]

I've implemented a new function RT_LumaPixelsDifferent() for this problem, gives a pixel count of number of pixels
that are different in any way (RGB converted to Luma_Y before compare).

Here output of test of a few frames from your original clip, 1st two numbers are frame numbers compared


00000076 432.54067993 [3936] RT_DebugF: 172:173 ] CDif=0.000000 LDif=0.000000 CNT=0
00000077 432.61959839 [3936] RT_DebugF: 172:174 ] CDif=0.000000 LDif=0.000000 CNT=0
00000078 432.69915771 [3936] RT_DebugF: 172:175 ] CDif=0.000340 LDif=0.000838 CNT=108
00000079 432.77612305 [3936] RT_DebugF: 172:176 ] CDif=0.000447 LDif=0.000912 CNT=189
00000080 432.85601807 [3936] RT_DebugF: 172:177 ] CDif=0.000433 LDif=0.000894 CNT=179
00000081 432.93295288 [3936] RT_DebugF: 172:178 ] CDif=0.000440 LDif=0.000899 CNT=188
00000082 433.00939941 [3936] RT_DebugF: 172:179 ] CDif=0.000348 LDif=0.000727 CNT=163
00000083 433.08856201 [3936] RT_DebugF: 172:180 ] CDif=0.001467 LDif=0.002735 CNT=659

So is comparing with frame 172, through to 180, seems that maybe anything above about 200 pixels changed
is an edit, 189 pixels changed just looks to me like a cursor move. Frame 180 is a smallish edit.

[eduardobedoya]

a particular solution for a particular problem?
Looks very accurate, Thanks StainlessS

The main cursor is a cross shape of four pixelsize white dots, I could get rid of it, but then I could not see where I am actually positioning the brush in the screen
Some times the cursor is shown as a circular or eliptical white shapes, I can easily set the program so they dont appear anymore, so that you can focus on dealing with the main cross shape cursor.

PD: I have a bunch of recorded painting sesions already trimmed by hand, I would like to test them out with your new script.

[StainlessS]

This shows changes from previous frame (including eg cursor movement)


SHOW=True

aviName="D:\avs\avi\in\painting test for avisynth.avi.AVI"
Avisource(AviName)
Prev=DeleteFrame(FrameCount-1).SelectEvery(1,-1)
Clipdelta(Prev,Last,true,SHOW)
ScriptClip("""RT_SubTitle("%d",current_frame)""")
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


EDIT: What looks like 4 white pixels of cursor, is more like a cloud of pixel changes.

EDIT:
Oh dear!, seems when you do an undo, it leaves some rubbish in the undo region.
This compares all frames with frame 372, play from 372 and see the rubbish left after undos.


FRAME=372

aviName="D:\avs\avi\in\painting test for avisynth.avi.AVI"
Avisource(AviName)

A=trim(FRAME,-1)
A=A.FreezeFrame(1,Last.FrameCount-1,0)
Clipdelta(a,Last,true)
ScriptClip("""RT_SubTitle("%d",current_frame)""")
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}

[eduardobedoya]

I never realize of that rubbish left areas
The circular and eliptical cursor will not appear anymore.

I am still looking for a way get rid of the crossshape cursor,
if the avs script will be more accurate by removing the crossshape cursor, then I will try find a way to get rid of it.

[StainlessS]

I am still looking for a way get rid of the crossshape cursor,
if the avs script will be more accurate by removing the crossshape cursor, then I will try find a way to get rid of it.

Yes very definitely more accurate, perhaps you could cajole the authors of your paint program to also fix the rubbish left after UNDO operation, there can be a lot of it, although might be invisible to the eye, is not so invisible to plugin, and messes up detection. Send the authors the script that shows the rubbish, perhaps they are unaware that their software has this problem.

I have not forgotten you, I am still working on a solution, hopefully within a few days I can have something that mostly works OK, even with the cursor and undo rubbish present.

[eduardobedoya]

of course, Ill wait until your next release
I am still looking for a way to get rid of the crossshape cursor, I have made a post in superuser and stackoverflow, asking for a third party appplication that could show a cursor (ico file) in the screen always, so I can disable the paint program crossshape cursor. Once I have only a ico file cursor I can tell camtasia to not record it. (I just dont know why this option just does not work with the crossshape cursor).
I will submit any news.
Thanks StainlessS

EDIT> I kinda get rid of the cloud crossshape cursor, message sent.

[StainlessS]

Saw that you were on-line yesterday.
I'm still working on this and have not forgotten you, (although I have had a few excursions doing other things just because
my head hurts when I keep banging it against the wall).
I have not as yet moved on to try your later clip with the single 3x2 pixel cursor, I'm still banging away at the more difficult
1st clip you provided.
Anyway, just thought I'de let you know that I have not (as yet) given up totally :)

[StainlessS]

Just had a look at the 3x2 cursor clip.
This shows differences from previous frame (already posted this previously)

SHOW=False

#aviName="D:\avs\avi\in\painting test for avisynth.avi.AVI"
aviName="D:\avs\avi\in\capture-1.avi.AVI"

Avisource(AviName)

Prev=DeleteFrame(FrameCount-1).SelectEvery(1,-1)
Clipdelta(Prev,Last,true,SHOW)
ScriptClip("""RT_SubTitle("%d",current_frame)""")
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


and this (already posted) shows difference between frame 2200 and current frame, (play from 2200)

#FRAME=372
FRAME=2200

SHOW=False
#aviName="D:\avs\avi\in\painting test for avisynth.avi.AVI"
aviName="D:\avs\avi\in\capture-1.avi.AVI"

Avisource(AviName)

A=trim(FRAME,-1)
A=A.FreezeFrame(1,Last.FrameCount-1,0)
Clipdelta(a,Last,true,SHOW)
ScriptClip("""RT_SubTitle("%d",current_frame)""")
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool "amp",bool "show") {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}


If you play from 2200, it shows rubbish being generated on the frame, some on the left (close but not that close to edits)
and a lot of rubbish on the right (nowhere near edits).
I was wondering if it might actually be CamStudio that is causing the rubbish problems (including cloud around the cursor problem),
it might be a good idea to try another screen cap software if you have access to one.
You can locate rubbish by using first script above, just watch for unexplained 'gunk' appearing, and then use 2nd script
setting FRAME to a few frames before the rubbish appears.

I'm not sure I can do anything so long as the rubbish keeps appearing, I think something really has to go, either the paint package, or
CamStudio, would also be good if you could find a way of not recording the cursor.

[eduardobedoya]

I came back, I will provide sample without cursor (I found a way to record without it), You may be right and perhaps is the recording app or even the painting app, lets try just without cursor, maybe you could find a way to get rid of 70% of the undos.

[StainlessS]

Hi there ed,
Sorry bout not getting any further, but is still on back burner, no further release of RT_stats since then, still working on it.
As I dont like continually battering my head against the wall, I have been doing a few other things in between, but keep going
back to both your problem, and a Scene Change detector which I am trying to perfect. I will not release another RT until I have achieved both.
Leave a link if your like and I'll DL and add it to my queue of things to do.

Good to hear about the cursor, 'sort of' looking forward to having another go at it.

[Wilbert]

removed off-topic posts. @Seedmanc if you have problems with the rules in some circumstances you know where to report them (and that's not here).

[StainlessS]

Hi Ed, just a sitrep.

Still banging away at your problem, using new version RT_Stats, not having a great deal of luck as yet because of the crud pixels
in UNDO's. Especially where there are several undo's in succession where one undo frame acquires 17,800+ crud pixels of error up to about 5 luma levels.
Its hard to tell difference bwetween crud and edit.

Anyway, just thought I'de say that you should not delete your account or anything like that, I am still busy.

Here a log of what I just got a few moments ago on your most recent sample, script completely different to previous, think I need to
go back and use some of the tactics in an earlier attempt.

Fsel_Eduardobedoya_Batch.Log
LumaTol=0.015000 EDIT_T = 0.002000 KeepMax=1
YV12=True PC709=True CROPPING=128,128,328,128

1/1 ] Processing D:\ED\Last capture without cursor.avi.AVI
QWIK Scan DBase creation = 0.00 Secs (0.00 Mins)
34,75 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=1.004695E-003 @ 34<-->76)
117,207 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=5.156109E-004 @ 117<-->208)
276,388 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=9.560631E-004 @ 276<-->389)
418,462 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=4.763737E-004 @ 418<-->463)
482,591 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.116602E-004 @ 482<-->592)
612,668 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=7.792190E-005 @ 612<-->669)
713,811 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=3.735278E-003 @ 713<-->812)
826,849 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=3.665093E-003 @ 826<-->850)
879,912 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.001653E-003 @ 879<-->913)
919,1011 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=1.656476E-002 @ 919<-->1012)
1118,1151 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=4.518918E-003 @ 1118<-->1152)
1211,1221 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=5.032871E-003 @ 1211<-->1222)
1259,1311 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=4.758210E-004 @ 1259<-->1312)
1391,1418 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=3.862937E-004 @ 1391<-->1419)
1553,1593 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.724504E-004 @ 1553<-->1594)
1608,1647 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.812926E-004 @ 1608<-->1648)
1690,1722 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=9.748528E-004 @ 1690<-->1723)
1727,1769 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=9.030099E-004 @ 1727<-->1770)
1784,1811 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.906874E-004 @ 1784<-->1812)
1826,1878 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=1.137328E-003 @ 1826<-->1879)
1879,1902 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=1.205303E-003 @ 1879<-->1903)
1923,1953 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=4.327153E-004 @ 1923<-->1954)
1957,1987 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=2.155287E-004 @ 1957<-->1988)
2016,2089 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=1.071564E-003 @ 2016<-->2090)
2095,2128 *** SKIPPING earlier duplicate sequence and UNDOS (Crud=6.532177E-004 @ 2095<-->2129)


Kept 409 of 2150 frames [1464x824 143.33 secs (2.39 Mins) @ 15.00 FPS]
File CrudMax = 1.656476E-002 CrudMaxFrame=1012 FramesSearched=395266
QWIK SCAN 420.76 Secs (7.01 Mins) InFPS=5.11 OutFPS=0.97 FramesSearchedFPS=939.41

fn="D:\ED\Last capture without cursor.avi.AVI"
Avisource(fn)
ScriptClip("Subtitle(String(current_frame))")
PathAndNode="D:\ED\Last capture without cursor.avi"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last

D:\ED\Last capture without cursor.avi.AVI Tot File Time = 421.20 Seconds (7.02 Mins)


TOTAL Time = 421.21 Seconds (7.02 Mins)

Global CrudMax =1.656476E-002


DONE


EDIT: I'm cropping off the thumbnail, dont need additional crud from that. Also note, that the new sample also has a lot of visible crud
at the bottom of frame that I have not noticed before, anyway, cropping it off for better detection.

[StainlessS]

@Eduardobedoya,
Here tis at last, hope you did not close you D9 account (or cut off your ear, tis the artistic way I hear [with my one good ear]).

Part 1 of 2 (glue them back together into a single script)


# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# Remove Undos from painting captures.
########################################################################
THRESH = 0.01 # (0.01) Max LumaTol due to crud between undone/undo frames. (set about double CrudMax)
# (CrudMax only valid as Check when results are satisfactory)
# Upper Limit for self tuning LumaTol.
#
LUMATOL_SCALE = 1.0 # (1.0) Range 1.0 -> 2.0. Should never need change from 1.0.
# LumaTol is self tuning but could possibly be in error where without this setting would fail miserably.
# If self tune LumaTol is set too low then will not find matching UNDONE frames. This setting allows
# to increase LumaTol as LumaTol is multiplied by it, and then restricted at upper limit by THRESH above.
# If ever need to be changed, suggest something like 1.00001. (temporary change only)
#
LUMATOL_ADD = 0.0 # (0.0) Range 0.0 -> 1.0. Additional adjustment added to LumaTol, probably never necessary, but if so then
# something like 0.000001. Applied before limiting to THRESH as above.
#
MIN_EDITLEN = 4 # (4) An edit has to be at least this many frames long (frames between UNDONE and UNDO, exclusive)
OVR_PIXCNT_THR = 4 # (4) Thresh for RT_LumaPixelsDifferentCount, only pixel differences greater than this are counted.
OVR_PIXCNT_LIM = 50 # (50)If RT_LumaPixelsDifferentCount(Thresh=OVR_PIXCNT_THR) greater than this then is OVERRIDDEN as false detection.
# Above OVR_ settings for detecting override where difference between undone/undo is too visible (ie not crud).
#
ScanAheadSecs = 10 * 60 # Search ahead range in seconds
ChromaWeight = 1.0/3.0 # (1.0/3.0) YUV Chroma Weighting, 0.0 -> 1.0
FPS = 15.0 # Play speed for output script.
VERBOSITY = 0 # (0) 0 to 3. Debug & logging verbosity.
########################################################################
# Chop off crud around outsides
GLOBAL CROP_L = 128 # Crop Left
GLOBAL CROP_T = 128 # Crop Top
GLOBAL CROP_R = 328 # Crop Right (Including that thumbnail with the delayed UNDO and additional crap)
GLOBAL CROP_B = 128 # Crop Bottom
########################################################################
# During Testing
GLOBAL CREATE = True
GLOBAL DELETE_DB = True
########################################################################
GLOBAL PC709 = True
GLOBAL CROP_L = (CROP_L / 4) * 4 GLOBAL CROP_T = (CROP_T / 4) * 4
GLOBAL CROP_R = (CROP_R / 4) * 4 GLOBAL CROP_B = (CROP_B / 4) * 4
FSEL_TITLE="Select AVI files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True
AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))
NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_WriteFile(LOG,"%s\n",LOG,Append=False)

S=RT_String("THRESH = %f LUMATOL_SCALE=%f LUMATOL_ADD=%f\nMIN_EDITLEN=%d OVR_PIXCNT_THR=%d OVR_PIXCNT_LIM=%d",
\ THRESH,LUMATOL_SCALE,LUMATOL_ADD,MIN_EDITLEN,OVR_PIXCNT_THR,OVR_PIXCNT_LIM)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

S=RT_String("ScanAheadSecs=%d ChromaWeight=%f OutFPS=%f",ScanAheadSecs,ChromaWeight,FPS)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

S=RT_String("PC709=%s CROPPING=%d,%d,%d,%d",PC709,CROP_L,CROP_T,CROP_R,CROP_B)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

GLOBAL Global_CrudMax = 0.0
Global Global_PixCntMax = 0

Crud_S="""
AMP=True
PCTHRESH=@
aviName=$@@$
PC709=@@@
Path=RT_FilenameSplit(AviName,3)
Node=RT_FilenameSplit(AviName,4)
CrudFrames = Path + $CrudInspect_Frames_$ + Node + $.TXT$
Avisource(AviName)
Crop(CROP_L,CROP_T,-CROP_R,-CROP_B)
ConvertToYV12(matrix=(PC709)?$PC.709$:$PC.601$)
NUMBERSCLIP=FrameSel(Cmd=CrudFrames,Ordered=True,SHOW=TRUE)
NUMBERSCLIP=NUMBERSCLIP.CROP(0,0,20*10,1*20).POINTResize(20*10*2,1*20*2)
NA=NUMBERSCLIP.SelectEven
NB=NUMBERSCLIP.SelectODD
FrameSel(Cmd=CrudFrames,Ordered=True)
A=SelectEven
B=SelectODD
DELTA = Clipdelta(A,B,AMP)
DELTA = DELTA.OVERLAY(NA,X=0,Y=DELTA.HEIGHT-NA.HEIGHT).OVERLAY(NB,X=DELTA.WIDTH-NB.WIDTH,Y=DELTA.HEIGHT-NA.HEIGHT)
DELTA
SC=$$$
pc=RT_LumaPixelsDifferentCount(A,B)
pct=RT_LumaPixelsDifferentCount(A,B,THRESH=PCTHRESH)
ld=RT_LumaDifference(A,B)
fd=RT_FrameDifference(A,B)
S=RT_String($UNDO=%d] LumaDif=%f FrameDif=%f LumaPixelsDifferent=%d LumaPixelsDifferentByMoreThan_%d=%d$,current_frame,ld,fd,pc,PCTHRESH,pct)
Subtitle(S,size=24,font=$Courier New$)
$$$
ScriptClip(SC)
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool $amp$,bool $show$) {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}
"""

GSCript("""
TOTSTART = RT_TimerHP
For(i=0,NFILES-1) {
START = RT_TimerHP
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n%d/%d ] Processing '%s'\n",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)

DropUndo(FN,Crud_S,ScanAheadSecs,thresh=THRESH,Min_EditLen=MIN_EDITLEN,
\ log=LOG,ChromaWeight=ChromaWeight,fps=FPS,
\ ovr_pixcnt_thr=OVR_PIXCNT_THR,ovr_pixcnt_lim=OVR_PIXCNT_LIM,
\ LumaTol_Scale=LUMATOL_SCALE,LumaTol_Add=LUMATOL_ADD,Verbosity=VERBOSITY)

T = RT_TimerHP - START
S=RT_String("%d/%d '%s' Tot File Time = %.2f Seconds (%.2f Mins)",i+1,NFILES,FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
T = RT_TimerHP - TOTSTART
S=RT_String("\nBATCH TOTAL Time = %.2f Seconds (%.2f Mins)\nGlobal CrudMax=%f\nGlobal MaxLumaPixelsDifferentBy_%d = %d",
\ T,T/60.0,GLOBAL_CrudMax,OVR_PIXCNT_THR,Global_PixCntMax)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("\nDONE\n\n%s",S)
S=RT_StrReplace(S,Chr(10),"\n")
""")


Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)

EDITED:

[StainlessS]

Part 2 of 2


Function DropUndo(String AviName,String Crud_S,Float "ScanAheadSecs",Float "THRESH",Int "Min_EditLen",
\ String "Log",Float "ChromaWeight",float "fps",Int "ovr_pixcnt_thr",Int "ovr_pixcnt_lim",
\ Float "LumaTol_Scale",Float "LumaTol_Add", Int "Verbosity") {
myName="DropUndo: "
ScanAheadSecs=Float(Default(ScanAheadSecs,10.0*60.0))
THRESH=Float(Default(THRESH,0.02))
MIN_EDITLEN=Default(MIN_EDITLEN,4)
LOG = Default(LOG,"DropUndo.LOG")
ChromaWeight = Float(Default(ChromaWeight,1.0/3.0))
FPS = Float(Default(FPS,FPS))
OVR_PIXCNT_THR=Default(OVR_PIXCNT_THR,4)
OVR_PIXCNT_LIM=Default(OVR_PIXCNT_LIM,50)
LumaTol_Scale = Float(Default(LumaTol_Scale,1.0))
LumaTol_Add = Float(Default(LumaTol_Add,0.0))
VERBOSITY=Default(VERBOSITY,0)

Assert(Exist(AviName),myName+AviName+" Does Not Exist")

Avisource(AviName)
ORG=Last

Crop(CROP_L,CROP_T,-CROP_R,-CROP_B) # Crop crud
ConvertToYV12(Matrix=(PC709)?"PC.709":"PC.601")

Assert(ScanAheadSecs>0.0,myName+"ScanAheadSecs Must be greater than zero")
Assert(THRESH>0.0, myName+"THRESH Must be greater than zero")
Assert(MIN_EDITLEN>0,myName+"Greater than zero please")
Assert(ChromaWeight>=0.0 && ChromaWeight<=1.0, myName+"ChromaWeight range 0.0 -> 1.0")
Assert(OVR_PIXCNT_THR>=0 && OVR_PIXCNT_THR<=8, myName+"OVR_PIXCNT_THR range 0 -> 8")
Assert(OVR_PIXCNT_LIM>=0 && OVR_PIXCNT_LIM<=100, myName+"OVR_PIXCNT_LIM range 0 -> 100")
Assert(LumaTol_Scale>=1.0 && LumaTol_Scale<=2.0, myName+"LUMATOL_SCALE range 1.0 -> 2.0")
Assert(LumaTol_Add>=0.0 && LumaTol_Add<=1.0, myName+"LUMATOL_ADD range 0.0 -> 1.0")
Assert(VERBOSITY>=0 && VERBOSITY<=3, myName+"VERBOSITY range 0 -> 3")

Path = RT_FilenameSplit(AviName,3) # Drive + Dir
Node = RT_FilenameSplit(AviName,4) # Name
PathAndNode = Path+Node # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.TXT"
Ranges=PathAndNode+"_Ranges.TXT"
ScriptFile=Path+"_AVS_"+Node+"_SelectFrames.AVS"
CrudScript = Path + "_CRUD_INSPECT_" + Node + ".AVS"
CrudFrames = Path + "CrudInspect_Frames_" + Node + ".TXT"
Fnd_S = RT_String("CROP_L\nCROP_T\nCROP_R\nCROP_B\n$\n@@@\n@@\n@\n")
Rep_S = RT_String("%d\n%d\n%d\n%d\n%c\n%s\n%s\n%d",CROP_L,CROP_T,CROP_R,CROP_B,34,PC709,RT_GetFullPathName(AviName),OVR_PIXCNT_THR)
Crud_S = RT_StrReplaceMulti(Crud_S,Fnd_S,Rep_S)

ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
DB=PathAndNode+".DB"
PREVDB=PathAndNode + "_Prev.DB"
NEXTDB=PathAndNode + "_Next.DB"

RT_FileDelete(CMDFrames) # Delete any existing frames file
RT_FileDelete(Ranges) # Delete any existing Ranges file
RT_FileDelete(CrudFrames)
RT_FileDelete(CrudScript)

### TEMPLATE script
FrameSel_Select="""
Show=False # Set True to show original Frame Number
fn="%s"
Avisource(fn)
(Show) ? ScriptClip("Subtitle(String(current_frame))") : NOP
PathAndNode="%s"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last.AssumeFPS(%f)
"""

###
Select_S = FrameSel_Select
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,PathAndNode,FPS) # Insert filenames

FramesSearched = 0
CrudMax = 0.0
CrudMaxFrame = -1
PixCntMax = 0
PixCntMaxFrame = -1
Kept = 0

GSCript("""
if(CREATE) {
START = RT_TimerHP
RT_QwikScanCreate(DB,prevdb="",nextdb=NEXTDB,debug=true)
T= RT_TimerHP - START
S = RT_String("QWIK Scan DBase creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_WriteFile(LOG,"%s",S,Append=True)
}
START = RT_TimerHP
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)

for(i=LastFrame,0,-1) {
SKIP = False
if(i > 0) {
PreUndoDif = RT_FrameDifference(Last,Last,n=i,n2=i-1,ChromaWeight=ChromaWeight)
FramesSearched = FramesSearched + 1
if(PreUndoDif > 0.0) {
StartFrame = Max(i-ScanAheadFrames,0)
EndFrame = (i - 1 - MIN_EDITLEN)

# We are looking for an UNDONE frame that is more similar to i frame than i-1.
# If i is an UNDO frame then UNDONE frame will be more similar to i than i-1, so LumaTol estimate will find it fast.
# Alternatively, if i is not an UNDO frame then LumaTol estimate will speed up NOT FINDING frame.
# If estimate greater than THRESH, then i could still be an UNDO frame just i-1 is quite a lot different to i frame, so we
# limit LumaTol to THRESH assuming well set for Crud max.
# Below two lines speed up whole script significantly (rather than just using THRESH setting for LumaTol).

# arg frame order IS SIGNIFICANT (we are matching to i frame, estimate is for i-1).
LumaTol = LumaTol_Scale * RT_QwikScanEstimateLumaTol(Last,Last,n=i,n2=i-1) + LumaTol_Add
LumaTol = (LumaTol>THRESH) ? THRESH : LumaTol

DifThresh = PreUndoDif*0.999999 # We are looking for BETTER match, NOT equal

if(VERBOSITY>=1) {
S = RT_String("%5d] NEWSEARCH: SearchStart=%d SearchEnd=%d Seeking FDif <= %E {LumaTol=%E}",
\ i,StartFrame,EndFrame,DifThresh,LumaTol)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
For(j=StartFrame,EndFrame) {
MaxDistance = EndFrame - j
if(VERBOSITY>=2) {
S = RT_String(" SEARCH: SearchStart=%d MaxDistance=%d",j,MaxDistance)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
# XP=3, strictly better than, ie left most frame (first found, furthest from i).
# MaxDistance limits search to exclude MIN_EDITLEN frames.
Result=RT_QwikScan(Last,j,Last,i,DB,NEXTDB,lumatol=LumaTol,Flags=$04,fd=DifThresh,maxdistance=MaxDistance,XP=3)
if(Result>=0) { # Exit Condition succeeds
FramesSearched = FramesSearched + (QWKS_BM_FD_FRM-j+1)
if(VERBOSITY>=2) {
S = RT_String(" MATCH: Frame=%5d FrameDifference=%E ",QWKS_BM_FD_FRM,QWKS_BM_FD)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
CrudPix=RT_LumaPixelsDifferentCount(Last,Last,n=i,n2=QWKS_BM_FD_FRM,Thresh=OVR_PIXCNT_THR)
FramesSearched = FramesSearched + 1
if(CrudPix <= OVR_PIXCNT_LIM) {
YCrudDif = RT_QwikScanEstimateLumaTol(Last,Last,n=i,n2=QWKS_BM_FD_FRM) # Crud measured by LumaTol estimate
FramesSearched = FramesSearched + 1
if(YCrudDif > CrudMax) {
CrudMax = YCrudDif
CrudMaxFrame = i
}
if(CrudPix > PixCntMax) {
PixCntMax = CrudPix
PixCntMaxFrame = i
}
RT_WriteFile(CrudFrames,"%d\n%d",QWKS_BM_FD_FRM,i,Append=True)
if(VERBOSITY==0) {
S=RT_String("%5d] UNDO: Matched %d -> %d : Skip %d to %d : (FDif=%E : LumaPixelsDifferentBy_%d = %d)",
\ i,QWKS_BM_FD_FRM,i,QWKS_BM_FD_FRM+1,i,QWKS_BM_FD,OVR_PIXCNT_THR,CrudPix)
} Else {
S=RT_String(" UNDO: Matched %d -> %d : Skip %d to %d : (FDif=%E : LumaPixelsDifferentBy_%d = %d)",
\ QWKS_BM_FD_FRM,i,QWKS_BM_FD_FRM+1,i,QWKS_BM_FD,OVR_PIXCNT_THR,CrudPix)
}
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
i = QWKS_BM_FD_FRM # Skip edited Frames up to and including UNDO frame
j = EndFrame # Break
} Else {
if(VERBOSITY>=2){
S=RT_String(" OVERRIDE: LumaPixelsDifferentBy_%d = %d (I -> Matched Frame)",OVR_PIXCNT_THR,CrudPix)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
j = QWKS_BM_FD_FRM # Continue search
}
} Else { # Result < 0, Exit condition fails
FramesSearched = FramesSearched + MaxDistance + 1 # Searched full extent
if(VERBOSITY>=3) {
if(QWKS_BM_FLAGS!=0) { # Got a Best Match Only
S = RT_String(" NOT FOUND: Best Match Frame=%5d FDif=%E",QWKS_BM_FD_FRM,QWKS_BM_FD)
} Else { # Did not even get a Best match
S = RT_String(" NOT FOUND: No Best Match:")
}
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
j = EndFrame # Break
}
} # End for j
} Else { # PreUndoDif <= 0.0
if(VERBOSITY>=2) {
d = i-1
for(k=d-1,0,-1) {
Dif = RT_FrameDifference(Last,Last,n=i,n2=k,ChromaWeight=ChromaWeight)
FramesSearched = FramesSearched + 1
if(Dif == 0.0) {
d = k
} Else {
k = - 1
}
}
S=RT_String("%5d] IDENTICAL: Frames %d to %d : Skipping Frames %d to %d",i,d,i,d+1,i)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
i = d+1 # Retry at d
}
SKIP = True # Dont write i
}
} # i == 0
if(!SKIP || i==0) {
if(VERBOSITY>=3) {
if(i==0) {S=RT_String("%5d] WRITING: Frame",i) }
Else {S=RT_String(" WRITING: Frame") }
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
RT_WriteFile(CMDFrames,"%d",i,Append=True)
Kept = Kept + 1
}
}

RT_WriteFile(ScriptFile,"%s",Select_S)
RT_WriteFile(CrudScript,"%s",Crud_S)
(DELETE_DB) ? RT_FileDelete(DB) : NOP
(DELETE_DB) ? RT_FileDelete(NEXTDB) : NOP
T = RT_TimerHP - START
FT=FrameCount / FrameRate
S=RT_String("\nKept %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]",Kept, FrameCount,ORG.Width,ORG.Height,FT,FT/60.0,FrameRate)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("File Max Successful CrudMax = %E CrudMaxFrame=%d",CrudMax,CrudMaxFrame)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("File Max Successful LumaPixelsDifferentBy_%d = %d @ Frame = %d",ovr_pixcnt_thr,PixCntMax,PixCntMaxFrame)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f FramesSearched=%d FramesSearchedFPS=%.2f",
\ T,T/60.0,FrameCount/T,Kept/T,FramesSearched,FramesSearched/T)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
Global Global_CrudMax = Max(Global_CrudMax,CrudMax)
Global Global_PixCntMax = Max(Global_PixCntMax,PixCntMax)
""")
Return 0
}

EDITED: Minor mods and a little tinkering, zip updated.

Only tested on your most recent clip without the cursor.
Sorry bout the wait, sick as a parrot of seeing your lovely artwork, gets a little tired after about 200+ processings.
Think it works pretty good and quite fast too.

Generates the output processing avs for each input clip and also generates a 'crud viewer' script so you can judge settings.

Here: script in zip: LINK REMOVED

And incase you do not still have the original clip without cursor, here tis: http://www.mediafire.com/download/9vxf957u29z9heo/Last+capture+without+cursor.avi
EDIT: TechSmith codec version, needs conversion to compressed RGB before script use, as in earlier posts.

Good luck.

[StainlessS]

Did this on Core Duo 2.4GHz whilst on-line and Antivirus running, VERBOSITY=0 (Too long for previous post).

Fsel_Eduardobedoya_Batch.Log
THRESH = 0.010000 LUMATOL_SCALE=1.000000 LUMATOL_ADD=0.000000
MIN_EDITLEN=4 OVR_PIXCNT_THR=4 OVR_PIXCNT_LIM=50
ScanAheadSecs=600 ChromaWeight=0.333333 OutFPS=15.000000
PC709=True CROPPING=128,128,328,128

1/1 ] Processing 'D:\ED\Last capture without cursor.avi.AVI'
QWIK Scan DBase creation = 94.49 Secs (1.57 Mins)
2124] UNDO: Matched 2095 -> 2124 : Skip 2096 to 2124 : (FDif=6.532177E-004 : LumaPixelsDifferentBy_4 = 0)
2085] UNDO: Matched 2073 -> 2085 : Skip 2074 to 2085 : (FDif=9.394840E-006 : LumaPixelsDifferentBy_4 = 0)
2072] UNDO: Matched 1992 -> 2072 : Skip 1993 to 2072 : (FDif=1.100854E-003 : LumaPixelsDifferentBy_4 = 9)
1984] UNDO: Matched 1957 -> 1984 : Skip 1958 to 1984 : (FDif=2.155287E-004 : LumaPixelsDifferentBy_4 = 4)
1951] UNDO: Matched 1923 -> 1951 : Skip 1924 to 1951 : (FDif=4.327153E-004 : LumaPixelsDifferentBy_4 = 2)
1900] UNDO: Matched 1816 -> 1900 : Skip 1817 to 1900 : (FDif=2.470290E-003 : LumaPixelsDifferentBy_4 = 14)
1809] UNDO: Matched 1784 -> 1809 : Skip 1785 to 1809 : (FDif=2.906874E-004 : LumaPixelsDifferentBy_4 = 1)
1768] UNDO: Matched 1727 -> 1768 : Skip 1728 to 1768 : (FDif=9.030099E-004 : LumaPixelsDifferentBy_4 = 7)
1716] UNDO: Matched 1693 -> 1716 : Skip 1694 to 1716 : (FDif=8.908519E-004 : LumaPixelsDifferentBy_4 = 4)
1643] UNDO: Matched 1608 -> 1643 : Skip 1609 to 1643 : (FDif=2.812926E-004 : LumaPixelsDifferentBy_4 = 0)
1588] UNDO: Matched 1553 -> 1588 : Skip 1554 to 1588 : (FDif=2.917927E-004 : LumaPixelsDifferentBy_4 = 0)
1431] UNDO: Matched 1426 -> 1431 : Skip 1427 to 1431 : (FDif=1.199224E-004 : LumaPixelsDifferentBy_4 = 1)
1411] UNDO: Matched 1391 -> 1411 : Skip 1392 to 1411 : (FDif=4.951633E-004 : LumaPixelsDifferentBy_4 = 0)
1309] UNDO: Matched 1259 -> 1309 : Skip 1260 to 1309 : (FDif=4.758210E-004 : LumaPixelsDifferentBy_4 = 1)
1219] UNDO: Matched 1186 -> 1219 : Skip 1187 to 1219 : (FDif=2.571976E-003 : LumaPixelsDifferentBy_4 = 11)
1149] UNDO: Matched 1112 -> 1149 : Skip 1113 to 1149 : (FDif=4.598498E-003 : LumaPixelsDifferentBy_4 = 13)
1048] UNDO: Matched 1039 -> 1048 : Skip 1040 to 1048 : (FDif=9.947477E-006 : LumaPixelsDifferentBy_4 = 0)
985] UNDO: Matched 938 -> 985 : Skip 939 to 985 : (FDif=6.108304E-003 : LumaPixelsDifferentBy_4 = 24)
928] UNDO: Matched 914 -> 928 : Skip 915 to 928 : (FDif=2.305052E-003 : LumaPixelsDifferentBy_4 = 9)
908] UNDO: Matched 870 -> 908 : Skip 871 to 908 : (FDif=2.127102E-003 : LumaPixelsDifferentBy_4 = 8)
850] UNDO: Matched 828 -> 850 : Skip 829 to 850 : (FDif=3.548486E-003 : LumaPixelsDifferentBy_4 = 12)
807] UNDO: Matched 713 -> 807 : Skip 714 to 807 : (FDif=3.715935E-003 : LumaPixelsDifferentBy_4 = 20)
668] UNDO: Matched 612 -> 668 : Skip 613 to 668 : (FDif=7.792190E-005 : LumaPixelsDifferentBy_4 = 1)
588] UNDO: Matched 482 -> 588 : Skip 483 to 588 : (FDif=2.116602E-004 : LumaPixelsDifferentBy_4 = 0)
460] UNDO: Matched 420 -> 460 : Skip 421 to 460 : (FDif=3.868463E-005 : LumaPixelsDifferentBy_4 = 0)
390] UNDO: Matched 370 -> 390 : Skip 371 to 390 : (FDif=3.039507E-005 : LumaPixelsDifferentBy_4 = 0)
368] UNDO: Matched 276 -> 368 : Skip 277 to 368 : (FDif=9.245627E-004 : LumaPixelsDifferentBy_4 = 3)
205] UNDO: Matched 117 -> 205 : Skip 118 to 205 : (FDif=5.156109E-004 : LumaPixelsDifferentBy_4 = 1)
65] UNDO: Matched 0 -> 65 : Skip 1 to 65 : (FDif=1.008011E-003 : LumaPixelsDifferentBy_4 = 2)

Kept 410 of 2150 frames [1920x1080 143.33 secs (2.39 Mins) @ 15.00 FPS]
File Max Successful CrudMax = 2.074689E-003 CrudMaxFrame=1149
File Max Successful LumaPixelsDifferentBy_4 = 24 @ Frame = 985
QWIK SCAN 107.41 Secs (1.79 Mins) InFPS=20.02 OutFPS=3.82 FramesSearched=424888 FramesSearchedFPS=3955.89 :)
1/1 'D:\ED\Last capture without cursor.avi.AVI' Tot File Time = 202.10 Seconds (3.37 Mins)

BATCH TOTAL Time = 202.10 Seconds (3.37 Mins)
Global CrudMax=0.002075
Global MaxLumaPixelsDifferentBy_4 = 24

EDIT: Redone without Antivirus, with updated script.
EDIT: Requires latest RT_Stats v1.43 (posted earlier today) for RT_QwikScanEstimateLumaTol().

EDIT: Will be quicker if you move the canvas from time to time, and the more you crop would also increase speed but have to ensure
edits are central.

[eduardobedoya]

Thanks StainlessS, very much, I've been very occupied, I have not had time even to paint,
I will check it out ASAP, I dont remember very well how I used to use Avisynth, and the scripts in batch, but I wil try to get back everything, do I have to paste those two scripts into a single one? or are two batchs steps?

please tell me, was it easier to delete the duplicated frames in a video recorded without cursor? was there still a cloudy mood in the painting strokes? was there still inequality between undo and redo images in the painting?
what porcentage of the duplicated frames were accuracy deleted by the script?

Thanks advanced man, I will post my test ASAP. thanks

[StainlessS]

eduardobedoya,
The two posts were only because the script would not fit in one post (16kb limit on D9 user forum, [20kb in devs]).
Just download the zip, is already a single avisynth script.

please tell me, was it easier to delete the duplicated frames in a video recorded without cursor?

Absolutely, but still difficult. (thankyou for that, I would still be banging my head against the wall if you had not removed cursor).

was there still inequality between undo and redo images in the painting?

Yes, The provided script also creates a Crud inspection script, ie a script that shows the "inequality" so you can judge if it got it wrong.
(A solid lump of 'crud' might signify that it got it wrong).
EDIT: You can change AMP=True to AMP=False in crud inspection script to view crud without amplification.
EDIT: Crud Inspection script, solid lump of crud might indicate bad detection and wrong UNDO, viewing result
script would show if any detection missed (UNDO not removed).

"what porcentage of the duplicated frames were accuracy deleted by the script?",

Well tis my belief that 100% were accurately deleted, and none deleted in error, I watched though entire clip at 1 FPS (without blinking, [mostly]
and I could not see any errors at all).

I will not be doing any further work on this I am currenly otherwise occupied doing other stuff, sorry.
EDIT: Its done and working with user configurable settings and so is pretty much complete, only other mods would be tinkering
with output formatted logs etc.

Hope you are happy with results and appologies for the obscene amount of time it took to arrive.
be gud. :)

EDIT: There were a number of frames that had only a few (grouped) pixels changed, I could not tell whether these were intentional
edits or more mistakes in the paint program, if they were not within an UNDONE/UNDO region they are left in as they also affect all
following frames.

[eduardobedoya]

Thanks man, I will try it asap. Thanks again.

EDIT: jajaja. I didn't cut my ear., just now Im having time to check this and remember how to use your radical script.
Yes I still have the without cursor video, testing it as soon as I get it back how to do it.
Thanks man, programmers are digital engineers, you rule.

[eduardobedoya]

Hi StainlessS
Finally I tested it, it worked like a charm

First I converted tsc2 codec into UtVideo RGB VCM
then I run the Second batch

here I post the Second batch log...


Fsel_Eduardobedoya_Batch.Log
THRESH = 0.010000 LUMATOL_SCALE=1.000000 LUMATOL_ADD=0.000000
MIN_EDITLEN=4 OVR_PIXCNT_THR=4 OVR_PIXCNT_LIM=50
ScanAheadSecs=600 ChromaWeight=0.333333 OutFPS=15.000000
PC709=True CROPPING=128,128,328,128

1/1 ] Processing 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Last capture without cursor UtVideo RGB VCM.avi'
QWIK Scan DBase creation = 74.33 Secs (1.24 Mins)
2124] UNDO: Matched 2095 -> 2124 : Skip 2096 to 2124 : (FDif=6.504545E-004 : LumaPixelsDifferentBy_4 = 1)
2085] UNDO: Matched 2073 -> 2085 : Skip 2074 to 2085 : (FDif=9.947477E-006 : LumaPixelsDifferentBy_4 = 0)
2072] UNDO: Matched 1992 -> 2072 : Skip 1993 to 2072 : (FDif=1.087038E-003 : LumaPixelsDifferentBy_4 = 9)
1984] UNDO: Matched 1957 -> 1984 : Skip 1958 to 1984 : (FDif=2.133181E-004 : LumaPixelsDifferentBy_4 = 5)
1951] UNDO: Matched 1923 -> 1951 : Skip 1924 to 1951 : (FDif=4.238731E-004 : LumaPixelsDifferentBy_4 = 4)
1900] UNDO: Matched 1816 -> 1900 : Skip 1817 to 1900 : (FDif=2.460895E-003 : LumaPixelsDifferentBy_4 = 12)
1809] UNDO: Matched 1784 -> 1809 : Skip 1785 to 1809 : (FDif=2.835031E-004 : LumaPixelsDifferentBy_4 = 1)
1768] UNDO: Matched 1727 -> 1768 : Skip 1728 to 1768 : (FDif=9.063257E-004 : LumaPixelsDifferentBy_4 = 9)
1716] UNDO: Matched 1693 -> 1716 : Skip 1694 to 1716 : (FDif=8.842202E-004 : LumaPixelsDifferentBy_4 = 6)
1643] UNDO: Matched 1608 -> 1643 : Skip 1609 to 1643 : (FDif=2.906874E-004 : LumaPixelsDifferentBy_4 = 0)
1588] UNDO: Matched 1553 -> 1588 : Skip 1554 to 1588 : (FDif=2.951085E-004 : LumaPixelsDifferentBy_4 = 0)
1431] UNDO: Matched 1426 -> 1431 : Skip 1427 to 1431 : (FDif=1.127381E-004 : LumaPixelsDifferentBy_4 = 1)
1411] UNDO: Matched 1391 -> 1411 : Skip 1392 to 1411 : (FDif=4.973739E-004 : LumaPixelsDifferentBy_4 = 0)
1309] UNDO: Matched 1259 -> 1309 : Skip 1260 to 1309 : (FDif=4.835579E-004 : LumaPixelsDifferentBy_4 = 1)
1219] UNDO: Matched 1186 -> 1219 : Skip 1187 to 1219 : (FDif=2.550423E-003 : LumaPixelsDifferentBy_4 = 10)
1149] UNDO: Matched 1112 -> 1149 : Skip 1113 to 1149 : (FDif=4.566997E-003 : LumaPixelsDifferentBy_4 = 14)
1048] UNDO: Matched 1039 -> 1048 : Skip 1040 to 1048 : (FDif=9.394840E-006 : LumaPixelsDifferentBy_4 = 0)
985] UNDO: Matched 938 -> 985 : Skip 939 to 985 : (FDif=6.128199E-003 : LumaPixelsDifferentBy_4 = 21)
928] UNDO: Matched 914 -> 928 : Skip 915 to 928 : (FDif=2.302288E-003 : LumaPixelsDifferentBy_4 = 7)
908] UNDO: Matched 870 -> 908 : Skip 871 to 908 : (FDif=2.147550E-003 : LumaPixelsDifferentBy_4 = 6)
850] UNDO: Matched 828 -> 850 : Skip 829 to 850 : (FDif=3.540197E-003 : LumaPixelsDifferentBy_4 = 13)
807] UNDO: Matched 713 -> 807 : Skip 714 to 807 : (FDif=3.668409E-003 : LumaPixelsDifferentBy_4 = 22)
668] UNDO: Matched 612 -> 668 : Skip 613 to 668 : (FDif=8.234301E-005 : LumaPixelsDifferentBy_4 = 1)
588] UNDO: Matched 482 -> 588 : Skip 483 to 588 : (FDif=2.149760E-004 : LumaPixelsDifferentBy_4 = 0)
460] UNDO: Matched 420 -> 460 : Skip 421 to 460 : (FDif=3.757936E-005 : LumaPixelsDifferentBy_4 = 0)
390] UNDO: Matched 370 -> 390 : Skip 371 to 390 : (FDif=2.818452E-005 : LumaPixelsDifferentBy_4 = 0)
368] UNDO: Matched 276 -> 368 : Skip 277 to 368 : (FDif=9.300892E-004 : LumaPixelsDifferentBy_4 = 2)
205] UNDO: Matched 117 -> 205 : Skip 118 to 205 : (FDif=5.106372E-004 : LumaPixelsDifferentBy_4 = 1)
65] UNDO: Matched 0 -> 65 : Skip 1 to 65 : (FDif=9.925372E-004 : LumaPixelsDifferentBy_4 = 2)

Kept 410 of 2150 frames [1920x1080 143.33 secs (2.39 Mins) @ 15.00 FPS]
File Max Successful CrudMax = 2.110001E-003 CrudMaxFrame=1900
File Max Successful LumaPixelsDifferentBy_4 = 22 @ Frame = 807
QWIK SCAN 41.09 Secs (0.68 Mins) InFPS=52.33 OutFPS=9.98 FramesSearched=424875 FramesSearchedFPS=10341.13
1/1 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Last capture without cursor UtVideo RGB VCM.avi' Tot File Time = 115.90 Seconds (1.93 Mins)

BATCH TOTAL Time = 115.90 Seconds (1.93 Mins)
Global CrudMax=0.002110
Global MaxLumaPixelsDifferentBy_4 = 22


Then I copied the avs files and run the third batch
and it get it done, just in the third batch black window, it says...
Dub: Input <decompression> format is: YUV420.
Dub: Output <decompression> format is: RGB888.
Ending Operation

It does mean that the selecting frame process is runing in YUV420 color mode? The source file was UtVideo RGB, what does it mean?
Is that is so, has YUV color mode got enought color information to detect subtle color differences between strokes in the selecting frame process?
by the way, I used to notice the difference between RGB and YUV in UtVideos in the version 13.3.1 but now in the version 14.2.0 I really cant tell the difference.
So, now what is the difference between RGB and YUV in Utvideo beside the filesize??? Do you have please any link about last UtVideo upgrades?

Guess that will be all, Great script StainlessS, thanks man.
I will perform further tests with larger clips, and post any feedback if it could be usefull to you. Thanks once again, thanks for all.

[StainlessS]

Copy ONLY the files beginning with "_AVS_" and not the "_CRUD_INSPECT_" files, they should be first in directory listing when sorted alphabetic ascending,
copy to the output batch directory.

Here, is output script for your test clip, it outputs whatever the source clip is, should be RGB. (The "_CRUD_INSPECT_" scripts output YV12, hence your problem, I think).


Show=False # Set True to show original Frame Number
fn="D:\ED\Last capture without cursor.avi.AVI"
Avisource(fn)
(Show) ? ScriptClip("Subtitle(String(current_frame))") : NOP
PathAndNode="D:\ED\Last capture without cursor.avi"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last.AssumeFPS(15.000000)


I use UT_Video 13.3.1, I think later versions demand more recent processor and as I use some old Pentium 4's I will not be updating to more recent versions and dont know what if any differences there are from previous.

Do search on-site for latest version, not hard to find

[eduardobedoya]

when I copy both the "_AVS_" and the "_CRUD_" files and render the third batch, it created two videos one with all the selected frames, an another with a grey screen with some kinda sparkles instead of the strokes and some text data. From now on I only copy the "_AVS_"

The source for the second batch file was UtVideo RGB, I have try to use UtVideo YUV420 instead, since I dont longer see any visual difference between RGB and YUV420 in UtVideo codec v14.
But I have found that using YUV420 video as source will produce unwanted results, almost does not drop any frame. So I keep using RGB.


I see that the Fsel_Eduardobedoya_Batch_Sept.avs is not editable, I guess is a full road script, Ive tested it with other shortlenght videos
I mean, in the previous video (the one with the subtle hair strokes) the script performed with great accuracy,
but perhaps by looking at different scanarios you could find the most accurate "tunning" for the script in order to work in all scenarios, hope I may not been abusing of your help,
I send you a 1:50min clip that had 85% accuracy
https://www.sendspace.com/file/50owfa

Thanks again man, it looks very promising for testing in 30min lenght videos.

PD: Here is the avs batch script Log of this last 1:50min clip


Fsel_Eduardobedoya_Batch.Log
THRESH = 0.010000 LUMATOL_SCALE=1.000000 LUMATOL_ADD=0.000000
MIN_EDITLEN=4 OVR_PIXCNT_THR=4 OVR_PIXCNT_LIM=50
ScanAheadSecs=600 ChromaWeight=0.333333 OutFPS=15.000000
PC709=True CROPPING=128,128,328,128

1/1 ] Processing 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Test 00 UtVideo RGB VCM.avi'
QWIK Scan DBase creation = 47.41 Secs (0.79 Mins)
1649] UNDO: Matched 1636 -> 1649 : Skip 1637 to 1649 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
1583] UNDO: Matched 1542 -> 1583 : Skip 1543 to 1583 : (FDif=5.426902E-004 : LumaPixelsDifferentBy_4 = 3)
1498] UNDO: Matched 1391 -> 1498 : Skip 1392 to 1498 : (FDif=3.659014E-003 : LumaPixelsDifferentBy_4 = 10)
1327] UNDO: Matched 1278 -> 1327 : Skip 1279 to 1327 : (FDif=5.140082E-003 : LumaPixelsDifferentBy_4 = 19)
1249] UNDO: Matched 1234 -> 1249 : Skip 1235 to 1249 : (FDif=5.526376E-006 : LumaPixelsDifferentBy_4 = 0)
1217] UNDO: Matched 1204 -> 1217 : Skip 1205 to 1217 : (FDif=1.105275E-005 : LumaPixelsDifferentBy_4 = 0)
1190] UNDO: Matched 1121 -> 1190 : Skip 1122 to 1190 : (FDif=4.747157E-004 : LumaPixelsDifferentBy_4 = 5)
1039] UNDO: Matched 1029 -> 1039 : Skip 1030 to 1039 : (FDif=8.179037E-005 : LumaPixelsDifferentBy_4 = 1)
957] UNDO: Matched 951 -> 957 : Skip 952 to 957 : (FDif=1.160539E-005 : LumaPixelsDifferentBy_4 = 0)
814] UNDO: Matched 805 -> 814 : Skip 806 to 814 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
758] UNDO: Matched 738 -> 758 : Skip 739 to 758 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
736] UNDO: Matched 703 -> 736 : Skip 704 to 736 : (FDif=6.405070E-004 : LumaPixelsDifferentBy_4 = 2)
678] UNDO: Matched 673 -> 678 : Skip 674 to 678 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
609] UNDO: Matched 557 -> 609 : Skip 558 to 609 : (FDif=3.404801E-003 : LumaPixelsDifferentBy_4 = 10)
518] UNDO: Matched 469 -> 518 : Skip 470 to 518 : (FDif=4.070729E-003 : LumaPixelsDifferentBy_4 = 14)
432] UNDO: Matched 418 -> 432 : Skip 419 to 432 : (FDif=8.276301E-003 : LumaPixelsDifferentBy_4 = 38)
395] UNDO: Matched 388 -> 395 : Skip 389 to 395 : (FDif=2.100023E-005 : LumaPixelsDifferentBy_4 = 0)
359] UNDO: Matched 324 -> 359 : Skip 325 to 359 : (FDif=6.114936E-003 : LumaPixelsDifferentBy_4 = 17)
312] UNDO: Matched 303 -> 312 : Skip 304 to 312 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
212] UNDO: Matched 207 -> 212 : Skip 208 to 212 : (FDif=1.050011E-005 : LumaPixelsDifferentBy_4 = 0)
205] UNDO: Matched 198 -> 205 : Skip 199 to 205 : (FDif=4.973739E-006 : LumaPixelsDifferentBy_4 = 0)
164] UNDO: Matched 111 -> 164 : Skip 112 to 164 : (FDif=1.195687E-002 : LumaPixelsDifferentBy_4 = 37)
95] UNDO: Matched 88 -> 95 : Skip 89 to 95 : (FDif=2.763188E-006 : LumaPixelsDifferentBy_4 = 0)
23] UNDO: Matched 0 -> 23 : Skip 1 to 23 : (FDif=7.456187E-003 : LumaPixelsDifferentBy_4 = 21)

Kept 733 of 1650 frames [1920x1080 110.00 secs (1.83 Mins) @ 15.00 FPS]
File Max Successful CrudMax = 5.654124E-003 CrudMaxFrame=432
File Max Successful LumaPixelsDifferentBy_4 = 38 @ Frame = 432
QWIK SCAN 135.26 Secs (2.25 Mins) InFPS=12.20 OutFPS=5.42 FramesSearched=557865 FramesSearchedFPS=4124.47
1/1 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Test 00 UtVideo RGB VCM.avi' Tot File Time = 183.12 Seconds (3.05 Mins)

BATCH TOTAL Time = 183.12 Seconds (3.05 Mins)
Global CrudMax=0.005654
Global MaxLumaPixelsDifferentBy_4 = 38


and here are the Debug View Logs of the same clip

second batch

00000001 0.00000000 [8224] Fsel_Eduardobedoya_Batch: THRESH = 0.010000 LUMATOL_SCALE=1.000000 LUMATOL_ADD=0.000000
00000002 0.00003637 [8224] Fsel_Eduardobedoya_Batch: MIN_EDITLEN=4 OVR_PIXCNT_THR=4 OVR_PIXCNT_LIM=50
00000003 0.00064265 [8224] Fsel_Eduardobedoya_Batch: ScanAheadSecs=600 ChromaWeight=0.333333 OutFPS=15.000000
00000004 0.00122880 [8224] Fsel_Eduardobedoya_Batch: PC709=True CROPPING=128,128,328,128
00000005 0.00385271 [8224] Fsel_Eduardobedoya_Batch:
00000006 0.00388379 [8224] Fsel_Eduardobedoya_Batch: 1/1 ] Processing 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Test 00 UtVideo RGB VCM.avi'
00000007 0.00391097 [8224] Fsel_Eduardobedoya_Batch:
00000008 0.08910672 [8224]
00000009 0.08910672 [8224] RT_QwikScanCreate: RT_QwikScanCreateDB() by StainlessS
00000010 0.08913533 [8224] RT_QwikScanCreate: DBaseAlloc DB
00000011 0.08966392 [8224] RT_QwikScanCreate: DBaseAlloc NextDB
00000012 0.09306854 [8224] RT_QwikScanCreate: Filling DB with FingerPrint data (Will take some time)
00000013 7.54825926 [8224] RT_QwikScanCreate: record(256) 15.5%
00000014 15.15915012 [8224] RT_QwikScanCreate: record(512) 31.0%
00000015 22.56482887 [8224] RT_QwikScanCreate: record(768) 46.5%
00000016 29.82000160 [8224] RT_QwikScanCreate: record(1024) 62.1%
00000017 37.05014801 [8224] RT_QwikScanCreate: record(1280) 77.6%
00000018 44.28882217 [8224] RT_QwikScanCreate: record(1536) 93.1%
00000019 47.48595428 [8224] RT_QwikScanCreate: record(1650) 100.0%
00000020 47.48601532 [8224] RT_QwikScanCreate: Filling NextDB with data
00000021 47.48653793 [8224] RT_QwikScanCreate: record(1536) 6.9%
00000022 47.48751450 [8224] RT_QwikScanCreate: record(1280) 22.4%
00000023 47.48849869 [8224] RT_QwikScanCreate: record(1024) 37.9%
00000024 47.48946381 [8224] RT_QwikScanCreate: record(768) 53.5%
00000025 47.49042892 [8224] RT_QwikScanCreate: record(512) 69.0%
00000026 47.49139023 [8224] RT_QwikScanCreate: record(256) 84.5%
00000027 47.49236298 [8224] RT_QwikScanCreate: record(0) 100.0%
00000028 47.49265671 [8224] RT_QwikScanCreate: Total time = 47.41 seconds (0.79 mins)
00000029 47.49498367 [8224] DropUndo: QWIK Scanning file ... Please Wait
00000030 47.54751968 [8224] DropUndo: 1649] UNDO: Matched 1636 -> 1649 : Skip 1637 to 1649 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
00000031 52.80663681 [8224] DropUndo: 1583] UNDO: Matched 1542 -> 1583 : Skip 1543 to 1583 : (FDif=5.426902E-004 : LumaPixelsDifferentBy_4 = 3)
00000032 54.31874847 [8224] DropUndo: 1498] UNDO: Matched 1391 -> 1498 : Skip 1392 to 1498 : (FDif=3.659014E-003 : LumaPixelsDifferentBy_4 = 10)
00000033 55.81215286 [8224] DropUndo: 1327] UNDO: Matched 1278 -> 1327 : Skip 1279 to 1327 : (FDif=5.140082E-003 : LumaPixelsDifferentBy_4 = 19)
00000034 56.89404297 [8224] DropUndo: 1249] UNDO: Matched 1234 -> 1249 : Skip 1235 to 1249 : (FDif=5.526376E-006 : LumaPixelsDifferentBy_4 = 0)
00000035 57.94061279 [8224] DropUndo: 1217] UNDO: Matched 1204 -> 1217 : Skip 1205 to 1217 : (FDif=1.105275E-005 : LumaPixelsDifferentBy_4 = 0)
00000036 59.25577927 [8224] DropUndo: 1190] UNDO: Matched 1121 -> 1190 : Skip 1122 to 1190 : (FDif=4.747157E-004 : LumaPixelsDifferentBy_4 = 5)
00000037 126.53381348 [8224] DropUndo: 1039] UNDO: Matched 1029 -> 1039 : Skip 1030 to 1039 : (FDif=8.179037E-005 : LumaPixelsDifferentBy_4 = 1)
00000038 150.00593567 [8224] DropUndo: 957] UNDO: Matched 951 -> 957 : Skip 952 to 957 : (FDif=1.160539E-005 : LumaPixelsDifferentBy_4 = 0)
00000039 167.14712524 [8224] DropUndo: 814] UNDO: Matched 805 -> 814 : Skip 806 to 814 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
00000040 171.00790405 [8224] DropUndo: 758] UNDO: Matched 738 -> 758 : Skip 739 to 758 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
00000041 171.04545593 [8224] DropUndo: 736] UNDO: Matched 703 -> 736 : Skip 704 to 736 : (FDif=6.405070E-004 : LumaPixelsDifferentBy_4 = 2)
00000042 172.28634644 [8224] DropUndo: 678] UNDO: Matched 673 -> 678 : Skip 674 to 678 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
00000043 174.27455139 [8224] DropUndo: 609] UNDO: Matched 557 -> 609 : Skip 558 to 609 : (FDif=3.404801E-003 : LumaPixelsDifferentBy_4 = 10)
00000044 175.56694031 [8224] DropUndo: 518] UNDO: Matched 469 -> 518 : Skip 470 to 518 : (FDif=4.070729E-003 : LumaPixelsDifferentBy_4 = 14)
00000045 176.35115051 [8224] DropUndo: 432] UNDO: Matched 418 -> 432 : Skip 419 to 432 : (FDif=8.276301E-003 : LumaPixelsDifferentBy_4 = 38)
00000046 177.06050110 [8224] DropUndo: 395] UNDO: Matched 388 -> 395 : Skip 389 to 395 : (FDif=2.100023E-005 : LumaPixelsDifferentBy_4 = 0)
00000047 177.57496643 [8224] DropUndo: 359] UNDO: Matched 324 -> 359 : Skip 325 to 359 : (FDif=6.114936E-003 : LumaPixelsDifferentBy_4 = 17)
00000048 177.83166504 [8224] DropUndo: 312] UNDO: Matched 303 -> 312 : Skip 304 to 312 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
00000049 179.93904114 [8224] DropUndo: 212] UNDO: Matched 207 -> 212 : Skip 208 to 212 : (FDif=1.050011E-005 : LumaPixelsDifferentBy_4 = 0)
00000050 179.97833252 [8224] DropUndo: 205] UNDO: Matched 198 -> 205 : Skip 199 to 205 : (FDif=4.973739E-006 : LumaPixelsDifferentBy_4 = 0)
00000051 180.84188843 [8224] DropUndo: 164] UNDO: Matched 111 -> 164 : Skip 112 to 164 : (FDif=1.195687E-002 : LumaPixelsDifferentBy_4 = 37)
00000052 181.51409912 [8224] DropUndo: 95] UNDO: Matched 88 -> 95 : Skip 89 to 95 : (FDif=2.763188E-006 : LumaPixelsDifferentBy_4 = 0)
00000053 182.70394897 [8224] DropUndo: 23] UNDO: Matched 0 -> 23 : Skip 1 to 23 : (FDif=7.456187E-003 : LumaPixelsDifferentBy_4 = 21)
00000054 182.75611877 [8224] DropUndo:
00000055 182.75619507 [8224] DropUndo: Kept 733 of 1650 frames [1920x1080 110.00 secs (1.83 Mins) @ 15.00 FPS]
00000056 182.75773621 [8224] DropUndo: File Max Successful CrudMax = 5.654124E-003 CrudMaxFrame=432
00000057 182.75927734 [8224] DropUndo: File Max Successful LumaPixelsDifferentBy_4 = 38 @ Frame = 432
00000058 182.76168823 [8224] DropUndo: QWIK SCAN 135.26 Secs (2.25 Mins) InFPS=12.20 OutFPS=5.42 FramesSearched=557865 FramesSearchedFPS=4124.47
00000059 183.11918640 [8224] Fsel_Eduardobedoya_Batch: 1/1 'F:\00 AVISYNTH SECOND BATCH\INPUT-OUTPUT\Test 00 UtVideo RGB VCM.avi' Tot File Time = 183.12 Seconds (3.05 Mins)
00000060 183.12054443 [8224] Fsel_Eduardobedoya_Batch:
00000061 183.12057495 [8224] Fsel_Eduardobedoya_Batch: BATCH TOTAL Time = 183.12 Seconds (3.05 Mins)
00000062 183.12060547 [8224] Fsel_Eduardobedoya_Batch: Global CrudMax=0.005654
00000063 183.12062073 [8224] Fsel_Eduardobedoya_Batch: Global MaxLumaPixelsDifferentBy_4 = 38


third batch
00000001 0.00000000 [7356] SHIMVIEW: ShimInfo(Complete)
00000002 13.06780338 [2400] SHIMVIEW: ShimInfo(Complete)
00000003 13.39500046 [6196] SHIMVIEW: ShimInfo(Complete)
00000004 13.60489368 [3548] SHIMVIEW: ShimInfo(Complete)
00000005 13.63414574 [6000] SHIMVIEW: ShimInfo(Complete)
00000006 13.63701153 [7712] SHIMVIEW: ShimInfo(Complete)
00000007 13.64167690 [4704] SHIMVIEW: ShimInfo(Complete)
00000008 13.64673519 [8492] SHIMVIEW: ShimInfo(Complete)
00000009 13.65096092 [8820] SHIMVIEW: ShimInfo(Complete)
00000010 13.65474319 [3208] SHIMVIEW: ShimInfo(Complete)
00000011 13.69241619 [3332] SHIMVIEW: ShimInfo(Complete)
00000012 13.82886028 [6356] SHIMVIEW: ShimInfo(Complete)
00000013 13.94016266 [8420] SHIMVIEW: ShimInfo(Complete)
00000014 13.94712830 [6584] SHIMVIEW: ShimInfo(Complete)
00000015 13.96794605 [7504] SHIMVIEW: ShimInfo(Complete)
00000016 14.05450535 [2100] SHIMVIEW: ShimInfo(Complete)
00000017 14.07977772 [9176] SHIMVIEW: ShimInfo(Complete)
00000018 14.25737667 [6212] SHIMVIEW: ShimInfo(Complete)
00000019 14.32255650 [2152] SHIMVIEW: ShimInfo(Complete)
00000020 14.34952068 [6296] SHIMVIEW: ShimInfo(Complete)
00000021 14.36253738 [5488] SHIMVIEW: ShimInfo(Complete)
00000022 14.37000656 [6876] SHIMVIEW: ShimInfo(Complete)
00000023 14.70396996 [5244] SHIMVIEW: ShimInfo(Complete)
00000024 14.83184052 [8504] SHIMVIEW: ShimInfo(Complete)
00000025 15.10169888 [7756] SHIMVIEW: ShimInfo(Complete)
00000026 15.12605572 [8996] SHIMVIEW: ShimInfo(Complete)
00000027 15.13991737 [8212] SHIMVIEW: ShimInfo(Complete)
00000028 16.45670891 [5332] SHIMVIEW: ShimInfo(Complete)
00000029 16.54409027 [8560] SHIMVIEW: ShimInfo(Complete)
00000030 16.57573700 [6716] SHIMVIEW: ShimInfo(Complete)

[StainlessS]

You did not say if was erroneously removing frames OR, not removing undone frames that it should have done.

I see that the Fsel_Eduardobedoya_Batch_Sept.avs is not editable

Why not ???, perhaps only read-only was set on it for some reason, it is intended that you are able to
edit the config settings at beginning of file.

The generated _CRUD_INSPECT script shows the rubbish left behind by your paint program after an undo, it is intended that
you inspect this to see if the script got it correct, if significant 'lump' of crud (rubbish) is found in
CRUD output, then that undo is probably wrong and would have to amend config settings to suite. (view with and without
amplification (AMP=True/False).


########################################################################
THRESH = 0.01 # (0.01) Max LumaTol due to crud between undone/undo frames. (set about double CrudMax)
# (CrudMax only valid as Check when results are satisfactory)
# Upper Limit for self tuning LumaTol.
#
LUMATOL_SCALE = 1.0 # (1.0) Range 1.0 -> 2.0. Should never need change from 1.0.
# LumaTol is self tuning but could possibly be in error where without this setting would fail miserably.
# If self tune LumaTol is set too low then will not find matching UNDONE frames. This setting allows
# to increase LumaTol as LumaTol is multiplied by it, and then restricted at upper limit by THRESH above.
# If ever need to be changed, suggest something like 1.00001. (temporary change only)
#
LUMATOL_ADD = 0.0 # (0.0) Range 0.0 -> 1.0. Additional adjustment added to LumaTol, probably never necessary, but if so then
# something like 0.000001. Applied before limiting to THRESH as above.
#
MIN_EDITLEN = 4 # (4) An edit has to be at least this many frames long (frames between UNDONE and UNDO, exclusive)
OVR_PIXCNT_THR = 4 # (4) Thresh for RT_LumaPixelsDifferentCount, only pixel differences greater than this are counted.
OVR_PIXCNT_LIM = 50 # (50)If RT_LumaPixelsDifferentCount(Thresh=OVR_PIXCNT_THR) greater than this then is OVERRIDDEN as false detection.
# Above OVR_ settings for detecting override where difference between undone/undo is too visible (ie not crud).
#
ScanAheadSecs = 10 * 60 # Search ahead range in seconds
ChromaWeight = 1.0/3.0 # (1.0/3.0) YUV Chroma Weighting, 0.0 -> 1.0
FPS = 15.0 # Play speed for output script.
VERBOSITY = 0 # (0) 0 to 3. Debug & logging verbosity.
########################################################################

important ones in BLUE

THRESH should be higher than Crudmax shown on ALL good detections (about double at least).
OVR_PIXCNT_THR and OVR_PIXCNT_LIM detect whether 'sparkles' are crud or real edit.

Perhaps OVR_PIXCNT_LIM need be set lower as this line seems to have high "LumaPixelsDifferentBy_4 = 38"


432] UNDO: Matched 418 -> 432 : Skip 419 to 432 : (FDif=8.276301E-003 : LumaPixelsDifferentBy_4 = 38)


Perhaps try at 30. (EDIT: I only had one good sample to play with [not counting moving cursor examples] and set to 50 by default,
if 30 works better over a number of samples then use that PERMANENTLY)

EDIT: You can temporarily set THRESH high (eg 0.1) whilst trying to find better setting for OVR_PIXCNT_THR and OVR_PIXCNT_LIM
and when working OK, set THRESH to about double maximum found CrudMax.

You need to see which undo's it got wrong and adjust OVR_PIXCNT_THR and OVR_PIXCNT_LIM to suite.
A Lower setting for THRESH just speeds up detections, so temporarily set it higher when finding best settings for the OVR
config settings. The OVR settings are an attempt to detect crud, very difficult because the number of crud pixels can vary between
a few tens of pixels up to tens of thousands of rubbish pixels (different to the UNDONE frame), you might even want to set OVR_PIXCNT_THR to 5 and reduce
OVR_PIXCNT_LIM based on log lines like "LumaPixelsDifferentBy_4 = 38", the "_4" is OVR_PIXCNT_THR and 38 is compared with
OVR_PIXCNT_LIM, the idea is to determine if the crud is visible or not.

[eduardobedoya]

Sorry I said so cuz I tried to open the avs batch script with notepad and all the code appeared messed up, like all the script was place in 6 lines only.

Now I have open it with microsoft word and it worked fine. What is the right software to open scripts, is there some software that could count the lines? should I be able to open it with notepad?

I read all you explained, I will perform further test with a bunch of "different scenario" clips next months.
Im trying to understand the different values in the script, I undestood that
I must read the generated crud avs file in order to know how to handle OVR settings, but is it not enought to just see the Log file?

I understand that the crud is the rubbish produced by the undos, but I dont understand why is the AMP(=true/false) for.
I mean, I edit the avs batch script changing AMP (=true or false) then I run the avs so _CRUD avs file is generated, right? So I need to run the avs batch script twice one with the AMP=true and another with the AMP=false, right? what is exactly the difference between AMP=true data and AMP=false data?

Thanks Advanced man, guess I could not config this script as if I made it, but hope I could understand it little more with your help, thanks for all.

[StainlessS]

Sorry, I guess I had PSPad (text editor) set to not write Carriage Returns or something.
You can use eg NotePad+ or PSPad to replace notepad and also to edit AVS files.
http://forum.doom9.org/showthread.php?p=1581965#post1581965

The numbers in log files tell you what was found, the crud inspect script shows whether settings worked well or not.
AMP = AMPlification, it makes crud much more visible, when off it should look like plain grey frame, if NOT grey then is a mistake.
Do not edit AMP in batcher script, only in the generated _CRUD_INSPECTxxxx.avs files.

EDIT: If not grey when AMP=False, then you need see the log values (for that frame) and adjust config settings.
I prefer to see results of rubbish when AMP=true, if you prefer to see grey then set permanently in batcher to false.

EDIT: As said in some previous post, the CRUD_INSPECT script shows if a detection was wrong, and the results from
the output script should be viewed to see if it missed some undos.

[StainlessS]

Just spotted your SendSpace sample, must have missed it first time around, downloaded, will take a look.

NOTE, here

432] UNDO: Matched 418 -> 432 : Skip 419 to 432 : (FDif=8.276301E-003 : LumaPixelsDifferentBy_4 = 38)


In this line that I suspect might be bad detection, there are only 13 frames between UNDONE and UNDO frames (EXCLUSIVE) so if was correct undo
then you executed an undo within less than 1 second of making a mistake (@15.0 FPS), if you can guarantee eg at least 1 second ie 15 frames
before an error is undone, then set MIN_EDITLEN = 15 (or more if you can guarantee), this would help avoid some erroneous detections.

EDIT: It seems that the greater the amount of time elapsed between edit/undo (including multiple undo's) increases the amount of crud produced.
With long clips and using your required 10 minutes search ahead, it may not be possible to tell the difference between crud and edit, no matter
what settings are used. It's all really a balancing act and I cannot guarantee that any perfect solution could ever be found, unless you could find
a paint program that does not leave crud pixels after an undo, if you can find such a program (or fix the one you use) then batch script should
work pretty much perfectly as it is.

EDIT: Some more on PSPad here:http://forum.doom9.org/showthread.php?p=1682010#post1682010

and here Avisynth.ini for PSPad in the DATA folder: (put in Syntax folder after extract, cant remember where I got it)
https://www.mediafire.com/folder/hb26mthbjz7z6/StainlessS

[eduardobedoya]

Thanks StainlessS, I installed Pspad (it tried to set some search engine as default in my browser) it looks huge, I almost feel like a programmer when looking at that interface xD, It opens avs fine, but still I don't understand why is the "Avisynth.ini" for? I guess I will only make a few adjustments to the batch avs.

Do not edit AMP in batcher script, only in the generated _CRUD_INSPECTxxxx.avs files.

Yes of course I only edit AMP in the generated _CRUD_INSPECT.avs file
The batch avs generates the _AVS and the _CRUD_INSPECT, and you said I only had to copy the _AVS to an INPUT folder in order to the run the third batch, if I copy also the _CRUD_INSPECT to that INPUT folder and run the third batch it will create two videos one with the selected frames and one with the grey screen and the cruds.
What I guessed is that I would need to run that third batch process two times if I wanted to try the two options of the _CRUD_INSPECT (one with AMP=true and another with AMP=false) am I right? I guess I should leave it in true.

In this line that I suspect might be bad detection, there are only 13 frames between UNDONE and UNDO frames (EXCLUSIVE) so if was correct undo
then you executed an undo within less than 1 second of making a mistake (@15.0 FPS), if you can guarantee eg at least 1 second ie 15 frames
before an error is undone, then set MIN_EDITLEN = 15 (or more if you can guarantee), this would help avoid some erroneous detections.

Usually undos are done repeatedly in less than one second, It happens that UNDOS come in groups consecutivily, then a couple of seconds or half minute without UNDOS, and then again another group of UNDOS, and so on. So I guess MIN_EDITLEN should be around 3 or 5.

EDIT: It seems that the greater the amount of time elapsed between edit/undo (including multiple undo's) increases the amount of crud produced.
With long clips and using your required 10 minutes search ahead, it may not be possible to tell the difference between crud and edit, no matter
what settings are used. It's all really a balancing act and I cannot guarantee that any perfect solution could ever be found, unless you could find
a paint program that does not leave crud pixels after an undo, if you can find such a program (or fix the one you use) then batch script should
work pretty much perfectly as it is.

What if we reduce the search ahead to 5 minutes, could it help to better tell the difference between crud and edit?
Actually, the last video I sent you was recorded using a new version of the software, perhaps it could have less crud.
If not, Do you think I should try to record the painting process with UtVideo instead of tsc2 and perform a test with the batchscript to see if it still finds crud, could tsc2 be the reason of the crud?
Yes man, exacly, balancing act, so what should get priority in that balancing act?
In think the script should allow any possible mistake (Undos that didn't get cutted), provided that it could detect and differentiate all subtle strokes as different frames,
I mean, the script could skip some UNDOS, but could not CUT subtle strokes frames cuz it thinks they are all the same frame.

Finally man, the script has 85% accuracy, it looks already well balanced, and it will be a really very usefull if it can keep its accuracy ratio and detect 99% of subtle strokes in 30min videos.
I guess I only need to learn how to tweak its different settings, or perhaps you could even add some new variable to it. Thanks for all man.

[StainlessS]

it tried to set some search engine as default in my browser

Yes some free software does that, it gives them a little cash income, just de-select it.

why is the Avisynth.ini for?

Allows to highlight Avisynth syntax + can eg press a key and run a program (eg MPlayer/VDub)on avs script.
See Here:- http://forum.doom9.org/showthread.php?t=170684

Dont bother with the batch thing on _CRUD_INSPECT scripts, just use them if you think it got something wrong and
only directly open in eg VDubMod, not batch processed.

With AMP = True, bad detection crud frame would have a 'solid' lump of crud, not like the 'sparkles' that are
normally seen in crud frames, cant explain any better, best to see for yourself (could try script on one of the
moving cursor clips and you are bound to see what crud lumps look like).

Actually, the line I thought might be crud looks like it is OK, I'm not sure but I think there may be a logic
problem in script (a kind of sequencing problem). I've also discovered a bug in the RT_FSelOpen() function, which
I am gonna have to look at first.
EDIT: False alarm for the RT_FSelOpen bug, seems to be something a bit strange in MediaPlayerClassic-Home Cinema.
If user clicks on Cancel in fileselector, and script throws error via Assert then MPC-HC has a 2nd go at opening the script
and the file selector jumps up for a second time.


When I mentioned MIN_EDITLEN = 15, I mean the number of frames between last click on group of undo's (frame nearest
end of clip) and the 1st frame before consecutive edits that will be undone (frame nearest start of clip),
undo sequences are removed as a single block, multiple consecutive undo groups all at once, so for an instance
where only a single UNDO will be processed, you could guarantee 15 frames for MIN_EDITLEN just by waiting for 1 second
before you click on UNDO, so long as number of frames between first UNDONE frame and last UNDO frame is over
MIN_EDITLEN then everything would be fine, the MIN_EDITLEN setting is for when you make a single mistake and
immediately UNDO it, the MIN_EDITLEN setting was set deliberately low as in your test clip you obviously had a
finger on the UNDO key so you could make an edit and immediately undo it again, in reality I dont think it would
need to be as low as default 4.(hope that makes some sense).

"What if we reduce the search ahead to 5 minutes, could it help to better tell the difference between crud and edit?"

I think I said from the beginning that a 10 minute search ahead was a little ambitious (but not so much if the
paint program did not introduce crud into the mix). As it is, the clips tested so far have been short clips and so
the 10 min search ahead has not been tested at all. Only time will tell when you try on 30 min clips whether will
work OK or not, but the real problem is I think multiple consecutive undo's with cascadingly accumulated crud
with each undo. In one of your clips (a moving cursor clip I think) I did see crud that WAS VISIBLE
in the _AVS_ result clip, on my monitor was I think a tear drop shape about 3mm across and clearly visible dark blob
which remained on-frame until end of clip (not an edit by you, results of crud by paint program).
In reality, it is probable that the dark blob is a lot less likely to appear, was probably down to your providing
an UNDO TORTURE TEST clip, with many repeated edits and undos in rapid succession.

Tsc2 could indeed be the reason for the crud rather than paint program, but paint program chief suspect I think.
If was Tsc2 and used different record s/w then would remove the crud problem altogether, that has already been
suggested earlier.

Viewing the _CRUD_INSPECT_ script I did not see any bad detections (even the "LumaPixelsDifferentBy_4 = 38" line
frame looked OK), it just looks like it missed removing a couple of undo's and that looks like it may be the
sequencing problem mentioned previously.

I'll get back.

EDIT: 'Crud lumps' are (EDIT: usually) not crud, they are undo mistakes (undone by mistake).
EDIT: The main problem is:, is it an UNDO with crud, OR, an edit. Without crud the problem is much simpler.

[StainlessS]

OK, here we go again: http://www.mediafire.com/download/2ggvi1i2jc4ocmi/Fsel_Eduardobedoya_Batch_17Dec2014.zip

Raised THRESH to 0.015.
Added ReScan after UNDO (correct sequencing problem).
Had to make RT_QwikScan and RT_QwikScanEstimateLumaTol a bit safer, slowed down a little but not so very much.


Log

Fsel_Eduardobedoya_Batch.Log
THRESH = 0.015000 LUMATOL_SCALE=1.000000 LUMATOL_ADD=0.000000
MIN_EDITLEN=4 OVR_PIXCNT_THR=4 OVR_PIXCNT_LIM=50
ScanAheadSecs=600 ChromaWeight=0.333333 OutFPS=15.000000
PC709=True CROPPING=128,128,328,128

1/2 ] Processing 'D:\ED\Last capture without cursor.avi.AVI'
QWIK Scan DBase creation = 130.94 Secs (2.18 Mins)
2124] UNDO: Matched 2095 -> 2124 : Skip 2096 to 2124 : (FDif=6.532177E-004 : LumaPixelsDifferentBy_4 = 0)
2085] UNDO: Matched 2073 -> 2085 : Skip 2074 to 2085 : (FDif=9.394840E-006 : LumaPixelsDifferentBy_4 = 0)
2072] UNDO: Matched 1992 -> 2072 : Skip 1993 to 2072 : (FDif=1.100854E-003 : LumaPixelsDifferentBy_4 = 9)
1984] UNDO: Matched 1957 -> 1984 : Skip 1958 to 1984 : (FDif=2.155287E-004 : LumaPixelsDifferentBy_4 = 4)
1951] UNDO: Matched 1923 -> 1951 : Skip 1924 to 1951 : (FDif=4.327153E-004 : LumaPixelsDifferentBy_4 = 2)
1900] UNDO: Matched 1816 -> 1900 : Skip 1817 to 1900 : (FDif=2.470290E-003 : LumaPixelsDifferentBy_4 = 14)
1809] UNDO: Matched 1784 -> 1809 : Skip 1785 to 1809 : (FDif=2.906874E-004 : LumaPixelsDifferentBy_4 = 1)
1768] UNDO: Matched 1727 -> 1768 : Skip 1728 to 1768 : (FDif=9.030099E-004 : LumaPixelsDifferentBy_4 = 7)
1716] UNDO: Matched 1693 -> 1716 : Skip 1694 to 1716 : (FDif=8.908519E-004 : LumaPixelsDifferentBy_4 = 4)
1643] UNDO: Matched 1608 -> 1643 : Skip 1609 to 1643 : (FDif=2.812926E-004 : LumaPixelsDifferentBy_4 = 0)
1588] UNDO: Matched 1553 -> 1588 : Skip 1554 to 1588 : (FDif=2.917927E-004 : LumaPixelsDifferentBy_4 = 0)
1431] UNDO: Matched 1426 -> 1431 : Skip 1427 to 1431 : (FDif=1.199224E-004 : LumaPixelsDifferentBy_4 = 1)
1411] UNDO: Matched 1391 -> 1411 : Skip 1392 to 1411 : (FDif=4.951633E-004 : LumaPixelsDifferentBy_4 = 0)
1309] UNDO: Matched 1259 -> 1309 : Skip 1260 to 1309 : (FDif=4.758210E-004 : LumaPixelsDifferentBy_4 = 1)
1219] UNDO: Matched 1186 -> 1219 : Skip 1187 to 1219 : (FDif=2.571976E-003 : LumaPixelsDifferentBy_4 = 11)
1149] UNDO: Matched 1112 -> 1149 : Skip 1113 to 1149 : (FDif=4.598498E-003 : LumaPixelsDifferentBy_4 = 13)
1048] UNDO: Matched 1039 -> 1048 : Skip 1040 to 1048 : (FDif=9.947477E-006 : LumaPixelsDifferentBy_4 = 0)
985] UNDO: Matched 938 -> 985 : Skip 939 to 985 : (FDif=6.108304E-003 : LumaPixelsDifferentBy_4 = 24)
928] UNDO: Matched 914 -> 928 : Skip 915 to 928 : (FDif=2.305052E-003 : LumaPixelsDifferentBy_4 = 9)
908] UNDO: Matched 870 -> 908 : Skip 871 to 908 : (FDif=2.127102E-003 : LumaPixelsDifferentBy_4 = 8)
850] UNDO: Matched 828 -> 850 : Skip 829 to 850 : (FDif=3.548486E-003 : LumaPixelsDifferentBy_4 = 12)
807] UNDO: Matched 713 -> 807 : Skip 714 to 807 : (FDif=3.715935E-003 : LumaPixelsDifferentBy_4 = 20)
668] UNDO: Matched 612 -> 668 : Skip 613 to 668 : (FDif=7.792190E-005 : LumaPixelsDifferentBy_4 = 1)
588] UNDO: Matched 482 -> 588 : Skip 483 to 588 : (FDif=2.116602E-004 : LumaPixelsDifferentBy_4 = 0)
460] UNDO: Matched 420 -> 460 : Skip 421 to 460 : (FDif=3.868463E-005 : LumaPixelsDifferentBy_4 = 0)
390] UNDO: Matched 370 -> 390 : Skip 371 to 390 : (FDif=3.039507E-005 : LumaPixelsDifferentBy_4 = 0)
368] UNDO: Matched 276 -> 368 : Skip 277 to 368 : (FDif=9.245627E-004 : LumaPixelsDifferentBy_4 = 3)
205] UNDO: Matched 117 -> 205 : Skip 118 to 205 : (FDif=5.156109E-004 : LumaPixelsDifferentBy_4 = 1)
65] UNDO: Matched 0 -> 65 : Skip 1 to 65 : (FDif=1.008011E-003 : LumaPixelsDifferentBy_4 = 2)

Kept 410 of 2150 frames [1920x1080 143.33 secs (2.39 Mins) @ 15.00 FPS]
File Max Successful CrudMax = 2.758771E-003 CrudMaxFrame=1900
File Max Successful LumaPixelsDifferentBy_4 = 24 @ Frame = 985
QWIK SCAN 130.94 Secs (2.18 Mins) InFPS=16.42 OutFPS=3.13 FramesSearched=459016 FramesSearchedFPS=3505.60
1/2 'D:\ED\Last capture without cursor.avi.AVI' Tot File Time = 262.12 Seconds (4.37 Mins)

2/2 ] Processing 'D:\ED\Test 00 tsc2.avi.AVI'
QWIK Scan DBase creation = 98.48 Secs (1.64 Mins)
1649] UNDO: Matched 1636 -> 1649 : Skip 1637 to 1649 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
1583] UNDO: Matched 1542 -> 1583 : Skip 1543 to 1583 : (FDif=5.360585E-004 : LumaPixelsDifferentBy_4 = 3)
1498] UNDO: Matched 1391 -> 1498 : Skip 1392 to 1498 : (FDif=3.694935E-003 : LumaPixelsDifferentBy_4 = 8)
1327] UNDO: Matched 1273 -> 1327 : Skip 1274 to 1327 : (FDif=5.472770E-003 : LumaPixelsDifferentBy_4 = 19)
1262] UNDO: Matched 1234 -> 1262 : Skip 1235 to 1262 : (FDif=3.011875E-004 : LumaPixelsDifferentBy_4 = 3)
1217] UNDO: Matched 1204 -> 1217 : Skip 1205 to 1217 : (FDif=8.842202E-006 : LumaPixelsDifferentBy_4 = 0)
1190] UNDO: Matched 1099 -> 1190 : Skip 1100 to 1190 : (FDif=2.953296E-003 : LumaPixelsDifferentBy_4 = 11)
1039] UNDO: Matched 1029 -> 1039 : Skip 1030 to 1039 : (FDif=8.068509E-005 : LumaPixelsDifferentBy_4 = 1)
957] UNDO: Matched 951 -> 957 : Skip 952 to 957 : (FDif=1.050012E-005 : LumaPixelsDifferentBy_4 = 0)
945] UNDO: Matched 910 -> 945 : Skip 911 to 945 : (FDif=3.842490E-003 : LumaPixelsDifferentBy_4 = 7)
814] UNDO: Matched 805 -> 814 : Skip 806 to 814 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
758] UNDO: Matched 738 -> 758 : Skip 739 to 758 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
736] UNDO: Matched 673 -> 736 : Skip 674 to 736 : (FDif=5.128477E-003 : LumaPixelsDifferentBy_4 = 13)
609] UNDO: Matched 557 -> 609 : Skip 558 to 609 : (FDif=3.419169E-003 : LumaPixelsDifferentBy_4 = 9)
536] UNDO: Matched 527 -> 536 : Skip 528 to 536 : (FDif=6.465860E-005 : LumaPixelsDifferentBy_4 = 0)
518] UNDO: Matched 469 -> 518 : Skip 470 to 518 : (FDif=4.058571E-003 : LumaPixelsDifferentBy_4 = 14)
432] UNDO: Matched 418 -> 432 : Skip 419 to 432 : (FDif=8.242038E-003 : LumaPixelsDifferentBy_4 = 40)
395] UNDO: Matched 388 -> 395 : Skip 389 to 395 : (FDif=1.989495E-005 : LumaPixelsDifferentBy_4 = 0)
359] UNDO: Matched 324 -> 359 : Skip 325 to 359 : (FDif=6.175726E-003 : LumaPixelsDifferentBy_4 = 16)
312] UNDO: Matched 303 -> 312 : Skip 304 to 312 : (FDif=0.000000E+000 : LumaPixelsDifferentBy_4 = 0)
212] UNDO: Matched 207 -> 212 : Skip 208 to 212 : (FDif=7.736927E-006 : LumaPixelsDifferentBy_4 = 0)
205] UNDO: Matched 198 -> 205 : Skip 199 to 205 : (FDif=8.842202E-006 : LumaPixelsDifferentBy_4 = 0)
164] UNDO: Matched 111 -> 164 : Skip 112 to 164 : (FDif=1.198560E-002 : LumaPixelsDifferentBy_4 = 36)
95] UNDO: Matched 88 -> 95 : Skip 89 to 95 : (FDif=4.421101E-006 : LumaPixelsDifferentBy_4 = 0)
23] UNDO: Matched 0 -> 23 : Skip 1 to 23 : (FDif=7.389318E-003 : LumaPixelsDifferentBy_4 = 14)

Kept 646 of 1650 frames [1920x1080 110.00 secs (1.83 Mins) @ 15.00 FPS]
File Max Successful CrudMax = 7.427458E-003 CrudMaxFrame=432
File Max Successful LumaPixelsDifferentBy_4 = 40 @ Frame = 432
QWIK SCAN 173.10 Secs (2.88 Mins) InFPS=9.53 OutFPS=3.73 FramesSearched=490864 FramesSearchedFPS=2835.73
2/2 'D:\ED\Test 00 tsc2.avi.AVI' Tot File Time = 271.88 Seconds (4.53 Mins)

BATCH TOTAL Time = 534.00 Seconds (8.90 Mins)
Global CrudMax=0.007427
Global MaxLumaPixelsDifferentBy_4 = 40


This is crud lump (ie not crud, detection mistake in CRUD_INSPECT script), see if you can spot it (can be any color).
https://s20.postimg.cc/vlutc4wal/0_zpsd1bd81e6.jpg (https://postimg.cc/image/faupftjsp/)

EDIT:
Make this alteration near end of script, (missed writing last frame 0 to log)

if(!SKIP || i==0) {
if(VERBOSITY>=3 || i==0) {
if(i==0) {S=RT_String("%5d] WRITING: Frame %d",i,i) }
Else {S=RT_String(" WRITING: Frame %d",i) }
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
RT_WriteFile(CMDFrames,"%d",i,Append=True)
Kept = Kept + 1
}


EDIT: Mobile now. Pub shuts in 100 minutes, gotta dash to get some more in.
Crud lump synthesised, not produced on your clips.
Think, 100% satisfactory on both clips.

[StainlessS]

Cute lil chipmonks here : https://www.youtube.com/watch?v=n3M7IR6jkpc

Dont hold that against me, it has been a difficult task.

So far as I am concerned, tis fixed (let me know how it goes on longer clips).

[StainlessS]

Post #1 of 2 (update script for RT_Stats v2.0, converted to using RT_ Arrays instead of DBase)

Fsel_Eduardobedoya_Batch_04Dec2016.avs Part 1


# ############################
# Fsel_Eduardobedoya_Batch.avs, by StainlessS
# Remove Undos from painting captures.
########################################################################
THRESH = 0.015 # (0.015) Max LumaTol due to crud between undone/undo frames. (set about double CrudMax)
# (CrudMax only valid as Check when results are satisfactory)
# Upper Limit for self tuning LumaTol.
#
LUMATOL_SCALE = 1.0 # (1.0) Range 1.0 -> 2.0. Should never need change from 1.0.
# LumaTol is self tuning but could possibly be in error where without this setting would fail miserably.
# If self tune LumaTol is set too low then will not find matching UNDONE frames. This setting allows
# to increase LumaTol as LumaTol is multiplied by it, and then restricted at upper limit by THRESH above.
# If ever need to be changed, suggest something like 1.00001. (temporary change only)
#
LUMATOL_ADD = 0.0 # (0.0) Range 0.0 -> 1.0. Additional adjustment added to LumaTol, probably never necessary, but if so then
# something like 0.000001. Applied before limiting to THRESH as above.
#
MIN_EDITLEN = 4 # (4) An edit has to be at least this many frames long (frames between UNDONE and UNDO, exclusive)
OVR_PIXCNT_THR = 4 # (4) Thresh for RT_LumaPixelsDifferentCount, only pixel differences greater than this are counted.
OVR_PIXCNT_LIM = 50 # (50)If RT_LumaPixelsDifferentCount(Thresh=OVR_PIXCNT_THR) greater than this then is OVERRIDDEN as false detection.
# Above OVR_ settings for detecting override where difference between undone/undo is too visible (ie not crud).
#
ScanAheadSecs = 10 * 60 # Search ahead range in seconds
ChromaWeight = 1.0/3.0 # (1.0/3.0) YUV Chroma Weighting, 0.0 -> 1.0
FPS = 15.0 # Play speed for output script.
VERBOSITY = 0 # (0) 0 to 3. Debug & logging verbosity.
########################################################################
# Chop off crud around outsides
GLOBAL CROP_L = 128 # Crop Left
GLOBAL CROP_T = 128 # Crop Top
GLOBAL CROP_R = 328 # Crop Right (Including that thumbnail with the delayed UNDO and additional crap)
GLOBAL CROP_B = 128 # Crop Bottom
########################################################################
# During Testing
GLOBAL CREATE = True
GLOBAL DELETE_ARR = false
########################################################################
GLOBAL PC709 = True
GLOBAL CROP_L = (CROP_L / 4) * 4 GLOBAL CROP_T = (CROP_T / 4) * 4
GLOBAL CROP_R = (CROP_R / 4) * 4 GLOBAL CROP_B = (CROP_B / 4) * 4
FSEL_TITLE="Select AVI files"
FSEL_DIR="."
FSEL_FILT="Avi files|*.avi"
FSEL_MULTI=True

AVIFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(AVIFILE_LIST.IsString,"RT_FSelOpen: Error="+String(AVIFILE_LIST))


NFILES=RT_TxtQueryLines(AVIFILE_LIST) # Query Number of lines in String ie number of files.
myName="Fsel_Eduardobedoya_Batch: "
LOG="Fsel_Eduardobedoya_Batch.Log"
RT_WriteFile(LOG,"%s\n",LOG,Append=False)

S=RT_String("THRESH = %f LUMATOL_SCALE=%f LUMATOL_ADD=%f\nMIN_EDITLEN=%d OVR_PIXCNT_THR=%d OVR_PIXCNT_LIM=%d",
\ THRESH,LUMATOL_SCALE,LUMATOL_ADD,MIN_EDITLEN,OVR_PIXCNT_THR,OVR_PIXCNT_LIM)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

S=RT_String("ScanAheadSecs=%d ChromaWeight=%f OutFPS=%f",ScanAheadSecs,ChromaWeight,FPS)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

S=RT_String("PC709=%s CROPPING=%d,%d,%d,%d",PC709,CROP_L,CROP_T,CROP_R,CROP_B)
RT_WriteFile(LOG,"%s",S,Append=True)
RT_DebugF("%s",S,name=myName)

GLOBAL Global_CrudMax = 0.0
Global Global_PixCntMax = 0

Crud_S="""
AMP=True
PCTHRESH=@
aviName=$@@$
PC709=@@@
Path=RT_FilenameSplit(AviName,3)
Node=RT_FilenameSplit(AviName,4)
CrudFrames = Path + $CrudInspect_Frames_$ + Node + $.TXT$
Avisource(AviName)
Crop(CROP_L,CROP_T,-CROP_R,-CROP_B)
ConvertToYV12(matrix=(PC709)?$PC.709$:$PC.601$)
NUMBERSCLIP=FrameSel(Cmd=CrudFrames,Ordered=False,SHOW=TRUE)
NUMBERSCLIP=NUMBERSCLIP.CROP(0,0,20*10,1*20).POINTResize(20*10*2,1*20*2)
NA=NUMBERSCLIP.SelectEven
NB=NUMBERSCLIP.SelectODD
FrameSel(Cmd=CrudFrames,Ordered=False)
A=SelectEven
B=SelectODD
DELTA = Clipdelta(A,B,AMP)
DELTA = DELTA.OVERLAY(NA,X=0,Y=DELTA.HEIGHT-NA.HEIGHT).OVERLAY(NB,X=DELTA.WIDTH-NB.WIDTH,Y=DELTA.HEIGHT-NA.HEIGHT)
DELTA
SC=$$$
pc=RT_LumaPixelsDifferentCount(A,B)
pct=RT_LumaPixelsDifferentCount(A,B,THRESH=PCTHRESH)
ld=RT_LumaDifference(A,B)
fd=RT_FrameDifference(A,B)
S=RT_String($UNDO=%d] LumaDif=%f FrameDif=%f LumaPixelsDifferent=%d LumaPixelsDifferentByMoreThan_%d=%d$,current_frame,ld,fd,pc,PCTHRESH,pct)
Subtitle(S,size=24,font=$Courier New$)
$$$
ScriptClip(SC)
return Last

# Return Clip Difference of input clips (amp==true = Amplified, show==true = show background)
Function ClipDelta(clip clip1,clip clip2,bool $amp$,bool $show$) {
amp=Default(amp,false)
show=Default(show,false)
c2=clip1.levels(128-32,1.0,128+32,128-32,128+32).greyscale()
c1=clip1.subtract(clip2)
c1=(amp)?c1.levels(127,1.0,129,0,255):c1
return (show)?c1.Merge(c2):c1
}
"""

GSCript("""
TOTSTART = RT_TimerHP
For(i=0,NFILES-1) {
START = RT_TimerHP
FN=RT_TxtGetLine(AVIFILE_LIST,i) # Filename of avi file i
S=RT_String("\n%d/%d ] Processing '%s'\n",i+1,NFILES, FN)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)

DropUndo(FN,Crud_S,ScanAheadSecs,thresh=THRESH,Min_EditLen=MIN_EDITLEN,
\ log=LOG,ChromaWeight=ChromaWeight,fps=FPS,
\ ovr_pixcnt_thr=OVR_PIXCNT_THR,ovr_pixcnt_lim=OVR_PIXCNT_LIM,
\ LumaTol_Scale=LUMATOL_SCALE,LumaTol_Add=LUMATOL_ADD,Verbosity=VERBOSITY)

T = RT_TimerHP - START
S=RT_String("%d/%d '%s' Tot File Time = %.2f Seconds (%.2f Mins)",i+1,NFILES,FN, T,T/60.0)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
T = RT_TimerHP - TOTSTART
S=RT_String("\nBATCH TOTAL Time = %.2f Seconds (%.2f Mins)\nGlobal CrudMax=%f\nGlobal MaxLumaPixelsDifferentBy_%d = %d",
\ T,T/60.0,GLOBAL_CrudMax,OVR_PIXCNT_THR,Global_PixCntMax)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("\nDONE\n\n%s",S)
S=RT_StrReplace(S,Chr(10),"\n")
""")


Return blankclip(length=24*60*60*24).Subtitle(S,Align=5,Y=100,lsp=0,Size=30)

[StainlessS]

Post #2 of 2

Fsel_Eduardobedoya_Batch_04Dec2016.avs Part 1



Function DropUndo(String AviName,String Crud_S,Float "ScanAheadSecs",Float "THRESH",Int "Min_EditLen",
\ String "Log",Float "ChromaWeight",float "fps",Int "ovr_pixcnt_thr",Int "ovr_pixcnt_lim",
\ Float "LumaTol_Scale",Float "LumaTol_Add", Int "Verbosity") {
myName="DropUndo: "
ScanAheadSecs=Float(Default(ScanAheadSecs,10.0*60.0))
THRESH=Float(Default(THRESH,0.02))
MIN_EDITLEN=Default(MIN_EDITLEN,4)
LOG = Default(LOG,"DropUndo.LOG")
ChromaWeight = Float(Default(ChromaWeight,1.0/3.0))
FPS = Float(Default(FPS,FPS))
OVR_PIXCNT_THR=Default(OVR_PIXCNT_THR,4)
OVR_PIXCNT_LIM=Default(OVR_PIXCNT_LIM,50)
LumaTol_Scale = Float(Default(LumaTol_Scale,1.0))
LumaTol_Add = Float(Default(LumaTol_Add,0.0))
VERBOSITY=Default(VERBOSITY,0)

Assert(Exist(AviName),myName+AviName+" Does Not Exist")

Avisource(AviName)
ORG=Last

Crop(CROP_L,CROP_T,-CROP_R,-CROP_B) # Crop crud
ConvertToYV12(Matrix=(PC709)?"PC.709":"PC.601")

Assert(ScanAheadSecs>0.0,myName+"ScanAheadSecs Must be greater than zero")
Assert(THRESH>0.0, myName+"THRESH Must be greater than zero")
Assert(MIN_EDITLEN>0,myName+"Greater than zero please")
Assert(ChromaWeight>=0.0 && ChromaWeight<=1.0, myName+"ChromaWeight range 0.0 -> 1.0")
Assert(OVR_PIXCNT_THR>=0 && OVR_PIXCNT_THR<=8, myName+"OVR_PIXCNT_THR range 0 -> 8")
Assert(OVR_PIXCNT_LIM>=0 && OVR_PIXCNT_LIM<=100, myName+"OVR_PIXCNT_LIM range 0 -> 100")
Assert(LumaTol_Scale>=1.0 && LumaTol_Scale<=2.0, myName+"LUMATOL_SCALE range 1.0 -> 2.0")
Assert(LumaTol_Add>=0.0 && LumaTol_Add<=1.0, myName+"LUMATOL_ADD range 0.0 -> 1.0")
Assert(VERBOSITY>=0 && VERBOSITY<=3, myName+"VERBOSITY range 0 -> 3")

Path = RT_FilenameSplit(AviName,3) # Drive + Dir
Node = RT_FilenameSplit(AviName,4) # Name
PathAndNode = Path+Node # Drive + Dir + Name
CMDFrames=PathAndNode+"_Frames.TXT"
Ranges=PathAndNode+"_Ranges.TXT"
ScriptFile=Path+"_AVS_"+Node+"_SelectFrames.AVS"
CrudScript = Path + "_CRUD_INSPECT_" + Node + ".AVS"
CrudFrames = Path + "CrudInspect_Frames_" + Node + ".TXT"
Fnd_S = RT_String("CROP_L\nCROP_T\nCROP_R\nCROP_B\n$\n@@@\n@@\n@\n")
Rep_S = RT_String("%d\n%d\n%d\n%d\n%c\n%s\n%s\n%d",CROP_L,CROP_T,CROP_R,CROP_B,34,PC709,RT_GetFullPathName(AviName),OVR_PIXCNT_THR)
Crud_S = RT_StrReplaceMulti(Crud_S,Fnd_S,Rep_S)

ScanAheadFrames = Int(ScanAheadSecs*FrameRate)
ARR=PathAndNode+".ARR"
PREV=PathAndNode + "_Prev.ARR"
NEXT=PathAndNode + "_Next.ARR"

RT_FileDelete(CMDFrames) # Delete any existing frames file
RT_FileDelete(Ranges) # Delete any existing Ranges file
RT_FileDelete(CrudFrames)
RT_FileDelete(CrudScript)
CRUD_DB="~"+RT_LocalTimeString+".DB"
RT_DBaseAlloc(CRUD_DB,0,"ii") # CRUD_DB int,int

### TEMPLATE script
FrameSel_Select="""
Show=False # Set True to show original Frame Number
fn="%s"
Avisource(fn)
(Show) ? ScriptClip("Subtitle(String(current_frame))") : NOP
PathAndNode="%s"
CmdFrames=PathAndNode+"_Frames.txt"
Ranges=PathAndNode+"_Ranges.txt"
Ex=Exist(CmdFrames)
(Ex) ? FrameSel_CmdReWrite(Ranges,cmd=CmdFrames,reject=False) : NOP # Informational ONLY
(Ex) ? FrameSel(cmd=CmdFrames,reject=False) : NOP
Return Last.AssumeFPS(%f)
"""

###
Select_S = FrameSel_Select
Select_S = RT_StrReplaceDeep(RT_StrReplace(Select_S,Chr(9)," ")," "," ") # TAB and SPACE compact
Select_S = RT_String(Select_S,AviName,PathAndNode,FPS) # Insert filenames

FramesSearched = 0
CrudMax = 0.0
CrudMaxFrame = -1
PixCntMax = 0
PixCntMaxFrame = -1
Kept = 0

GSCript("""
if(CREATE) {
START = RT_TimerHP
RT_QwikScanCreate(ARR,prev="",next=NEXT,debug=true)
T= RT_TimerHP - START
S = RT_String("QWIK Scan ARRAY creation = %.2f Secs (%.2f Mins)",T,T/60.0)
RT_WriteFile(LOG,"%s",S,Append=True)
}
START = RT_TimerHP
LastFrame=FrameCount-1
RT_DebugF(" QWIK Scanning file ... Please Wait",name=myName)

for(i=LastFrame,0,-1) {
SKIP = False
if(i > 0) {
PreUndoDif = RT_FrameDifference(Last,Last,n=i,n2=i-1,ChromaWeight=ChromaWeight)
FramesSearched = FramesSearched + 1
if(PreUndoDif > 0.0) {
StartFrame = Max(i-ScanAheadFrames,0)
EndFrame = (i - 1 - MIN_EDITLEN)

# We are looking for an UNDONE frame that is more similar to i frame than i-1.
# If i is an UNDO frame then UNDONE frame will be more similar to i than i-1, so LumaTol estimate will find it fast.
# Alternatively, if i is not an UNDO frame then LumaTol estimate will speed up NOT FINDING frame.
# If estimate greater than THRESH, then i could still be an UNDO frame just i-1 is quite a lot different to i frame, so we
# limit LumaTol to THRESH assuming well set for Crud max.
# Below two lines speed up whole script significantly (rather than just using THRESH setting for LumaTol).

# arg frame order IS SIGNIFICANT (we are matching to i frame, estimate is for i-1).
LumaTol = RT_QwikScanEstimateLumaTol(Last,Last,n=i,n2=i-1)
LumaTol = LumaTol_Scale * LumaTol + LumaTol_Add
LumaTol = (LumaTol>THRESH) ? THRESH : LumaTol

DifThresh = PreUndoDif*0.999999 # We are looking for BETTER match, NOT equal

if(VERBOSITY>=1) {
S = RT_String("%5d] NEWSEARCH: SearchStart=%d SearchEnd=%d Seeking FDif <= %E {LumaTol=%E}",
\ i,StartFrame,EndFrame,DifThresh,LumaTol)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
For(j=StartFrame,EndFrame) {
MaxDistance = EndFrame - j
if(VERBOSITY>=2) {
S = RT_String(" SEARCH: SearchStart=%d MaxDistance=%d",j,MaxDistance)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
# XP=3, strictly better than, ie left most frame (first found, furthest from i).
# MaxDistance limits search to exclude MIN_EDITLEN frames.
Result=RT_QwikScan(Last,j,Last,i,ARR,NEXT,lumatol=LumaTol,Flags=$04,fd=DifThresh,maxdistance=MaxDistance,XP=3)
if(Result>=0) { # Exit Condition succeeds
FramesSearched = FramesSearched + (QWKS_BM_FD_FRM-j+1)
if(VERBOSITY>=2) {
S = RT_String(" MATCH: Frame=%5d FrameDifference=%E ",QWKS_BM_FD_FRM,QWKS_BM_FD)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
CrudPix=RT_LumaPixelsDifferentCount(Last,Last,n=i,n2=QWKS_BM_FD_FRM,Thresh=OVR_PIXCNT_THR)
FramesSearched = FramesSearched + 1
if(CrudPix <= OVR_PIXCNT_LIM) {
YCrudDif = RT_QwikScanEstimateLumaTol(Last,Last,n=i,n2=QWKS_BM_FD_FRM) # Crud measured by LumaTol estimate
FramesSearched = FramesSearched + 1
if(YCrudDif > CrudMax) {
CrudMax = YCrudDif
CrudMaxFrame = i
}
if(CrudPix > PixCntMax) {
PixCntMax = CrudPix
PixCntMaxFrame = i
}
RT_DBaseAppend(CRUD_DB,QWKS_BM_FD_FRM,i)
if(VERBOSITY==0) {
S=RT_String("%5d] UNDO: Matched %d -> %d : Skip %d to %d : (FDif=%E : LumaPixelsDifferentBy_%d = %d)",
\ i,QWKS_BM_FD_FRM,i,QWKS_BM_FD_FRM+1,i,QWKS_BM_FD,OVR_PIXCNT_THR,CrudPix)
} Else {
S=RT_String(" UNDO: Matched %d -> %d : Skip %d to %d : (FDif=%E : LumaPixelsDifferentBy_%d = %d)",
\ QWKS_BM_FD_FRM,i,QWKS_BM_FD_FRM+1,i,QWKS_BM_FD,OVR_PIXCNT_THR,CrudPix)
}
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
i = QWKS_BM_FD_FRM+1 # Skip edited Frames up to and including UNDO frame: Rescan @ i - 1
SKIP = True # Dont write i
if(VERBOSITY>=2) {
S=RT_String(" RESCAN: @ %d",i-1)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
j = EndFrame # Break
} Else {
if(VERBOSITY>=2){
S=RT_String(" OVERRIDE: LumaPixelsDifferentBy_%d = %d (I -> Matched Frame)",OVR_PIXCNT_THR,CrudPix)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
j = QWKS_BM_FD_FRM # Continue search
}
} Else { # Result < 0, Exit condition fails
FramesSearched = FramesSearched + MaxDistance + 1 # Searched full extent
if(VERBOSITY>=3) {
if(QWKS_BM_FLAGS!=0) { # Got a Best Match Only
S = RT_String(" NOT FOUND: Best Match Frame=%5d FDif=%E",QWKS_BM_FD_FRM,QWKS_BM_FD)
} Else { # Did not even get a Best match
S = RT_String(" NOT FOUND: No Best Match:")
}
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
j = EndFrame # Break
}
} # End for j
} Else { # PreUndoDif <= 0.0
if(VERBOSITY>=2) {
d = i-1
for(k=d-1,0,-1) {
Dif = RT_FrameDifference(Last,Last,n=i,n2=k,ChromaWeight=ChromaWeight)
FramesSearched = FramesSearched + 1
if(Dif == 0.0) {
d = k
} Else {
k = - 1
}
}
S=RT_String("%5d] IDENTICAL: Frames %d to %d : Skipping Frames %d to %d",i,d,i,d+1,i)
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
i = d+1 # Retry at d
}
SKIP = True # Dont write i
}
} # i == 0
if(!SKIP || i==0) {
if(VERBOSITY>=3 || i==0) {
if(i==0) {S=RT_String("%5d] WRITING: Frame %d",i,i) }
Else {S=RT_String(" WRITING: Frame %d",i) }
RT_DebugF("%s",S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
}
RT_WriteFile(CMDFrames,"%d",i,Append=True)
Kept = Kept + 1
}
}

# Delayed write of crud frames (cannot now use FrameSel(Ordered=True) in Crud_Inspect script due to RESCAN duplicate frames Nos)
for(i=RT_DBaseRecords(CRUD_DB)-1,0,-1) {
RT_WriteFile(CrudFrames,"%d\n%d",RT_DBaseGetField(CRUD_DB,i,0),RT_DBaseGetField(CRUD_DB,i,1),Append=True)
}
RT_FileDelete(CRUD_DB)

RT_WriteFile(ScriptFile,"%s",Select_S)
RT_WriteFile(CrudScript,"%s",Crud_S)
(DELETE_ARR) ? RT_FileDelete(ARR) : NOP
(DELETE_ARR) ? RT_FileDelete(NEXT) : NOP
T = RT_TimerHP - START
FT=FrameCount / FrameRate
S=RT_String("\nKept %d of %d frames [%dx%d %.2f secs (%.2f Mins) @ %.2f FPS]",Kept, FrameCount,ORG.Width,ORG.Height,FT,FT/60.0,FrameRate)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("File Max Successful CrudMax = %E CrudMaxFrame=%d",CrudMax,CrudMaxFrame)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("File Max Successful LumaPixelsDifferentBy_%d = %d @ Frame = %d",ovr_pixcnt_thr,PixCntMax,PixCntMaxFrame)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
S=RT_String("QWIK SCAN %.2f Secs (%.2f Mins) InFPS=%.2f OutFPS=%.2f FramesSearched=%d FramesSearchedFPS=%.2f",
\ T,T/60.0,FrameCount/T,Kept/T,FramesSearched,FramesSearched/T)
RT_DebugF(S,name=myName)
RT_WriteFile(LOG,"%s",S,Append=True)
Global Global_CrudMax = Max(Global_CrudMax,CrudMax)
Global Global_PixCntMax = Max(Global_PixCntMax,PixCntMax)
""")
Return 0
}