RT_Stats, Compile-time/Runtime Functions v1.43 - 08 Oct 2014

[real.finder]

Abridgement.avs

ColorBars().ShowFrameNumber

SCMD="""
2,198
199,242
243, 2410
2411, 20767
20768, 39916
39917, 42634
42635, 43513
"""

outFn = ".\Abridgement.txt"

FrameSel_CmdReWrite(Last,Outfn, scmd=SCmd, Reject=False,Ordered=True,Range=True,Space=False,Prune=False)

Return Last


Abridgement.txt

2,43513


EDIT: Or this (Same output result)

ColorBars().ShowFrameNumber

INCMD="""
2,198
243, 2410
20768, 39916
42635, 43513
"""

OUTCMD="""
199,242
2411, 20767
39917, 42634
"""

SCMD = INCMD + OUTCMD

outFn = ".\Abridgement.txt"

FrameSel_CmdReWrite(Last,Outfn, scmd=SCmd, Reject=False,Ordered=True,Range=True,Space=False,Prune=False)

Return Last

Above, Orders and coalesces all the ranges, outputting only distinct ranges, in above case only a single output range.

EDIT: What is the end result of all of this, give complete description of ALL required [and take your time :) ].

thanks, I was working on this too

function abtrim(string "trimline") {
Assert(defined(trimline), "'intrim' trimline is needed")
abtrimloop(LCase(trimline))
}

function abtrimloop(string "trimline", int "remainder") {
remainder = default(remainder, -1)
otrimline = trimline
trimline = remainder > 0 ? RightStr(trimline, remainder) : trimline
ofindtrim = FindStr(trimline, "trim")
sishtrim = ofindtrim != 0
trimstart = FindStr(trimline, "(")
findtrimstart = trimstart + 1
trimend = FindStr(trimline, ")")
findtrimend = trimend - 1
taketrim = MidStr(trimline, findtrimstart, findtrimend)
trims = FindStr(taketrim, ",")
strim = !sishtrim ? nop() : MidStr(taketrim,1,trims-1).Value.Int
etrim = !sishtrim ? nop() : MidStr(taketrim,trims+1).Value.Int
ntrimr = RightStr(trimline,StrLen(trimline)-trimend)
ofindntrim = FindStr(ntrimr, "trim")
sishntrim = ofindntrim != 0
ntrimstart = sishntrim ? FindStr(ntrimr, "(") : nop()
findntrimstart = !sishntrim ? nop() : ntrimstart + 1
ntrims = !sishntrim ? nop() : FindStr(ntrimr, ",")
sntrim = !sishntrim ? nop() : MidStr(ntrimr,findntrimstart,ntrims-1).Value.Int
ntrimline=LeftStr(otrimline,trims+findtrimstart+(remainder > 0 ? remainder : 0))+(((etrim+1)==sntrim) ? MidStr(ntrimr,ntrims+1) : MidStr(trimline,trims+findtrimstart+1))
pointer = !sishtrim ? 0 : remainder != 0 ? remainder == -1 ? ntrimline != otrimline ? FindStr(ntrimline, "+") != 0 ? trims+findtrimstart-1 : 0 : FindStr(ntrimline, "+") : remainder + FindStr(RightStr(ntrimline, StrLen(ntrimline) - remainder), "+") : nop()
pointer != 0 ? abtrimloop(ntrimline, pointer) : ntrimline
}

it work with abtrim("trim(2, 198)++trim(199, 242)++trim(243, 2410)++Trim(2411, 20767)++Trim(20768, 39916)++Trim(39917, 42634)++Trim(42635, 43513)")

but still not for this (need fix, still working on it) abtrim("trim(2, 198)++trim(243, 2410)++Trim(2411, 20767)++Trim(20768, 39916)++Trim(39917, 42634)++Trim(42635, 43513)")

and it need nothing but avs

[StainlessS]

I'll look again at all of this tomorrow.
Would be a lot easier (I think) if had frames list source (String or file), and processed that, instead of screwing around trying to parse
list with "Trim(xxx, yyy) ++ ..." etc, much easier to create trim command from frames string/file than
to parse trims command and extract frames list from that.

Also, making it all work in Avs script alone [and properley] is not gonna be easy, nor small.
EDIT: Although with recent AVS+ arrays at least can implement frame range sorting and coalescing with less difficulty.
(I'm not using array supporting versions of Avs+, it dont work with XP MeGUI, so I'm pretty much sticking with avs+ 3.5, r3106)

EDIT:
and it need nothing but avs
Is RT_ usage OK ? [you use it for RT_writeFile() ].
For your abridgement thing, you need to parse and verify correctness of your frame ranges, eg end frame same or after start frame,
for coalescing, you also need sorting all ranges beforehand. If you dont do it properley, it will bite your ass the first time
somebody screws up the trims/ranges list.
Without proper parsing/sorting/checking, will run risk of eg Infinite Recursion, stack exhaustion, where script/avisynth will just disappear
without any error message or indication of what went wrong, not a good way to crash, or, it might just badly mangle your result clip.
The things you are attempting are quite tricky even in C, in basic script is gonna be less easy.

EDIT: Also, before you use RT_WriteFile with Append=true, is best to use RT_FileDelete on the target file [in the intrim() func].
Otherwise on multiple use eg

0,1
199,242
2411,20767
39917,42634
43514,107891
0,1
199,242
2411,20767
39917,42634
43514,107891
...
etc

[real.finder]

Is RT_ usage OK ? [you use it for RT_writeFile() ].
yes, in fact abtrim is for helping intrim

EDIT: Also, before you use RT_WriteFile with Append=true, is best to use RT_FileDelete on the target file [in the intrim() func].

yes, I note this, will use it when I done with abtrim

edit: I think this did it!

function abtrim(string "trimline") {
Assert(defined(trimline), "'intrim' trimline is needed")
abtrimloop(LCase(trimline))
}

function abtrimloop(string "trimline", int "remainder") {
remainder = default(remainder, -1)
otrimline = trimline
sisending = !(remainder < StrLen(trimline))
trimline = remainder > 0 ? !sisending ? RightStr(trimline, StrLen(trimline) - remainder) : trimline : trimline
ofindtrim = sisending ? nop() : FindStr(trimline, "trim")
sishtrim = sisending ? false : ofindtrim != 0
trimstart = !sishtrim ? nop() : ofindtrim + FindStr(MidStr(trimline, ofindtrim + 1), "(")
findtrimstart = !sishtrim ? nop() : trimstart + 1
trimend = !sishtrim ? nop() : ofindtrim + FindStr(MidStr(trimline, ofindtrim + 1), ")")
findtrimend = !sishtrim ? nop() : trimend - 1
taketrim = !sishtrim ? nop() : MidStr(trimline, findtrimstart, findtrimend - findtrimstart + 1)
trims = !sishtrim ? nop() : FindStr(taketrim, ",")
strim = !sishtrim ? nop() : MidStr(taketrim,1,trims-1).Value.Int
etrim = !sishtrim ? nop() : MidStr(taketrim,trims+1).Value.Int
ntrimr = !sishtrim ? nop() : RightStr(trimline,StrLen(trimline)-trimend)
ofindntrim = !sishtrim ? nop() : FindStr(ntrimr, "trim")
sishntrim = !sishtrim ? nop() : ofindntrim != 0
ntrimstart = !sishtrim ? nop() : sishntrim ? FindStr(ntrimr, "(") : nop()
findntrimstart = !sishtrim ? nop() : !sishntrim ? nop() : ntrimstart + 1
ntrims = !sishtrim ? nop() : !sishntrim ? nop() : FindStr(ntrimr, ",")
sntrim = !sishtrim ? nop() : !sishntrim ? nop() : MidStr(ntrimr,findntrimstart,ntrims-1).Value.Int
domerge = !sishtrim ? nop() : sishntrim ? ((etrim+1)==sntrim) : false
ntrimline=!sishtrim ? nop() : !domerge ? otrimline : LeftStr(otrimline,(remainder > 0 ? remainder : 0) + findtrimstart + trims - 1) + MidStr(ntrimr,ntrims+1)
remainder = !sishtrim ? nop() : domerge ? remainder : max(0,remainder) + trimend
!sishtrim ? otrimline : sishntrim ? abtrimloop(ntrimline, remainder) : ntrimline
}

edit: even better for both

function abtrim(string "trimline") {
Assert(defined(trimline), "'intrim' trimline is needed")
function abtrimloop(string "trimline", int "remainder") {
remainder = default(remainder, -1)
otrimline = trimline
sisending = !(remainder < StrLen(trimline))
trimline = remainder > 0 ? !sisending ? RightStr(trimline, StrLen(trimline) - remainder) : trimline : trimline
ofindtrim = sisending ? nop() : FindStr(trimline, "trim")
sishtrim = sisending ? false : ofindtrim != 0
trimstart = !sishtrim ? nop() : ofindtrim + FindStr(MidStr(trimline, ofindtrim + 1), "(")
findtrimstart = !sishtrim ? nop() : trimstart + 1
trimend = !sishtrim ? nop() : ofindtrim + FindStr(MidStr(trimline, ofindtrim + 1), ")")
findtrimend = !sishtrim ? nop() : trimend - 1
taketrim = !sishtrim ? nop() : MidStr(trimline, findtrimstart, findtrimend - findtrimstart + 1)
trims = !sishtrim ? nop() : FindStr(taketrim, ",")
strim = !sishtrim ? nop() : MidStr(taketrim,1,trims-1).Value.Int
etrim = !sishtrim ? nop() : MidStr(taketrim,trims+1).Value.Int
ntrimr = !sishtrim ? nop() : RightStr(trimline,StrLen(trimline)-trimend)
ofindntrim = !sishtrim ? nop() : FindStr(ntrimr, "trim")
sishntrim = !sishtrim ? nop() : ofindntrim != 0
ntrimstart = !sishtrim ? nop() : sishntrim ? FindStr(ntrimr, "(") : nop()
findntrimstart = !sishtrim ? nop() : !sishntrim ? nop() : ntrimstart + 1
ntrims = !sishtrim ? nop() : !sishntrim ? nop() : FindStr(ntrimr, ",")
sntrim = !sishtrim ? nop() : !sishntrim ? nop() : MidStr(ntrimr,findntrimstart,ntrims-1).Value.Int
domerge = !sishtrim ? nop() : sishntrim ? ((etrim+1)==sntrim) : false
ntrimline=!sishtrim ? nop() : !domerge ? otrimline : LeftStr(otrimline,(remainder > 0 ? remainder : 0) + findtrimstart + trims - 1) + MidStr(ntrimr,ntrims+1)
remainder = !sishtrim ? nop() : domerge ? remainder : max(0,remainder) + trimend
!sishtrim ? otrimline : sishntrim ? abtrimloop(ntrimline, remainder) : ntrimline
}
abtrimloop(LCase(trimline))
}

function intrim(clip i, string "trimline", string "output") {
Assert(defined(trimline), "'intrim' trimline is needed")
i
lastframe = framecount() - 1
TrimFile=defined(output) ? output : "TrimFile.TXT"
Exist(TrimFile) ? RT_FileDelete(TrimFile) : nop()
function intrimloop(clip i, string "trimline", string "output", int "lastframe", bool "firstcall") {
firstcall = default(firstcall, false)
ofindtrim = FindStr(trimline, "trim")
sishtrim = ofindtrim != 0
trimstart = !sishtrim ? nop() : FindStr(trimline, "(")
findtrimstart = !sishtrim ? nop() : trimstart + 1
trimend = !sishtrim ? nop() : FindStr(trimline, ")")
findtrimend = !sishtrim ? nop() : trimend - 1
taketrim = !sishtrim ? nop() : MidStr(trimline, findtrimstart, findtrimend-findtrimstart+1)
trims = !sishtrim ? nop() : FindStr(taketrim, ",")
strim = !sishtrim ? nop() : MidStr(taketrim,1,trims-1).Value.Int
etrim = !sishtrim ? nop() : MidStr(taketrim,trims+1).Value.Int
etrim = !sishtrim ? nop() : etrim == 0 ? lastframe : etrim
ntrimr = !sishtrim ? nop() : rightStr(trimline,StrLen(trimline)-trimend)
ofindntrim = !sishtrim ? nop() : FindStr(ntrimr, "trim")
sishntrim = !sishtrim ? nop() : ofindntrim != 0
ntrimstart = !sishtrim ? nop() : sishntrim ? FindStr(ntrimr, "(") : nop()
findntrimstart = !sishtrim ? nop() : !sishntrim ? nop() : ntrimstart + 1
ntrims = !sishtrim ? nop() : !sishntrim ? nop() : FindStr(ntrimr, ",")
sntrim = !sishtrim ? nop() : !sishntrim ? nop() : MidStr(ntrimr,findntrimstart,ntrims-1).Value.Int
istrim = !sishtrim ? nop() : !sishtrim ? nop() : strim !=0 && firstcall ? 0 : etrim + 1
ietrim = !sishtrim ? nop() : !sishtrim ? nop() : firstcall && istrim == 0 ? strim-1 : sishntrim ? sntrim - 1 : lastframe
writeline = sishtrim ? sishntrim ? strim != 0 : etrim < lastframe : false
writeline ? RT_WriteFile(output, \
"%d,%d\n", istrim, ietrim, append=true) : nop()
sishtrim ? intrimloop(i, firstcall && istrim == 0 ? trimline : rightStr(trimline,StrLen(trimline)-trimend), output, lastframe) : i
}
intrimloop(abtrim(trimline), TrimFile, lastframe, true)
eval(trimline)
}

I think it's ok, unless there are hidden bug, for now only RT_Stats is needed for intrim

edit: I added them in https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/AnimeIVTC.avsi

if there any update in future it will be there

[real.finder]

Nope cant be done unless arrays implemented in script.
[No idea if arrays version of 3.6 would work, I have been sticking to 3.52 noarray version].
Assuming that arrays as args to script functions do work, then try to just pass the array to the dll. [see what happens]

EDIT: Actually I'm using 3.5 r3106.

avs+ 3.6 and array work, I did it in UniversalPluginsFolders recently (in plugins64+ folder)

[StainlessS]

I really should not be spending any time on D9, too busy.
But, when I do get time, I'll dissect UniversalPluginsFolders and steal all of the good bits.

[StainlessS]

RT_Stats 2.0 Beta 13, Update:- https://www.mediafire.com/file/xa3t1wx234gyzfq/RT_Stats_25%252626_x86_x64_dll_v2.00Beta13_20201229.zip/file
Also on SendSpace below this post.

[real.finder]

lets say that there are comments (with #) like this:-

#~ Trim(0, 9006)++Trim(9007, 23602)++Trim(23603, 40776)++Trim(40777, 43472)++Trim(43473, 44402)
Trim(44403, 48808)++Trim(48809, 64002)++Trim(64003, 85162)++Trim(85163, 87858)++Trim(87859, 88788)
#~ ++Trim(88789, 94064)++Trim(94065, 108480)++Trim(108481, 129550)++Trim(129551, 132246)++Trim(132247, 133356)

is there a way to delete comment lines from string? so it will be as

Trim(44403, 48808)++Trim(48809, 64002)++Trim(64003, 85162)++Trim(85163, 87858)++Trim(87859, 88788)

need this since those lines will break inTrim() (https://github.com/realfinder/AVS-Stuff/blob/85aa25a36c9cb9a17b5ad941eb9be9d3b0f6d76b/avs%202.5%20and%20up/AnimeIVTC.avsi#L634)

[StainlessS]

Does this sort you out OK R.F., ?

(I've left in non required Chr testing funcs)


Function ChrIsNul(String S) {return RT_Ord(S)== 0} # End of String
Function ChrIsTAB(String S) {return RT_Ord(S)== 9} # TAB
Function ChrIsLF(String S) {return RT_Ord(S)==10} # Line Feed (New Line)
Function ChrIsCR(String S) {return RT_Ord(S)==13} # Carriage Return
Function ChrIsSPACE(String S) {return RT_Ord(S)==32} # SPACE
Function ChrIsBang(String S) {return RT_Ord(S)==33} # !
Function ChrIsDQuote(String S) {return RT_Ord(S)==34} # "
Function ChrIsHash(String S) {return RT_Ord(S)==35} # #
Function ChrIsDollar(String S) {return RT_Ord(S)==36} # $
Function ChrIsPercent(String S) {return RT_Ord(S)==37} # %
Function ChrIsAmp(String S) {return RT_Ord(S)==38} # &
Function ChrIsSQuote(String S) {return RT_Ord(S)==39} # '
Function ChrIsOParen(String S) {return RT_Ord(S)==40} # (
Function ChrIsCParen(String S) {return RT_Ord(S)==41} # )
Function ChrIsStar(String S) {return RT_Ord(S)==42} # *
Function ChrIsPlus(String S) {return RT_Ord(S)==43} # +
Function ChrIsComma(String S) {return RT_Ord(S)==44} # ,
Function ChrIsHyphen(String S) {return RT_Ord(S)==45} # -
Function ChrIsDot(String S) {return RT_Ord(S)==46} # .
Function ChrIsSlash(String S) {return RT_Ord(S)==47} # /
Function ChrIsColon(String S) {return RT_Ord(S)==58} # :
Function ChrIsSColon(String S) {return RT_Ord(S)==59} # ;
Function ChrIsLT(String S) {return RT_Ord(S)==60} # <
Function ChrIsEqual(String S) {return RT_Ord(S)==61} # =
Function ChrIsGT(String S) {return RT_Ord(S)==62} # >
Function ChrIsQuery(String S) {return RT_Ord(S)==63} # ?
Function ChrIsAt(String S) {return RT_Ord(S)==64} # @
Function ChrIsOSquare(String S) {return RT_Ord(S)==91} # [
Function ChrIsBSlash(String S) {return RT_Ord(S)==92} # /
Function ChrIsCSquare(String S) {return RT_Ord(S)==93} # ]
Function ChrIsHat(String S) {return RT_Ord(S)==94} # ^
Function ChrIsUScore(String S) {return RT_Ord(S)==95} # _
Function ChrIsOCurly(String S) {return RT_Ord(S)==123} # {
Function ChrIsPipe(String S) {return RT_Ord(S)==124} # |
Function ChrIsCCurly(String S) {return RT_Ord(S)==125} # }
Function ChrIsTilde(String S) {return RT_Ord(S)==126} # ~
#
Function ChrIsWhite(String S) {C=RT_Ord(S) return C==32||C>=8&&C<=13} # SPC, BackSPC, TAB, LF, VT, FF, CR
Function ChrIsDigit(String S) {C=RT_Ord(S) return C>=48&&C<=57}
Function ChrIsUCase(String S) {C=RT_Ord(S) return C>=65&&C<=90}
Function ChrIsLCase(String S) {C=RT_Ord(S) return C>=97&&C<=122}
Function ChrIsAlpha(String S) {C=RT_Ord(S) return C>=65&&C<=90||C>=97&&C<=122}
Function ChrIsHex(String S) {C=RT_Ord(S) return C>=48&&C<=57||C>=65&&C<=70||C>=97&&C<=102}
# Req GScript or AVS+
Function ChrEatWhite(String S) {i=1 C=RT_Ord(S,i) While(C==32||C>=8&&C<=13) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}
Function ChrEatAlpha(String S) {i=1 C=RT_Ord(S,i) While(C>=65&&C<=90||C>=97&&C<=122) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}
Function ChrEatDigits(String S) {i=1 C=RT_Ord(S,i) While(C>=48&&C<=57) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}
Function ChrEatHex(String S) {i=1 C=RT_Ord(S,i) While(C>=48&&C<=57||C>=65&&C<=70||C>=97&&C<=102) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}

#################################### Above, we only use ChrEatWhite

SSS="""
#~ Trim(0, 9006)++Trim(9007, 23602)++Trim(23603, 40776)++Trim(40777, 43472)++Trim(43473, 44402)
Trim(44403, 48808)++Trim(48809, 64002)++Trim(64003, 85162)++Trim(85163, 87858)++Trim(87859, 88788)
#~ ++Trim(88789, 94064)++Trim(94065, 108480)++Trim(108481, 129550)++Trim(129551, 132246)++Trim(132247, 133356)
"""

NewS = ""
LineCount = SSS.RT_TxtQueryLines
for(i=0,LineCount-1) { # Will Ignore Blank lines and lines where first non-white char is hash '#'.
TmpS = SSS.RT_TxtGetLine(i).ChrEatWhite # Get line of text from multi-line string and remove leading whitespace.
if(TmpS.RT_Ord != 35 && TmpS.RT_Ord != 0) { # NOT '#' or End Of Line
NewS = (NewS=="") ? TmpS : NewS + Chr(10) + TmpS
}
}

BlankClip(Width=1024,height=640,length=1).RT_Subtitle("Stripped =\n%s",NewS)

[real.finder]

yes, thanks StainlessS

:goodpost:

[Dogway]

StainlessS is RT_QwikScan() aimed to align matching shots or only frames (ala MatchFrames). I spend a lot of time syncing videos (to match audio actually) and I grew tired of it, some shots are longer, shorter, slided, added or omitted, as such an example manually crafted script would look like this:

v=ffvideosource("E:\video01.mkv")
a=LWLibavaudioSource("E:\video01.mkv")

v2=ffvideosource("E:\video02.mkv")
a2=LWLibavaudioSource("E:\video02.mkv").ConvertAudioToFloat()

a.ConvertAudioToFloat()

audiodub(v,last)
assumefps(24000,1001)

av2=audiodub(v2,a2)

TimeStretch(rate=((24000.0/1001.0)*100.0)/25.0)

trim(746,0)
trim(0,28046)+trim(av2,28046+1,28502)+trim(28202+1,0)
trim(0,30886)+trim(av2,30886+1,30886+14*24+4)+trim(30886+1,0)
trim(0,37583)+trim(av2,37583+1,37583+13*24+3)+trim(37583+1,0)
trim(0,39334)+trim(av2,39334+1,39334+18*24-7)+trim(39334+1,0)
trim(0,43695)+trim(av2,43695+1,43695+5*24)+trim(43695+1,0)
(...)
trim(0,137057)+trim(av2,137057+1,137666-4)+trim(137057+1,0)
trim(0,137907)+trim(av2,137907+1,138531-1)+trim(137907+1,0)
trim(0,139372)+trim(av2,139372+1,139665)+trim(139372+1,0)
trim(0,141478)+trim(av2,141478+1,142295)+trim(141478+1,0)
trim(0,145026)+trim(av2,145026+1,145411+8)+trim(145026+1,0)
trim(0,149794)+trim(av2,149794+1,150601+48+19)+trim(149794+1,0)
trim(0,151913)+trim(av2,151913+1,151913+24*13+38)+trim(151913+1,0)
trim(0,155769)+trim(av2,155769+1,155769+24*32-14)+trim(155769+1,0)
trim(0,157588)+trim(av2,157588+1,158418)+trim(157683,0)
trim(0,158766)+trim(av2,158766+1,158986)+trim(158766+1,0)
trim(0,160356)+trim(av2,160356+1,160356+24*14-6)+trim(160356+1,0)
trim(0,161050)+trim(av2,161050+1,161976)+trim(161050+1,0)
trim(0,162076)+trim(av2,162076+1,162205+24+25)+trim(162076+1,0)

delayaudio(32*0.0417)
killvideo()

Is it possible to automatize this or do any of your tools facilitate the process?

[StainlessS]

RT_QwikScan, was a bit of a nightmare.
Is to find similar frames only.
It started off good, where spec was to find IDENTICAL frames only, it did this pretty much perfectly, and after creation
of DBase, did so at an incredible ~3 millions frames per second searched [or thereabouts, on an XP Q6600 2.40GHz CPU].
Sadly, it turned out frames required were not identical at all, had moving cursor [art package thingy] which made not identical,
and then it was found out that art package itself did not do a perfect UNDO operation, so frames to seek [UNDO] were different to
the frames they were supposed to UNDO to. There began my nightmare 9 months getting it to work at all, and it also dramatically
slowed search to something like 25,000 frames per second. If provided identical frames then should still be quite fast but
providing for error made it slower, maybe something like 250,000 FPS searched or thereabouts.
DBase on Q6600 took about 18 minutes to create on 200,000 frame clip [DVD], but it took about same to do single averageluma scan on same length clip.
It is also not certain that frame will indeed be matched, if too different or if LumaTol [luma tolerance] is set too low.

If clips are spatially aligned, then might help to look at SirPlant(), first posted here:- https://forum.doom9.org/showthread.php?p=1787898#post1787898
Its in the current RT_Stats 2.0Beta13 AVS directory.

EDIT: Link above is only description, not script. Is 540+ lines script. EDIT: 556 lines, 41KB.
EDIT: It uses RT_QwikScan, which I'de forgotten about.

[Dogway]

Thanks a lot, I will have a look, yes I typically align them to easily spot frame differences.

[Dogway]

Hey, I had some time to test RT_QwikScan thanks to carefully reading this post (https://forum.doom9.org/showpost.php?p=1787899&postcount=376) of yours and let me tell you, it works magnificiently. I tested on a 6000 frames chunk and it worked great, it's going to save me a lot of time.
I admit it took me a while to get used to the switches workflow but once it clicks in your brain it makes it all the way easier.

A few remarks I wanted frame accuracy to the frame, so I had to comment one assert line:
# Assert(16 <= MatchLen,RT_string("%s16 <= MatchLen(%d)",myName,MatchLen))
Another thing is I had the clips reverted, the source clip is the one to replace, the extended version so to speak, in the end the prune call will replace everything in source except for those unique in it.


I will post here my code as a reminder and help for other people:

LOG Files creation

r0=ffvideosource("Extended version.mkv")
r1=ffvideosource("Theatrical version.mkv") # both should be temporal and spatially aligned

src = r0.trim(917,7000) # Source clip
rep = r1.trim(917,7000) # Replace clip, Holds sequences that will replace those in source clip.
# Sequences in Replace clip should be in SAME order as in source clip.
# Clips must be same size, colorspace.
# When splicing sequences with Prune (later) everything must be similar, audio etc,
# also audio should both be normalized for similar voume levels, this detection stage ignores audio.
# In later script, Prune will replace with both video and audio from replace clip.
SRCFN = "Source_"
REPFN = "Replac_"
srcARR = SRCFN+".ARR" # Name of RT_QwikScan FingerPrint Array
srcNxtARR = SRCFN+".Next.ARR" # Name of RT_QwikScan Locator Next Array (Next == scan forward Array)
repARR = REPFN+".ARR" # Name of RT_QwikScan FingerPrint Array
repNxtARR = REPFN+".Next.ARR" # Name of RT_QwikScan Locator Next Array (Next == scan forward Array)

#
Th = 0.8 # Default 0.9(not close matching frames maybe about 0.8)
Th2 = 0.65 # Default 0.75(not close matching frames maybe about 0.65)
LUMATOL = 16.0 # Default 8.0(not close matching frames maybe up to about 16.0)
MatchLen = 1 # Default 60(Minimum sequence length that must match (frames). Default 60(not close matching frames maybe about 32)
SEARCHSTEP = 1 # Default MatchLen/4(not close matching frames maybe about 4) Frames to step after match failure.
STRICT = True # Default True=strict 1:1 matching of frames, False=allow slight mismatch ie missing/extra frames in replacement clip.
VERBOSITY = 5 # Debug output verbocity level, to DebugView.
INSMODE = 2 # Default 0=Replace Only, 1=Insert Extras Only, 2=Replace and Insert Extras
BLKW = 64 # default 64
BLKH = 64 # default BLKW
OLAPX = 0 # default BLKW/2
OLAPY = 0 # default BLKH/2

X = 8 # Avoid edge noise or logos. (All default 0)
Y = 8
W = -8
H = -8

### !!! MUST SET TRUE FIRST, THEN FALSE to RUN !!! ###
CREATE = True # Set True to Create QWikScan arrays for SirPlant. (View progress in DebugView)


(CREATE) ? RT_QwikScanCreate(Src,srcARR,"",srcNxtARR,debug=true,x=X,y=Y,w=W,h=H) : NOP # Only need create these once, Re-Create if change coords or clips.
(CREATE) ? RT_QwikScanCreate(rep,RepARR,"",repNxtARR,debug=true,x=X,y=Y,w=W,h=H) : NOP

### !!! USE DEBUGVIEW !!! ###
(!CREATE) ? SirPlant(src,rep,srcARR,srcNxtARR,repARR,repNxtARR,
\ Th=Th,Th2=Th2,MatchLen=MatchLen,SearchStep=SEARCHSTEP,LumaTol=LUMATOL,Strict=STRICT,InsMode=INSMODE,VerBosity=VERBOSITY,
\ Blkw=BLKW,BlkH=BLKH,OLapX=-OLAPX,OLapY=OLAPY
\ ) : NOP

return Messageclip("All Done" + ((Create)?", Set Create=False":""))



DEBUG Result

vSrc="Extended version.mkv"
vRep="Theatrical version.mkv"

Project="Default"
Cmd=Project+"_Command.txt"
SHOW = True
SRC_AUDIO = False # Use only Source audio. ONLY if STRICT and not inserting EXTRA replacement ranges, else out of sync audio likely
STACK = False # Same for STACK, only if Strict and no EXTRA replacement ranges.

A=ffvideosource(vSrc).trim(917,7000)
B=A C=A
D=ffvideosource(vRep).trim(917,7000)
E=D F=D

A = (SHOW) ? A.AddBorders(0,0,0,16,$000000) : A
B = (SHOW) ? B.AddBorders(0,0,0,16,$404040) : B
C = (SHOW) ? C.AddBorders(0,0,0,16,$808080) : C
D = (SHOW) ? D.AddBorders(0,0,0,16,$00FF00) : D
E = (SHOW) ? E.AddBorders(0,0,0,16,$FF0000) : E
F = (SHOW) ? F.AddBorders(0,0,0,16,$0000FF) : F

SrcAudio=A.KillVideo

Nicks="""
InterveningSource=0
MatchingSource=1
EndRangeSource=2
MatchingReplacement=3
ExtraReplacement=4
EndRangeReplacement=5
"""

Prune(A,B,C,D,E,F,cmd=Cmd,NickName=Nicks,Show=SHOW)

(SRC_AUDIO) ? AudioDub(SrcAudio) : NOP

return (STACK) ? StackVertical(A,Last) : Last


Result Script

A=ffvideosource(vSrc).trim(917,7000)
B=A C=A
D=ffvideosource(vRep).trim(917,7000)
E=D F=D

Nicks="""
InterveningSource=0
MatchingSource=1
EndRangeSource=2
MatchingReplacement=3
ExtraReplacement=4
EndRangeReplacement=5
"""

Prune(A,B,C,D,E,F,cmd="Default_Command.txt",NickName=Nicks)

I understand Prune will honor the audio of each clip right?

[StainlessS]

Prune splices both video and audio, so both source clips need to be normalised so as not to have jumps in volume level.

Glad you figured out how to use it, not so easy to understand, nor to write, still baffles the hell out of me :)

Maybe I should remove that Assert.

EDIT:

Project="Default"
Cmd=Project+"_Command.txt"
SHOW = True
SRC_AUDIO = False # Use only Source audio. ONLY if STRICT and not inserting EXTRA replacement ranges, else out of sync audio likely
STACK = False

As above, audio will also be replaced together with video.

If True, then will use only original audio from source [ie src audio is audiodubbed over the spliced result].
[possible audio out-of-sync if not 1:1 matches or inserting extra sequences that exist in rep but not in src].

[StainlessS]

BUMP, see above edit.

[Dogway]

I comitted a mistake and lost 3h of processing :(
In any case here are my observations.

It took 3h to process 60000 frames (didn't use Prefetch though), about 1/3 of a 2h film on a i7-4790K, at which point my RAM usage reached 32Gb and had to terminate it.
It fails on fades to dark, I had to go and manually add some frames there on result clip and it eventually matches again so no big issue, probably only it has to scan more time in preprocess stage.
I had some minor issue on a scene where shots were a bit shuffled, and had to manually fix that part, but again minor issue counting that I'm saving to do hundreds of edits.
Clips need to have same number of frames, but in this case (extended vs theatrical) they don't, so I appended a blankclip to the theatrical, not sure if this is fine.

Also I'm wondering if I can lower resolution from 1080p for the prepass stage to avoid the above RAM issue.

[StainlessS]

Th = 0.8 # Default 0.9(not close matching frames maybe about 0.8)
Th2 = 0.65 # Default 0.75(not close matching frames maybe about 0.65)
LUMATOL = 16.0 # Default 8.0(not close matching frames maybe up to about 16.0)
MatchLen = 1 # Default 60(Minimum sequence length that must match (frames). Default 60(not close matching frames maybe about 32)
SEARCHSTEP = 1 # Default MatchLen/4(not close matching frames maybe about 4) Frames to step after match failure.
STRICT = True # Default True=strict 1:1 matching of frames, False=allow slight mismatch ie missing/extra frames in replacement clip.
VERBOSITY = 5 # Debug output verbocity level, to DebugView.
INSMODE = 2 # Default 0=Replace Only, 1=Insert Extras Only, 2=Replace and Insert Extras
BLKW = 64 # default 64
BLKH = 64 # default BLKW
OLAPX = 0 # default BLKW/2
OLAPY = 0 # default BLKH/2



MatchLen = 1 # Default 60(Minimum sequence length that must match (frames). Default 60(not close matching frames maybe about 32)
# Default might be better

SEARCHSTEP = 1 # Default MatchLen/4(not close matching frames maybe about 4) Frames to step after match failure.
# It does bi-directional scan anyway, defaults may be faster

Big LumaTol would severely affect times, much much slower. [but if is very different frames, then has to be high-ish]
Fails on fade to dark
Not really surprising, esp if one fade and other dont.
I had some minor issue on a scene where shots were a bit shuffled, and had to manually fix that part,
MatchLen = 1, and SEARCHSTEP=1 probably did not help.
also, INSMOD=2, 2=Replace and Insert Extras. Only if rep clip has sequences which are not in src, otherwise could cause problems with out of order
seqs where matches fail [where Default 0=Replace Only, might be better].

Clips need to have same number of frames, but in this case (extended vs theatrical) they don't, so I appended a blankclip to the theatrical, not sure if this is fine.

NOPE, same length dont matter. [would just make it slower adding blankclip, more to fingerprint, more to search]

Also I'm wondering if I can lower resolution from 1080p for the prepass stage to avoid the above RAM issue.
Dont think would be a problem [maybe even eg 320,240 would work, MAYBE, but 640x480 or thereabout a bit safer].
The arrays are not used after scan script, so size would not be problem to Prune splicing, Prune command file already created.

think these are biggest hogs for memory [defaults might also be faster]

MatchLen = 1 # Default 60(Minimum sequence length that must match (frames). Default 60(not close matching frames maybe about 32)
SEARCHSTEP = 1 # Default MatchLen/4(not close matching frames maybe about 4) Frames to step after match failure.

[Dogway]

Thanks for the help.

[Morzzz]

After butchering the source code quite hard, I got this to compile on Linux. Seems to be running Duplicity2/DropDeadGorgeous just fine, but it'll probably set the house on fire when trying anything else.

It does contain some more portable alternatives to some of the really Win32/VSC centered code though. If anyone interested (StainlessS?) drop me a PM.

[StainlessS]

Thanks.
I'm not using Avs+ on Linux just yet, but when I do, I may bug you a bit :)

[real.finder]

After butchering the source code quite hard, I got this to compile on Linux. Seems to be running Duplicity2/DropDeadGorgeous just fine, but it'll probably set the house on fire when trying anything else.

It does contain some more portable alternatives to some of the really Win32/VSC centered code though. If anyone interested (StainlessS?) drop me a PM.

isn't better to post the changes directly?

[StainlessS]

I have no problem is Morzzz wants to post the updates.

[Morzzz]

Ok, I found out I'm in need of a far less intelligent dedup algorithm, so I have no need for this after all :-D

Anyways, here is the rudimentary code:

https://github.com/mischa85/rt_stats

[StainlessS]

Thanks.

[VoodooFX]

I want to write one line to existing txt file, but I want it to be the first line. Is it possible?

[StainlessS]

Nope sorry, nothing like that implemented.
Best I can suggest [so long as only rarely used], is to read entire file, then re-write with something like

FN=".\test.txt"
Preamble = "Something something somethng"
S=RT_ReadTxtFromFile(FN)
RT_WriteFile(FN,"%s\n%s",Preamble,S) # Will create anew, NO Append

Untested.
EDIT: Added "\n" between %s strings formatting, assumes that Preamble does not end in n/l.
EDIT: If n/l status at end of Preamble UNKNOWN, could use

FN=".\test.txt"
Preamble = "Something something somethng"
S=RT_ReadTxtFromFile(FN)
RT_WriteFile(FN,"%s",Preamble) # Will create anew, NO Append [with guaranteed n/l at end {EDIT: Add n/l if not already present}]
RT_WriteFile(FN,"%s",S,Append=true) # glue on the rest. [with guaranteed n/l at end]

[VoodooFX]

I thought that to read a whole file and then write it is the only option.
Your script works well.

[StainlessS]

Glad it works for U. Another peculiar one added in one of the more recent betas was

RT_FileDuplicate(String Filename1,String Filename2,bool "Overwrite"=False)
Makes a copy of the source Filename1, to a second destination Filename2, automatically overwriting existing Filename2 if
Overrwrite is true (default false). Fails in existing Filename2 if Overrwrite= false (default).

Cant remember why I added it.
No reply necessary.

EDIT: I think I added above for binary duplicate, ie dont have to contain text only. [original requirement probably for short-ish template clip duplicate, so multi-megabytes, probably multi-GigaBytes OK]

[StainlessS]

Test for above mentioned RT_FileDuplicate()

# Infile:- SHA-512: c15605364db905631581032b32f5984b85d2ada7fcdf75058a1e72d7685e1920b01aeff4dc5ccea49519fd54638e052906184ee1a640e9e54d25e9ef43324747
# OutFile:- SHA-512: c15605364db905631581032b32f5984b85d2ada7fcdf75058a1e72d7685e1920b01aeff4dc5ccea49519fd54638e052906184ee1a640e9e54d25e9ef43324747

BlankClip(Width=520,height=128)
FN1="F:\Infile.mp4"
FN2="E:\Outfile.mp4"
TS = RT_TimerHP()
RT_FileDuplicate(FN1,FN2,OverWrite=True)
TE = RT_TimerHP()

T = TE - TS

Subtitle(RT_string("2.50 GB (2,693,697,922 bytes)\\nFile Duplicated in %f seconds",T),lsp=0,size=32)

https://i.postimg.cc/6pSWGnqm/filesize.jpg (https://postimages.org/)
SHA-512 for both at top of code block. [time for two {in and out} physical hard drives]

EDIT: Did not time it but windows copies same file in about 2+ seconds.

EDIT: And there is also this one in latest beta.

RT_FileRename(String OldFilename,String NewFileName)
Renames a file from OldFilename to NewFilename.
This function renames the file or directory specified by OldFilename to the name given by NewFilename.
The OldFilename must be the path of an existing file or directory.
The NewFilename must not be the name of an existing file or directory.
You can use the function to move a file from one directory or device to another by giving a different path in the NewFileName argument.
However, you cannot use the function to move a directory. Directories can be renamed, but not moved.

Return 0 on success.
On Error, returns
-1, Access denied or read only, or NewFileName already exists or could not be created (invalid path),
or oldname is a directory and newname specifies a different path.
-2, OldFilename File or Path not found.
-3, NewFileName contains invalid characters.
-4, Unknown error.

[VoodooFX]

I wish DBASE could store masks. :)

[StainlessS]

I wish DBASE could store masks. :)

Thats always been in the back of my mind.

EDIT: Well not for DBase, but for RT_ Arrays, would likely also have to implement type UShort [maybe as BIN16] in both Array and DBase, for unsigned 16 bit int.
Currently implements non standard types BIN [8 bit unsigned int] and [EDIT: Intended] for internal RT_stats use only, Double [returned to AVS as 32 bit float].

[VoodooFX]

For some reason I thought that it supports it and I'll make like logodata storage out of it.
I can store stuff to ebmp but there wouldn't be coords and other settings.

[StainlessS]

Roughly whats implemented for array [additional to the array itself] (same for DBase)


*****************************************************
********** RT_ARRAY FUNCTIONS ***********
*****************************************************

RT_Stats ARRAY functions allow fast access to a file based Array of up to 3 dimensions of fixed element type,
Bool, Int, Float, String, or BIN. BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in the Array
as an unsigned 8 bit Int, upper 24 bits ignored.
The maximum possible file size is about 1TB, and depends on number and size of dimensions and element type and size, also
depends upon avilable disk space.
Element types Bool and BIN require 1 byte, Int and Float 4 bytes, and String is of fixed maximum length (user chosen) up to
256KB (256 * 1024, no nul terminating character is stored in an Array String).
Maximum possible size of dimension 1 is $7FFFFFFE but depends upon available disk space and also limited to max 1TB.
RT_ArrayAlloc, allows you to create an array file with a fixed number of dimensions and where the 2nd and/or 3rd dimensions
of a multi-dimensional array are fixed in size but the size of the first dimension can 'grow', single dimension arrays can
'grow' too. You can create an array where the first dimension is 0 (any other dimensions fixed), ie no elements pre-allocated
in array file, in this case you can use RT_ArrayExtend to 'grow' the first dimension of the array so that it can be used. In addition,
single dimension arrays can also use RT_ArrayAppend to add elements to the end of the array.
An Array file allows up to 1024 Attributes of types Int or Float, which can be used for whatever a user desires, these
attributes are stored in the Array file header which is 32768 bytes, so the actual data starts at $8000 hex.
In addition to user attributes, an Array has currently 10 user String Attributes that can be be used for whatever you want, the
string attributes are currently a maximum of 1024 characters in length.
In addition to user attributes and user string attributes, there are currently 128 ID's (Int of Float) that are intended to be used
by either RT_Stats itself, or script developers, intended function of ID's are to in some way eg link an array to a particular
clip, by storing maybe Width, Height, Framecount and perhaps other values, the stored ID's might be used to Validate an array
against a clip.
You could use the Array functions to return multiple results from a script function to the caller via a caller supplied Array
filename, perhaps with some kind of status variable returned directly.


EDIT: IIRC, in pre AVS+ [not sure about avs+], Array element and DBase field reads were faster than access to Global Vars. [and that was probably on Core Duo with SATA I drive].

[StainlessS]

Here a few functions than might be handy. [RT_Stats required]


# ChrEatWhite is posted elsewhere, used in Chrxxxxx funcs library.
Function ChrEatWhite(String S) {i=1 C=RT_Ord(S,i) While(C==32||C>=8&&C<=13) {i=i+1 C=RT_Ord(S,i)} return i>1?MidStr(S,i):S}
Function StrSQuote(string S) {Return "'" + S + "'" } # Surrounded String with Single Quotes.
Function StrDQuote(string S) {Return Chr(34) + S + Chr(34) } # Surrounded String with Double Quotes.
Function StrTrim(String S, Int "Md") {Md=Default(Md,0) S=((Md!=2)?S.RevStr.ChrEatWhite.RevStr:S) Return((Md!=1)?S.ChrEatWhite():S) } # Md=1 Trim WhiteSpace from right : Md=2 Trim WhiteSpace from Left : Md=0(Default) or anything else, Left + Right Trim
Function StrLeftAlign(String S,int "PadLen") {PadLen=Default(PadLen,S.StrLen) Return S.StrTrim(0).RT_StrPad(PadLen)} # Left Align S, result SPACE padded on Right to at least PadLen(default StrLen(S)) length.
Function StrRightAlign(String S,int "PadLen") {PadLen=Default(PadLen,S.StrLen) S = S.StrTrim(0) Return RT_StrPad("",PadLen-S.StrLen) + S} # Right Align S, result SPACE padded on Left to at least PadLen(default StrLen(S)) length.
Function StrCenterAlign(String S,int "PadLen") {PadLen=Default(PadLen,S.StrLen) S=S.StrTrim(0) Return(RT_StrPad("",(PadLen-StrLen(S))/2)+S).RT_StrPad(PadLen)} # Center Align S, result SPACE padded evenly on both Left & Right to at least PadLen(default StrLen(S)) length.
Function StrAlign(String S,Int "Md",Int "PadLen") {Md=Default(Md,0) Return(Md==1)?S.StrRightAlign(PadLen):(Md==2)?S.StrLeftAlign(PadLen):S.StrCenterAlign(PadLen)}# Md=1 Right Align : Md=2 Left Align : Md=0(Default) or anything else, Center Align. Result is L/R/Center padded to at least PadLen(default StrLen(S)) Length.


Demo client


############ DEMO STUFF ###################
#Import(.\SomethingOrOther.avsi")

Src =" some test text "

RT_DebugF("LEN=%d",Src.Strlen)

Global SBUF = StrSQuote(Src).RT_StrPad(48) + "SOURCE STRING # (len between quotes=32)" + "\n" + Chr(10)

Add2BUF(Src, "StrTrim(S) # Left & Right Trim : Default " )
Add2BUF(Src, "StrTrim(S,0) # Left & Right Trim" )
Add2BUF(Src, "StrTrim(S,1) # Right Trim" )
Add2BUF(Src, "StrTrim(S,2) # Left Trim" )

Add2BUF(Src, "StrLeftAlign(S) # Left Align, Default Pad to original length" )
Add2BUF(Src, "StrLeftAlign(S,42) # Left Align, Pad to 42" )
Add2BUF(Src, "StrLeftAlign(S,20) # Left Align, Pad to 20" )

Add2BUF(Src, "StrRightAlign(S) # Right Align, Default Pad to original length" )
Add2BUF(Src, "StrRightAlign(S,42) # Right Align, Pad to 42" )
Add2BUF(Src, "StrRightAlign(S,20) # Right Align, Pad to 20" )

Add2BUF(Src, "StrCenterAlign(S) # Center align, Default Pad to original length" )
Add2BUF(Src, "StrCenterAlign(S,42) # Center Align, pad to 42" )
Add2BUF(Src, "StrCenterAlign(S,20) # Center Align, pad to 20" )

Add2BUF(Src, "StrAlign(S) # Center Align, Default Pad to original length" )
Add2BUF(Src, "StrAlign(S,0) # Center Align, Default Pad to original length" )
Add2BUF(Src, "StrAlign(S,1) # Right Align, Default Pad to original length" )
Add2BUF(Src, "StrAlign(S,2) # Left Align, Default Pad to original length" )

Add2BUF(Src, "StrAlign(S,0,42) # Center Align, Pad to 42" )
Add2BUF(Src, "StrAlign(S,1,42) # Right Align, Pad to 42" )
Add2BUF(Src, "StrAlign(S,2,42) # Left Align, Pad to 42" )

Add2BUF(Src, "StrAlign(S,0,20) # Center Align, Pad to 20" )
Add2BUF(Src, "StrAlign(S,1,20) # Right Align, Pad to 20" )
Add2BUF(Src, "StrAlign(S,2,20) # Left Align, Pad to 20" )

BlankClip(width=1152,Height=640)

Subtitle(SBUF,Font="Courier New",lsp=0)

Return Last

Function Add2BUF(String S, String FnS) {
s3 = Eval(Fns)
Global SBUF = SBUF + StrSQuote(s3).RT_StrPad(48) + FnS + "\n" + Chr(10)
}


Result
https://i.postimg.cc/fygRfzs5/TESTING-00.jpg (https://postimages.org/)

EDIT: There are String Trim type functions implemented in AVS+, this does not use them. {TrimLeft, TrimRight, TrimAll}
EDIT: You can also use PadLen=0 with some, to not pad at all.

[rgr]

Hi.
How big of an array can RT_ArrayAlloc hold?
Avisynth works for a few tens of thousands, but for a few hundred thousand it crashes :)

[StvG]

Are you sure you didn't hit your memory limit? In this case also RT_ArrayAlloc will crash/fail.

[StainlessS]

RT_Array's are file based, nothing to do with Avs+ arrays.
Same for RT_DBase's.


*****************************************************
********** RT_ARRAY FUNCTIONS ***********
*****************************************************

RT_Stats ARRAY functions allow fast access to a file based Array of up to 3 dimensions of fixed element type,
Bool, Int, Float, String, or BIN. BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in the Array
as an unsigned 8 bit Int, upper 24 bits ignored.
The maximum possible file size is about 1TB, and depends on number and size of dimensions and element type and size, also
depends upon avilable disk space.
Element types Bool and BIN require 1 byte, Int and Float 4 bytes, and String is of fixed maximum length (user chosen) up to
256KB (256 * 1024, no nul terminating character is stored in an Array String).
Maximum possible size of dimension 1 is $7FFFFFFE but depends upon available disk space and also limited to max 1TB.
RT_ArrayAlloc, allows you to create an array file with a fixed number of dimensions and where the 2nd and/or 3rd dimensions
of a multi-dimensional array are fixed in size but the size of the first dimension can 'grow', single dimension arrays can
'grow' too. You can create an array where the first dimension is 0 (any other dimensions fixed), ie no elements pre-allocated
in array file, in this case you can use RT_ArrayExtend to 'grow' the first dimension of the array so that it can be used. In addition,
single dimension arrays can also use RT_ArrayAppend to add elements to the end of the array.
An Array file allows up to 1024 Attributes of types Int or Float, which can be used for whatever a user desires, these
attributes are stored in the Array file header which is 32768 bytes, so the actual data starts at $8000 hex.
In addition to user attributes, an Array has currently 10 user String Attributes that can be be used for whatever you want, the
string attributes are currently a maximum of 1024 characters in length.
In addition to user attributes and user string attributes, there are currently 128 ID's (Int of Float) that are intended to be used
by either RT_Stats itself, or script developers, intended function of ID's are to in some way eg link an array to a particular
clip, by storing maybe Width, Height, Framecount and perhaps other values, the stored ID's might be used to Validate an array
against a clip.
You could use the Array functions to return multiple results from a script function to the caller via a caller supplied Array
filename, perhaps with some kind of status variable returned directly.

***
***
***

RT_ArrayAlloc(string FileName,int "Type"=1,int "Dim1"=0,int "Dim2"=0,int "Dim3"=0,int "StringLenMax"=256)

Creates a file to hold a one, two or three dimensional Array of elements of type Bool, Int, Float, String, or BIN.
BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in array as an unsigned 8 bit int, upper 24 bits ignored.

Filename, Array Filename, no default.

Type, int, default 1(0->4). 0=Bool, 1=Int, 2 = Float, 3 = String, 4 = BIN. Type of the array elements, fixed for lifetime of array.
v1.40, NOTE, There is also a type 5 (double) which is not of use to the user, although a type 5 array will accept Float
(as double is not an Avisynth type). Type 5 is implemented for use by RT_Stats internal functions to store higher precision results.
Array type double (5) will be returned as Float to Avisynth script.

Dim1, Dim2, Dim3, all Default 0, range 0 or greater. Number of array elements per dimension (max 3 dimensions).
Where Dim1 & Dim2 & Dim3 all at default 0, will be 1 dimensional unallocated Array.
Dim1 governs the pre-allocated size of the array first dimension.
The first dimension can 'grow' for 1, 2 or 3 dimensional arrays. (RT_ArrayExtend, or for single dimension only RT_ArrayAppend)
Dim2 and Dim3 fix the number of array dimensions for the lifetime of the array, and also the size of dimensions 2 and 3.

Dim1 Dim2 Dim3 Dimension 1 is [EDIT: default] 0, NO elements allocated, can use RT_ArrayExtend to 'grow' size of first dimension for use.
0 0 0 SINGLE dimension array. Can ALSO use RT_ArrayAppend to add elements to end of the single dimension array
0 0 >0 ILLEGAL, Throws error condition.
0 >0 0 TWO dimension array, Dim2 fixes size of 2nd dimension. CANNOT use RT_ArrayAppend.
0 >0 >0 THREE dimension array, Dim2 & Dim3 fix size of 2nd & 3rd dimensions. CANNOT use RT_ArrayAppend.

Dim1 Dim2 Dim3 Dimension 1 is Dim1 in size, elements ARE pre-allocated, initialized to zeros and can be used. Can also 'grow'.
>0 0 0 SINGLE dimension array. Can ALSO use RT_ArrayAppend as well as RT_ArrayExtend.
>0 0 >0 ILLEGAL, Throws error condition.
>0 >0 0 TWO dimension array, Dim2 fixes size of 2nd dimension. Can ALSO use RT_ArrayExtend.
>0 >0 >0 THREE dimension array, Dim2 & Dim3 fix size of 2nd & 3rd dimensions. Can ALSO use RT_ArrayExtend.
(initialized to zeros, Bool=False,Float=0.0,String="").

StringLenMax, default 256(1 -> (256 *1024)). Fixed Maximum string length excluding nul term. Error if string length exceeds.
Only used for Type=3 (string).

Returns maximum size of dim1 dimension given current flesystem and disk space available to the user.
All RT_ArrayXXXX() errors produce an error alert.

It makes little sense to set either Dim2 or Dim3 to 1 although this will not cause an error condition, just be less efficient,
a 3 dimensional array with all dimensions 1, refers only to a single element but will take slightly longer to calculate the
linear array offset.

There are quite a few additional RT_Array functions.

Also, RT_DBase,

*****************************************************
********** DBASE FUNCTIONS ***********
*****************************************************

RT_Stats DBASE functions allow fast access to a file based DataBase where each record can have up to 1024 fields of variable type,
Bool, Int, Float, String, or BIN. BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in the DBase field
as an unsigned 8 bit Int, upper 24 bits ignored. there is also a type double used in DBase, this is only of use to internal RT_Stats
functions, setting or getting type double takes or returns type Float (as Avisynth cannot handle double precision float).
The maximum possible DBase file size is about 1TB, and the number of records is restricted to $7FFFFFFE, about 2 Billion.
Field types Bool and BIN require 1 byte, Int and Float 4 bytes, and String is of fixed maximum length (user chosen) up to
256KB ((256 * 1024), no nul terminating character is stored in a DBase String). The private type Double requires 8 bytes.
A DBase file allows up to 1024 Attributes of types Int or Float, which can be used for whatever a user desires, these
attributes are stored in the DBase file header which is 32768 bytes, so the actual data starts at $8000 hex.
In addition to user attributes, a DBase has currently 10 user String Attributes that can be be used for whatever you want, the
string attributes are currently a maximum of 1024 characters in length.
In addition to user attributes and user string attributes, there are currently 128 ID's (Int of Float) that are intended to be used
by either RT_Stats itself, or script developers, intended function of ID's are to in some way eg link a DBase to a particular
clip, by storing maybe Width, Height, Framecount and perhaps other values, the stored ID's might be used to Validate a DBase
against a clip.
You could use the DBase functions to return multiple results from a script function to the caller via a caller supplied DBase
filename. The DBase functions also make it more likely that script functions could be written to eg find where edits between
two clips have been made, or previously impossible/implausible tasks performed.

***
***
***

RT_DBaseAlloc(String Filename,Int Records,String TypeString,Int "StringLenMax"=256)
Creates a file to hold DataBase records. Each record can have up to 1024 fields of varying type, Bool, Int, Float, String, or BIN.
BIN is a BYTE (8 bit) sized Int where only lowest 8 bits of Int are stored in DBase field as an unsigned 8 bit int, upper 24 bits ignored.
v1.40, NOTE, There is also a type double which is not of use to the user, although a type double field will accept Float
(as double is not an Avisynth type). Type double is implemented for use by RT_Stats internal functions to store higher precision results.
Type double will be returned as Float to Avisynth script.

Filename, DBASE Filename, no default.

Records, int zero or more. Number of records to create, initalized to zeros (initialized to zeros, Bool=False,Float=0.0,String="").
If you dont know the final size of your DBase, you can create a DBase with 0 records, and thereafter use RT_DBaseAppend to add
records to the end of the DBase.

TypeString, string holding record construction data. One character for each field, valid characters are "bifsn" or Uppercase.
'b' = BOOL, 'i' = INT, 'f' = FLOAT, 's' = STRING, 'n' = BIN. (the private type double uses 'e' or 'E' in the TypeString)
The 's' STRING construction character can be followed by numeric digits describing fixed maximum string length (excluding
any nul term character), otherwise StringLenMax is used by default instead.
For no particular reason, I have limited the maximum string length to 256K, ie 256 * 1024, throws an error if greater.

StringLenMax Default=256. Default length of String fields if no explicit string length defined following TypeString S constructor char.

Example TypeString:-
"ifffs1024bsn", 1st field Int, 3 Float fields, 1 string field of length=1024, 1 Bool field, 1 string field using length of
StringLenMax, and 1 BIN field. 8 Fields in all (0->7).

Returns number of records created.
All RT_DBaseXXXX() errors produce an error alert.


RT_ARRAY_Functions:- http://avisynth.nl/index.php/RT_Stats#RT_ARRAY_Functions
DBASE_Functions:- http://avisynth.nl/index.php/RT_Stats#DBASE_Functions

EDIT: RT_ Arrays and DBase files cannot store clip variables; but as file based, can be re-used via another script (if not deleted).

[rgr]

Thanks -- 1TB should be enough.
What will be faster -- RT_Stats ARRAY or DBASE?

[StainlessS]

About the same.
Dont know why, but I nearly always prefer to use DBase. [unless 3 dimensional array]

EDIT:
Timings on earlier version RT_Stats, AVS 2.6 Std, and on Core 2 machine (Q9550 CPU) with SATA 2,
Was quicker to access DBase record/field than to access same type var as AVS Global variable.
(but slower than access AVS Local var)

[rgr]

RT_ArrayAlloc(string FileName,int "Type"=1,int "Dim1"=0,int "Dim2"=0,int "Dim3"=0,int "StringLenMax"=256)

Dim1 governs the pre-allocated size of the array first dimension.
The first dimension can 'grow' for 1, 2 or 3 dimensional arrays. (RT_ArrayExtend, or for single dimension only RT_ArrayAppend)

After "RT_ArrayAlloc(arrname,2,1000)" array size (RT_ArrayGetDim(arrname)) will be 0 or 1000? (If 1000, then they will be filled with zeros?)

Edit: OK, I see: "Dimension 1 is Dim1 in size, elements ARE pre-allocated, initialized to zeros and can be used."

[StainlessS]

Edit: OK, I see: "Dimension 1 is Dim1 in size, elements ARE pre-allocated, initialized to zeros and can be used."
Right on brother.
"initialized to zeros", for float <type=2> init to 0.0. (int=0, string="").

[rgr]

Right on brother.
"initialized to zeros", for float <type=2> init to 0.0. (int=0, string="").

It would be nice to be able to choose what element it should be filled with, because for an array of ~100k elements it takes 22mins :)

[StainlessS]

because for an array of ~100k elements it takes 22mins
Not for me, for array took ~1:51 and for DB ~1:52.
ArrayTest.avs

ARR="D:\Z\Test.Arr"
SZ=100*1000
F=42.0
RT_ArrayAlloc(ARR,2,SZ)
Start=RT_TimerHP()
for(i=0,SZ-1) { RT_ArraySet(ARR,F,i) }
End=RT_TimerHP()
Tim = End-Start
M=int(Tim/60) S=Tim-(m*60)
Msg=RT_String("Done: Time=%d:%02.3f",M,S)
MessageClip(Msg)


DBaseTest.avs

DB="D:\Z\Test.DB"
SZ=100*1000
F=42.0
TypeString="f"
RT_DBaseAlloc(DB,SZ,TypeString)
Start=RT_TimerHP()
for(i=0,SZ-1) { RT_DBaseSet(DB,i,F) }
End=RT_TimerHP()
Tim = End-Start
M=int(Tim/60) S=Tim-(m*60)
Msg=RT_String("Done: Time=%d:%02.3f",M,S)
MessageClip(Msg)


Mine, on i7-8700 CPU. <not that fast on single core, Base Freq 3.2GHz, Max Turbo Frequency 4.6GHz>

But I shall keep your comment in mind on next update (but dont hold your breath, last update was 2018, I think)

EDIT: Storage on Gen 3 NVMe. [EDIT: Due to <EDIT: OS and physical drive> disk caching, spinning drive should not be much different]


RT_ArraySet(string FileName,var Var,int ix1,int "ix2",int "ix3")
Filename, Array Filename, no default.
Var, a variable of Type as set in RT_ArrayAlloc.
ixl, int, zero relative. 1st subscript of array. Valid range 0 -> RT_ArrayGetDim(FileName,1)-1.
ix2, int, zero relative. 2nd subscript of array. Valid range 0 -> RT_ArrayGetDim(FileName,2)-1.
Only valid if 2 or 3 dimensional array. For single dimension array, can omit, supply RT_Undefined(), or -1.
ix3, int, zero relative. 3rd subcript of array. Valid range 0 -> RT_ArrayGetDim(FileName,3)-1.
Only valid if 3 dimensional array. For 1 or 2 dimension array, can omit, supply RT_Undefined(), or -1.
Returns total number of elements in array.

If single dimension array, only specifiy the 1st dimension ix1, dont supply ix2 and ix3 as 0. {for 2 dim array, only supply ix1 and ix2}
EDIT: From Array docs for RT_ArrayAlloc,
It makes little sense to set either Dim2 or Dim3 to 1 although this will not cause an error condition, just be less efficient,
a 3 dimensional array with all dimensions 1, refers only to a single element but will take slightly longer to calculate the
linear array offset.

EDIT: Also, if you hard code F as 42.0 instead of using Var F, then will not have to read F local variable at every iteration setting the array element <or DB field>.

EDIT: Output single dim array to a text file [quite slow, tested only on Float array]

ARR="D:\Z\Test.Arr"
FILE="D:\Z\TestArr.txt"
NDIMS = RT_ArrayGetDim(ARR,Dim=0) # Number of dimensions in array
Assert(NDIMS==1, "Error, single dimension array only please")
SZ = RT_ArrayGetDim(ARR,Dim=1) # Num Array elements in single dimension array
TYPE=RT_ArrayGetType(ARR) # 0=Bool, 1=Int, 2 = Float, 3 = String, 4 = BIN. # NOTE, There is also a type 5 (double)
Fmt = "%d ] %" + Select(TYPE, "b","d","f","s","d","f") + "\n"
RT_FileDelete(FILE)
Start=RT_TimerHP()
for(i=0,SZ-1) { RT_WriteFile(FILE,Fmt, i, RT_ArrayGet(ARR,i), Append=True) }
End=RT_TimerHP()
Tim = End-Start
M=int(Tim/60) S=Tim-(m*60)
TYPE_S = RT_ArrayTypeName(Type) # Name of type
Msg=RT_String("Written:- Time=%d:%02.3f single Dim array[%d] of type='%s' to file='%s'",M,S,SZ,TYPE_S,FILE)
MessageClip(Msg)


eg Produces msg clip, "Written:- Time=29:11.995 single Dim array[100000] of type='Float' to file='D:\Z\TestArr.txt'"

0 ] 42.000000
1 ] 42.000000
2 ] 42.000000
3 ] 42.000000
4 ] 42.000000
5 ] 42.000000
6 ] 42.000000
7 ] 42.000000
8 ] 42.000000
9 ] 42.000000
10 ] 42.000000
11 ] 42.000000
12 ] 42.000000
13 ] 42.000000
# etc

A bit on the slow side, maybe the formatted string production is what takes the time.
About 57 elements per second written to file.
Result text file, 1,845KB in size.

[rgr]

Not for me, for array took ~1:51 and for DB ~1:52.
ArrayTest.avs

ARR="D:\Z\Test.Arr"
SZ=100*1000
F=42.0
RT_ArrayAlloc(ARR,2,SZ)
Start=RT_TimerHP()
for(i=0,SZ-1) { RT_ArraySet(ARR,F,i) }
End=RT_TimerHP()
Tim = End-Start
M=int(Tim/60) S=Tim-(m*60)
Msg=RT_String("Done: Time=%d:%02.3f",M,S)
MessageClip(Msg)



Hmmm....

"Done: Time=15:5.778"

Ryzen5700G+Win10+NVME
CPU usage was around 0% :)

[StainlessS]

Done it again, this time only 1:39, with Comodo Antivirus running.

https://i.postimg.cc/MTGhQprJ/time.jpg (https://postimg.cc/67DjstXM)image sharing (https://postimages.org/)
AVS running 100% on single core, other stuff file writing [Drive D: 1%] + other system stuff.

EDIT: I used W10, + Vdub2 as app.

Does your Ryzen5700G thingy have similar to 12 gen intel i7,
performance cores + efficiency cores ?
Is it perhaps running on efficiency core ?

EDIT: In performace graph, right click "change graph to logical cores".
EDIT: I think with mine, every 2nd graph is a logical core, the physical cores are the ones doing something.

EDIT: With 12th gen intel i7, 16 performance cores [EDIT: half of those being logical] shown on upper graph, bottom 4 cores [EDIT: half of those being logical] are efficiency thingies.

EDIT: On your Task Manager Details TAB, is your RightClick Player_appName/Set_Priority set to anything other than Normal ?

[rgr]

https://gcdnb.pbrd.co/images/eCTifyufvmFY.png?o=1
Using VDub changes nothing.
My Ryzen has all the performance cores :)

[rgr]

After raising priority to high:
"Done: Time=9:56.085"

[StainlessS]

No idea what you are showing me there (ffmpeg.exe ???).
Show the CPU cores graph.

Also,
EDIT: On your Task Manager Details TAB, is your RightClick Player_appName/Set_Priority set to anything other than Normal ?
answer above.

EDIT: Gone t' pub.

[rgr]

Yes, I'm running the script through ffmpeg.
Now I've running compression, so I'll paste it later. But don't expect 0% to be visible there :)

EDIT: On your Task Manager Details TAB, is your RightClick Player_appName/Set_Priority set to anything other than Normal ?

After raising priority to high:
"Done: Time=9:56.085"

[StainlessS]

Now I've running compression, so I'll paste it later. But don't expect 0% to be visible there
I expect your AVS instance to be running on a single graph core, your CPU % will be for total ALL cores.
[Mine was 100% on single core [top left core graph], but 16% total CPU]

Yes, I'm running the script through ffmpeg.
Cant for the life of me figure out why.