Memory problem opening multiple clips

[fvisagie]

Hi All,

I'm running into what seems to be a memory problem when cutting and splicing together AVCHD camera files. The behaviour is the same as when previous versions of L-SMASH WORKS leaked out of memory. I'm using AvsPmod 2.5.1 with Avisynth 2.6.0 Alpha 4 and L-SMASH WORKS r725's LWLibavAudio/VideoSource() on Windows 7 Pro SP1 32-bit.

Because the camera creates one file per scene, I want to first open all files assigning each to a unique variable, and then use the variable names to more conveniently reorder and Trim() them. As it stands the script therefore looks like this (just for now, later I plan to use more meaningful clip variable names ;)):

# ===================
# Files
# ===================
c20140510154014 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154014.mts")
c20140510154111 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154111.mts")
c20140510154135 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154135.mts")
c20140510154151 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154151.mts")
c20140510154217 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154217.mts")
c20140510154243 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154243.mts")
c20140510154322 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154322.mts")
c20140510154429 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154429.mts")
c20140510154445 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154445.mts")
c20140510154458 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154458.mts")
c20140510154521 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154521.mts")
c20140510154558 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154558.mts")
c20140510154744 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154744.mts")
c20140510154854 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154854.mts")
c20140510155012 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155012.mts")
c20140510155050 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155050.mts")
c20140510155133 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155133.mts")
c20140510155144 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155144.mts")
c20140510155152 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155152.mts")
c20140510155207 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155207.mts")
c20140510155218 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155218.mts")
c20140510155231 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155231.mts")
c20140510155252 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155252.mts")
c20140510155338 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155338.mts")
c20140510155412 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155412.mts")
c20140510155427 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155427.mts")
c20140510155451 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155451.mts")
c20140510155550 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155550.mts")
c20140510155634 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155634.mts")
c20140510155703 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155703.mts")
c20140510155732 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155732.mts")
c20140510155848 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155848.mts")
c20140510155931 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510155931.mts")
c20140510160017 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160017.mts")
c20140510160100 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160100.mts")
c20140510160129 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160129.mts")
c20140510160226 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160226.mts")
c20140510160437 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160437.mts")
c20140510160606 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160606.mts")
c20140510160721 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160721.mts")
c20140510160729 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160729.mts")
c20140510160916 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510160916.mts")
c20140525171805 = LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140525171805.mts")

# ===================
# Timeline editing
# ===================
# ...

# ===================
# Final output
# ===================

MessageClip("No edited output available yet")


LWOpen() is defined in the auto-include utility script below as:

function LWOpen(string source) {
return(AudioDub(LWLibavVideoSource(source), LWLibavAudioSource(source)))
}


When I run my script, I get:

LWLibavAudioSource: failed to get the audio track.
(FV_utils-1.9.0.avsi, line 116)
(C:\Users\fvisagie\Videos\Home Videos\20140510 Collene & Kallie Pre-shoot\1. Footage.avs, line 38)


This error would occur anywhere from line 38 to line 41, whenever AvsPmod's memory usage reaches around 1666 MB RAM. At that point total physical memory utilisation across all processes is only ~75%. The machine has 4 GB.

Setting higher-than-default values with SetMemoryMax() makes no difference I could notice. When I use FFVideo/AudioSource() instead of LWLibavAudio/Source(), I get similar behaviour:

FFVideoSource: Insanity detected: decoder returned an empty frame
(FV_utils-1.9.0.avsi, line 184)
(C:\Users\fvisagie\Videos\Home Videos\20140510 Collene & Kallie Pre-shoot\1. Footage.avs, line 45)

This happens at about 1620 MB memory usage.

When I do NOT assign variables to the opened clips e.g.

LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154014.mts")
LWOpen("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154111.mts")
...

I do not get this problem. I know the filtergraph etc. probably look different in this case, but even so I confirmed that each file is still opened and indexed.

This suggests I would be able to complete this task by opening the files and editing the timeline in-line, e.g. LWOpen("file1").Trim(a, b) ++ LWOpen("file2").Trim(c, d). However I find this way of working without variables much less convenient.

Am I doing something wrong, or is this a bug? Is there a way to get around this memory problem so I can use my preferred approach, i.e. assigning a variable to each opened clip?

Many thanks,
Francois

[Gavino]

If I understand what's going on here, it's not the use of variables per se which is causing the problem. In your script without variables, each source filter instance is no longer referenced once the line following has been evaluated (overwriting the variable last). They will therefore be immediately freed by Avisynth (the filters' destructors will be called). In the 'variables' script, because distinct variables are used, each one holds a reference so nothing is freed.

Using the source filters (or wrapper function) directly (instead of variables) won't gain anything because the filter graph itself will hold a reference to each one used.

The only thing I can suggest is to render your script in sections, saving intermediate results to a lossless format.

[StainlessS]

In my experience, all video (not image) source filters screw up at a certain number of open clips,
I think I decided that 28 (maybe it was 23) opened files was usually as many as could be reliably opened at once, so
if I had more than this I would do them in several multiclip sections, saving to lossles as Gavino suggests.

[fvisagie]

Thanks for your feedback, guys.

if I had more than this I would do them in several multiclip sections, saving to lossles as Gavino suggests.

That's a daunting prospect - with one of my recent projects the camera generated 449 clips/files!

Some more results.

As you suspect, even when I open and splice files in-line with no intermediate functions or variables, e.g.

...
AudioDub(LWLibavVideoSource("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154111.mts"), LWLibavAudioSource("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154111.mts")) ++ \
AudioDub(LWLibavVideoSource("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154135.mts"), LWLibavAudioSource("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140510 Collene & Kallie Pre-shoot\20140510154135.mts")) ++ \
...

I get the same problem.

When I set the LWLibavVideoSource() threads parameter to values in the range 1 - 3 inclusive, the problem does not occur. The lower the threads value, the less memory used and the slower the processing. My machine has a quad-core i5-2540M.

So, conversely, the higher the number of threads used the more memory used until ultimately the plugin fails. This begs the question of why the plugin errors out when not even the physical memory has been exhausted yet at that point - ~75%?

[raffriff42]

In my experience, all video (not image) source filters screw up at a certain number of open clips,That's been my experience as well. However you can use FFmpeg to join your videos together (http://trac.ffmpeg.org/wiki/How%20to%20concatenate%20%28join,%20merge%29%20media%20files) into a few (1 to 20) longer clips, then work with those in Avisynth, cutting them up as much as you like*

*edit - I don't know what the limit is, but I have just tested 200 clip variables (trimmed from a single Source) with no trouble.

[StainlessS]

You could do worse than take a look here to see a work in progress for joining multiple clips (althoug you would not need most
of the functionality of the scripts given)
http://forum.doom9.org/showthread.php?t=166820&highlight=avisynthesizer_mod

[fvisagie]

I assume there is no existing or easy solution to the issue of too many open clips then.

You could do worse than take a look here to see a work in progress for joining multiple clips (althoug you would not need most
of the functionality of the scripts given)
http://forum.doom9.org/showthread.php?t=166820&highlight=avisynthesizer_mod

Thanks for the suggestion. Apart from the purpose of RoboSplice which I managed to figure out, what does the rest of that project do? The original project and documentation seem to have been removed from the Tangentsoft site.

However you can use FFmpeg to join your videos together (http://trac.ffmpeg.org/wiki/How%20to%20concatenate%20%28join,%20merge%29%20media%20files)
...
I have just tested 200 clip variables (trimmed from a single Source) with no trouble.

FFmpeg's been my preferred back-up plan for joining, thanks, primarily because that approach preserves stream metadata. Thanks also for the reassuring test.

The downside of joining source files (in addition to more time and disk space) is that metadata is sacrificed. In this case, once joined the start and end points of clips are lost. Typically in the projects I work with this adds a whole new dimension of effort.

What existing solutions are there to this issue other than the non-precise approach of image-based scene change detection? What I would find very elegant is concatenating in Avisynth to lossless output with a runtime script that records timecode discontinuities in a scene file/AvsPmod bookmark file etc. The concept could conceivably be extended to continuously record any metadata such as date and timestamp to e.g. subtitle format. As opposed to extracting metadata on-the-fly (which I haven't yet found a solution for) I currently foresee parsing pre-generated metadata text files. Therefore this approach could even be used to apply timeline edits to existing subtitle text files without needing to parse potentially complex script timeline edits. I've found a way of providing AVCHD metadata such as timecode and timestamp, and current_frame very nicely identifies source file frame number at runtime. However, I've found no way of identifying the current input file(s) at runtime. Any suggestions for that (bearing in mind I'm in no position to write my own plugins ;))?

[StainlessS]

The original TangentSoft doc is included in the DOC folder inside the zip, "Videographica AVISynthesizer Manual.htm".
Looks like he might have been getting unwanted communications about what was for him a dead project.

The original Avisynthesizer project allows group selection of files in Windows Explorer, right click and 'Send To" Avisynthesizer
will output an avs file which concatenates selected files using a template script.

If you chose to use Avisynthesizer_Mod, it would be no trouble at all to capture filenames, ie in equivalent of my DoFile()
function, could update a global variable representing File number, writing filename and extracted data to a file, or maybe even
an RT_Stats database (you can have up to 256 fields in current RT_Stats Dbase, 1024 in the Beta version recently posted in that thread).
You can also use eg RT_Call() to call an external function (eg MediaInfo cmd line version) that extracts metadata to a text file, parse and extract from that,
and maybe add it to the Dbase file.
The Dbase can hold all Avisynth variable types (excluding clip), so eg first field of a record (clip) might be filename, 2nd frame count, etc.
Reading writing and parsing extracted text files can reasonably easily done using RT_Stats functions.
One user (Filker I think) used RT_Stats and RT_Call to extract EXIF data from jpegs using a command line utility that extracts EXIF data from those
jpegs, might give you a few ideas if you look up the concerning posts.
You could even eg have a master DBase, holding eg names of a sub DBase, where sub Dbase holds individual clip data, and perhaps eg name of text file
holding subtitles text. Master Dbase holding name of Sub Dbase file, and data relating to the complete assembled final clip.
You dont need to be able to write your own plugins, RT_Stats probably provides pretty much everything you need, apart from external
commands to extract the various types of data you are interested in keeping.

EDIT: So this Avisynthesizer_mod template

#ASYNTHER RoboSplice

Global G_FILECOUNT=___FILECNT___

Global G_DEBUG=True
Global G_QBC=True Global G_SAMPLES=32 Global G_QBC_THRESH=-32.0 Global G_INTERLACED=False
Global G_AL_STRENGTH=0.95 Global G_GAMMA=1.0 Global G_AUTOGAIN=False

Global G_DEBLOCK=20
Global G_RPOW2=false
Global G_DISPWID=0 Global G_DISPHIT=0

Global G_FRNUM=0 Global G_FRDEN=0
Global G_USE_INTERFRAME=False

Global G_ASPECT_DARX=0 Global G_ASPECT_DARY=0

Global G_MCD=0 Global G_SIGMA=1.65 Global G_SHARPEN=0.0
Global G_CHANNELS=2 Global G_SAMPLERATE=44100 Global G_BITS=16 Global G_NORMALIZE=(G_FILECOUNT>1)

[GetTrim("___FILE___")]


would produce this script (if "Send To" the 4 filenames in blue):

#ASYNTHER RoboSplice

Global G_FILECOUNT=4

Global G_DEBUG=True
Global G_QBC=True Global G_SAMPLES=32 Global G_QBC_THRESH=-32.0 Global G_INTERLACED=False
Global G_AL_STRENGTH=0.95 Global G_GAMMA=1.0 Global G_AUTOGAIN=False

Global G_DEBLOCK=20
Global G_RPOW2=false
Global G_DISPWID=0 Global G_DISPHIT=0

Global G_FRNUM=0 Global G_FRDEN=0
Global G_USE_INTERFRAME=False

Global G_ASPECT_DARX=0 Global G_ASPECT_DARY=0

Global G_MCD=0 Global G_SIGMA=1.65 Global G_SHARPEN=0.0
Global G_CHANNELS=2 Global G_SAMPLERATE=44100 Global G_BITS=16 Global G_NORMALIZE=(G_FILECOUNT>1)

GetTrim("D:\AVS\AVI\TEST2YV12.avi") ++ \
GetTrim("D:\AVS\AVI\TEST2RGB.avi") ++ \
GetTrim("D:\AVS\AVI\TESTYV12.avi") ++ \
GetTrim("D:\AVS\AVI\TESTRGB.avi")


Where GetTrim() can do anything it wants with the filename, resulting clips being joined together.
EDIT: There is facility to order the given filenames within Avisynthesizer_mod.

[fvisagie]

@StainlessS,

Thanks!
RT_Stats probably provides pretty much everything you need, apart from external commands

Are you also implying that it can identify the current input clip at runtime? Just wanted to check before diving into it ;).

[StainlessS]

Where GetTrim() can do anything it wants with the filename

As it says above. Read my previous post again.

[StainlessS]

fvisagie,

OK, spent a few hours on the stuff below, it's not very pretty but works OK, still would need a lot of
work to perfect such usage, and some kind of function to do a binary search on the Dbase when
user jumps about on the timeline (to find which trim it is in and current trim frame and filename).
You cannot jump about, MUST only play straight though.
It is in the form of an Avisynthesizer Template that has already been initialized by Avisynthesizer,
ie has had "Global G_FILECOUNT" and the input filenames aready set. You just need to delete those lines
and uncomment the original template lines to restore it as a template.
If you want to try it out, suggest cut out 5 clips from an avi, named "0.avi" to "4.avi" with either
1000 or 100 frames (I used 1000).
It will create a single Dbase and enter in a little data, just frame numbers really and filename (you can add whatever else you like
and change the source filter to whatever).
During ScriptClip processing it subtitles:- whole movie frame number, Trim number, Trim frame number, and trim Filename.
Probably not all globals were necessary, but I gotta get some sleep sometime soon.
Here tis:

#ASYNTHER Test
############################################
#Global G_FILECOUNT=___FILECNT___ # Total clips being processed, filled in by Avisynthesizer_Mod
Global G_FILECOUNT=5 # Total clips being processed
############################################
Global G_CURRENT_TRIM=0 # Count of clips done so far : MUST BE 0
############################################
Global G_PROJECTNAME="Test" # Change to whatever you want
Global G_DB=G_PROJECTNAME+".DB" # DBase
Global G_TYPESTRING="s512s512iii" # Fields:-
# 0) String(512) ClipFileName
# 1) String(512) ClipSubTitles (not used)
# 2) Int ClipFrameCount
# 3) Int MovieStartFrame
# 4) Int MovieEndFrame,
#############################################
RT_DBaseAlloc(G_DB,G_FILECOUNT,G_TYPESTRING) # Create Movie DB
###
#[ProcessTrim("___FILE___")] # filled in by Avisynthesizer_Mod
ProcessTrim("0.avi") ++ \
ProcessTrim("1.avi") ++ \
ProcessTrim("2.avi") ++ \
ProcessTrim("3.avi") ++ \
ProcessTrim("4.avi")

Global G_CURRENT_TRIM = -1 # Init for play
Global G_MOVIETRIM_START= -1 # Init for play
Global G_MOVIETRIM_END = -1 # Init for play
Global G_FN="" # Init for play

ScriptClip("""
DoInit = (current_Frame > G_MOVIETRIM_END)
Global G_CURRENT_TRIM = (DoInit) ? G_CURRENT_TRIM + 1 : G_CURRENT_TRIM
Global G_MOVIETRIM_START= (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,3) : G_MOVIETRIM_START
Global G_MOVIETRIM_END = (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,4) : G_MOVIETRIM_END
Global G_FN= (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,0) : G_FN
RT_SubTitle("%d] {%d:%d} %s",current_frame,G_CURRENT_TRIM,current_frame-G_MOVIETRIM_START,G_FN)
return Last
""")
Return Last

#############################################
### Contents of Process.avsi
Function ProcessTrim(string fn) {
RT_DebugF("Processing %d ] %s",G_CURRENT_TRIM,fn)
c = Avisource(fn)
FC = c.FrameCount
SubTitleFN="" # We dont have subtitle file, maybe we could get it from some external program.
S = (G_CURRENT_TRIM == 0) ? 0 : RT_DBaseGetField(G_DB,G_CURRENT_TRIM-1,4) + 1 # Previous MovieEndFrame + 1
E = S + FC - 1
RT_DBaseSet(G_DB,G_CURRENT_TRIM,fn,SubTitleFN,FC,S,E)
Global G_CURRENT_TRIM = G_CURRENT_TRIM + 1 # Increment current trim number. NOTE Assign Global
return c
}


And this is a little script to show contents of DBase, might come in handy to check out if working OK.
Results sent to DebugView.

DB="TEST.DB"
###
TYPENAMES=RT_String("Bool\nInt\nFloat\nString\nBin\n")
GSCript("""
Records=RT_DBaseRecords(DB)
Fields=RT_DBaseFields(DB)
RT_DebugF("\n%s\n",DB)
RT_DebugF("Records = %d",RT_DBaseRecords(DB))
RT_DebugF("RecordSize = %d",RT_DBaseRecordSize(DB))
RT_DebugF("RecordsMax = %d ($%X)",RT_DBaseRecordsMax(DB),RT_DBaseRecordsMax(DB))
RT_DebugF("Fields = %d",FIELDS)
for(i=0,Fields - 1) {
TYPE_S=RT_TxtGetLine(TYPENAMES,RT_DBaseFieldType(DB,i))
RT_DebugF(" %2d ) Type = %d(%6s) Size = %d",i,RT_DBaseFieldType(DB,i),TYPE_S,RT_DBaseFieldSize(DB,i))
}

RT_debugF("\nShowing Contents\n")
QUOT=Chr(34)
for(i=0,Records-1) {
RT_DebugF("%d] ----------------",i)
for(j=0,fields-1) {
Typ=RT_DBaseFieldType(DB,j)
RT_DebugF(" %d] %s%s%s",j,Typ==3?QUOT:"",String(RT_DBaseGetField(DB,i,j)),Typ==3?QUOT:"")
}
}
""")
colorbars.killaudio
return last


EDIT: You will have to edit path to filenames, I used "D:\AVS\DD\".
EDIT: Forget above, I've edited to use current directory eg just "0.avi".

[fvisagie]

OK, spent a few hours on the stuff below

This is valuable, thanks for the effort. Hope you got some sleep :).

Cheers,
Francois

[StainlessS]

Hope you got some sleep
Not yet, I'll have to hurry because I've got to get up half an hour ago :)

OK, done the findTrim() thing, you can now jump about on the timeline.

#ASYNTHER Test
############################################
#Global G_FILECOUNT=___FILECNT___ # Total clips being processed, filled in by Avisynthesizer_Mod
Global G_FILECOUNT=5 # Total clips being processed
############################################
Global G_CURRENT_TRIM=0 # Count of clips done so far : MUST BE 0
############################################
Global G_PROJECTNAME="Test" # Change to whatever you want
Global G_DB=G_PROJECTNAME+".DB" # DBase
Global G_TYPESTRING="s512s512iii" # Fields:-
# 0) String(512) ClipFileName
# 1) String(512) ClipSubTitles (not used)
# 2) Int ClipFrameCount
# 3) Int MovieStartFrame (of trim)
# 4) Int MovieEndFrame (of trim)
#############################################
RT_DBaseAlloc(G_DB,G_FILECOUNT,G_TYPESTRING) # Create Movie DB
###
#[ProcessTrim("___FILE___")] # filled in by Avisynthesizer_Mod
ProcessTrim("0.avi") ++ \
ProcessTrim("1.avi") ++ \
ProcessTrim("2.avi") ++ \
ProcessTrim("3.avi") ++ \
ProcessTrim("4.avi")

Global G_PREV_FRAME = -2 # Force Init
Global G_CURRENT_TRIM = -1 # Init for play
Global G_MOVIETRIM_START = -1 # Init for play
Global G_MOVIETRIM_END = -1 # Init for play
Global G_FN="" # Init for play

ScriptClip("""
DoInit = (G_PREV_FRAME + 1 != current_frame || current_frame > G_MOVIETRIM_END)
Global G_CURRENT_TRIM = (DoInit) ? FindTrim(G_DB,3,4,current_frame) : G_CURRENT_TRIM
Global G_MOVIETRIM_START= (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,3) : G_MOVIETRIM_START
Global G_MOVIETRIM_END = (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,4) : G_MOVIETRIM_END
Global G_FN = (DoInit) ? RT_DBaseGetField(G_DB,G_CURRENT_TRIM,0) : G_FN
RT_SubTitle("%d] {%d:%d} %s",current_frame,G_CURRENT_TRIM,current_frame-G_MOVIETRIM_START,G_FN)
G_PREV_FRAME = current_frame # Remember previous frame visited for next iteration
return Last
""")
Return Last

#############################################
### Contents of Process.avsi
Function ProcessTrim(string fn) {
RT_DebugF("Processing %d ] %s",G_CURRENT_TRIM,fn)
c = Avisource(fn)
FC = c.FrameCount
SubTitleFN="" # We dont have subtitle file, maybe we could get it from some external program.
S = (G_CURRENT_TRIM == 0) ? 0 : RT_DBaseGetField(G_DB,G_CURRENT_TRIM-1,4) + 1 # Previous MovieEndFrame + 1
E = S + FC - 1
RT_DBaseSet(G_DB,G_CURRENT_TRIM,fn,SubTitleFN,FC,S,E)
Global G_CURRENT_TRIM = G_CURRENT_TRIM + 1 # Increment current trim number. NOTE Assign Global
return c
}

Function FindTrim(String DB,Int S_Field,Int E_Field,Int Frame) {
# DB Find Trim (record) containing movie frame Frame, S_Field = MovieStartFrame, E_Field = MovieEndFrame
result = -1 # Init NOT FOUND
low = 0
high = RT_DBaseRecords(DB) - 1
GSCript("""
while(low <= high) {
mid = (low + high) / 2
if(RT_DBaseGetField(DB,mid,E_Field) < Frame) {
low = mid + 1
} Else If (RT_DBaseGetField(DB,mid,S_Field) > Frame) {
high = mid - 1
} Else {
low = high + 1 # Force exit
Result = mid
}
}
""")
return result
}

[fvisagie]

Thanks muchly.

OK, done the findTrim() thing

So having digested what RT-Stats, Avisynthesizer_Mod and Gscript are all about, let alone the logic, lemme see if I get this straight, big breath... (my only real coding work was M68K assembler), this works by:

storing clip data in a DB,
having knowledge of the script's timeline editing (in this case UnalignedSplices and the order in which they occur)
frame-counting at runtime,
and using that to infer from the DB the current input clip and frame (the latter also being the same as current_frame?)

Feel free to correct ;).

[StainlessS]

1) Yep.

2) Nope, Aligned Splice. (I refer to them as trims probably because RoboSplice also originally trimmed black frames from start/end of each input clip).
(Avisynthesizer used UnAligned Splice, I modded because I did not like that).

3) Yep.
4) Yep.

Having Global data available in ScriptClip makes in unnecessary to interogate the DBase for everything at every frame, eg getting
Filename of clip would swallow a chunk of memory at every frame (not released until avisynth closure). In the given instance we subtitle splice
info on every frame, and so we need to know it at every frame. We only need to interogate DBase when user jumps about, or when we enter a new splice region.
If you only needed to do something when entering a new splice, then it would probably suffice to track Global G_PREV_FRAME, and Global G_MOVIETRIM_END,
the 1st to see if user jumped about, 2nd to detect entering of new splice region.
EDIT And Global G_MOVIETRIM_START if you needed to know Splice frame number.

EDIT: Note, The Dbase also has 256 (currently or 1024 in beta) Int or Float attributes that can be read/written,
during DBase creation phase, could use one of them to track current splice/trim, but not much different to using a Global.

EDIT: It might be better to move fields MovieStartFrame and MovieEndFrame to fields 0 and 1, ie movie data at start of DBase
and after that everything belongs to trim/splice.

[StainlessS]

Here modified FselShowDB.avs

# FselShowDB.avs
FSEL_TITLE="Select DBase file"
FSEL_DIR="."
FSEL_FILT="DB files|*.DB"
FSEL_MULTI=False
DBFILE_LIST = RT_FSelOpen(title=FSEL_TITLE,dir=FSEL_DIR,filt=FSEL_FILT,multi=FSEL_MULTI)
Assert(DBFILE_LIST.IsString,"FselShowDB.avs: RT_FSelOpen Error="+String(DBFILE_LIST))
DB=RT_TxtGetLine(DBFILE_LIST,0)
###
TYPENAMES=RT_String("Bool\nInt\nFloat\nString\nBin\n")
GSCript("""
Records=RT_DBaseRecords(DB)
Fields=RT_DBaseFields(DB)
RT_DebugF("\n%s\n",DB)
RT_DebugF("Records = %d",RT_DBaseRecords(DB))
RT_DebugF("RecordSize = %d",RT_DBaseRecordSize(DB))
RT_DebugF("RecordsMax = %d ($%X)",RT_DBaseRecordsMax(DB),RT_DBaseRecordsMax(DB))
RT_DebugF("Fields = %d",FIELDS)
for(i=0,Fields - 1) {
TYPE_S=RT_TxtGetLine(TYPENAMES,RT_DBaseFieldType(DB,i))
RT_DebugF(" %2d ) Type = %d(%6s) Size = %d",i,RT_DBaseFieldType(DB,i),TYPE_S,RT_DBaseFieldSize(DB,i))
}
RT_debugF("\nShowing Contents\n")
QUOT=Chr(34)
for(i=0,Records-1) {
RT_DebugF("%d] ----------------",i)
for(j=0,fields-1) {
Typ=RT_DBaseFieldType(DB,j)
RT_DebugF(" %d] %s%s%s",j,Typ==3?QUOT:"",String(RT_DBaseGetField(DB,i,j)),Typ==3?QUOT:"")
}
}
""")
colorbars.killaudio.Subtitle("See DebugView Output")
return last



Uses file selector to pick DB file.

AvsPMod (just installed) is a bit wierd with this script, sometimes swallows DebugView output, sometimes interferes with the file selector
mainly the FSEL_FILT string (changes it to ""), but sometimes OK, External Player (F6) is OK though (windows Mplayer2 for me).
EDIT: Also inserts a DirectShowSource("Test.DB") into the script when it plays up, whats with that then.

Here output

00000001 0.00000000 RT_DebugF:
00000002 0.00002762 RT_DebugF: D:\AVS\DD\Test.DB
00000003 0.00005642 RT_DebugF:
00000004 0.00047348 RT_DebugF: Records = 5
00000005 0.00088868 RT_DebugF: RecordSize = 1036
00000006 0.00162306 RT_DebugF: RecordsMax = 2072844 ($1FA10C)
00000007 0.00172965 RT_DebugF: Fields = 5
00000008 0.00277997 RT_DebugF: 0 ) Type = 3(String) Size = 512
00000009 0.00384017 RT_DebugF: 1 ) Type = 3(String) Size = 512
00000010 0.00488063 RT_DebugF: 2 ) Type = 1( Int) Size = 4
00000011 0.00592725 RT_DebugF: 3 ) Type = 1( Int) Size = 4
00000012 0.00698488 RT_DebugF: 4 ) Type = 1( Int) Size = 4
00000013 0.00709123 RT_DebugF:
00000014 0.00712013 RT_DebugF: Showing Contents
00000015 0.00714840 RT_DebugF:
00000016 0.00743387 RT_DebugF: 0] ----------------
00000017 0.00846935 RT_DebugF: 0] "0.avi"
00000018 0.00944840 RT_DebugF: 1] ""
00000019 0.01044262 RT_DebugF: 2] 1000
00000020 0.01141452 RT_DebugF: 3] 0
00000021 0.01237941 RT_DebugF: 4] 999
00000022 0.01248989 RT_DebugF: 1] ----------------
00000023 0.01347254 RT_DebugF: 0] "1.avi"
00000024 0.01463834 RT_DebugF: 1] ""
00000025 0.01563924 RT_DebugF: 2] 1000
00000026 0.01661687 RT_DebugF: 3] 1000
00000027 0.01759837 RT_DebugF: 4] 1999
00000028 0.01770981 RT_DebugF: 2] ----------------
00000029 0.01875268 RT_DebugF: 0] "2.avi"
00000030 0.01972829 RT_DebugF: 1] ""
00000031 0.02070023 RT_DebugF: 2] 1000
00000032 0.02167538 RT_DebugF: 3] 2000
00000033 0.02266695 RT_DebugF: 4] 2999
00000034 0.02277710 RT_DebugF: 3] ----------------
00000035 0.02374526 RT_DebugF: 0] "3.avi"
00000036 0.02472806 RT_DebugF: 1] ""
00000037 0.02570513 RT_DebugF: 2] 1000
00000038 0.02668370 RT_DebugF: 3] 3000
00000039 0.02765870 RT_DebugF: 4] 3999
00000040 0.02776932 RT_DebugF: 4] ----------------
00000041 0.02874646 RT_DebugF: 0] "4.avi"
00000042 0.02990422 RT_DebugF: 1] ""
00000043 0.03091367 RT_DebugF: 2] 1000
00000044 0.03188236 RT_DebugF: 3] 4000
00000045 0.03285628 RT_DebugF: 4] 4999

[fvisagie]

2) Nope, Aligned Splice.
Hehe, exhaustion takes its toll ;).

Here modified FselShowDB.avs

Great stuff.


mainly the FSEL_FILT string (changes it to "")

When I had something similar it was due to AvsPmod parsing incorrectly. It will probably not work but just in case, see if turning off Options.Program settings.Save/Load.Save *.avs scripts etc. turns off AvsPmod's editing of scripts also.


EDIT: Also inserts a DirectShowSource("Test.DB") into the script when it plays up, whats with that then.

Sounds like an AvsPmod template for the ".db" extension that somehow got triggered.

Anyhow, many thanks for all this effort, sincerely appreciated.

[fvisagie]

This works quite nicely too, attaching the necessary knowledge to the clip's filtergraph.

function Open(string source, bool "trace") {
trace = Default(trace, True)
LWOpen(source)
trace ? ScriptClip("""source = """" + source + """"
Subtitle(source + "\n" + String(current_frame), x=0, y=0, size=24, lsp=1)""", after_frame=true) : NOP()
}

c1 = Open("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140221 Kam'Bati\2014-02-22_11-20-24.mp4")
c2 = Open("C:\Users\fvisagie\Videos\Home Videos\Camera Tapes\20140221 Kam'Bati\2014-02-22_12-09-10.mp4")

c1.Trim(10, 20) ++ c2.Trim(30, 40)
ShowFrameNumber(x=Width()-60, y=20)


Whenever a clip's filtergraph is accessed, its ScriptClipt() code now knows the corresponding source file name and frame number :).

For completeness, LWOpen() is

function LWOpen(string source, int "threads") {
threads = Default(threads, 0)
return(AudioDub(LWLibavVideoSource(source, threads=threads), LWLibavAudioSource(source)))
}