Getting around multiple AVISource limitation?

[ShawnFumo]

Hey, I have kind of an odd problem here (though it wouldn't be at all unusual if avisynth was being used as a general editing engine)...

I want to edit together an instructional video using multiple avi clips on my drive. I had captured DV directly to my hard drive (almost like a webcam) and then used Virtualdub to save out the good takes (using direct stream copy). This significantly cut down on the hard drive space needed to store the footage and let me give a descriptive name to each clip.

Now, Avisynth is convenient for this project, since DGBob gives wonderful slowmotion, I could easily add numbering to the clips as reference, etc. Especially since I'm not editing to music or anything. Just straight cuts. I was already able to create batch file to generate an avs file which loads them all in and cuts them together as a preview.

The problem is that while Avisynth can easily open a couple of large files, it is severely limited when it comes to multiple smaller files. It can load around 70 files before it croaks, giving an error of not finding the decompressor for dvsd. I'm using the panasonic codec. I'm also using 2.5.6 of Avisynth.

First, is this a limitation of windows itself, or something internal to Avisynth? Anyone know of an easy way around this?

The only way I could think of at the moment would be to use some program (avimaster maybe?) to concatinate all the files into one large file and (somehow) during that process record the frame count of each file and create an avs file with trims for each clip so I can re-arrange them. Any thoughts on if that is practical? I'd hate to double the hard drive space needed and make a big 10gig avi, but I will if I have to.

A last resort would be to create avs files for each clip and then bring them into premiere (if I remember right, there's a plugin for premiere that lets it use avs files directly), but I'd rather just use avisynth if at all possible..

Thanks!
Shawn

[rfmmars]

Hey, I have kind of an odd problem here (though it wouldn't be at all unusual if avisynth was being used as a general editing engine)...

I want to edit together an instructional video using multiple avi clips on my drive. I had captured DV directly to my hard drive (almost like a webcam) and then used Virtualdub to save out the good takes (using direct stream copy). This significantly cut down on the hard drive space needed to store the footage and let me give a descriptive name to each clip.

Now, Avisynth is convenient for this project, since DGBob gives wonderful slowmotion, I could easily add numbering to the clips as reference, etc. Especially since I'm not editing to music or anything. Just straight cuts. I was already able to create batch file to generate an avs file which loads them all in and cuts them together as a preview.

The problem is that while Avisynth can easily open a couple of large files, it is severely limited when it comes to multiple smaller files. It can load around 70 files before it croaks, giving an error of not finding the decompressor for dvsd. I'm using the panasonic codec. I'm also using 2.5.6 of Avisynth.

First, is this a limitation of windows itself, or something internal to Avisynth? Anyone know of an easy way around this?

The only way I could think of at the moment would be to use some program (avimaster maybe?) to concatinate all the files into one large file and (somehow) during that process record the frame count of each file and create an avs file with trims for each clip so I can re-arrange them. Any thoughts on if that is practical? I'd hate to double the hard drive space needed and make a big 10gig avi, but I will if I have to.

A last resort would be to create avs files for each clip and then bring them into premiere (if I remember right, there's a plugin for premiere that lets it use avs files directly), but I'd rather just use avisynth if at all possible..

Thanks!
Shawn

Would you be kind enough to post your script. I have never been able to do more than 28 or so myself.

richard
photorecall.net

[ShawnFumo]

Ok, well it looks like I have a possible solution, using virtualdub. It looks like one can create a sylvia script which appends multiple clips together. If I set that to direct stream copy and then frameserve it out, I can open it all in avisynth using one avisource command. I can make a batch file which creates the sylvia file, though I'm not sure if there's a way to automate frameserving in virtualdub. Not a huge deal if not, since I only have to set that up once at the start..

Now, I need to automate the trim points. I think I can do that in avisynth itself by opening each avi file one at a time and writing out the in/out points to another avs file. I think if I overwrite the same clip, it won't run into the barrier since there'll only be one file open by the time it gets to the output.

We'll see how it goes... I just wish there was an easier way of doing this. I know the programs are different, but you'd think that if virtualdub can frameserve an arbitrary number of avi files at once, that avisynth could handle it (especially since it's a dedicated frameserver).

Shawn

[ShawnFumo]

Richard,

I wasn't doing anything special. I'm using functions, but it basically comes down to something like:

output = output+avisource("file.avi")

Maybe the amount is dependant on hardware or memory or something...

Shawn

[Leak]

Maybe the amount is dependant on hardware or memory or something...
Nah, it's got more to with AviSynth opening all files used with AviSource at the same time, and Windows loading a codec instance for each of them - and blowing up when it hits some internal limit of loaded (codec) DLLs...

A solution would be to only open a file when it's frames are first accessed and start closing the oldest files again when a certain number of files have been opened - that would work fine unless you really need frames from <limit> different clips at the same time - but it's better than not working at all, and of course the limit could be adjustable.

It's just that nobody implemented this yet...

np: The Dolls - Sunbird (The Dolls)

[ShawnFumo]

Leak, that'd have to be implemented internal to avisynth right? That would definitely be cool feature to have for those that wanted to use avisynth for editing lots of files together.

In any case, I did finally manage to implement the solution I talked about above. I'm now able to work with over 250 files directly from inside of avisynth while keeping them as separate files on the hard drive... :)

It is all a bit crazy (three batch files, frameserving from virtualdub, writing avs from avs, and a total of four avs files in the end), but it actually works and seems to use very little memory.

If anyone is interested in the code, let me know and I'll post it..

Shawn

[gzarkadas]

If anyone is interested in the code, let me know and I'll post it..
We are always interested (ok, at least I am interested :) )!

[ShawnFumo]

Haha.. ok, here we go. Keep in mind this is still way messy and needs some cleaning up. All my scripting type files start with underscores so they sort at the top of the directory.

First up is "_gen_virtualdub.bat":

@echo off
echo creating stuff

del _append.syl

for %%I in (*.avi) do (
echo VirtualDub.Append^(U"%%~fI"^);
) >> "_append.syl"


All that does is create the script "_append.syl" (deleting it if it already exists, since it'd append it otherwise) with something like this:

VirtualDub.Open(U"E:\DV\wrist mount\combo_dismount_traphang_suicide1.avi");
VirtualDub.Append(U"E:\DV\wrist mount\combo_dismount_traphang_suicide2.avi");
VirtualDub.Append(U"E:\DV\wrist mount\combo_infinitelindy_curlmounts1.avi");
VirtualDub.Append(U"E:\DV\wrist mount\combo_nickmount1.avi");


Now, since I'm using a batch file instead of being smart and using python or something, everything will be Appends. So you need to open it up and change the first line to "Open" as I have shown above. Also, the "U" isn't a mistake. That let's you have backlashes without having to escape out.

Now fire up Virtualdub, and in the File menu select Run Script. Then be sure it is Direct Stream Copy under video and do Start Frame Server. If you've never done VD frameserving, be sure you run auxsetup and install AVIFile support first. I'm saving out "_fs.vdr" as the frameserver file.

Next up is another batch file (could probably be combined into the last one actually) called "_gen_avs_pre.bat":


@echo off
echo creating stuff

del _files.avs

for %%I in (*.avi) do (
echo findedit^("%%~nI.avi"^)
) >> "_files.avs"


That creates something like this in "_files.avs":

findedit("combo_dismount_traphang_suicide1.avi")
findedit("combo_dismount_traphang_suicide2.avi")
findedit("combo_infinitelindy_curlmounts1.avi")
findedit("combo_nickmount1.avi")


That file is imported into the following avs file (which also has the function findedit) "_genedits.avs":


# Where the files are. Good for previewing.
SetWorkingDir("e:\dv\wrist mount\")

global frameoffset=0

import("_files.avs")

function findedit(string clipname) {
numframes = framecount(avisource(clipname))
thisStart = string(frameoffset)
global frameoffset = frameoffset+numframes
thisEnd = string(frameoffset-1)
shortname = LeftStr(clipname, StrLen(clipname)-4)

filetext = "# "+clipname+Chr(13)
\+shortname+"=trim(bigclip,"+thisStart+","+thisEnd+")"+Chr(13)

WriteFileEnd(blankclip(), "_trims.avs", "filetext", append=true)
}


As you call findedit, it keeps track of how many frames have been in clips of previous calls and uses that as an offset to the current one. It only opens one file at a time and so avoids the avisource limit. Also, my saving grace was that WriteFileEnd can be called multiple times and it'll write everything out from all calls (in semi-random order, but that doesn't matter in this case). Just play this file and after some chugging, a blank clip will play. As soon as it starts playing, it has written out the file.

That gives us "_trims.avs":

# combo_dismount_traphang_suicide2.avi
combo_dismount_traphang_suicide2=trim(bigclip,274,512)

# combo_infinitelindy_curlmounts1.avi
combo_infinitelindy_curlmounts1=trim(bigclip,513,803)

# combo_dismount_traphang_suicide1.avi
combo_dismount_traphang_suicide1=trim(bigclip,0,273)

# combo_nickmount1.avi
combo_nickmount1=trim(bigclip,804,1298)


The way I have it set right now, it actually creates an avisynth clip using the name of the original file (minus the extension), which is very convenient. Keep in mind that if you want to use it this way, your original files can't have characters like "-" or "+" or spaces, all of which would confuse avs. I originally used dashes instead of underscores and had to use Oscar's File Renamer to fix them. This file will be imported into another one (which contains "bigclip").

That final file is "_edit.avs":

# For slow motion and/or de-interlacing
LoadPlugin("c:\program files\avisynth\plugins\DGBob.dll")

# Where the files are. Good for previewing.
SetWorkingDir("e:\dv\wrist mount\")

# Get the massive append of videos from Virtualdub
bigclip=avisource("_fs.vdr")
# Get access to these files by way of trims of bigclip
import("_trims.avs")

# Edit clips together. Slowmo and further trimming are possible. Adds clips to
# the global "output" clip.
addClip(combo_dismount_traphang_suicide2, slowmo=false)
addClip(combo_dismount_traphang_suicide1, slowmo=true, start=0, end=50)

# Correct aspect ratio from DV to Computer.
#output = Lanczos4Resize(output, 720, 540)

return output


############ Functions ###############

function addClip(clip base, bool "slowmo", int "start", int "end") {
# Show the entire clip unless otherwise specified.
start = default(start,0)
end = default(end,0)

#base = AVISource(name)
base = killaudio(base)
base = Trim(base,start,end)

# Convert the color space for eventual XviD compression. Need this since
# Panasonic DV codec only outputs RGB.
base = ConvertToYUY2(base, interlaced=true)

# De-interlace and maybe do slowmo. Currently using bob for
# faster preview of non-slowmo.
base = slowmo ? DGBob(base, 0, mode=2, thresh=0) : bob(base)
base = changefps(base, 29.97)

# This is probably better quality for non-slowmo, but too slow
# to really preview correctly.
#base = slowmo ? DGBob(base, 0, mode=2, thresh=0) : DGBob(base, 0,mode=0, thresh=0)

# Creates or appends to the global "output" clip
try {global output = output+base}
catch(err_msg) {global output = base}
}


As you can see, I can now refer to each of the original files by way of their clip alias. All I'm doing right now is adding them together with possible slowmo and additional cuts. I'll probably fuss with dissolves and such after, but the main problem is solved. I may also use a third batch file to generate "addclip" calls for every file with the default settings that I can then modify.

Hope that helps someone!
Shawn

P.S. If you're wondering on my bizarre file names, it is for a yo-yo video. ;)

[gzarkadas]

Well documented solutions always help; thanks :D

How many clips did you process in one go with this set of scripts?

[ShawnFumo]

I added a couple more recently, so I think I'm up to over 270 files. I'm not using them all in my final video, but I don't think there'd any issues doing so. I think if there was a limit on the number of clip names, an error would have occured when importing the trims file. So far I've got a 12 minute output video going, all with dissolves, so it seems to be working quite well. :)

Shawn