Stitching lots of videos together

JonE · 19th October 2011, 09:21

If I have, say, 1000 videos of short duration - 10 seconds each for example - is there any way of efficiently stitching the whole lot together in AviSynth ?

The brute force method of DirectShowSource(file1)+DirectShowSource(file2) + ..... fails ; I'm using DirectShow (GraphStudio actually) to play the script and once if I stitch more than ~~120 files together it simply wont open the script. Looking at the memory usage in task manager, I can see it ramping up as it adds each file and suspect the ~120 limit is probably a memory issue.

So, is there an alternative way of doing this ? There are more efficient ways of stitching _image_ files, as discussed in this thread but I'm not sure how to adapt this to video files.

I want to use the script to re-code the large number of files to a single file, so don't care if there is a pause when switching from one file to another.

Any ideas ?

TTFN,
Jon

Gavino · 19th October 2011, 09:38

You might find you get further using ffms2 instead of DirectShowSource, though even then you will still eventually hit a limit.

See this recent thread, where a similar question was asked.

Bloax · 19th October 2011, 14:02

Well, you could stitch a lot of them together, encode them losslessly to a single file, and keep doing that.
And then stitch all of those larger clips together and encode them as you want to.
Slow, yes. Tedious, yes. Will it work? Most likely! (Unless something goes very, very wrong!)

JonE · 19th October 2011, 15:10

Unfortunately DirectShowSource is the only method I have as the video format comes from a DVR recorder that uses a proprietary h264-based file format but they do supply appropriate DirectShow components.

I was hoping there was a clever trick whereby a source was only created at the point where the individual clip was actually used instead of preparing all clips at the start but my scripting abilities just aren't up to the job of working out if that is even possible :-(

Otherwise, I guess stitching a smaller group together, then stitching the smaller groups together might just work; using my example of 1000 files, if I restricted groups to 10 then, joined them to make groups of 100, then finally grouped them to make the final group of 1000, total processing would be only 3 times longer (plus a bit for overheads) than doing them all in one go and I could run it as an overnight job or something.

However, 2hrs worth of clips, 720x576, 25fps would require 224GB of storage for the intermediate files (gulp!).

TTFN,
Jon

PS : Thanks for link Gavino, missed it on me searches, D'oh!

JonE · 19th October 2011, 15:24

Additional : Just a thought, generating a script to join all these files together would be done by a separate utility. Well, that same utility can access the image size, duration and frame rates of all the individual clips being added to the script and hence the total duration of the resulting joined clip would also be known. Would knowing this in advance help make a clever 'loads-small-clip-only-when-needed' script feasible?

TTFN,
Jon

johnmeyer · 19th October 2011, 16:24

I treat AVISynth as just another tool, rather than the only video editing tool. So, when I am faced -- and I do this a lot -- with having to process several hundred files, I drop them on the timeline in Sony Vegas (my video editing program) and then frameserve this into an AVISynth script as a single file. It takes about two seconds to drop the files into Vegas; I then start the frameserver, and within another ten seconds, I am viewing the results of my script in VirtualDub.

If you don't have Vegas, take a look at whatever editing program you use and see if it too supports some sort of frameserving.

JonE · 20th October 2011, 09:01

I have several alternative editors but they appear to use DirectShow Editing services. If a file has audio and video then it creates TWO seperate instances of the decoding path for each clip, one for audio and one for video, so I'm guessing I'd be two times worse off ; another issue is many commercial editors use their own codec sets so although I have a codec for the footage it doesn't mean the editor will utilise it.

As regards AviSynth, its still an interesting exercise though, as if someone were to say develop an editor based on the AviSynth core (a subject touched on in a number of threads here) then knowing the limitations will be important.

TTFN,
Jon

PS: The actual solution in this one specific case is GMFBridge as I can create just one clip at a time and will therefore have no limit to the number of input clips - at the expense of having to actually do some work [cold shudder]. But still interesting if AviSynth can do it.

19th October 2011, 09:21	#1 \| Link
JonE Registered User Join Date: Sep 2008 Location: Birmingham (UK) Posts: 69	Stitching lots of videos together If I have, say, 1000 videos of short duration - 10 seconds each for example - is there any way of efficiently stitching the whole lot together in AviSynth ? The brute force method of DirectShowSource(file1)+DirectShowSource(file2) + ..... fails ; I'm using DirectShow (GraphStudio actually) to play the script and once if I stitch more than ~~120 files together it simply wont open the script. Looking at the memory usage in task manager, I can see it ramping up as it adds each file and suspect the ~120 limit is probably a memory issue. So, is there an alternative way of doing this ? There are more efficient ways of stitching _image_ files, as discussed in this thread but I'm not sure how to adapt this to video files. I want to use the script to re-code the large number of files to a single file, so don't care if there is a pause when switching from one file to another. Any ideas ? TTFN, Jon

19th October 2011, 09:38	#2 \| Link
Gavino Avisynth language lover Join Date: Dec 2007 Location: Spain Posts: 3,431	You might find you get further using ffms2 instead of DirectShowSource, though even then you will still eventually hit a limit. See this recent thread, where a similar question was asked. __________________ GScript and GRunT - complex Avisynth scripting made easier

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

19th October 2011, 14:02	#3 \| Link
Bloax The speed of stupid Join Date: Sep 2011 Posts: 317	Well, you could stitch a lot of them together, encode them losslessly to a single file, and keep doing that. And then stitch all of those larger clips together and encode them as you want to. Slow, yes. Tedious, yes. Will it work? Most likely! (Unless something goes very, very wrong!)

19th October 2011, 15:10	#4 \| Link
JonE Registered User Join Date: Sep 2008 Location: Birmingham (UK) Posts: 69	Unfortunately DirectShowSource is the only method I have as the video format comes from a DVR recorder that uses a proprietary h264-based file format but they do supply appropriate DirectShow components. I was hoping there was a clever trick whereby a source was only created at the point where the individual clip was actually used instead of preparing all clips at the start but my scripting abilities just aren't up to the job of working out if that is even possible :-( Otherwise, I guess stitching a smaller group together, then stitching the smaller groups together might just work; using my example of 1000 files, if I restricted groups to 10 then, joined them to make groups of 100, then finally grouped them to make the final group of 1000, total processing would be only 3 times longer (plus a bit for overheads) than doing them all in one go and I could run it as an overnight job or something. However, 2hrs worth of clips, 720x576, 25fps would require 224GB of storage for the intermediate files (gulp!). TTFN, Jon PS : Thanks for link Gavino, missed it on me searches, D'oh!

19th October 2011, 15:24	#5 \| Link
JonE Registered User Join Date: Sep 2008 Location: Birmingham (UK) Posts: 69	Additional : Just a thought, generating a script to join all these files together would be done by a separate utility. Well, that same utility can access the image size, duration and frame rates of all the individual clips being added to the script and hence the total duration of the resulting joined clip would also be known. Would knowing this in advance help make a clever 'loads-small-clip-only-when-needed' script feasible? TTFN, Jon

19th October 2011, 16:24	#6 \| Link
johnmeyer Registered User Join Date: Feb 2002 Location: California Posts: 2,695	I treat AVISynth as just another tool, rather than the only video editing tool. So, when I am faced -- and I do this a lot -- with having to process several hundred files, I drop them on the timeline in Sony Vegas (my video editing program) and then frameserve this into an AVISynth script as a single file. It takes about two seconds to drop the files into Vegas; I then start the frameserver, and within another ten seconds, I am viewing the results of my script in VirtualDub. If you don't have Vegas, take a look at whatever editing program you use and see if it too supports some sort of frameserving.

20th October 2011, 09:01	#7 \| Link
JonE Registered User Join Date: Sep 2008 Location: Birmingham (UK) Posts: 69	I have several alternative editors but they appear to use DirectShow Editing services. If a file has audio and video then it creates TWO seperate instances of the decoding path for each clip, one for audio and one for video, so I'm guessing I'd be two times worse off ; another issue is many commercial editors use their own codec sets so although I have a codec for the footage it doesn't mean the editor will utilise it. As regards AviSynth, its still an interesting exercise though, as if someone were to say develop an editor based on the AviSynth core (a subject touched on in a number of threads here) then knowing the limitations will be important. TTFN, Jon PS: The actual solution in this one specific case is GMFBridge as I can create just one clip at a time and will therefore have no limit to the number of input clips - at the expense of having to actually do some work [cold shudder]. But still interesting if AviSynth can do it.