Decompressor error after too many calls to AVISource?

[cayblood]

I'm splicing a bunch of 1GB dv files together like this:

switched = \
AVISource(aviroot + "\switched_001.avi") + \
AVISource(aviroot + "\switched_002.avi") + \
AVISource(aviroot + "\switched_003.avi") + \
AVISource(aviroot + "\switched_004.avi") + \
AVISource(aviroot + "\switched_005.avi") + \
AVISource(aviroot + "\switched_006.avi") + \
etc.

to make a single stream. If I make this too long, I get an error like this:

AVISource: couldn't locate a decompressor for fourcc dvsd

Does anyone know if there is a way I can do this that can work for any number of files? I am trying to edit multiple camera angles and I have a lot of files.

Thanks,
Carl Youngblood

[Dark Shikari]

I found that AVIsource craps out over a certain number of calls... DirectShowSource, however, doesn't.

[squid_80]

The dll for the decompression codec is loaded separately for every AVISource call. Eventually an OS-imposed limit is reached, the codec can't be loaded and you get that error message.

[Fizick]

probably we must put this info to docs,
but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?

[IanB]

but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?

Very good questions, that are very hard to answer.

It is mostly a per process limit (i.e per script). Generally most people can open 50 but cannot open 100. It usually depends on how much else is being loaded.

Last time I tested, with the Panasonic DV codec I could open 57 and I have had it fail with even less.

[TrueIndigo]

Hey,
I was hoping to off-line a 10-minute film using a single avs file, which might involve around 200 separate avi clips. Does this mean I can’t use Avisynth as I hoped? If DirectShowSource is an answer, what is the essential difference between it and AviSource? Why isn't everyone using DirectShowSource? Yes, I am a newbie!

[IanB]

@TrueIndigo,

You may have to prebatch your scripts into segments of approx 50 input clips, output intermediate lossless workfiles .AVI's then do your master edit, inputing from the 4 intermediate workfiles .AVI's.

Until you try on your PC with your codecs you won't know if you will have problems.

AviSource() uses Video for Windows (VfW) codecs, some formats do not have a VfW codec. It is guarenteed to provide frame accurate seeking. If you have an appropriate VfW codec it just works.

DirectShowSource() uses M$ DirectShow API's intended for writing Media Players. It is NOT guarenteed to provide frame accurate seeking, although many formats actually do get it right. Nearly all formats provide DirectShow codecs, so you can actually decode your stream without a VfW codec. If your format does not implement frame accurate seeking and your encoder app plus your script require it, then you may need to preprocess your stream out to a lossless intermediate format like Huffyuv or raw uncompressed .AVI's.

People do not use DirectShowSource() always because it so often sucks. There are many format specific *Source() filters available like Mpeg2Source(), etc. Generally if your format has a custom source pluggin use it.

[TrueIndigo]

IanB – thanks for your explanation about the differences between DirectShowSource and AviSource. It seems a shame that perhaps the only solution would be to “output intermediate lossless workfiles”. Is there a way of somehow joining avs files together? Or have an avs file referencing other avs files (or would that still be the same as a single avs file with more than 50 avi’s being referenced, and therefore still invoke the error?).
Thanks.

[squid_80]

If you do avisource("a script.avs")++avisource("another script.avs") etc. you're still going to run into the same problem.

IanB: Would this really be so hard to fix? Can't we make a static array somewhere in the avisource class to keep track of what codecs are already loaded and reuse them?

[Leak]

You could also try ffmpegsource (search the forum for it) that uses ffmpeg's libavcodec to read all kinds of files. I'm not totally sure, but I wouldn't rule out the possibility that it also won't run into any kind of DLL load limit here...

[IanB]

@TrueIndigo,

If you are really averse to using workfiles, assuming the files all match in format you could use VirtualDub's append segment to join all the files, then either do a directstream copy to a single file or frame serve the whole lots as one stream. this has been discussed before.

First, test your limits before you start looking for complicated alternatives. You might not have a problem.

@Squid_80,

There was a developement thread discussing various codec caching schemes to get around this limit. I think the consensus was it was doable but a big job.

Restoring SegmentedAviSource to the original VDub model could be much less difficult. I always thought it a pox that the multi-segment avi file capabilities were arbitarily discarded, particularly for re-joining delta frame formats where the segments do not start with a key frame. However it only supported 100 files (00..99).

@Leak,

Good thought, ffmpegsource does stand a good chance of beating the odds.

[TrueIndigo]

IanB -- thanks for your input as usual, though unfortunately I don’t know enough to understand much of the advice!

"Use search. This has been discussed before." -- I am an absolute newbie and I’m not sure what I am searching for. I only just recognised the significance (to me) of the title of this thread. I was hoping to use Avisynth/VirtualDubMod to off-line a short film. If there is a limitation on the number of avi clips (maybe 50+) in a single script, this means a script will run into trouble after about 3-5 minutes (depending on your style of editing). That’s not very long for editing. I started reading about applications on the forum like AvsFilmCutter and assumed some people were editing with Avisynth without any problems.

"Averse to using workfiles" -- I’d like to keep the on-going work in progress in the form of an avs file. This is the thing I like about Avisynth: it is a self-logging EDL and you can access it directly, make changes, refresh, and see the results.

The work-a-rounds instead of using a single script I don’t fully understand. Maybe I’ve misunderstood, but do you mean this workflow:

1. Use a number of avs scripts, each keeping below the problematic limit of avi calls.
2. Export an avi copy of each and in VirtualDub join them together in order to view the combined work.

This allows you to view the current edit, but you can’t make instant, interactive changes as you could by playing an avs file. You no longer have access to the original in/out times of individual clips which are in the avs scripts. So I would have to make manual notes about the changes, implement them in the various avs scripts, export another round of avi workfiles, view the group again, possibly decide on more changes, make more notes, transfer the changes to the various avs scripts, etc, etc.

"First, test your limits...You might not have a problem." -- do you mean that some people have a problem after 50+ AviSource calls in a single script, and some have no problem with maybe 300+ calls? I was a bit disappointed because I thought that maybe 50+ will indeed turn out to be about the limit for everyone (because it’s an OS-related problem). I’m not clear about the nature of this problem.

IanB, thanks for your thoughts. My lack of understanding frustrates me as much as it must mystify you!
Thanks,
TrueIndigo.

[IanB]

I was a bit disappointed because I thought that maybe 50+ will indeed ...

Why would you be disappointed if the limit does not effect you

Waiter, waiter, there is a fly in my soup. Sir! Shhh, everyone will want one

The limit comes from how many .DLL's a single process can open. Avisynth is not in control of the process it is just one .DLL of many. The encoder application is the controlling process and we do not know what your encoder application is and what needs it has, it could be anything, VirtualDub, VirtualDubMod, MeGUI, TMpegEnc, ... We also do not know what codec your video uses and how many .DLL's it uses or how it shares those .DLL's.

There is a limit and people have run into it with as few as 50 AviSource statements. You need to check your video application with your video codec with your video clips running on your PC.

The best workaround I can offer is do not segment your input video clips in the first place or append the raw segments together into fewer and bigger meaningful lumps.

AviSource("Lumps1to10.avi")
Trim( 42, 142) + Trim(242, 342) + Trim(442, 542) + \
Trim(642, 742) + Trim(842, 942) + ...

Is a perfectly usable model.

And yes learn to use search

Search for the words and terms I and others have already used in this thread. They will keep occuring in the threads that also discuss similar issues.

And when the internal search is to restricted use Google and add the term "site:doom9.org".

[TrueIndigo]

IanB, thanks for your reply, I appreciate it.

"The limit comes from how many .DLL's a single process can open..." -- yes, I said I wasn't clear about the nature of this problem, and your explanation has made it much clearer. When I get more of my material together for a test I'll find out more about my particular setup. Many thanks.

"Do not segment your input video clips in the first place" -- I like the HDVSplit method of splitting up clips according to shot, because it saves space on the HDD. For example, in a worst case scenario, you might need to keep a single half-hour long clip on the HDD, from which you might only want to use a few short shots. As far as efficient space usage goes, it's much better to have a few clips each a few minutes long, rather than a single half-hour clip. This space issue becomes more of a problem if you have a lot of material to choose from.

"Append the raw segments together into fewer and bigger...lumps" -- Yes, this solves the possible problem for Avisynth. After a process of elimination I could join the required short clips together into just a few large files, and these would be saved out as a high quality uncompressed video in order not to lose quality (because using the online/offline method, they have to become the new master footage -- you need the absolute relationship between the low res proxy and high res master which it represents). I do have a few reservations about this though, compared to staying with the original separate m2t files:

(1) Uncompressed files (even efficient lossless compression formats) take up much more HDD space than the equivalent m2t files they replace -- considering they are HD masters, m2t files only take up the same space as SD DV.

(2) It's an extra process which takes more time -- accumulates into more of an issue when a lot of material is involved. For example, my last short film had an edited duration of 28 minutes which, because of the informal nature of the shooting, was derived from over 40 hours of material.

(3) When I append raw segments together as a large file, I can't know the final order they will be in because I haven't started editing yet. They could be joined in an arbitrary way or just as it comes from the camera (clips a, b, c...m, n, o, etc). I might insert a few seconds of black blank space between each clip so there is no confusion over where one clip ends and another starts. Then the clips could be trimmed and spliced using the example coding you gave. But editing is normally non-linear, so my final cut will include the clips in a mixed up order (eg. clip c, k, j...f, a, l, etc). Then I might want to transpose sound from one part to another. From a newbie's point of view the scripting for this seems like a nightmare (even if I knew how to do it), compared to being able to do the same thing with single clips.

cayblood -- I hope you don't think I've hijacked your thread for a discussion about editing. Clearly, I am interested in the viability of Avisynth as a NLE, but my queries are in the context of possible problems with large avi calls in a single script, and this is the nature of your thread. IanB and the others have made some great points here and I hope the discussion might be of interest to others. I'm trying to learn from you guys, my intention is not to try your patience -- life is hard enough as it is!

Many thanks.

[Ebobtron]

If we are talking m2t (HD MPEG2), can you not combine all your source files using DGINDEX, they would remain as individual files imported to AviSynth with one *.d2v file. But I am not sure about this at all.

[foxyshadis]

You don't even need to do that - you can batch index all of your files into individual d2vs, then load them all with multiple calls to mpeg2source without running into the dll limit, that way you don't have to waste time splitting them back up in the script.

Indigo, if you don't find a satisfactory solution, you might also save time/money by buying a much larger hard drive, 500G to 1T are so cheap now. It's a brute force method, but it beats wasting your time and your customers'.

[IanB]

From Post #7

Originally Posted by ianb

Generally if your format has a custom source pluggin use it.

[TrueIndigo]

Foxyshadis -- sensible advice, thanks.

"multiple calls to mpeg2source without running into the dll limit" -- I originally tried playing the m2t/d2t files, which would be the most elegant workflow of all (and using ReduceBy2 in case that helped performance), but the playback was too choppy on my machine for viable editing. I have a P4 3.0 GHz, 2 Gb RAM, 2x 250 GB HDDs. This machine is probably too slow for realtime HDV editing, but I don't mind the offline/online process (using SD sized avi proxies). However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script. As IanB says, I need to test it with my particular setup. I should really get a new machine which would be powerful enough to play the m2t/d2t files, but I'm trying to make a no-budget feature-length project next year, so money is tight.

"save time/money by buying a much larger hard drive" -- buying a 1 TB drive is exactly what I intend to do. Waiting for Western Digital (my preferred maker) to put out such a drive. Buying a large drive is cheaper than buying a new computer with a small drive (though I wish I could aford both).

Thanks for your messages Ebobtron.

[Leak]

Originally Posted by TrueIndigo

However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script.

Any reason you didn't try ffmpegSource? It's able to load all kinds of video *without* resorting to VfW.

The "mpeg" part of it's name doesn't mean that it's for MPEG video only, but only means that it is based on the ffmpeg project's libavcodec that is used in players/codecs like mplayer, xine, VLC and/or ffdshow...

np: Bola - AMNiON (Soup)

[foxyshadis]

Ah, I understand now. You could probably get away with still using mpeg2source if you batch convert them to SD mpeg-2 instead of DV, or you can try ffmpegsource on the avi as leak says.