Decompressor error after too many calls to AVISource?

[cayblood]

I'm splicing a bunch of 1GB dv files together like this:

switched = \
AVISource(aviroot + "\switched_001.avi") + \
AVISource(aviroot + "\switched_002.avi") + \
AVISource(aviroot + "\switched_003.avi") + \
AVISource(aviroot + "\switched_004.avi") + \
AVISource(aviroot + "\switched_005.avi") + \
AVISource(aviroot + "\switched_006.avi") + \
etc.

to make a single stream. If I make this too long, I get an error like this:

AVISource: couldn't locate a decompressor for fourcc dvsd

Does anyone know if there is a way I can do this that can work for any number of files? I am trying to edit multiple camera angles and I have a lot of files.

Thanks,
Carl Youngblood

[Dark Shikari]

I found that AVIsource craps out over a certain number of calls... DirectShowSource, however, doesn't.

[squid_80]

The dll for the decompression codec is loaded separately for every AVISource call. Eventually an OS-imposed limit is reached, the codec can't be loaded and you get that error message.

[Fizick]

probably we must put this info to docs,
but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?

[IanB]

but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?Very good questions, that are very hard to answer. :p

It is mostly a per process limit (i.e per script). Generally most people can open 50 but cannot open 100. It usually depends on how much else is being loaded.

Last time I tested, with the Panasonic DV codec I could open 57 and I have had it fail with even less.

[TrueIndigo]

Hey,
I was hoping to off-line a 10-minute film using a single avs file, which might involve around 200 separate avi clips. Does this mean I can’t use Avisynth as I hoped? If DirectShowSource is an answer, what is the essential difference between it and AviSource? Why isn't everyone using DirectShowSource? Yes, I am a newbie!

[IanB]

@TrueIndigo,

You may have to prebatch your scripts into segments of approx 50 input clips, output intermediate lossless workfiles .AVI's then do your master edit, inputing from the 4 intermediate workfiles .AVI's.

Until you try on your PC with your codecs you won't know if you will have problems.

AviSource() uses Video for Windows (VfW) codecs, some formats do not have a VfW codec. It is guarenteed to provide frame accurate seeking. If you have an appropriate VfW codec it just works.

DirectShowSource() uses M$ DirectShow API's intended for writing Media Players. It is NOT guarenteed to provide frame accurate seeking, although many formats actually do get it right. Nearly all formats provide DirectShow codecs, so you can actually decode your stream without a VfW codec. If your format does not implement frame accurate seeking and your encoder app plus your script require it, then you may need to preprocess your stream out to a lossless intermediate format like Huffyuv or raw uncompressed .AVI's.

People do not use DirectShowSource() always because it so often sucks. There are many format specific *Source() filters available like Mpeg2Source(), etc. Generally if your format has a custom source pluggin use it.

[TrueIndigo]

IanB – thanks for your explanation about the differences between DirectShowSource and AviSource. It seems a shame that perhaps the only solution would be to “output intermediate lossless workfiles”. Is there a way of somehow joining avs files together? Or have an avs file referencing other avs files (or would that still be the same as a single avs file with more than 50 avi’s being referenced, and therefore still invoke the error?).
Thanks.

[squid_80]

If you do avisource("a script.avs")++avisource("another script.avs") etc. you're still going to run into the same problem.

IanB: Would this really be so hard to fix? Can't we make a static array somewhere in the avisource class to keep track of what codecs are already loaded and reuse them?

[Leak]

You could also try ffmpegsource (search the forum for it) that uses ffmpeg's libavcodec to read all kinds of files. I'm not totally sure, but I wouldn't rule out the possibility that it also won't run into any kind of DLL load limit here...

[IanB]

@TrueIndigo,

If you are really averse to using workfiles, assuming the files all match in format you could use VirtualDub's append segment to join all the files, then either do a directstream copy to a single file or frame serve the whole lots as one stream. :search: this has been discussed before.

First, test your limits before you start looking for complicated alternatives. You might not have a problem.

@Squid_80,

There was a developement thread discussing various codec caching schemes to get around this limit. I think the consensus was it was doable but a big job.

Restoring SegmentedAviSource to the original VDub model could be much less difficult. I always thought it a pox that the multi-segment avi file capabilities were arbitarily discarded, particularly for re-joining delta frame formats where the segments do not start with a key frame. However it only supported 100 files (00..99).

@Leak,

Good thought, ffmpegsource does stand a good chance of beating the odds.

[TrueIndigo]

IanB -- thanks for your input as usual, though unfortunately I don’t know enough to understand much of the advice!

"Use search. This has been discussed before." -- I am an absolute newbie and I’m not sure what I am searching for. I only just recognised the significance (to me) of the title of this thread. I was hoping to use Avisynth/VirtualDubMod to off-line a short film. If there is a limitation on the number of avi clips (maybe 50+) in a single script, this means a script will run into trouble after about 3-5 minutes (depending on your style of editing). That’s not very long for editing. I started reading about applications on the forum like AvsFilmCutter and assumed some people were editing with Avisynth without any problems.

"Averse to using workfiles" -- I’d like to keep the on-going work in progress in the form of an avs file. This is the thing I like about Avisynth: it is a self-logging EDL and you can access it directly, make changes, refresh, and see the results.

The work-a-rounds instead of using a single script I don’t fully understand. Maybe I’ve misunderstood, but do you mean this workflow:

1. Use a number of avs scripts, each keeping below the problematic limit of avi calls.
2. Export an avi copy of each and in VirtualDub join them together in order to view the combined work.

This allows you to view the current edit, but you can’t make instant, interactive changes as you could by playing an avs file. You no longer have access to the original in/out times of individual clips which are in the avs scripts. So I would have to make manual notes about the changes, implement them in the various avs scripts, export another round of avi workfiles, view the group again, possibly decide on more changes, make more notes, transfer the changes to the various avs scripts, etc, etc.

"First, test your limits...You might not have a problem." -- do you mean that some people have a problem after 50+ AviSource calls in a single script, and some have no problem with maybe 300+ calls? I was a bit disappointed because I thought that maybe 50+ will indeed turn out to be about the limit for everyone (because it’s an OS-related problem). I’m not clear about the nature of this problem.

IanB, thanks for your thoughts. My lack of understanding frustrates me as much as it must mystify you!
Thanks,
TrueIndigo.

[IanB]

I was a bit disappointed because I thought that maybe 50+ will indeed ...Why would you be disappointed if the limit does not effect you :confused:

Waiter, waiter, there is a fly in my soup. Sir! Shhh, everyone will want one

The limit comes from how many .DLL's a single process can open. Avisynth is not in control of the process it is just one .DLL of many. The encoder application is the controlling process and we do not know what your encoder application is and what needs it has, it could be anything, VirtualDub, VirtualDubMod, MeGUI, TMpegEnc, ... We also do not know what codec your video uses and how many .DLL's it uses or how it shares those .DLL's.

There is a limit and people have run into it with as few as 50 AviSource statements. You need to check your video application with your video codec with your video clips running on your PC.

The best workaround I can offer is do not segment your input video clips in the first place or append the raw segments together into fewer and bigger meaningful lumps.AviSource("Lumps1to10.avi")
Trim( 42, 142) + Trim(242, 342) + Trim(442, 542) + \
Trim(642, 742) + Trim(842, 942) + ...Is a perfectly usable model.

And yes learn to use search :search:

Search for the words and terms I and others have already used in this thread. They will keep occuring in the threads that also discuss similar issues.

And when the internal search is to restricted use Google and add the term "site:doom9.org".

[TrueIndigo]

IanB, thanks for your reply, I appreciate it.

"The limit comes from how many .DLL's a single process can open..." -- yes, I said I wasn't clear about the nature of this problem, and your explanation has made it much clearer. When I get more of my material together for a test I'll find out more about my particular setup. Many thanks.

"Do not segment your input video clips in the first place" -- I like the HDVSplit method of splitting up clips according to shot, because it saves space on the HDD. For example, in a worst case scenario, you might need to keep a single half-hour long clip on the HDD, from which you might only want to use a few short shots. As far as efficient space usage goes, it's much better to have a few clips each a few minutes long, rather than a single half-hour clip. This space issue becomes more of a problem if you have a lot of material to choose from.

"Append the raw segments together into fewer and bigger...lumps" -- Yes, this solves the possible problem for Avisynth. After a process of elimination I could join the required short clips together into just a few large files, and these would be saved out as a high quality uncompressed video in order not to lose quality (because using the online/offline method, they have to become the new master footage -- you need the absolute relationship between the low res proxy and high res master which it represents). I do have a few reservations about this though, compared to staying with the original separate m2t files:

(1) Uncompressed files (even efficient lossless compression formats) take up much more HDD space than the equivalent m2t files they replace -- considering they are HD masters, m2t files only take up the same space as SD DV.

(2) It's an extra process which takes more time -- accumulates into more of an issue when a lot of material is involved. For example, my last short film had an edited duration of 28 minutes which, because of the informal nature of the shooting, was derived from over 40 hours of material.

(3) When I append raw segments together as a large file, I can't know the final order they will be in because I haven't started editing yet. They could be joined in an arbitrary way or just as it comes from the camera (clips a, b, c...m, n, o, etc). I might insert a few seconds of black blank space between each clip so there is no confusion over where one clip ends and another starts. Then the clips could be trimmed and spliced using the example coding you gave. But editing is normally non-linear, so my final cut will include the clips in a mixed up order (eg. clip c, k, j...f, a, l, etc). Then I might want to transpose sound from one part to another. From a newbie's point of view the scripting for this seems like a nightmare (even if I knew how to do it), compared to being able to do the same thing with single clips.

cayblood -- I hope you don't think I've hijacked your thread for a discussion about editing. Clearly, I am interested in the viability of Avisynth as a NLE, but my queries are in the context of possible problems with large avi calls in a single script, and this is the nature of your thread. IanB and the others have made some great points here and I hope the discussion might be of interest to others. I'm trying to learn from you guys, my intention is not to try your patience -- life is hard enough as it is!

Many thanks.

[Ebobtron]

If we are talking m2t (HD MPEG2), can you not combine all your source files using DGINDEX, they would remain as individual files imported to AviSynth with one *.d2v file. But I am not sure about this at all.

[foxyshadis]

You don't even need to do that - you can batch index all of your files into individual d2vs, then load them all with multiple calls to mpeg2source without running into the dll limit, that way you don't have to waste time splitting them back up in the script.

Indigo, if you don't find a satisfactory solution, you might also save time/money by buying a much larger hard drive, 500G to 1T are so cheap now. It's a brute force method, but it beats wasting your time and your customers'.

[IanB]

From Post #7Generally if your format has a custom source pluggin use it.

[TrueIndigo]

Foxyshadis -- sensible advice, thanks.

"multiple calls to mpeg2source without running into the dll limit" -- I originally tried playing the m2t/d2t files, which would be the most elegant workflow of all (and using ReduceBy2 in case that helped performance), but the playback was too choppy on my machine for viable editing. I have a P4 3.0 GHz, 2 Gb RAM, 2x 250 GB HDDs. This machine is probably too slow for realtime HDV editing, but I don't mind the offline/online process (using SD sized avi proxies). However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script. As IanB says, I need to test it with my particular setup. I should really get a new machine which would be powerful enough to play the m2t/d2t files, but I'm trying to make a no-budget feature-length project next year, so money is tight.

"save time/money by buying a much larger hard drive" -- buying a 1 TB drive is exactly what I intend to do. Waiting for Western Digital (my preferred maker) to put out such a drive. Buying a large drive is cheaper than buying a new computer with a small drive (though I wish I could aford both).

Thanks for your messages Ebobtron.

[Leak]

However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script.
Any reason you didn't try ffmpegSource? It's able to load all kinds of video *without* resorting to VfW.

The "mpeg" part of it's name doesn't mean that it's for MPEG video only, but only means that it is based on the ffmpeg project's libavcodec that is used in players/codecs like mplayer, xine, VLC and/or ffdshow...

np: Bola - AMNiON (Soup)

[foxyshadis]

Ah, I understand now. You could probably get away with still using mpeg2source if you batch convert them to SD mpeg-2 instead of DV, or you can try ffmpegsource on the avi as leak says.

[TrueIndigo]

Foxyshadis and Leak -- some great ideas there for me to try with both SD avi’s and SD mpeg-2’s. Been busy but will try next week and see how I get on. Must say, this is great forum support, thanks.

[TrueIndigo]

Just to follow up on the avi option: I mocked up some editing to find out what my particular AviSource limit would be. For off-line editing I am using the Microsoft mpeg-4 V2 avi codec, and the maximum number of calls in one script turned out to be 75. For typical editing this gives me about 8 minutes duration. A short film can be edited with two avs scripts, and this is not much of an inconvenience.
Thanks.

[squid_80]

The limit comes from how many .DLL's a single process can open. Avisynth is not in control of the process it is just one .DLL of many. The encoder application is the controlling process and we do not know what your encoder application is and what needs it has, it could be anything, VirtualDub, VirtualDubMod, MeGUI, TMpegEnc, ... We also do not know what codec your video uses and how many .DLL's it uses or how it shares those .DLL's.
I've been having a play around and I'm not sure this is where the problem lies... I loaded a codec (xvid) using loadlibrary, got the address of DriverProc via GetProcAddress and repeatedly used ICOpenFunction to get a HIC. Surprisingly it still died at the same number of calls. I think there's some other limit inside ICOpen/ICLocate (possibly heap related?). So the only way I can see around it is to write replacement functions for all used IC* calls... or alternatively "borrow" them from wine :devil:

Re restoring SegmentedAVISource: Doesn't look hard at all, just add a boolean "append" parameter to avisource, if it's true search for incrementing filenames and call IAVIReadHandler::AppendFile for each one.

[squid_80]

I rewrote the IC* functions so avisynth can load/talk to the video codec directly, and it works - sort of. It gets past the original loading limit but hits another one...
http://216.75.63.164/andrew/codec_mem.jpg

That's with 113 1280x720 xvid files loaded. One copy of the codec dll in memory and a DRV_OPEN call made for each avisource instance. Then I tried adding some more and hit F2 in vdub to reload and not surprisingly copped an unknown exception. Probably because my pagefile's set at 2gb.

Trying to run a script like this would be next to useless so I think I'm going to forget it and fix the appending instead.

[cweb]

I had a problem today with multiple avisource's (the length in minutes of the avs would be an incredibly large and incorrect size), so after reading this thread I used ffmpegsource for a few of the clips, which solved the problem.

[a1s2d3f4]

Except that there is a limit with FFMpegSource also - at least for me

[TomArrow]

With 7 years having passed and AVISynth+ and 64 bit being a thing, is there some way to address this issue? I just ran into it and I feel like there should be some solution. Can AVISource be rewritten? Does passing multiple filenames to single AVISource calls help reduce the amount of dll calls?

[StainlessS]

Maybe not the answer that you are looking for, but,
TWriteAVI plugin [cannot use MT and x86 only], has a demo provided to splice 1000 20 frame clips, if you are really pushed maybe you wanna give it a bash (requires numbered clips).
The demo has script to make 1000 numbered clips, and additional couple of script to splice and show the result.

Demo here (is included in TWriteAVI zip):- https://forum.doom9.org/showthread.php?p=1743422#post1743422
TWriteAVI here:- https://forum.doom9.org/showthread.php?t=172837&highlight=twriteAVI

[poisondeathray]

Was it a memory issue, or a "windows number of calls" issue ?

Another workaround might be to mix several source filters eg. AVISource for 74, FFMS2 for 74, Lsmash for 74 (if "75" was the magic number)

How do other Windows programs (e.g. NLE's) get around this ? I know they use custom internal decoders for some formats (e.g. DV-AVI) , but some 3rd party ones are still routed through VFW if that codec does not come with the NLE

[TomArrow]

Thanks StainlessS! I ended up using ffvideosource and it worked in my case, but it might crap out with more files too, as it only ended up working in 64 bit. Might give this a go next time.

[StainlessS]

How do other Windows programs (e.g. NLE's) get around this ?
I imagine that they open(and close) files as required (same as my demo) rather than all at once, Avs needs to know
output number of frames before frameserving starts.

[Emulgator]

NLEs: Just having a VHS restoration project using 8 intermediate MAGY fíles here in Vegas Pro 13.
Eight instances of MagicVUV are loaded sequentially as the project is loaded.
These eight stay loaded permanently from opening until closing, so one per source stream, no matter how many instances/clips/edit per stream I cut/edit.

[johnmeyer]

I wasn't part of the original thread, but looking at the OP's original problem, he was trying to "splice together" a bunch of AVIs using AVISynth syntax. My solution to this -- as well as my solution to pretty much every other problem of getting stuff into AVISynth -- is to load the files onto the Vegas timeline and then use Debubmode's frameserver to serve these into my AVISynth script. This solve dozens of problems, including the one posed by the OP. You just have to remember to set the Vegas project properties to match exactly the source video properties so that it doesn't conform the video to some other standard. Since Vegas provides a one-click method to make sure that happens (the new versions do it automatically) this is not a big issue.

[StainlessS]

NLEs: Just having a VHS restoration project using 8 intermediate MAGY fíles here in Vegas Pro 13.
Eight instances of MagicVUV are loaded sequentially as the project is loaded.
These eight stay loaded permanently from opening until closing, so one per source stream, no matter how many instances/clips/edit per stream I cut/edit.
Perplexing.