Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 11th November 2007, 06:17   #1  |  Link
cayblood
Registered User
 
Join Date: Apr 2007
Posts: 36
Decompressor error after too many calls to AVISource?

I'm splicing a bunch of 1GB dv files together like this:

switched = \
AVISource(aviroot + "\switched_001.avi") + \
AVISource(aviroot + "\switched_002.avi") + \
AVISource(aviroot + "\switched_003.avi") + \
AVISource(aviroot + "\switched_004.avi") + \
AVISource(aviroot + "\switched_005.avi") + \
AVISource(aviroot + "\switched_006.avi") + \
etc.

to make a single stream. If I make this too long, I get an error like this:

AVISource: couldn't locate a decompressor for fourcc dvsd

Does anyone know if there is a way I can do this that can work for any number of files? I am trying to edit multiple camera angles and I have a lot of files.

Thanks,
Carl Youngblood
cayblood is offline   Reply With Quote
Old 11th November 2007, 07:05   #2  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,688
I found that AVIsource craps out over a certain number of calls... DirectShowSource, however, doesn't.
Dark Shikari is offline   Reply With Quote
Old 11th November 2007, 10:08   #3  |  Link
squid_80
Registered User
 
Join Date: Dec 2004
Location: Melbourne, AU
Posts: 1,963
The dll for the decompression codec is loaded separately for every AVISource call. Eventually an OS-imposed limit is reached, the codec can't be loaded and you get that error message.
squid_80 is offline   Reply With Quote
Old 11th November 2007, 20:20   #4  |  Link
Fizick
AviSynth plugger
 
Fizick's Avatar
 
Join Date: Nov 2003
Location: Russia
Posts: 2,183
probably we must put this info to docs,
but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?
__________________
My Avisynth plugins are now at http://avisynth.org.ru and mirror at http://avisynth.nl/users/fizick
I usually do not provide a technical support in private messages.
Fizick is offline   Reply With Quote
Old 11th November 2007, 21:59   #5  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
Quote:
but what is limit? 10 ? 20 ? 50 ?
Is this limit per script (env?) or per Windows?
Very good questions, that are very hard to answer.

It is mostly a per process limit (i.e per script). Generally most people can open 50 but cannot open 100. It usually depends on how much else is being loaded.

Last time I tested, with the Panasonic DV codec I could open 57 and I have had it fail with even less.
IanB is offline   Reply With Quote
Old 20th November 2007, 17:36   #6  |  Link
TrueIndigo
Registered User
 
Join Date: Nov 2007
Location: UK
Posts: 16
Hey,
I was hoping to off-line a 10-minute film using a single avs file, which might involve around 200 separate avi clips. Does this mean I canít use Avisynth as I hoped? If DirectShowSource is an answer, what is the essential difference between it and AviSource? Why isn't everyone using DirectShowSource? Yes, I am a newbie!
TrueIndigo is offline   Reply With Quote
Old 20th November 2007, 21:47   #7  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
@TrueIndigo,

You may have to prebatch your scripts into segments of approx 50 input clips, output intermediate lossless workfiles .AVI's then do your master edit, inputing from the 4 intermediate workfiles .AVI's.

Until you try on your PC with your codecs you won't know if you will have problems.

AviSource() uses Video for Windows (VfW) codecs, some formats do not have a VfW codec. It is guarenteed to provide frame accurate seeking. If you have an appropriate VfW codec it just works.

DirectShowSource() uses M$ DirectShow API's intended for writing Media Players. It is NOT guarenteed to provide frame accurate seeking, although many formats actually do get it right. Nearly all formats provide DirectShow codecs, so you can actually decode your stream without a VfW codec. If your format does not implement frame accurate seeking and your encoder app plus your script require it, then you may need to preprocess your stream out to a lossless intermediate format like Huffyuv or raw uncompressed .AVI's.

People do not use DirectShowSource() always because it so often sucks. There are many format specific *Source() filters available like Mpeg2Source(), etc. Generally if your format has a custom source pluggin use it.
IanB is offline   Reply With Quote
Old 21st November 2007, 11:41   #8  |  Link
TrueIndigo
Registered User
 
Join Date: Nov 2007
Location: UK
Posts: 16
IanB Ė thanks for your explanation about the differences between DirectShowSource and AviSource. It seems a shame that perhaps the only solution would be to ďoutput intermediate lossless workfilesĒ. Is there a way of somehow joining avs files together? Or have an avs file referencing other avs files (or would that still be the same as a single avs file with more than 50 aviís being referenced, and therefore still invoke the error?).
Thanks.
TrueIndigo is offline   Reply With Quote
Old 21st November 2007, 12:14   #9  |  Link
squid_80
Registered User
 
Join Date: Dec 2004
Location: Melbourne, AU
Posts: 1,963
If you do avisource("a script.avs")++avisource("another script.avs") etc. you're still going to run into the same problem.

IanB: Would this really be so hard to fix? Can't we make a static array somewhere in the avisource class to keep track of what codecs are already loaded and reuse them?
squid_80 is offline   Reply With Quote
Old 21st November 2007, 12:33   #10  |  Link
Leak
ffdshow/AviSynth wrangler
 
Leak's Avatar
 
Join Date: Feb 2003
Location: Austria
Posts: 2,441
You could also try ffmpegsource (search the forum for it) that uses ffmpeg's libavcodec to read all kinds of files. I'm not totally sure, but I wouldn't rule out the possibility that it also won't run into any kind of DLL load limit here...
__________________
now playing: [artist] - [track] ([album])
Leak is offline   Reply With Quote
Old 21st November 2007, 13:27   #11  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
@TrueIndigo,

If you are really averse to using workfiles, assuming the files all match in format you could use VirtualDub's append segment to join all the files, then either do a directstream copy to a single file or frame serve the whole lots as one stream. this has been discussed before.

First, test your limits before you start looking for complicated alternatives. You might not have a problem.

@Squid_80,

There was a developement thread discussing various codec caching schemes to get around this limit. I think the consensus was it was doable but a big job.

Restoring SegmentedAviSource to the original VDub model could be much less difficult. I always thought it a pox that the multi-segment avi file capabilities were arbitarily discarded, particularly for re-joining delta frame formats where the segments do not start with a key frame. However it only supported 100 files (00..99).

@Leak,

Good thought, ffmpegsource does stand a good chance of beating the odds.
IanB is offline   Reply With Quote
Old 22nd November 2007, 20:31   #12  |  Link
TrueIndigo
Registered User
 
Join Date: Nov 2007
Location: UK
Posts: 16
IanB -- thanks for your input as usual, though unfortunately I donít know enough to understand much of the advice!

"Use search. This has been discussed before." -- I am an absolute newbie and Iím not sure what I am searching for. I only just recognised the significance (to me) of the title of this thread. I was hoping to use Avisynth/VirtualDubMod to off-line a short film. If there is a limitation on the number of avi clips (maybe 50+) in a single script, this means a script will run into trouble after about 3-5 minutes (depending on your style of editing). Thatís not very long for editing. I started reading about applications on the forum like AvsFilmCutter and assumed some people were editing with Avisynth without any problems.

"Averse to using workfiles" -- Iíd like to keep the on-going work in progress in the form of an avs file. This is the thing I like about Avisynth: it is a self-logging EDL and you can access it directly, make changes, refresh, and see the results.

The work-a-rounds instead of using a single script I donít fully understand. Maybe Iíve misunderstood, but do you mean this workflow:

1. Use a number of avs scripts, each keeping below the problematic limit of avi calls.
2. Export an avi copy of each and in VirtualDub join them together in order to view the combined work.

This allows you to view the current edit, but you canít make instant, interactive changes as you could by playing an avs file. You no longer have access to the original in/out times of individual clips which are in the avs scripts. So I would have to make manual notes about the changes, implement them in the various avs scripts, export another round of avi workfiles, view the group again, possibly decide on more changes, make more notes, transfer the changes to the various avs scripts, etc, etc.

"First, test your limits...You might not have a problem." -- do you mean that some people have a problem after 50+ AviSource calls in a single script, and some have no problem with maybe 300+ calls? I was a bit disappointed because I thought that maybe 50+ will indeed turn out to be about the limit for everyone (because itís an OS-related problem). Iím not clear about the nature of this problem.

IanB, thanks for your thoughts. My lack of understanding frustrates me as much as it must mystify you!
Thanks,
TrueIndigo.
TrueIndigo is offline   Reply With Quote
Old 22nd November 2007, 23:17   #13  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
Quote:
I was a bit disappointed because I thought that maybe 50+ will indeed ...
Why would you be disappointed if the limit does not effect you

Waiter, waiter, there is a fly in my soup. Sir! Shhh, everyone will want one

The limit comes from how many .DLL's a single process can open. Avisynth is not in control of the process it is just one .DLL of many. The encoder application is the controlling process and we do not know what your encoder application is and what needs it has, it could be anything, VirtualDub, VirtualDubMod, MeGUI, TMpegEnc, ... We also do not know what codec your video uses and how many .DLL's it uses or how it shares those .DLL's.

There is a limit and people have run into it with as few as 50 AviSource statements. You need to check your video application with your video codec with your video clips running on your PC.

The best workaround I can offer is do not segment your input video clips in the first place or append the raw segments together into fewer and bigger meaningful lumps.
Code:
AviSource("Lumps1to10.avi")
Trim( 42, 142) + Trim(242, 342) + Trim(442, 542) + \
Trim(642, 742) + Trim(842, 942) + ...
Is a perfectly usable model.

And yes learn to use search

Search for the words and terms I and others have already used in this thread. They will keep occuring in the threads that also discuss similar issues.

And when the internal search is to restricted use Google and add the term "site:doom9.org".

Last edited by IanB; 22nd November 2007 at 23:22. Reason: Add joke!
IanB is offline   Reply With Quote
Old 24th November 2007, 11:46   #14  |  Link
TrueIndigo
Registered User
 
Join Date: Nov 2007
Location: UK
Posts: 16
IanB, thanks for your reply, I appreciate it.

"The limit comes from how many .DLL's a single process can open..." -- yes, I said I wasn't clear about the nature of this problem, and your explanation has made it much clearer. When I get more of my material together for a test I'll find out more about my particular setup. Many thanks.

"Do not segment your input video clips in the first place" -- I like the HDVSplit method of splitting up clips according to shot, because it saves space on the HDD. For example, in a worst case scenario, you might need to keep a single half-hour long clip on the HDD, from which you might only want to use a few short shots. As far as efficient space usage goes, it's much better to have a few clips each a few minutes long, rather than a single half-hour clip. This space issue becomes more of a problem if you have a lot of material to choose from.

"Append the raw segments together into fewer and bigger...lumps" -- Yes, this solves the possible problem for Avisynth. After a process of elimination I could join the required short clips together into just a few large files, and these would be saved out as a high quality uncompressed video in order not to lose quality (because using the online/offline method, they have to become the new master footage -- you need the absolute relationship between the low res proxy and high res master which it represents). I do have a few reservations about this though, compared to staying with the original separate m2t files:

(1) Uncompressed files (even efficient lossless compression formats) take up much more HDD space than the equivalent m2t files they replace -- considering they are HD masters, m2t files only take up the same space as SD DV.

(2) It's an extra process which takes more time -- accumulates into more of an issue when a lot of material is involved. For example, my last short film had an edited duration of 28 minutes which, because of the informal nature of the shooting, was derived from over 40 hours of material.

(3) When I append raw segments together as a large file, I can't know the final order they will be in because I haven't started editing yet. They could be joined in an arbitrary way or just as it comes from the camera (clips a, b, c...m, n, o, etc). I might insert a few seconds of black blank space between each clip so there is no confusion over where one clip ends and another starts. Then the clips could be trimmed and spliced using the example coding you gave. But editing is normally non-linear, so my final cut will include the clips in a mixed up order (eg. clip c, k, j...f, a, l, etc). Then I might want to transpose sound from one part to another. From a newbie's point of view the scripting for this seems like a nightmare (even if I knew how to do it), compared to being able to do the same thing with single clips.

cayblood -- I hope you don't think I've hijacked your thread for a discussion about editing. Clearly, I am interested in the viability of Avisynth as a NLE, but my queries are in the context of possible problems with large avi calls in a single script, and this is the nature of your thread. IanB and the others have made some great points here and I hope the discussion might be of interest to others. I'm trying to learn from you guys, my intention is not to try your patience -- life is hard enough as it is!

Many thanks.
TrueIndigo is offline   Reply With Quote
Old 24th November 2007, 13:01   #15  |  Link
Ebobtron
Errant Knight
 
Ebobtron's Avatar
 
Join Date: Oct 2004
Location: St Louis, M0 US
Posts: 364
If we are talking m2t (HD MPEG2), can you not combine all your source files using DGINDEX, they would remain as individual files imported to AviSynth with one *.d2v file. But I am not sure about this at all.
Ebobtron is offline   Reply With Quote
Old 24th November 2007, 13:33   #16  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,532
You don't even need to do that - you can batch index all of your files into individual d2vs, then load them all with multiple calls to mpeg2source without running into the dll limit, that way you don't have to waste time splitting them back up in the script.

Indigo, if you don't find a satisfactory solution, you might also save time/money by buying a much larger hard drive, 500G to 1T are so cheap now. It's a brute force method, but it beats wasting your time and your customers'.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Old 24th November 2007, 14:19   #17  |  Link
IanB
Avisynth Developer
 
Join Date: Jan 2003
Location: Melbourne, Australia
Posts: 3,173
From Post #7
Quote:
Originally Posted by ianb View Post
Generally if your format has a custom source pluggin use it.
IanB is offline   Reply With Quote
Old 24th November 2007, 17:18   #18  |  Link
TrueIndigo
Registered User
 
Join Date: Nov 2007
Location: UK
Posts: 16
Foxyshadis -- sensible advice, thanks.

"multiple calls to mpeg2source without running into the dll limit" -- I originally tried playing the m2t/d2t files, which would be the most elegant workflow of all (and using ReduceBy2 in case that helped performance), but the playback was too choppy on my machine for viable editing. I have a P4 3.0 GHz, 2 Gb RAM, 2x 250 GB HDDs. This machine is probably too slow for realtime HDV editing, but I don't mind the offline/online process (using SD sized avi proxies). However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script. As IanB says, I need to test it with my particular setup. I should really get a new machine which would be powerful enough to play the m2t/d2t files, but I'm trying to make a no-budget feature-length project next year, so money is tight.

"save time/money by buying a much larger hard drive" -- buying a 1 TB drive is exactly what I intend to do. Waiting for Western Digital (my preferred maker) to put out such a drive. Buying a large drive is cheaper than buying a new computer with a small drive (though I wish I could aford both).

Thanks for your messages Ebobtron.
TrueIndigo is offline   Reply With Quote
Old 24th November 2007, 22:38   #19  |  Link
Leak
ffdshow/AviSynth wrangler
 
Leak's Avatar
 
Join Date: Feb 2003
Location: Austria
Posts: 2,441
Quote:
Originally Posted by TrueIndigo View Post
However, it was the choice of the avi format (which I thought would be a safe choice) that may be problematic if you want a large number of individual calls in a single avs script.
Any reason you didn't try ffmpegSource? It's able to load all kinds of video *without* resorting to VfW.

The "mpeg" part of it's name doesn't mean that it's for MPEG video only, but only means that it is based on the ffmpeg project's libavcodec that is used in players/codecs like mplayer, xine, VLC and/or ffdshow...

np: Bola - AMNiON (Soup)
__________________
now playing: [artist] - [track] ([album])
Leak is offline   Reply With Quote
Old 25th November 2007, 03:35   #20  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,532
Ah, I understand now. You could probably get away with still using mpeg2source if you batch convert them to SD mpeg-2 instead of DV, or you can try ffmpegsource on the avi as leak says.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order.
foxyshadis is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:05.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.