Processing many AVIs - a bug ?

[Shubin]

I've made a script that processes a LOT of DV fragments. About 50 or more. After adding another couple of avisource(...) the script can not be opened in DubMod. It reports a strange error : Could not find codec for YV12 format. Only Direct stream copy will be available. Then it shows garbage.
The same script, encoded in QuEnc shows another message : Memory at xxxxx can not be read.
Using another output format (skipping conversion from YUY2 to YV12) or splitting the script into smaller parts cures the problem.
Increasing memory for Avisynth using SetMemoryMax does nothing.
Avisynt is v. 2.54

[Fizick]

Why 2.5.4?
What about 2.5.5 or 2.5.6beta?

[Wilbert]

Could not find codec for YV12 format. Only Direct stream copy will be available.
... means you don't have a YV12 codec installed (like XviD/DivX). But, it's a bit strange because DV shouldn't output YV12. What kind of DV codec are you using?

[stickboy]

He said if he keeps the number down, his scripts open fine. He's only getting that error because he's opening many, many clips. I think it's a known issue that we discussed before.

Try it yourself and see:
AVISource("someClip.avi")
\ + AVISource("someClip.avi")
# Copy and paste the above line many times... 76 seems to be the magic
# number on my system.

[kingmob]

What happens if you use: setmemorymax(128) or something like that?

[sh0dan]

You've hit the maximum number of dll's a process can load - it's about 72, and there isn't much to do about it unfortunately. You could try loading additional videos by using DirectShowSource, but I'm not sure how much it'll help.

[Shubin]

Setmemorymax does nothing.
New version of Avisynth does not help.
2 sh0dan : why there's a need of loading a separate dll for every
avi file ? And if this is unavoidable problem, then what could be
done ? Would Import() launch another process that will be able to load its own 72 files then ?

[IanB]

Shubin,

As sh0dan say's there are limits. The other limit around the 50 files mark is the VFW codec table space. I have been caught by this before and I have a note to myself to add an option to SegmentedAviSource() and SegmentedDirectShowSource() to "stack" the input files into a single codec like VirtualDub does. The bad news it probably won't happen until 2.5.7.

IanB

[vinetu]

You've hit the maximum number of dll's a process can load - it's about 72, and there isn't much to do about it unfortunately. You could try loading additional videos by using DirectShowSource, but I'm not sure how much it'll help.
will that limit remain in Avisynth 3.0?
Also when the host Application is loading many dlls itself (Let's say Adobe Premiere) the "magic number" decreases to 34-37 IIRC.

[Leak]

will that limit remain in Avisynth 3.0?
Also when the host Application is loading many dlls itself (Let's say Adobe Premiere) the "magic number" decreases to 34-37 IIRC.

Shouldn't it be possible for AviSource to open the files on demand only, closing the least recently used file if it can't use any new files, or after a given number of files already has been opened?

np: Luomo - Market (Vocalcity)

[IanB]

Shouldn't it be possible for AviSource to open the files on demand only, closing the least recently used file if it can't use any new files, or after a given number of files already has been opened?Keen idea.

As file handles are not the limit, it is the number of loaded Dll's and VFW codec table space. So caching codec VFW object instances should be all that is needed. A single VFW object could be used to sequentially decompress all video streams with the same fourCC, output dimensions and colorspace.

Unfortunatly as the VFW objects have state (current frame for next delta frame) this would make parallel processing of 2 such files performance challanged. So some intelligence would be required.

Can we move this thread into the Developer forum.

IanB

[Leak]

Unfortunatly as the VFW objects have state (current frame for next delta frame) this would make parallel processing of 2 such files performance challanged. So some intelligence would be required.

That's why I suggested only closing files after a certain (user specifiable?) number of files have been opened...

[Bidoche]

will that limit remain in Avisynth 3.0?3.0 uses the same logic for loading than 2.5, so it has the same flaws.

Unfortunatly as the VFW objects have state (current frame for next delta frame) this would make parallel processing of 2 such files performance challanged. So some intelligence would be required.I think we can use the same codec handle if we correctly uses DecompressBegin and DecompressEnd to switch between source files.

The hard part would be to not do it systematically, but only when too much sources are openened.

[Shubin]

I see a possible solution : it is possible to detect that the given fragment (clip) will be used only in one place of a resulting clip. Thus it is possible to close it, freeing the slot.

Processing many files is so far the best way to make video out of home-made DV source. Programs for downloading video can split it into scenes. This makes processing much easier.

[IanB]

The hard part would be to not do it systematically, but only when too much sources are openened.Waiting until resources are exhausted before interviening can sometime lead to deadlock. Fortunatly with all key frame formats like Huffyuv, MJPEG and DV there is very little overhead in using DecompressBegin and DecompressEnd even on a per frame basis. As for delta frame intensive formats like DivX and Xvid I see much merit in Leaks idea of a user specified size of cached codec handles. The mode of operation could be keyed off codec info ALLKEYFRAMES...

IanB

[Bidoche]

Fortunatly with all key frame formats like Huffyuv, MJPEG and DV there is very little overhead in using DecompressBegin and DecompressEnd even on a per frame basis.So those could always use the same codec handle cached somewhere


As for delta frame intensive formats like DivX and Xvid I see much merit in Leaks idea of a user specified size of cached codec handles.ie releasing handles when there are too much.
Isn't that worse than trying to share them ?
On the other hand it avoids the issue of them being the same codec


Here is some (pseudo) code about those ideas
//somewhere
//circular buffer of codec handle (HIC is my 3.0 wrapper around codec handle)
boost::circular_buffer<boost::shared_ptr<HIC> > codecList;

class Decompressor
{
boost::weak_ptr<HIC> codec_; //non owning ref to a codec handle

void Decompress(/*data passed here*/)
{
boost::shared_ptr<HIC> codec = codec_.lock(); //try obtening an owning ref

if ( ! codec ) //it got invalided
{
//create new one, update circular buffer
}

//do decompression
}
};

[Leak]

ie releasing handles when there are too much.
Isn't that worse than trying to share them ?
On the other hand it avoids the issue of them being the same codec

I guess it would only be a problem if you're using more clips at the same time than the limit allows (i.e. blending the frames from >limit clips together or something like that), but if you just splice them together you only need a few clips at a time; if you need many clips at different points in time it'll hardly make a dent performance wise.

np: The Notwist - Moron (Shrink)

[Buckaroo]

To remedy and even experience an initial loading efficiency boost; Pop these dependencies extracted from a recent ReactOS (http://www.reactos.org/) distribution into the directory or direct path of your application(s) that will (also) be hosting avisynth's dynamic link library.

msvfw32.dll
winmm.dll