AviSynth+ 64b: General strategies to reduce memory consumption?

[LigH]

I wrote a script that loads a lot of half to nearly hour long FullHD videos in MP4 (AVC+AAC) via LSMASH{Video|Audio}Source and concatenates trimmed clips, overlaid with SubtitleEx and with filtered audio, with crossfades using Dissolve.

After adding about 80 clips from almost 30 files in an automating user function, AviSynth+ 64b crashes with a "Evaluate: Unhandled C++ exception!" after consuming ~15 GB RAM of my physically available 16 GB.

Switching to LwLibav{Video|Audio}Source does not help; creating indexes crashed once prematurely but finished in a second attempt.

Can you imagine any strategies to reduce the memory consumption? (No, DirectShowSource won't be considered.)

[DTL]

After adding about 80 clips from almost 30 files in an automating user function, AviSynth+ 64b crashes with a "Evaluate: Unhandled C++ exception!"


It may be simply some used plugin (or even AVS+ core) design issue and better to narrow it and post an issue to fix. Like some memory allocation non-handled exception (expected never-fail by designer).

As a simple (but not very nice for MT performance) solution you may start from simply reduction of AVS+ cache frames to 1 (or even 0 ?) with Prefetch(N,1). Though it will about disable any benefit from frame-based MT in AVS+ but it may help from crash.

Other possible workarounds is to use temporal files and split frame to areas (or YUV/RGB planes if possible) and process with 1-plane only format or parts of frame and combine output colour or full frame sized clip from temporal files after main processing into temporal files.

In theory with x64 address mode application should never run out of virtual address space but practical OS will die forever with RAM pages swapping to HDD/SSD. SSD case will work somehow faster. So it is better to found and fix that exception bug.

[coolgit]

I once did about 100 clips and had no issue. With limited info, it is hard to spot the problem. Are all the plugins 64 bits and not 32?
I never use audio in avisynth. I put the audio back in at the final compression. Try that for starters.

[wonkey_monkey]

Can you imagine any strategies to reduce the memory consumption? (No, DirectShowSource won't be considered.)

It would seem worth considering temporarily if for no reason other than to rule out an issue with LSMASH...

[LigH]

I put the audio back in at the final compression. Try that for starters.

Not possible as the audio is filtered (e.g. crossfaded) together with the video.

... to rule out an issue with LSMASH...

Might try FFMS2 instead.

For now I try to split the video into several shorter "episodes" of ~2 hours each; all together it would have produced an 8-10 hour long video.

[ChaosKing]

Like wonkey_monkey said I would another Source Plugin like BestSource https://forum.doom9.org/showthread.php?t=185408&highlight=bestsource
Works with vs and avs.

BS should have better frame accuracy anyway.

[wonkey_monkey]

BS should have better frame accuracy anyway.

Not just better, but perfect. The downside to this is that indexing is slower, since it decodes the whole video in full and generates a hash for every frame.

Bear in mind, though, that depending on the version, it may allocate up to 1Gb in cache (changed to 100Mb in the latest versions), and (undocumented) it may use up to 4 such caches per file. I always set cachesize = 0.

(It rather confusingly also uses "cache" to refer to the index file, in its "cachemode" setting, and by default it puts index files in user folders instead of the same folder as the file. So I always set cachemode = 4 so it behaves most like FFMS2)

[StainlessS]

Sounds like this old problem.
avisynth crashes when using to many input streams :- https://forum.doom9.org/showthread.php?t=170888

A system resources {windows} problem. (about 30 files open simultaneously)

An app like Premiere can open and close files as an when required, avisynth requires all files open at once.
(think thats how it must work for premiere)

You will likely have the same problem no matter which Source filter you use.

[LigH]

My main concern was: What does it need to keep in RAM all the time?

How much RAM does AviSynth need to keep 1 frame in FullHD resolution in YV12? It has nearly 2 MPx, average 12 bit per pixel, so the YUV 4:2:0 data in 8 bit per component depth needs about 3 MB/frame; let's be generous and say 4 MB with any possible meta data plus audio?

My last script processing about 242000 frames in total uses 8 GB RAM. That would mean AviSynth needs to keep at least 2000 frames in RAM all the time, despite sliding a processing window linearly over mostly independent clips only overlapping in crossfades of a second each. Could it be that it caches all the Dissolve spans all the time? That sounds a lot like a naive fallacy to me ... but might match the observation that videos with more crossfaded clips consume more RAM.

PS: The RAM consumption already goes up to the maximum before the first frame is delivered. It does not grow gradually while loading one source video after another when they are due.

[wonkey_monkey]

PS: The RAM consumption already goes up to the maximum before the first frame is delivered.

That definitely sounds like a source filter problem. Replace them all with another source filter or BlankClips with the same pixel format and resolution.

[LigH]

L-SMASH-Works r1266
LSMASH{A/V}Source: ~11,700 MiB RAM
LwLibav{A/V}Source: ~11,750 MiB RAM - adding ~400 MB per indexed source file, more or less, and it seems to correlate with ~10× the filesize of the index file

FFMS2 5.0: ~16,300 MiB RAM used, then suddenly: Could not allocate video frame. Out of memory. memory_max = 4294967296, memory_used = 114661892 Request=6220800 in AVSMeter64

DirectShowSource: ~1,240 MiB RAM used, then suddenly: Crop: Destination width is 0 or less. in AVSMeter64 (I guess it returned an invalid video frame)

BestSource R12: already pre-loading for half an hour, not sure when the processing will eventually start / after more than an hour: "BestVideoSource: Could not open video codec"

[LigH]

Reduced the number of clips to process.

L-SMASH-Works r1266
LSMASH{A/V}Source: ~6.5 GiB RAM
LwLibav{A/V}Source: ~6.4 GiB RAM

FFMS2 5.0: ~9 GiB RAM

DirectShowSource: unable to deliver a useable video stream (supposedly using LAV Filters 0.80 to decode 4K UHD AVC in MP4), not even in a single file call

BestSource R12: ~12 GiB RAM working set, ~24 GiB virtual (Private Bytes)

[poisondeathray]

Also avs trim cache consumes lots of memory . Consider using Trim calls with (cache=false)

see pinterf's comments
https://github.com/AviSynth/AviSynthPlus/issues/274
https://github.com/AviSynth/AviSynthPlus/issues/274#issuecomment-1065836642


cache=true: 24397 fps, 96 MiB memory
cache=false: 2088 fps, 44 MiB memory


clip=BlankClip(length=1000000000, pixel_type = "RGB32")
clp = clip.ConvertToRGB24
a = clp.Trim(20,-30, cache=false) + clp.Trim(2000,-30, cache=false)
a = a+a+a+a+a+a+a+a+a+a
a = a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a
a = a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a+a
return a





I think SS's Prune and ClipClop might be better at memory management too

[LigH]

Today I Learned ... will try, thanks.

No, doesn't seem to help.

[wonkey_monkey]

BestSource R12: ~12 GiB RAM working set, ~24 GiB virtual (Private Bytes)

Did you set cachesize = 0?

[LigH]

BSSource needs vcachesize=0, acachesize=0 ... but then it eats even 17 GiB working set!

[wonkey_monkey]

Results with my own big project - around 54x 45-60 minute Blu-ray rips with some 350 splices (I wrote my own C++ function* instead of using trim), opened in VirtualDub2:

Original project, using BestSource2 (RC10, cachesize = 0): 12Gb peak working set
Replacing sources with ColourBars or BlankClip: 780Mb peak working set

Don't ffms2, BestSource, and LSMASHSource all use ffmpeg internally?

*my splicing plugin has parameters for audio crossover and offsetting, if it's of any interest. It's not been written with general release in mind, though.

[ChaosKing]

They shoud all use ffmeg under the hood. BS is a bit different. For best frame accuracy it actually decodes the video while seeking to frame xy internally (I guess it starts from a keyframe).

[LigH]

I don't mind using ffmpeg core (libav), I don't mind an only nearly-perfect accuracy (even relying on ISO Media keyframe chunks inside MP4 is fine for me).

But I really wonder why it caches Gigabytes, what this cache contains, why it is so much, and how I can reduce that.

And also I am curious why decoding 4K-UHD AVC in MP4 with LAV Filters returns "no video stream"...

[wonkey_monkey]

Maybe ffmpeg just puts away ~200Mb for each file. It presumably has to keep track of at least a GOP's worth of pictures to construct the next one.

For best frame accuracy it actually decodes the video while seeking to frame xy internally (I guess it starts from a keyframe).

I think it asks ffmpeg for the requested frame, but if the hash doesn't match what it previously stored in the index using linear decoding (that's what makes indexing take so long), it adopts some other strategy to find it (asking for different frame numbers perhaps, or resetting ffmpeg somehow).

[rgr]

After adding about 80 clips from almost 30 files in an automating user function, AviSynth+ 64b crashes with a "Evaluate: Unhandled C++ exception!" after consuming ~15 GB RAM of my physically available 16 GB.

Have you considered creating an intermediate video file containing all these clips and then cutting just parts from it?

[LigH]

No. Can you imagine how large a hardly compressed FullHD video of 8-10 hours may get? Terabytes?

[rgr]

No. Can you imagine how large a hardly compressed FullHD video of 8-10 hours may get? Terabytes?

My hour-long FHD lossless FLV1 YUV444P12 takes up 1.5TB :) But I didn't limit myself because it's 1/10 of my HDD...
You can concatenate MP4 files losslessly, although I don't know how reliable that is. Or use an H264/H265 encoder with a low CRF and fast preset.

[real.finder]

did you try LSMASHSource_indexing to pre-indexing in CMD? maybe it will help

[LigH]

LSMASHVideoSource does not use separate index files but the index chunks already contained in MP4/MOV... files.

LwLibavVideoSource uses about as much RAM as LSMASHVideoSource once the index files are present.

[pbristow]

Suggestion: Use two copies of your script. The first one is edited to just compile the first half of the clips, and only opens the files needed to do that: Run that and save the result with minimal compression (try x264's lossless mode, but if you really can't afford that much disk space then I'd try a very slightly lossy setting like CQP=3, or CQP=5).

The second script then compiles the second half, *starting with* the temporary file produced by script 1.

With this approach, the total number of files you need to open at once is halved, and you only need enough temporary space for half the project.

Good luck! =:o}

[LigH]

I gave up trying to produce an annual collage and restricted it to 4 seasonals per year.

Still curious what exactly is kept in RAM. The full keyframe index for all imported media, possibly?

Regarding "visually transparent" intermediates, I am pretty confident that x264 with CRF 12 may be sufficient for my demands... anyway, too much trouble with questionable advantage.

[wonkey_monkey]

Still curious what exactly is kept in RAM.

The current GOP, required for decoding. I expect you'd find RAM usage would go up with longer GOPs or higher resolutions.

I suppose theoretically someone might be able to write a wrapper filter that constructed and destructed source filters as and when frames from each source were requested, though it'd probably be difficult to make it "nice."

[StvG]

Still curious what exactly is kept in RAM. The full keyframe index for all imported media, possibly?

There is initial buffer. More threads->more cached frames. E.g., threads=1 -> one cached frame. threads=16 -> 16 cached frames. This buffer can vary with one/few frames during decoding but considered it constant all the time. So use threads=1 for minimal initial buffer.

In addition of this initial buffer when decoding P/B frames (frames that depends on other frames data) more caching is involved. With threads=1 you limit the initial buffer to 1 cached frame but the total buffer depends on the frame types too. You also have things like b-pyramid...

[StainlessS]

Sounds like this old problem.
avisynth crashes when using to many input streams :- https://forum.doom9.org/showthread.php?t=170888

A system resources {windows} problem. (about 30 files open simultaneously)

An app like Premiere can open and close files as an when required, avisynth requires all files open at once.
(think thats how it must work for premiere)

You will likely have the same problem no matter which Source filter you use.

The opened clip count seems a pretty much constant (over time, over the years),
why is this not the same problem, has that 30 file open count been fixed ?

related threads

setmemorymax question
https://forum.doom9.org/showthread.php?t=171909

AviSynth Newbie - Using FadeIn/IO/Out
https://forum.doom9.org/showthread.php?t=176944

there are more threads on same-ish thing, but have not been able to find them.

EDIT: Not sure, but think if 30+ postage stamp sized files are opened, then same problem.

[real.finder]

LSMASHVideoSource does not use separate index files but the index chunks already contained in MP4/MOV... files.

LwLibavVideoSource uses about as much RAM as LSMASHVideoSource once the index files are present.

what about merge all clips in one file using something like mkvtoolnix? maybe not all but every like 5 or something to one file

[wonkey_monkey]

Working on a solution for this but currently hitting stumbling blocks.

[StainlessS]

From Google ["file" NEAR "open" NEAR "limit" site:forum.doom9.org]

Decompressor error after too many calls to AVISource?
https://forum.doom9.org/showthread.php?t=131687

The limit comes from how many .DLL's a single process can open. Avisynth is not in control of the process it is just one .DLL of many. The encoder application is the controlling process and we do not know what your encoder application is and what needs it has, it could be anything, VirtualDub, VirtualDubMod, MeGUI, TMpegEnc, ... We also do not know what codec your video uses and how many .DLL's it uses or how it shares those .DLL's.



From Google ["avisynth" NEAR "file" NEAR "open" NEAR "limit" site:forum.doom9.org] # reduce hit count by also requiring "avisynth"

Is there a limit to how many clips can be loaded simultaneously?
https://forum.doom9.org/showthread.php?t=139190

The limit comes from how many .dlls can be loaded into a process - avisource loads a codec for each call. Before anyone responds with "Why not fix it/make it work differently?" I suggest they look at the code and see what can be done. I had a go and managed to avoid reloading codecs that were already in use, but soon ran into another obstacle - setting up decompression processes for ~100 files makes memory usage go through the roof.

A much better solution, if suitable, is segmentedavisource. But it's only designed for joining segments that have been cut from one original file and it joins using unalignedsplice.



Processing many AVIs - a bug ?
https://forum.doom9.org/showthread.php?t=95482
You've hit the maximum number of dll's a process can load - it's about 72, and there isn't much to do about it unfortunately. You could try loading additional videos by using DirectShowSource, but I'm not sure how much it'll help.

Shubin,

As sh0dan say's there are limits. The other limit around the 50 files mark is the VFW codec table space. I have been caught by this before and I have a note to myself to add an option to SegmentedAviSource() and SegmentedDirectShowSource() to "stack" the input files into a single codec like VirtualDub does. The bad news it probably won't happen until 2.5.7.

IanB

Keen idea.

As file handles are not the limit, it is the number of loaded Dll's and VFW codec table space. So caching codec VFW object instances should be all that is needed. A single VFW object could be used to sequentially decompress all video streams with the same fourCC, output dimensions and colorspace.

Unfortunatly as the VFW objects have state (current frame for next delta frame) this would make parallel processing of 2 such files performance challanged. So some intelligence would be required.

Can we move this thread into the Developer forum.

IanB


joining many clips
https://forum.doom9.org/showthread.php?t=174357

Got sick of looking at links.

Good luck Wonkey.

EDIT: maybe better search,
Google [avisynth" NEAR "file" NEAR "open" NEAR ("limit" OR "max") site:forum.doom9.org]

EDIT: A couple more.

avisynth crashes when using to many input streams
https://forum.doom9.org/showthread.php?t=170888

Memory problem opening multiple clips
https://forum.doom9.org/showthread.php?p=1682093

[wonkey_monkey]

I got the maximum memory usage of a script sourcing and joining one second from each of 54 blu-ray rips down from 13Gb to under 450Mb.

Plugin coming soon.

[StainlessS]

"The Difficult We Do Immediately. The Impossible Takes a Little Longer."

(attributed to many, mutated over the centuries)

Nice one Wonkey.

[wonkey_monkey]

OnDemandFilter v0.1 (https://forum.doom9.org/showthread.php?t=186440)