Out of RAM with 316 UHD Drone footages

[FranceBB]

Hi there folks,
I have a 64 GB DDR4 server and I've been asked to make a masterclip out of 316 UHD Drone footages of a thing we're shooting which I cannot disclose (sorry). The "problem" is that at some point I believe it runs out of RAM 'cause it quickly reaches 64 GB and starts using pagefiles. It encoded the first 15 minutes out of 8 hours, however then it crashed. Is there a way to tell Avisynth: hey, you don't need to store 316 videos in your memory, you can index one, encode one and THEN append it rather than doing everything at once as it now seems to be indexing everything, appending everything in RAM and then start filtering stuff.
Of course I could just say: ok, I can split it into several AVS Script and then use the "Concat" function in FFMpeg or whatever to create a single masterclip, but... do I really wanna do that? Shouldn't there be a way to do this in Avisynth without having unlimited RAM? Another way would be to increase the paging file to something insane like 512 GB, but... do I really wanna do that?

This is the Avisynth Script: https://pastebin.com/LPb1p9MX

[wonkey_monkey]

of a thing we're shooting which I cannot disclose (sorry).

Blink twice if it's aliens.

[Zarxrax]

Curious if it still runs out of memory if you remove the degrain and denoise functions?

[FranceBB]

Blink twice if it's aliens.

hahahahahahaha no no, far less interesting xD

Curious if it still runs out of memory if you remove the degrain and denoise functions?

I'm gonna try with SetMemoryMax(1) and without the filters and save to lossless HuffYUV.
If it works I'll then re-encode the lossless HuffYUV to XDCAM-50.

I'll let you know.

[StainlessS]

There was always a problem concatenating multiple [separate input file] clips in Avisynth,
each source filter craps out at different input count.
Some kind of machine resources limit (some as low as about 24 input clips, and we're not talking super huge dimension clips, SD same).

[DTL]

It looks the design limit of the current cache system. It is made for faster processing and faster seek but uses too large memory. Increasing pagefile should be done by windows automatically if it not set to fixed size but will not help in speed.
If I calculate right the 316 of 4k rgb frames and 6 frames per clip require 46GB memory. May you can look into sources and build special version of avisynth with as low number of cached frames as possible and may be 316 clips with 1 frame per clip will fit in 64GB. But it may be cache with several frames per each framebuffer in a filtergraph - like 6 frames per each filter input and output.

[FranceBB]

It looks the design limit of the current cache system. It is made for faster processing and faster seek but uses too large memory. Increasing pagefile should be done by windows automatically if it not set to fixed size but will not help in speed.
If I calculate right the 316 of 4k rgb frames and 6 frames per clip require 46GB memory. May you can look into sources and build special version of avisynth with as low number of cached frames as possible and may be 316 clips with 1 frame per clip will fit in 64GB. But it may be cache with several frames per each framebuffer in a filtergraph - like 6 frames per each filter input and output.

Well, I've put SetMemoryMax(1), commented out the filters, increased the pagefile to 436 GB and launched the encode as HuffYUV lossless. Let's see how it goes:

https://i.imgur.com/bkLslFp.png

and still raising steadily.


EDIT: What about that, it's going! With as little as 149 GB of RAM! XD

https://i.imgur.com/aOFnCbe.png

[DTL]

Also simple idea - set threads to 1. Avisynth make large copy of data per each thread.
So set Prefetch(1) at the end.

Also thSAD=800 for MDegrain possibly will cause lots of blurred textures. Is the footage really as large damaged by grain ? For typical HD ENG/EFP 3-chip camera footage with gain about 0 dB I found the thSAD > 200 already cause some visible fine texture degradation. In the best case for every shot (every camera type and gain-dB setting) it is better to adjust thSAD individually. So if 316 clips where shot with different cam and different gain settings (may be auto-gain even) - the thSAD=800 may too more blur the low-gain and lownoise clips.

To workaround the memory issue with many-clips avisynth scripts I think about some 'piping' between different processes in the operating system. But still no working examples.

Like a set of 'source' avisynth scripts with partial set of clips feed the 'output' process with final long footage assembling via logical pipes without immediate files creation. May be even old VirtualDub in frameserving mode may be driver for 'source' avs srcipts with partial processing.

Also there were AVFS plugin for general 'virtual file' software system Pismo File Mount - https://www.videohelp.com/software/Avisynth-Virtual-File-System . It can generate virtual uncompressed AVI files in the filesystem from avisynth scripts and it can be source to final assembling avisynth script (via AVISource()). Not tested - just an idea.

So each avisynth enviroment in a working pool will operate with limited number of clips and may be in a sequence instead of parallel access.

[qyot27]

Did you try SetCacheMode? (http://avisynth.nl/index.php/Internal_functions#SetCacheMode)

Or running it on Fedora?

[StainlessS]

Try below, but still think will crap out due to input clip count.
Because there is a minimum value, you cannot disable the cache completely using SetMemoryMax().
However, in v2.60, you can effectively disable it by redefining the Cache() function (in your script) as follows:
function Cache(clip c) { return c }

[FranceBB]

Try below, but still think will crap out due to input clip count.

Will try! :)


Or running it on Fedora?

Fedora is the host to my Windows VMs, including the ones I use at work to RDP into the real bare metal server eheheheh

TL;DR I don't have Avisynth installed on Fedora, but even if I did it would run on my puny i7 4c/8th rather than on my 56c/112th server, so it wouldn't be worth it.

Also simple idea - set threads to 1. Avisynth make large copy of data per each thread.
So set Prefetch(1) at the end.

I'm not using Prefetch at all in the script. Will using Prefetch(1) make any difference at all?



Also thSAD=800 for MDegrain possibly will cause lots of blurred textures.


Yep, the idea is to blur... a lot. The reason is that the drone footage is not the grain in the shadows, but rather the fact that it's already pretty damn sharp in UHD and when I bring it down to FULL HD using SinpowResize() it creates so many high frequency components that everything becomes aliased once I flag it as interlaced (I need to output interlaced).
Remember the discussion we had about resampling kernels? Well, this is it, essentially I need to blur everything out so that it doesn't get an aliased mess once I bring it to XDCAM-50. Besides, blurring vertically only using Blur(0, 1.58) apparently isn't enough with this sort of footage (word of the director, not mine).




To workaround the memory issue with many-clips avisynth scripts I think about some 'piping' between different processes in the operating system. But still no working examples.


Yeah I was thinking about things like MPPipeline which can split a script into as many individual processes as someone wants, but not in the way it currently works but rather the one you described.

[FranceBB]

Cross your fingers, guys, it's almost over! :D

https://i.imgur.com/zEWAiQu.png

[DTL]

"(I need to output interlaced)."

There are many ways how to decrease vertical frequencies in Avisynth (simple but may be slow is twice resize like Resize(width, height/2) + Resize(width, height*2)) .

But if Avisynth already overloaded with processing and you use ffmpeg as output engine there is
interlace->lowpass->complex
or
tinterlace->interleave_top/bottom->flags->complex_filter
to try for fighting interlaced aliasing.

[FranceBB]

"(I need to output interlaced)."

There are many ways how to decrease vertical frequencies in Avisynth (simple but may be slow is twice resize like Resize(width, height/2) + Resize(width, height*2)) .

But if Avisynth already overloaded with processing and you use ffmpeg as output engine there is
interlace->lowpass->complex
or
tinterlace->interleave_top/bottom->flags->complex_filter
to try for fighting interlaced aliasing.

Well, it's a bit more complicated than that.
Let's put it this way, the stream will be 25p, so 25 progressive, but due to the compatibility stuff we all know and broadcasting, it MUST be flagged as 25i. What happens then is that the TVs around the country will receive the feed, trust the metadata, bob-deinterlace it to 50p (duplicating frames 'cause it's actually 25p) and this will introduce aliasing, so essentially I generally just use vertical blur, but in this very case, there are still too many high frequencies in the form of grass and leaves that create aliasing when the TVs assume interlaced and bob it.

Oh, by the way, the encode to lossless HuffYUV completed just fine! :D

https://i.imgur.com/Pu7paWS.png


General
Complete name : W:\00_INGEST_MAM\INGEST_SKY_ORIGINAL\00_COMPLETATO\Drone\Drones_Masterclip.avi
Format : AVI
Format/Info : Audio Video Interleave
Format profile : OpenDML
File size : 1.52 TiB
Duration : 1 h 34 min
Overall bit rate : 2 372 Mb/s
Writing application : Lavf59.0.101

Video
ID : 0
Format : HuffYUV
Format version : Version 2
Codec ID : FFVH
Duration : 1 h 34 min
Source duration : 1 h 34 min
Bit rate : 389 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 7.512
Stream size : 256 GiB (16%)

[kedautinh12]

1.52tb, lol

[FranceBB]

1.52tb, lol

Don't worry, I've got space:

https://i.imgur.com/bFtj7zp.png

https://i.imgur.com/BMHhNRp.png