8K Video encoding with x264

[Xiaopang]

I stumbled over some 8k content and wanted to re-encode it with the latest x264. However, I always run into malloc-errors even though the memory usage doesn't really seem that high.

The video in question is only 1 second long (60 fps) and I have 8GB of memory. I tested this after a fresh reboot. Is it really possible that this is not enough for such a short encode or is it a limitation in x264?

According to this thread (http://forum.doom9.org/archive/index.php/t-167021.html) there's probably no resolution limitation in the encoder, yet it also says that Level 5.2 only supports up to 4k. Since the encoder resorts to using this level even though the video has a resolution of 7680 x 4320 could this be the reason for this error?

This is my command line btw:

--threads auto --crf 0 --keyint 600 --ref 16 --no-fast-pskip --bframes 16 --b-adapt 2 --direct auto --me tesa --merange 64 --subme 11 --trellis 2 --partitions all --rc-lookahead 100 --no-dct-decimate --input-range pc --range pc

I also tried it with a lookahead value of 10 to no avail.

If you're wondering why I'm wasting time with such an apparently useless endeavor: The creator of this file used Lagarith to compress it losslessly and the original file has a size of a whooping 2,4GB. I just want to see how much better x264's lossless mode fares against Lagarith.

The creator didn't use much compression so I was able to recompress the source to only 1,67 GB, saving 750MB. It looks like x264 could get this down to 500MB, but there's no way to find out other than trying :)

In case someone wants to give the video a shot, you can download it here. (http://www.perfecthandproductions.com/video-crysis-3-8k-render)

[LoRd_MuldeR]

Do you use a 32-Bit or 64-Bit binary of x264?

I the former case, the memory that can be allocated by x264 will be at most 4 GB or, in some circumstances, only 2 GB:
http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

So you definitely want to try with a 64-Bit binary here!

Also, how do you load the video? If, for example, you load it via Avisynth, then Avisynth will also occupy a significant amount of memory within the x264 process.

You can try piping the video into x264 using Avs2YUV to avoid Avisynth occupying memory in the x264 process...

[foxyshadis]

Because x264 allocates for qpel, you need hundreds of megs allocated per reference frame, and you have 16. 32-bit memory is exhausted almost immediately due to memory fragmentation; even on 1080p video memory fragmentation can cause enough problems to error out with high enough settings. You really need 64-bit.

Beyond that, tesa, ref 16, and 16 b-frames won't help nearly as much as you're hoping, but I guess you might as well give it a shot.

[Xiaopang]

Thank you for the replies.

Yeah, I should have mentioned that I'm of course using the x64 version ;)

Thanks for the explanation regarding the reference frames. Is there a formula to calculate x264's memory requirement beforehand so that I can modify the settings accordingly?

[Atak_Snajpera]

3840x2160p can easily eat 6 GiB of ram using very slow preset. In theory 8K should need even 24 GiB of free ram.

[Xiaopang]

Back to answer my own question: The malloc errors occured way before exhausting the memory because my memory was actually faulty.

Once I had it replaced, a ref and rc-lookahead value of 5 was enough to keep memory consumption below 7600MB allowing me to re-encode the sample losslessly to 399MB :)

[Bloax]

Well technically it's only lossless if you encode it in the same --output-csp as the input. :c

[NikosD]

Back to answer my own question: The malloc errors occured way before exhausting the memory because my memory was actually faulty.


That lossless 7680x4320@60fps at 20.6 GBs video rate clip, is a real stress test for the whole system.

Using signature system Core i7-4790@3.8GHz and 8GB RAM I tried DXVA Checker to test it.

With null renderer I got 0/3/7 Min (fps)/ Avg(fps)/ Max (fps) with a 84% CPU utilization and ~160 MB/s HDD rate.

Then I removed a potential bottleneck of HDD by putting it on the SSD and I got ~240MB/s rate, but the whole performance didn't improve much.

With EVR renderer, performance was even worst of course:

Null renderer

0/3/7 CPU 84 % Total time 19,5 sec (total time for 1 sec video) (HDD)

0/3/7 CPU 90% Total time 18,3 sec (SSD)


EVR renderer

0/2/3 CPU 70 % Total time 28,6 sec (HDD)

0/2/4 CPU 73% Total time 26,5 sec (SSD)

[NikosD]

Back to answer my own question: The malloc errors occured way before exhausting the memory because my memory was actually faulty.

Once I had it replaced, a ref and rc-lookahead value of 5 was enough to keep memory consumption below 7600MB allowing me to re-encode the sample losslessly to 399MB :)

I tried latest nightly Handbrake and latest beta Vidcoder and they both didn't recognised the source properly.

The Vidcoder crashed and the Handbrake didn't allow me to encode it.

Have you tried any GUIs of x264 that work on that 8K sample ?