x265 encoding on very high core count machines?

[melp]

I would like to learn more about the multithreaded capabilities of x265, especially on very high core-count machines (60-80+ threads). I've read x265's docs on multithreading (http://x265.readthedocs.io/en/default/threading.html) and understand that HEVC is harder to parallelize than AVC, but I would like to better understand exatly what the limits are.

For example, could x265 saturate 60-80+ threads on a 4K encode with the veryslow or placebo preset with pmode and/or pme enabled? Has anyone done any tests like this on high core-count machines?

The only info I've found online is for 32-thread CPUs (https://forum.doom9.org/showthread.php?t=170690) and for 36-thread CPUs (https://www.anandtech.com/show/11839/intel-core-i9-7980xe-and-core-i9-7960x-review/11). In both cases, x265 seems to be able to take advantage of all cores for a 4K encode.

I would greatly appreciate any input!

[Atak_Snajpera]

For archiving purposes You can always run multiple x265 instances simultaneously like this
http://i.cubeupload.com/e8JoyV.png

This way you can saturate probably hundreds of CPU threads

[melp]

That's not a bad idea, assuming a single encode won't be able to saturate the CPUs. Still, I'd be interested to know what a single encode would look like on a high core count machine.

[poisondeathray]

Do you mean "look like" in terms of quality or just sheer performance ? The more threads, the more splits , the lower the quality / worse compression efficienccy

[melp]

Do you mean "look like" in terms of quality or just sheer performance ? The more threads, the more splits , the lower the quality / worse compression efficienccy
I'm interested in what an encode looks like in terms of performance, as I stated in the original post. And I think you're mistaken about more threads resulting in lower quality and worse compression efficiency in x265. More info on that topic is here: http://x265.readthedocs.io/en/default/threading.html

[Atak_Snajpera]

According to my tests single x265 (using default encoding settings) can saturate only 12 threads while encoding 1920x1080 footage.
3840x2160 should be enough to saturate 24 threads.

[melp]

According to my tests single x265 (using default encoding settings) can saturate only 12 threads while encoding 1920x1080 footage.
3840x2160 should be enough to saturate 24 threads.

As I said in the OP, I've found tests online where it can saturate 36 threads. I'm interested in performance on 80+ thread machines.

[Atak_Snajpera]

Just divide height by 64. This will give you estimated max threads for specific footage (1080p/2160p)

[melp]

It's not that simple. I would encourage you to read through the article I linked on x265 threading: http://x265.readthedocs.io/en/default/threading.html

[Atak_Snajpera]

64x64 CTUs are big! there are much fewer rows than with H.264 and similar codecs
I've done multiple tests on my 16 thread Xeon E5-2690 and single x265 can saturate to about ~75-80% on 1920x1080 footage.

2160p has two times more rows so you can easily estimate cpu usage.

[poisondeathray]

And I think you're mistaken about more threads resulting in lower quality and worse compression efficiency in x265. More info on that topic is here: http://x265.readthedocs.io/en/default/threading.html

Nope, this is a fact. I'm taking about frame threading

It definitely negatively impact compression efficiency. They even imply as much in the docs


--frame-threads
Number of concurrently encoded frames. Using a single frame thread gives a slight improvement in compression, since the entire reference frames are always available for motion compensation,


In real life tests, it's a measurable, repeatable difference that you can demonstrate on any test, any metric, any video. Plot threds vs. quality (whatever metric) and you will see inverse relationship.

But whether or not the quality hit is significant depends on the scenario. If you're using high bitrates, it's going to be negligible. But on low bitrate range scenarios, it can be significant to the point where human eye can see problems


But is using -F 1 reasonable ? Can you saturate your HW setup with -F 1 on a single encode ?. I doubt it .