x264 encoding on high core count systems

[Stereodude]

I know there are limits to how fast you can make x264 go by throwing logical cores at it, and the advice is to do multiple x264 jobs at once. So, you do multiple simultaneous encoding instances...

Is that still the best method with more than one physical CPU, or are there additional command line considerations? It seems like there would be a performance hit if Windows rolls threads from one physical CPU to another since not all the system memory is attached to any given CPU core and threads sharing information across cores on different physical CPUs would presumably suffer.

Can you control the affinity of which logical cores x264 will use from the command line so you can task one x264 job to one physical CPU and it's logical cores while tasking another job to another CPU, etc, etc... Or is it totally unnecessary and x264 and Windows (7 or 10) will play nice and sort it out for close enough to optimal results?

[Groucho2004]

Or is it totally unnecessary and x264 and Windows (7 or 10) will play nice and sort it out for close enough to optimal results?
That. Just let the Windows thread scheduler do its thing.

[Stereodude]

That. Just let the Windows thread scheduler do its thing.
What about limiting the number of threads for each x264 instance?

[Groucho2004]

What about limiting the number of threads for each x264 instance?
I never noticed any benefit from doing that and I sometimes run 4 encodes simultaneously on a 4 core CPU.

Here are two threads that are probably of interest to you:
http://forum.doom9.org/showthread.php?t=169695
http://forum.doom9.org/showthread.php?t=168875

[kolak]

4 cores system is not a big deal, but 72 one is bit different and multiple instances work way more efficient in such a case.

[Stereodude]

4 cores system is not a big deal, but 72 one is bit different and multiple instances work way more efficient in such a case.
Well I'm looking at using a dual CPU system with 16 physical cores, 32 logical cores (total). I guess I will do some benchmarking and find out once it is together.

[jpsdr]

There was several old posts saying that too much threads can have negative effects on picture quality. Basicaly until around 16 it's good (for 1080p), after, it can begin to have a not negligeable effect on quality. One of the reasons is that the picture parts begin to be too small over the search motion area.

[Groucho2004]

There was several old posts saying that too much threads can have negative effects on picture quality. Basicaly until around 16 it's good (for 1080p), after, it can begin to have a not negligeable effect on quality. One of the reasons is that the picture parts begin to be too small over the search motion area.
Lots of info about these issues in the two threads I linked to above.

[Stereodude]

Lots of info about these issues in the two threads I linked to above.
That says 27 for 1080p, but I came across another forum post yesterday when I was searching about this from Dark Shikari where he said both b frames and lookahead would / could use more threads safely. I can't find the post now though.

[Blue_MiSfit]

The rule of thumb I've heard is that you're fine to use up to 1 thread per 20 vertical pixels. Beyond that there is some noticeable quality loss.

[Stereodude]

The rule of thumb I've heard is that you're fine to use up to 1 thread per 20 vertical pixels. Beyond that there is some noticeable quality loss.
I'm pretty sure it's 40 pixels (lines), not 20 (per one of the earlier linked threads).

[Groucho2004]

Again, this post (http://forum.doom9.org/showthread.php?p=1646307#post1646307) explains it very well and akupenguin (Loren Merrit) certainly has extensive knowledge about it.

[Stereodude]

Again, this post (http://forum.doom9.org/showthread.php?p=1646307#post1646307) explains it very well and akupenguin (Loren Merrit) certainly has extensive knowledge about it.
Yes, that was the post I was referring to in my reply. The post is partially ambiguous IMHO. The way it's written it sounds like x264 will automatically restrict the number of threads it uses based on the vertical resolution of the video and the --mvrange-thread parameter to keep quality from suffering. That doesn't exactly agree with the understanding that x264 will use: frame based threads: 1.5 * logical processors, rounded down; slice based threads: 1 * logical processors. Up to a total limit of 128 threads.

[benwaggoner]

Pinning one encode per CPU (so, one each per socket) can help some with x264, so you don't get NUMA issues where the threads of one process might run on different cores.

[Stereodude]

What happened to the 2nd post of the thread that was a link about using start (http://ss64.com/nt/start.html)?

Pinning one encode per CPU (so, one each per socket) can help some with x264, so you don't get NUMA issues where the threads of one process might run on different cores.
Are you pinning them with start and the /NODE switch? Or, are you using /NODE & /AFFINITY? Or skipping /NODE and just using /AFFINITY?

[benwaggoner]

What happened to the 2nd post of the thread that was a link about using start (http://ss64.com/nt/start.html)?


Are you pinning them with start and the /NODE switch? Or, are you using /NODE & /AFFINITY? Or skipping /NODE and just using /AFFINITY?
I don't have the .bat file I was using handy, but I think I used /NODE.

[Stereodude]

I don't have the .bat file I was using handy, but I think I used /NODE.
Okay, per the help for start /NODE isn't not necessarily strictly enforced unless you add /AFFINITY with it.
Specifying /NODE allows processes to be created in a way that leverages memory locality on NUMA systems. For example, two processes that communicate with each other heavily through shared memory can be created to share the same preferred NUMA node in order to minimize memory latencies. They allocate memory from the same NUMA node when possible, and they are free to run on processors outside the specified node.
I guess I will try it each way. /NODE 0 & /NODE 1 vs /NODE 0 /AFFINITY FFFF & /NODE 1 /AFFINITY FFFF. I don't see how the latter case is functionally any different from using /AFFINITY FFFF & /AFFINITY FFFF0000 without the /NODE switch.

Did you also use --threads or --lookahead-threads in the x264 command line to reduce the number of threads x264 spawned? It seems like that would be necessary otherwise x264 will see a high number of logical cores and spawn a pile of threads even though it is restricted to only a subset of the logical cores.

[Stereodude]

Is there a way to tell how many threads x264 is using either from the encoded output, or by getting a more verbose output on the command line while it's running?

I only just got AVIsynth and all that jazz running on my first v1 E5 Xeon box (single processor). I have yet to tackle the dual processor one.

[sneaker_ger]

x264 writes the encoding settings incl. thread number into the output. You can read that using MediaInfo.

[Stereodude]

x264 writes the encoding settings incl. thread number into the output. You can read that using MediaInfo.
Okay, thanks.

As expected, it uses threads=24 / lookahead_threads=2 for an 8 physical core / 16 logical core processor. AFAIK, that should be okay for 1080p video since it stays above 40 vertical lines per thread "limit".

[Stereodude]

Hmmm... So adding --threads 24 --lookahead-threads 2 to the command line for the first pass noticeably slowed things down on my dual v1 Xeon E5-2670 box. I used the /NODE and /AFFINITY switch to pin the encode on only 1 NUMA node (1 physical CPU). Without any restriction on --threads and --lookahead-threads it runs in the mid 80FPS. With the thread restrictions it's in the low 50's. My single v1 Xeon E5-2689 box runs mid 90's on the same video with the same version of x264, same OS, equally fast decode engine, etc. It would seem that the first pass is spawning more threads on it's own than 24 / 2.

Edit: A little trial and error shows that --threads 24 isn't the parameter slowing it down, it's the --lookahead-threads 2
Edit2: Increasing lookahead-threads to 8 seems to give the same approximate performance as when it chooses on system with a single physical CPU. --threads 24 --lookahead-threads 8 is giving me mid 80 FPS for the first pass. Lower values of --lookahead-threads did not get there (tried 2, 3, 4, & 6)

[Stereodude]

Well, this isn't what I was expecting...

x264 will restrict its thread usage based on the /NODE and /AFFINITY switches. If I only give it 8 logical cores to play with it spawns 12 / 1. If I give it 16 logical cores it spawns 24 / 2. If I give it 32 logical cores it spawns 48 / 4

I also was surprised to see it effectively saturate 100% CPU on both physical processors if I let it run wild. Better than real time encoding using preset very slow in the 2nd pass. My command line:
--bitrate 18438 --preset veryslow --tune film --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 --pass 2

[Stereodude]

Even a crf 18 encode of the same Blu-ray source will saturate both CPUs. :confused:

--crf 18 --preset veryslow --tune film --bluray-compat --vbv-maxrate 40000 --vbv-bufsize 30000 --level 4.1 --keyint 24 --open-gop --slices 4 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1

[Stereodude]

Does anyone know how the visual quality loss from using too many threads will manifest itself? What visually should I be looking for in the output? Should the QP values also show deterioration (higher numbers for the same encode settings)?

[Stereodude]

Here are the results of my testing using the 2nd pass of a 2 pass Blu-ray re-encode (1920x1080) of a 2.35:1 movie with black bars on the top and bottom. The numbers are threads / lookahead-threads.

i7-4770k @ 4.2gHz [12 / 1]
encoded 195503 frames, 10.39 fps, 18445.55 kb/s
[info]: frame I:9295 Avg QP:10.37 size:185709
[info]: frame P:69175 Avg QP:14.49 size:118257
[info]: frame B:117033 Avg QP:15.16 size: 75998
E5-2670 dual CPU pinned to only one CPU (/NODE 0 & /AFFINITY FFFF) [24 / 2]
encoded 195503 frames, 14.06 fps, 18445.64 kb/s
[info]: frame I:9274 Avg QP:10.37 size:185136
[info]: frame P:70206 Avg QP:14.49 size:117965
[info]: frame B:116023 Avg QP:15.16 size: 75866
E5-2670 dual CPU 27 / 2 forced via command line (no node or affinity restrictions)
encoded 195503 frames, 20.31 fps, 18445.60 kb/s
[info]: frame I:9274 Avg QP:10.38 size:185082
[info]: frame P:70206 Avg QP:14.49 size:117979
[info]: frame B:116023 Avg QP:15.16 size: 75861
E5-2670 dual CPU no restrictions [48 / 4]
encoded 195503 frames, 27.42 fps, 18445.61 kb/s
[info]: frame I:9274 Avg QP:10.39 size:184313
[info]: frame P:70206 Avg QP:14.49 size:118064
[info]: frame B:116023 Avg QP:15.16 size: 75871