Multi-core Encoding Question

[DreadStarX]

I've done a fair bit of Googling but nothing seemed to answer my question.

I've got a Dell R810 with 40 Cores (4x 10 Core CPUs). If I allow my encoder to use all 40 cores, will I sacrifice quality? I want speed, but quality and a decent size.

:stupid:

[sneaker_ger]

Yes, unless you increase the bitrate a bit to make up for it.
http://forum.doom9.org/showthread.php?t=173277

[LoRd_MuldeR]

DreadStarX, your question is pretty generic. You didn't even say which encoder you are referring to.

It totally depends on the individual encoder software being used how well it scales to many CPU cores. Also I think it will be pretty hard to fully "saturate" a 40 cores system with a single encoding process, no matter what. Even software encoders like x264 and x265, which are known to scale extremely well to a large number of threads, still have a certain limit in this regard! Finally, as sneaker_ger indicated, at very high thread count there can be implications on quality. At "usual" thread counts the effect is negligible. As a rule of thumb, the higher the resolution, the more threads you can use before quality may degrade. So, there's a point where you should consider running multiple encoder processes in parallel to saturate your hardware...

[Nevilne]

If possible, split file in 4 parts, encode each in parallel with 10 threads, mux back together.

[GMJCZP]

In this matter, I Think StainlessS made a script, I'm interested.

[DreadStarX]

Alright. Well, I left the x264 / x265 blank, but I did post in the x264 because I don't know diddly about x265. I'm told to stay away from it because it's "under developed".

I'll see how much of a struggle the server has running ESXi with 10 cores per VM.

Would you say;
720p & Lower - 8 Cores or less
1080p - 8+ Cores
2K/4K - 16+ Cores

[jpsdr]

From what i've read on several posts on this topic, basicaly the safe limit is around 40 lines/thread, under, quality may begin to suffer (and i think very recently a patch on x264 internaly prevent it to go under). So, for a 1080p, you gan go safe until 22-24 (40 lines means 27 btw on 1080p).

[Groucho2004]

but I did post in the x264
Not quite, there is no x264 forum. This forum is called "MPEG-4 AVC / H.264" which covers a whole bunch of encoders.

Anyway, the min. 40 vertical lines/thread suggested by jpsdr is sound advice.

[StainlessS]

In this matter, I Think StainlessS made a script, I'm interested.

MakeMultiPartScripts.avs
posted here:- http://forum.doom9.org/showthread.php?p=1714171#post1714171

[Atak_Snajpera]

If you are lazy you can just fire Ripbot264 with enabled distributed encoding. Whole process is basically "fire and forget" ;)
https://www.youtube.com/watch?v=b-BICnj62iE

[GMJCZP]

MakeMultiPartScripts.avs
posted here:- http://forum.doom9.org/showthread.php?p=1714171#post1714171

Thank you, tin man. :thanks:

DreadStarx, I suggest enter the link provided by StainlessS.

[Stereodude]

If possible, split file in 4 parts, encode each in parallel with 10 threads, mux back together.
That doesn't work for 2 pass encoding. It works okay for crf. You may need to use --stitchable

[benwaggoner]

From what i've read on several posts on this topic, basicaly the safe limit is around 40 lines/thread...
Do we know where that came from? It's not even Mod16, and presumably all operations are at least at the macroblock row level. So 48 or 32 seem more likely.

Since the per core performance gets worse as number of cores and sockets go up, I think a 40 core system is probably not going to have net throughput as good as a simpler dual Xeon config, which would also be a lot cheaper. Certainly for x264 @ 1080p. x265 @ 2160p may be different, as that can certainly saturate 18 physical cores (36 logical).

[Groucho2004]

Do we know where that came from? It's not even Mod16, and presumably all operations are at least at the macroblock row level. So 48 or 32 seem more likely.
It was a post from Loren Merrit here (http://forum.doom9.org/showthread.php?p=1646307#post1646307).

[LoRd_MuldeR]

It was a post from Loren Merrit here (http://forum.doom9.org/showthread.php?p=1646307#post1646307).

If I read that right, he said that 40 pixels is actually the minimum "distance" that x264 enforces (with default settings) between concurrent threads.

Consequently, the maximum number of concurrent threads (disregarding B-Frames) is restricted to the vertical resolution divided by 40 - which means you effectively can't use more than 27 threads for 1080p input.

Thus, using even more threads than that won't make any difference, in terms of quality. And also won't make a difference in terms of speed.

[Asmodian]

27 may be the max number of useful threads for speed but we also use 12-18 threads as a rule of thumb for the maximum number of threads before it starts to impact quality (1080p). I wonder how many for 4K. :D

Using 10 threads each with 4 simultaneous encodes sounds like a good idea to me. :)

[DreadStarX]

Awesome. For my other servers, which are 8 - 16, those should be fine for encoding with a single executable running.

Awesome, simply awesome. I'm stoked to see how things turn out. Now to configure storage for encoding! Also, love the community, thanks for not being elitist jerks. Makes learning that much easier!

[benwaggoner]

Awesome. For my other servers, which are 8 - 16, those should be fine for encoding with a single executable running.
Pinning each encode to a single NUMA node will probably improve throughput as well. And will automatically constrain threads to the number of logical cores.

x264 doesn't have x265's nice "--pools" option, but if the process is launched pinned to a particular socket, you'll get the same effect.