Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
![]() |
#1 | Link |
Registered User
Join Date: Sep 2013
Posts: 38
|
Too many threads when set to auto
Hi all,
I'm just trying to clear up a little confusion about setting threads in x.264. From all the advice I've read and received here and elsewhere, large numbers of threads is ill-advised, and even has a negative impact on quality. Everyone seems to have a different opinion about how many threads is too many, but most advice is keep them under 20. However, when threads is set to auto (recommended default), x264 seems to set the number of threads to its default formula (1.5*logical processors). In many modern multiprocessor/core servers, this will almost always result in a number of threads considered anecdotally by most x.264 users as "too many". For example... We are running our x264 encodes on a Dell R710 server with dual 6-core processors for a total of 12 cores (24 processor threads using hyperthreading). When we set threads to "auto" in x264, the number of threads reported by Mediainfo is 36. That makes sense (so long as you consider a processor thread to be equivalent to a "logical processor" (more on that later). So while x264 seems to be working according to its formula, my question is why does the default "auto" setting permit the threads to rise above a threshold that most everyone seems to be saying will have a negative impact on quality? Or is it the case that the quality impact only comes into play when you set the number of threads higher than the actual number of logical processors on your machine (times 1.5)? In other words, do people see a reduction in quality with a large number of threads, even if their machine has enough processor threads to support the thread number calculated according to x264s formula? Thanks, Gregleto |
![]() |
![]() |
![]() |
#3 | Link |
Registered User
Join Date: Feb 2002
Location: San Jose, California
Posts: 4,454
|
As I understand it (imperfectly at best) the reduction in quality is due to the individual threads not sharing information with each other and when they get too high they are all working on such a small section that this lack of information starts to become significant. Exactly what "information" they are not sharing I don't know but I think it is some aspect of the results from each thread.
I think the auto threads was designed a while ago and with more normal systems in mind, with a multiprocessor many core system I suggest you set threads manually if you want quality over speed. @detmek Which resolution was that clip? I believe the higher the resolution the less threads have an impact on quality. |
![]() |
![]() |
![]() |
#5 | Link |
x264 developer
Join Date: Sep 2004
Posts: 2,392
|
There's two ways that lots of threads can lose quality:
|
![]() |
![]() |
![]() |
#6 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
|
|
![]() |
![]() |
![]() |
#7 | Link |
Registered User
Join Date: Sep 2013
Posts: 38
|
So based on the feedback here, below is the formula for properly setting threads that seems to make sense to me.
Threads should be equal to the lesser result of the following two calculations… 1. The number of logical processors multiplied by 1.5 2. The number of vertical pixels of video divided by 40 Examples A. Video encoded to 640x360 on a machine with a total of 8-cores and hyperthreading enabled. (8 cores with hyperthreading equals 16 logical processors) 1. 16*1.5 = 24 2. 360/40 = 9 Set threads to 9 (the lower result) B. Video encoded to 1280x720 on a machine with a total of 4-cores and hyperthreading enabled. (4 cores with hyperthreading equals 8 logical processors) 1. 8*1.5 = 12 2. 720/40 = 18 Set threads to 12 (the lower result) C. Video encoded to 1920x1080 on a machine with a total of 12-cores and hyperthreading enabled. (12 cores with hyperthreading equals 24 logical processors) 1. 24*1.5 = 36 2. 1080/40 = 27 Set threads to 27 (the lower result) Based on recent feedback and my (limited) understanding, with threads set to “auto” in x264, examples A and C would both result in an excessively high number of threads that could/may hinder performance and/or negatively affect quality. If anyone sees any problems with this approach please let me know. Thanks, Gregleto |
![]() |
![]() |
![]() |
#8 | Link |
Registered User
Join Date: Sep 2013
Posts: 38
|
Thanks from me as well. However, I just want to confirm one thing in your response that will add to my understanding. Near the end of your answer you state....
"And a 304p video won't use more than 8 threads for a sequence of P-frames." Did you mean to say "304p"? Thanks, Gregleto |
![]() |
![]() |
![]() |
#14 | Link |
Moderator
![]() Join Date: Jan 2006
Location: Portland, OR
Posts: 4,997
|
Is there much overhead in terms of CPU or RAM use with multithreading? For example, if I was encoding 36 individual identical streams simultaneously, would I see total fps throughput and RAM utilization go down significantly if I encoded each with --threads 1 instead of the implicit default of 36 threads for each instance (1296 threads! I'd like THAT workstation...)
|
![]() |
![]() |
![]() |
#15 | Link | |
Registered User
Join Date: Sep 2013
Posts: 38
|
Quote:
http://forum.doom9.org/showthread.php?t=168848 It's at the bottom of the thread and it received no replies. I think it's more appropriate for this thread. Could it be moved from there to here? Thanks, Gregleto |
|
![]() |
![]() |
![]() |
#16 | Link | |
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
|
Quote:
1 thread: 5.5 fps @ 25% CPU usage Auto (6 threads): 20.2 fps @ 100% CPU usage So, running 4 simultaneous encodings with "--threads 1" is more efficient but uses more memory - however - in separate processes. Running 36 simultaneous processes is probably more a problem for the hard drive serving your sources (although with a fast SSD maybe not so much). Last edited by Groucho2004; 16th October 2013 at 01:24. |
|
![]() |
![]() |
![]() |
#18 | Link | |
Registered User
Join Date: Oct 2002
Location: France
Posts: 2,434
|
Quote:
|
|
![]() |
![]() |
![]() |
Thread Tools | Search this Thread |
Display Modes | |
|
|