x264 multi-core (4+) threading optimization - Page 11

morph166955 · 26th May 2007, 17:42

@aku
so seeing as thread_pool_03 is faster then (or at least equal too) whats in the svn now, any chance we can get that to be committed to the svn or do you still want to do more testing of some sort?

ATT · 29th May 2007, 08:37

how does this translate to real world

2 hour 1080i/p HD movie MPEG2 to XH264 does it still get 120FPS or 200FPS im sure that's on SD right

with a quadcore/octcore system does it still take 2 days or more ?

right now i am sitting on a X24400 Noobish attempts doing some encoding a first pass on some anime: @ 6.20FPS

CLI: --pass 1 --stats .stats --bitrate 2000 --sar 1:1 -A all --aud --level 4.1 --ref 3 --mixed-refs --bframes 3 --direct auto --analyse none --me dia --subme 1 --threads 2 --no-ssim --no-psnr --progress
avis [info]: 1920x1088 @ 23.98 fps (150945 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!

basically for PS3 the second pass going to take 21 hours last time i tried. ( i had an error and i lost everything just minutes before it was about to finish)

has anyone done before and after of 1-2 hour film an have before and after times ? dualcore/quadcore/octcore

morph166955 · 29th May 2007, 15:53

Hows this...I'll let ya know! The system has been "down" while I've been working on my custom linux distro on it for the past week or so. I've got a nice long 1080p movie that I'm planning on testing it on overnight tonight. While it doesn't run 200fps, it does run what one would expect it to run being that the size of the image is different. For example, I've been running the tests on a cropped dvd image thats 848x352. Lets assume for math purposes that it ran exactly 200fps. Lets also assume for a moment that I was maxing out the CPU's which I'm not currently able to do. For a 1080p image you wouldn't expect 200fps since the 1920x1080 image is ~7 times larger then the 848x352 is. However, you would expect it to be around 200/7 which is ~28.5. When I was running some very quick tests on it (5000 frames max), I was seeing frame rates in the low 20's. While its not perfect scaling, it was adequate enough for my tastes considering that most current systems doing full 1080p run below 5fps if even that. I do remember at one point saying "wow I'm encoding 1080p in insane quality in real time" so I must assume that since it was a 24000/1001 frame rate that I was getting around 24-25fps for that block.

I'm recompiling x264 tonight to work on this and I'll run some tests once thats all done and I've gotten a few other small things tweaked. So as I said before, I'll let ya know!

708145 · 30th May 2007, 17:28

Quote:

Originally Posted by morph166955

i just ran a test to see what would happen if i loaded 4 ram buffers with 500 frames each, and then simultaniously forked 4 x264 (vanilla r655 build) threads running @ threads=2 (i also tried @ threads=3 the effect was negligible) , i was able to run the system at >90% cpu usage since each x264 thread was using 185-190% cpu of the 200% max it should be able to do. each thread ran at around 105fps giving me a total of about 420fps.

With just 8 cores I think 3 GOPs with 4threads will result in the same speedup with less memory usage. With more cores you'd want to find a better sweet spot (For 16 cores probably 5 GOPs with 5 threads each... I need to check this).

If you want to use GOP level encoding be aware that the main problem is not scaling performance but to get I frame positions right (as you already do) and also get the bit distribution right. Both x264farm and ELDER (my pet) come very close in that respect

bis besser,
T0B1A5

morph166955 · 18th August 2007, 15:02

In case anyone is still wanting to play around with this I have created a new diff based on the current svn. There were some things moved from common.h to osdep.h so I have fixed that as well as altered version.sh to generate a version that has "_THREADPOOL04" at the end (eg mine is now 0.56.669_THREADPOOL04). I just ran a two pass on some 1080p stuff and it gave me a gooooooood boost in speed.

Just to note, there is nothing different functionality-wize between this and the tp03 patch, I just fixed it to work with the current svn (r670).

http://www.benswebs.com/x264_thread_pool.04a.r670.diff

EDIT: oops uploaded it to the wrong place, link works now

morph166955 · 23rd September 2007, 17:32

Thread pool patch was updated to r678 and moved to http://www.benswebs.com/public/x264/....04b.r678.diff

Post #1 updated to have new link also

delacroixp · 24th September 2007, 10:22

It's always nice to see some movement in the world of multi-core threading optimization... especially in light of AMD plans to release a triple core in early 2008 and the need for a multi-core killer app ...

" Most of the software in the market cannot take advantage of multi-core chips, so why buy machines that boast to have them and pay the extra price ? " ( Sramana Mitra).

Pascal

708145 · 24th September 2007, 12:43

Quote:

Originally Posted by delacroixp

especially in light of AMD plans to release a triple core in early 2008.

Ah finally. A Tri-Athlon

The fitting CPU for the best sport in the world

But on-topic: There are apps around AI that definitely benefit from more cores. Think "decision helpers", "memory expanders" and "information filters" that adjust to your preferences. Video encoding is already as parallel as it need to be.

squid_80 · 24th September 2007, 15:57

Quote:

Originally Posted by delacroixp

" Most of the software in the market cannot take advantage of multi-core chips, so why buy machines that boast to have them and pay the extra price ? " ( Sramana Mitra).

Doesn't this guy ever run more than one program at once?

Sharktooth · 24th September 2007, 16:10

she is an idiot. she did a very superficial analysis and draw even some false conclusions.

lexor · 28th September 2007, 03:41

quick question, I see that Cef's page lists this patch (and others), but does Cef's build actually have this patch? (or any of the ones in patch directory?)

Adub · 28th September 2007, 04:50

Yeah, I believe Cef is fullfilling that hole that is Sharktooth in his absence.

Aka. He is building x264 with patches.

morph166955 · 28th September 2007, 21:07

well considering that the current revision of the patch wont apply cleanly to r680 i'm not sure how yes doing it. I'm going to update the patch this weekend to work with r680's naming conventions.

Cef · 28th September 2007, 22:45

Well I just added "x264_" here and there to compile with r680. I'm not even sure what's this patch suppsoed to do though, so I don't know if it's actually done right. diff is here

burfadel · 29th September 2007, 06:54

Cef, separate issue, but will you be adding the --me-prepass patch in future builds? its a great option! Its good now, in another post it was talked that it will be modified to reduce its performance impact (which is not excessive at the moment).

Dark Shikari · 29th September 2007, 07:11

Quote:

Originally Posted by burfadel

Cef, separate issue, but will you be adding the --me-prepass patch in future builds? its a great option! Its good now, in another post it was talked that it will be modified to reduce its performance impact (which is not excessive at the moment).

I need to fix a few things about it--Aku noticed a few problems with it that may both reduce its speed and its effectiveness below what it could be.

burfadel · 29th September 2007, 07:48

I know

maybe if the performance impact with the modifications is reduced enough it can be enabled by default, and then have an option of --nome-prepass or something?

Also, isn't imh supposed to replace umh? and also 'subme 8' replacing 'subme 7'?

morph166955 · 29th September 2007, 18:56

My patch is updated and uploaded to work with r680. Link on the main page has been updated.

NaturalThoughts · 27th August 2009, 09:04

I have query regarding QP in case of multithreading enabled.

As in the case of single thread, each next frame QP is calculated based upon previous frame stats.

IPPPP.... case
e.g. 0th Frame (QP =26)
1st frame (QP modifies based on 0th frame stats),
2st frame (QP modifies based on 1th frame stats)
and so on......
But in the case of more than one thread, how QP is assigned for next frame as present frame or previous frame is being encoded.

0th Frame (QP =26)
1st frame (QP = ?),
2st frame (QP = ?)
and so on......

Thanks in advance.

akupenguin · 27th August 2009, 10:40

1pass CRF/CQP: completely ignore previous frames' stats.
1pass ABR/2pass: QP adapts by stats from whichever frames have finished encoding. Any sane number of threads makes little difference, as the adaption time scale is much longer than the threading latency.
VBV: latency is important, so QP adapts by stats from whichever portions of each frame have finished encoding.

29th May 2007, 08:37	#202 \| Link
ATT Registered User Join Date: May 2007 Posts: 18	sorry for my ignorance but how does this translate to real world 2 hour 1080i/p HD movie MPEG2 to XH264 does it still get 120FPS or 200FPS im sure that's on SD right with a quadcore/octcore system does it still take 2 days or more ? right now i am sitting on a X24400 Noobish attempts doing some encoding a first pass on some anime: @ 6.20FPS CLI: --pass 1 --stats .stats --bitrate 2000 --sar 1:1 -A all --aud --level 4.1 --ref 3 --mixed-refs --bframes 3 --direct auto --analyse none --me dia --subme 1 --threads 2 --no-ssim --no-psnr --progress avis [info]: 1920x1088 @ 23.98 fps (150945 frames) x264 [info]: using SAR=1/1 x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow! basically for PS3 the second pass going to take 21 hours last time i tried. ( i had an error and i lost everything just minutes before it was about to finish) has anyone done before and after of 1-2 hour film an have before and after times ? dualcore/quadcore/octcore Last edited by ATT; 29th May 2007 at 08:40.

18th August 2007, 15:02	#205 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	In case anyone is still wanting to play around with this I have created a new diff based on the current svn. There were some things moved from common.h to osdep.h so I have fixed that as well as altered version.sh to generate a version that has "_THREADPOOL04" at the end (eg mine is now 0.56.669_THREADPOOL04). I just ran a two pass on some 1080p stuff and it gave me a gooooooood boost in speed. Just to note, there is nothing different functionality-wize between this and the tp03 patch, I just fixed it to work with the current svn (r670). http://www.benswebs.com/x264_thread_pool.04a.r670.diff EDIT: oops uploaded it to the wrong place, link works now Last edited by morph166955; 18th August 2007 at 15:14.

24th September 2007, 10:22	#207 \| Link
delacroixp Emperor building empire Join Date: Mar 2007 Location: ZAR Posts: 674	Ta... and tx for all the fish !!! It's always nice to see some movement in the world of multi-core threading optimization... especially in light of AMD plans to release a triple core in early 2008 and the need for a multi-core killer app ... " Most of the software in the market cannot take advantage of multi-core chips, so why buy machines that boast to have them and pay the extra price ? " ( Sramana Mitra). Pascal __________________ DualRomeEPYC . Storinator

24th September 2007, 16:10	#210 \| Link
Sharktooth Mr. Sandman Join Date: Sep 2003 Location: Haddonfield, IL Posts: 11,768	she is an idiot. she did a very superficial analysis and draw even some false conclusions. __________________ MPEG-4 ASP Custom Matrices: EQM V1(old), EQM AutoGK Sharpmatrix (aka EQM V2), EQM V3HR (updated 01/10/2004), EQM V3LR, EQM V3ULR (updated 04/02/2005), EQM V3UHR (updated 17/12/2004) and EQM V3EHR (updated 05/10/2004) Info about my ASP matrices. MPEG-4 AVC Custom Matrices: EQM AVC-HR Info about my AVC matrices My x264 builds. Mooo!!! Last edited by Sharktooth; 24th September 2007 at 16:18.

28th September 2007, 03:41	#211 \| Link
lexor Registered User Join Date: Jan 2004 Posts: 849	quick question, I see that Cef's page lists this patch (and others), but does Cef's build actually have this patch? (or any of the ones in patch directory?) __________________ Geforce GTX 260 Windows 7, 64bit, Core i7 MPC-HC, Foobar2000

26th May 2007, 17:42	#201 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	@aku so seeing as thread_pool_03 is faster then (or at least equal too) whats in the svn now, any chance we can get that to be committed to the svn or do you still want to do more testing of some sort?

29th May 2007, 15:53	#203 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	Hows this...I'll let ya know! The system has been "down" while I've been working on my custom linux distro on it for the past week or so. I've got a nice long 1080p movie that I'm planning on testing it on overnight tonight. While it doesn't run 200fps, it does run what one would expect it to run being that the size of the image is different. For example, I've been running the tests on a cropped dvd image thats 848x352. Lets assume for math purposes that it ran exactly 200fps. Lets also assume for a moment that I was maxing out the CPU's which I'm not currently able to do. For a 1080p image you wouldn't expect 200fps since the 1920x1080 image is ~7 times larger then the 848x352 is. However, you would expect it to be around 200/7 which is ~28.5. When I was running some very quick tests on it (5000 frames max), I was seeing frame rates in the low 20's. While its not perfect scaling, it was adequate enough for my tastes considering that most current systems doing full 1080p run below 5fps if even that. I do remember at one point saying "wow I'm encoding 1080p in insane quality in real time" so I must assume that since it was a 24000/1001 frame rate that I was getting around 24-25fps for that block. I'm recompiling x264 tonight to work on this and I'll run some tests once thats all done and I've gotten a few other small things tweaked. So as I said before, I'll let ya know!

23rd September 2007, 17:32	#206 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	Thread pool patch was updated to r678 and moved to http://www.benswebs.com/public/x264/....04b.r678.diff Post #1 updated to have new link also

28th September 2007, 04:50	#212 \| Link
Adub Fighting spam with a fish Join Date: Sep 2005 Posts: 2,699	Yeah, I believe Cef is fullfilling that hole that is Sharktooth in his absence. Aka. He is building x264 with patches. __________________ FAQs:Bond's AVC/H.264 FAQ Site:Adubvideo

28th September 2007, 21:07	#213 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	well considering that the current revision of the patch wont apply cleanly to r680 i'm not sure how yes doing it. I'm going to update the patch this weekend to work with r680's naming conventions.

28th September 2007, 22:45	#214 \| Link
Cef Registered User Join Date: Mar 2007 Posts: 25	Well I just added "x264_" here and there to compile with r680. I'm not even sure what's this patch suppsoed to do though, so I don't know if it's actually done right. diff is here

29th September 2007, 06:54	#215 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	Cef, separate issue, but will you be adding the --me-prepass patch in future builds? its a great option! Its good now, in another post it was talked that it will be modified to reduce its performance impact (which is not excessive at the moment).

29th September 2007, 07:48	#217 \| Link
burfadel Registered User Join Date: Aug 2006 Posts: 2,229	I know maybe if the performance impact with the modifications is reduced enough it can be enabled by default, and then have an option of --nome-prepass or something? Also, isn't imh supposed to replace umh? and also 'subme 8' replacing 'subme 7'?

29th September 2007, 18:56	#218 \| Link
morph166955 Registered User Join Date: Mar 2006 Posts: 443	My patch is updated and uploaded to work with r680. Link on the main page has been updated.

27th August 2009, 09:04	#219 \| Link
NaturalThoughts H.264 freak Join Date: Aug 2009 Location: In front of Computer Posts: 21	I have query regarding QP in case of multithreading enabled. As in the case of single thread, each next frame QP is calculated based upon previous frame stats. IPPPP.... case e.g. 0th Frame (QP =26) 1st frame (QP modifies based on 0th frame stats), 2st frame (QP modifies based on 1th frame stats) and so on...... But in the case of more than one thread, how QP is assigned for next frame as present frame or previous frame is being encoded. 0th Frame (QP =26) 1st frame (QP = ?), 2st frame (QP = ?) and so on...... Thanks in advance.

27th August 2009, 10:40	#220 \| Link
akupenguin x264 developer Join Date: Sep 2004 Posts: 2,392	1pass CRF/CQP: completely ignore previous frames' stats. 1pass ABR/2pass: QP adapts by stats from whichever frames have finished encoding. Any sane number of threads makes little difference, as the adaption time scale is much longer than the threading latency. VBV: latency is important, so QP adapts by stats from whichever portions of each frame have finished encoding. Last edited by akupenguin; 27th August 2009 at 10:43.