x264 revision 610 - sliceless threading - what's new

[bob0r]

------------------------------------------------------------------------
r610 | pengvado | 2006-12-16 01:32:38 +0100 (Sat, 16 Dec 2006) | 2 lines

more win32threads -> pthreads

------------------------------------------------------------------------
r609 | pengvado | 2006-12-16 00:08:57 +0100 (Sat, 16 Dec 2006) | 2 lines

cosmetics: rename list operators to be consistent with Perl, and move them to common/

------------------------------------------------------------------------
r608 | pengvado | 2006-12-16 00:06:21 +0100 (Sat, 16 Dec 2006) | 2 lines

win32: use pthreads instead of win32threads. for some reason, pthreads is much faster.

------------------------------------------------------------------------
r607 | pengvado | 2006-12-16 00:03:36 +0100 (Sat, 16 Dec 2006) | 13 lines

New threading method:
Encode multiple frames in parallel instead of dividing each frame into slices.
Improves speed, and reduces the bitrate penalty of threading.

Side effects:
It is no longer possible to re-encode a frame, so threaded scenecut detection
must run in the pre-me pass, which is faster but less precise.
It is now useful to use more threads than you have cpus. --threads=auto has
been updated to use cpus*1.5.
Minor changes to ratecontrol.

New options: --pre-scenecut, --mvrange-thread, --non-deterministic

------------------------------------------------------------------------


when multithreading "--threads auto" recommended


Here are some results showing cpu*1.5:
x264_610.exe --threads # -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720
x264_606.exe --threads # -B5000 -m6 -r5 --direct=temporal --me=hex -b2 -w --qcomp=0.10 -A"p8x8,i8x8,i4x4" -8 --fps=25 --output NUL 720p50_mobcal_ter.yuv 1280x720

Intel D930 3.0GHz (2 cores, no HT):
610 2 threads: 3.89
610 3 threads: 5.41
606 2 threads: 4.63
606 3 threads: 4.46

P4 xeon 3.06GHz (2 cores, HT):
610 4 threads: 4.24
610 5 threads: 4.63
606 4 threads: 3.98
606 5 threads: 4.01

AMD X2 3800+ 2.55GHz (2 cores):
610 2 threads: 4.07
610 3 threads: 5.24
606 2 threads: 4.85
606 3 threads: 4.78

C2D E6400 3.4GHz (2 cores):
610 2 threads: 8.78
610 3 threads: 12.35
606 2 threads: 9.52
606 3 threads: 8.34

I have more results, of earlier patches:
http://x264.nl/results.txt (cv = pthreads)

Not sure why, but Intel Core 2 Duo users can be happy with the big speed up.

Note as the changelog says, New options: --pre-scenecut, --mvrange-thread, --non-deterministic,
Go MeGUI .... and other guis!

[Thunderbolt8]

what about the aspect akupenguin once mentioned that, with more splitting up into more threads the quality also decreases (slightly) ? does this still apply now ?

[akupenguin]

Yes, but not as much as before. Hence the "reduces the bitrate penalty".

cpu: 4 core Xeon 5160

threads speed psnr loss
r606 r611 r606 r611
1: 1.000x 1.000x 0.000 0.000
2: 1.540x 1.739x -0.036 -0.004
3: 1.838x 2.384x -0.065 -0.002
4: 2.043x 3.224x -0.077 -0.005
5: 2.028x 3.512x -0.110 -0.009
6: 2.034x 3.629x -0.132 -0.009
7: 1.988x 3.680x -0.151 -0.015
8: 1.953x 3.702x -0.188 -0.017
9: 2.016x 3.729x -0.210 -0.020
10: 1.995x 3.742x -0.233 -0.031
11: 1.954x 3.749x -0.255 -0.030
12: 1.909x 3.765x -0.268 -0.040
13: 1.895x 3.770x -0.286 -0.045
14: 1.936x 3.759x -0.313 -0.046
15: 1.897x 3.781x -0.335 -0.045
16: 1.845x 3.765x -0.349 -0.046

scaling efficiency (speed / #cores):
r606: 51%
r611: 94%

The above was measured on Linux. We had much trouble getting efficient thread synchronization on Windows, and the performance still might be not quite as good.

[Sharktooth]

So... the megui automatic no. of threads detection should be updated.
also what's the exact usage for the new options? what do they actually do?
is the haali's AQ patch working correctly after those changes (it applies almost correctly btw...)?

for builders: pthread for win32 is here -> ftp://sources.redhat.com/pub/pthreads-win32

[bob0r]

@builders
or here:
cvs -d :pserver:anoncvs@sources.redhat.com:/cvs/pthreads-win32 checkout pthreads
make clean GC-static
copy pthreads/*.h to include dir
copy libpthreadGC2.a to lib dir

@sharktooth:
You can just use --threads auto for megui, as x264.exe will detect the # of cpu to use, and set threads by *1.5 (Using HT might give one or to extra threads, but the quality will not be affected much at all)

Exact usage of new options no clue here, from x264.exe:
--pre-scenecut Faster, less precise scenecut detection.
Required and implied by multi-threading.

--mvrange-thread <int> Minimum buffer between threads [-1 (auto)]

--non-deterministic Slightly improve quality of SMP, at the cost of repeatability

[akupenguin]

Short answer: you shouldn't need to use any of the new options.
--pre-scenecut might possibly be useful for fast 1pass single-threaded encodes, but is mostly just so I can compare the two scenecut algorithms without invoking the other threading stuff.

[Sharktooth]

im just updating megui... i removed the ability to set the number of threads within the x264 profile (updown control grayed out).
so --threads auto will be enforced when "Automatically set the number of threads" global option is enabled in megui settings.
It's just a workaround but should work as expected.

[Sharktooth]

Im reorganizing the MeGUI version numbers. A new MeGUI build is coming. Just be patient...:)

[Adub]

Uh, sharktooth? I just downloaded the new version, and those settings aren't greyed out. In addition MeGUI still sets the threads manually. Like I am using 2 threads right now according to the log for the first pass.

Edit: look at the post times! That should explain it.

[Audionut]

Thanks. This build is alot faster on a core 2.

[Zerofool]

Theoretically will this thing have any (positive) effect on single core CPUs (set at 2 threads) ? Does it make any sense using it that way?
(I'll try it tonight anyway :).)

[Sharktooth]

megui 0.2.3.2193 is up and running. Enjoy.
Tomorrow i'll update the Trunk version (0.2.4.0000) with a more elegant solution ;)

@bobor and other builders: DO A SVN CHECKOUT and use the sources in Tags/2193 for building.

[Audionut]

And working like a charm. Thanks.

[Adub]

I can't update to 2193. I mean I downloaded it, it asks me to restart the program. It does, but it still says 2192 at the top and the x264 options aren't greyed out.

[Sharktooth]

uhm... reboot. .NET cache is doing something wrong.
now im going to sleep (5:21AM) ;)
if you have still problems unistall megui and install a fresh version. then let it autoupdate.

[woah!]

i use ELDER now for my encodes but i see this "bug" in the release i thought might need a look at:

x264.nl REV611: http://mirror01.x264.nl/x264/revision611/x264.exe

H:\MCHD encodes\ELDER RATIO.1.77.1\905029>..\x264.exe -o test3.264 --ref 5 --8x8
dct --mixed-refs --no-fast-pskip --bframes 3 --bime --weightb --direct auto --me
range 12 --subme 6 --analyse all --me umh --colormatrix bt709 --filter -2:-1 --p
ass 2 --stats stageC-3.stats --progress -B 1377 stageA-3.avs
avis [info]: 720x400 @ 23.98 fps (357 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:5 Avg QP:20.60 size: 16678 PSNR Mean Y:46.04 U:47.30
V:47.27 Avg:46.40 Global:45.94
x264 [info]: slice P:268 Avg QP:22.98 size: 8262 PSNR Mean Y:44.48 U:46.08
V:45.76 Avg:44.89 Global:44.22
x264 [info]: slice B:84 Avg QP:23.46 size: 3583 PSNR Mean Y:44.30 U:46.71
V:46.87 Avg:44.96 Global:44.21
x264 [info]: mb I I16..4: 22.0% 63.3% 14.7%
x264 [info]: mb P I16..4: 15.7% 25.7% 4.5% P16..4: 30.2% 7.1% 2.0% 0.2% 0
.1% skip:14.5%
x264 [info]: mb B I16..4: 1.7% 4.0% 2.2% B16..8: 31.7% 1.6% 2.5% direct:
2.8% skip:53.5%
x264 [info]: 8x8 transform intra:56.0% inter:50.3%
x264 [info]: direct mvs spatial:0.0% temporal:100.0%
x264 [info]: ref P 73.7% 13.5% 8.0% 2.7% 2.2%
x264 [info]: ref B 86.8% 6.1% 3.7% 1.9% 1.5%
x264 [info]: SSIM Mean Y:0.9766733
x264 [info]: PSNR Mean Y:44.460 U:46.243 V:46.042 Avg:44.927 Global:44.239 kb/s:
1396.13

encoded 357 frames, 5.57 fps, 1396.55 kb/s
profiling:F:\msys\home\user\x264/x264.gcda:Cannot open
profiling:F:\msys\home\user\x264/matroska.gcda:Cannot open
profiling:F:\msys\home\user\x264/muxers.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/encoder.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/common.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/mdate.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/set.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/macroblock.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/set.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/ratecontrol.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/frame.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/pixel.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/macroblock.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/cpu.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/analyse.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/mc.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/cabac.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/cavlc.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/cabac.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/dct.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/quant.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/csp.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/predict.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/eval.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/i386/predict-c.gcda:Cannot open
profiling:F:\msys\home\user\x264/encoder/me.gcda:Cannot open
profiling:F:\msys\home\user\x264/common/i386/mc-c.gcda:Cannot open

H:\MCHD encodes\ELDER RATIO.1.77.1\905029>echo.1>stageC-result3.ready




===========================================



Sharktooths REV611: http://mirror05.x264.nl/Sharktooth/?dir=./x264


H:\MCHD encodes\ELDER RATIO.1.77.1\311401>..\x264.exe -o test3.264 --ref 5 --8x8
dct --mixed-refs --no-fast-pskip --bframes 3 --bime --weightb --direct auto --me
range 12 --subme 6 --analyse all --me umh --colormatrix bt709 --filter -2:-1 --p
ass 2 --stats stageC-3.stats --progress -B 1377 stageA-3.avs
avis [info]: 720x400 @ 23.98 fps (357 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2
x264 [info]: slice I:5 Avg QP:20.60 size: 16678 PSNR Mean Y:46.04 U:47.30
V:47.27 Avg:46.40 Global:45.94
x264 [info]: slice P:268 Avg QP:22.98 size: 8262 PSNR Mean Y:44.48 U:46.08
V:45.76 Avg:44.89 Global:44.22
x264 [info]: slice B:84 Avg QP:23.46 size: 3583 PSNR Mean Y:44.30 U:46.71
V:46.87 Avg:44.96 Global:44.21
x264 [info]: mb I I16..4: 22.0% 63.3% 14.7%
x264 [info]: mb P I16..4: 15.7% 25.7% 4.5% P16..4: 30.2% 7.1% 2.0% 0.2% 0
.1% skip:14.5%
x264 [info]: mb B I16..4: 1.7% 4.0% 2.2% B16..8: 31.7% 1.6% 2.5% direct:
2.8% skip:53.5%
x264 [info]: 8x8 transform intra:56.0% inter:50.3%
x264 [info]: direct mvs spatial:0.0% temporal:100.0%
x264 [info]: ref P 73.7% 13.5% 8.0% 2.7% 2.2%
x264 [info]: ref B 86.8% 6.1% 3.7% 1.9% 1.5%
x264 [info]: SSIM Mean Y:0.9766733
x264 [info]: PSNR Mean Y:44.460 U:46.243 V:46.042 Avg:44.927 Global:44.239 kb/s:
1396.13

encoded 357 frames, 6.19 fps, 1396.55 kb/s

H:\MCHD encodes\ELDER RATIO.1.77.1\311401>echo.1>stageC-result3.ready


Sharktooth's version works well but the other one as you can see isnt built correctly (i think)

anyways...

[bob0r]

Updated revision 611 coming right up

wrong 611 x264.exe md5: 2d738a38420fb2d92154556a1aa29c70
correct 611 x264.exe md5: b64eb9012bd24e2cd4f616be482c5a31

In about 30 minutes a proper version should be online, without the half-as*-ed make fprofiled version :)

[huang_ch]

Great job, thanks all the x264 developers for this. Below is a quick test from me:
Hardware: Conroe E6600@2.4GHz

FPS Kbps PSNR
b606(1t) 9.76 1200.92 47.015
b606(2t) 14.28 1214.10 47.008
b611(1t) 9.37 1229.90 47.203
b611(2t) 15.30 1242.54 47.248
b611(auto) 17.45 1242.29 47.248
b611(4t) 17.97 1241.65 47.247

It's good to see no PSNR penalty now and the CPU is mostly >90% usage. :thanks:

[Adub]

your getting >90%. I am only reaching about 83%. But that is still first pass. Plus I am Folding in the background.

[Sharktooth]

Updated revision 611 coming right up

wrong 611 x264.exe md5: 2d738a38420fb2d92154556a1aa29c70
correct 611 x264.exe md5: b64eb9012bd24e2cd4f616be482c5a31

In about 30 minutes a proper version should be online, without the half-as*-ed make fprofiled version :)

my build is fprofiled ...

@Merlin: .NET cache sometimes does weird things... and when it happens it's always a PITA :)

[MatMaul]

Hello !
Thanks for your great job.
On a single core CPU (pentium M), I noticed a little decrease of the speed, also with one thread.

command line :
x264 --bitrate 1000 --threads nb_of_threads --ref 3 --bframes 2 --direct auto --subme 5 --partitions all --8x8dct --me hex --progress --output NUL "dd.avs"

v606 :

1 threads : 11.56 fps, 1006.55 kb/s, PSNR 48.655
2 threads : 10.61 fps, 1007.08 kb/s, PSNR 48.626
3 threads : 10.63 fps, 1007.75 kb/s, PSNR 48.599

v611 :

1 threads : 11.20 fps, 1008.27 kb/s, PSNR 48.683
2 threads : 11.26 fps, 1007.95 kb/s, PSNR 48.687
3 threads : 11.20 fps, 1008.29 kb/s, PSNR 48.686

Why the 1 thread's ouputs are not identical ?

[Sharktooth]

what build you used?

[MatMaul]

bobor builds, I can't find rev606 of your builds

[Sharktooth]

http://megui.org/auto/x264_r606.zip
bobor's 611 is not fprofiled... that could be the cause of the slowdown vs 606....

[huang_ch]

Sharktooth, in my previous post, I also met a slight performance loss with 1 thread mode(9.76fps->9.37fps), all uses your builds.

[Sharktooth]

uhm... interesting...
with 1 thread speed is slightly lower but PSNR is slightly higher... (from 0.03 to 0.19db according to your results)

[MatMaul]

same results with your builds :

v606 :

1 threads : 11.57 fps, 1006.55 kb/s, PSNR 48.655
2 threads : 10.55 fps, 1007.08 kb/s, PSNR 48.626

v611 :

1 threads : 11.30 fps, 1008.27 kb/s, PSNR 48.683
2 threads : 11.37 fps, 1007.95 kb/s, PSNR 48.687

the PSNR is higher but the bitrate too.

[Sharktooth]

Well, however the speed difference is not so big :)
Could you please test if AQ is working properly?

[MatMaul]

AQ seems to works good (test with --aq-strength 0.5 --aq-sensitivity 15, video is ok).
The speed isn't very problematic (just 2%).

[huang_ch]

Well, however the speed difference is not so big :)
Could you please test if AQ is working properly?

I'm trying AQ this whole day to fight again those blocks(which is posted in another thread), but since I didn't use AQ before, so what I could tell is that AQ seems to really work, the output is really much bigger than no-AQ, and no obvious error.:)

[bob0r]

@Sharktooth,
No you got it wrong, i was propering my own build, and yes all my x264.exe files are make fprofiled, just the wrong 611, x264.exe was the one used by x264 source to collect it's information, then the final x264.exe failed (i need to fix gtk), and thus the half-as*-ed version of fprofiled was online :)

@MatMaul,
Yes, others have reported the same problem also, must be said, so far i have only noticed this with .avs input.

@akupenguin,
Should 606 and 611 with 1 thread give the same speed?
Or is 606 using 1 win32 thread and 611 is using 1 pthreads thread?

[vanger]

your getting >90%. I am only reaching about 83%. But that is still first pass. Plus I am Folding in the background.
On my AMD x2 3800+ 2500Mhz the cpu load is near 80% (85% max) at 1st pass... And near 95% (max 99%) at 2nd...

[MatMaul]

@MatMaul,
Yes, others have reported the same problem also, must be said, so far i have only noticed this with .avs input.


You talk about speed or not identical outputs with 1 threads ?

[bob0r]

On a single core, also try --threads 2. --threads 3, --threads 4, see how the speed fps and cpu load change, still on single core, it seems avisynth can limit an encode.

All test with big .yuv files show huge improvements on many computers.

[deets]

holy crap!!!

us PSP owners will be eternally thankful :) now i can use my centrino duo to its full power and after a quick test, i think my encoding fps speed has just gone up by a crazy amount!!

this is amazing, thanks very much for all your work, bobor and sharktooth

edit: i shall have to update my site, this is the best news for PSP owners since AVC was added :D

[Dreassica]

hmm switched from build 600 to 611 and i get artifacts and blocking on playback now.

command line used on both:

C:\x264.exe --qp 17 --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,2 --subme 6 --trellis 2 --analyse all --8x8dct --me esa --threads auto --thread-input --progress --no-dct-decimate --no-psnr --output "E:\sample.mkv" "E:\sample.avs"

[Thunderbolt8]

is it actually possible to use 4 threads with a core 2 duo ?

[emmel]

Very impressive.
Thanks.

[bob0r]

...

this is amazing, thanks very much for all your work, bobor and sharktooth

...

Since the x264 developers now get paid to work on x264, i guess Sharktooth and me can take full credit of Loren "pengvado" Merritt and co their work :devil:

So here goes: No problem, it was our pleassure!

akupenguin :sly:

[bob0r]

is it actually possible to use 4 threads with a core 2 duo ?

yes use --threads 4 (instead of --threads auto)
new threads limit is 16.

[Thunderbolt8]

yes use --threads 4 (instead of --threads auto)
new threads limit is 16.
so I could make my own new default with 16 threads each time then and it would be fastest ? is the loss of quality with 16 threads really as low as it seems on paper (i mean o,35% is like nothing)?

[akupenguin]

On a single core, also try --threads 2. --threads 3, --threads 4, see how the speed fps and cpu load change, still on single core, it seems avisynth can limit an encode.
If speed is limited by reading a big yuv file from hdd, then --thread-input will fix it. Load could also be limited if you put x264 on low priority, and then threads would be counterproductive. There is no other reason for load to not be 100% on a single core, and --threads should never be used on a single core.

Should 606 and 611 with 1 thread give the same speed?
Yes. But that's "should" as in a failure to do so would be a regression, not as in I'm sure its true.
Or is 606 using 1 win32 thread and 611 is using 1 pthreads thread?
1 thread invokes neither win32threads nor pthreads.

so I could make my own new default with 16 threads each time then and it would be fastest ?
I've seen quality loss of up to 1% for 16 threads. But yes, you could use lots.

[Adub]

wait x264 devs are getting paid now to work on the project? When did this happen?

[MatMaul]

@ akupenguin : is it normal I obtain a different output than with rev606 with 1 thread (or with multithreading disabled at the compilation) ?

v606, 1 thread : 1006.55 kb/s, PSNR 48.655

v611, 1 thread (or multithreading disabled at the compilation) : 1008.27 kb/s, PSNR 48.683

[akupenguin]

@MatMaul
Yes, there were minor changes in ratecontrol and B-adapt.
In your case, +.17% bitrate and +.028 psnr is a net gain, since as a rule of thumb .05 psnr is worth 1% bitrate.

[MatMaul]

Thanks, good news !

[emmel]

Small test on a four-core smp (two dual core Opterons) with default settings for x264 (--threads N -o test_720x576.264 test_720x576.y2m):

1 threads: 40.835 dB, 13.61 fps, 1743.59 kb/s
2 threads: 40.829 dB, 24.55 fps, 1740.32 kb/s
3 threads: 40.829 dB, 34.20 fps, 1740.32 kb/s
4 threads: 40.829 dB, 44.19 fps, 1740.25 kb/s
8 threads: 40.829 dB, 44.53 fps, 1739.77 kb/s

Scales almost linearly up to four threads without any significant loss in quality. Very nice indeed.

[Zero1]

hmm switched from build 600 to 611 and i get artifacts and blocking on playback now.

command line used on both:

C:\x264.exe --qp 17 --ref 16 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter 1,2 --subme 6 --trellis 2 --analyse all --8x8dct --me esa --threads auto --thread-input --progress --no-dct-decimate --no-psnr --output "E:\sample.mkv" "E:\sample.avs"
Same here.

Also, impressive improvements this time, my cores are 100% for a lot of the time with the occasional dip. It's nice that both cores are being put to good use (A64 Turion x2 TL-60).

[DeeGee]

Artifacting here too. But after ironing out those bugs, this is most exellent update. My Core 2 Duo is liking pthreads very much :)

[bob0r]

Can you guys post some screenshots of this artifacting?
Or even better, video samples.