Improvements to existing x264 options: Test thread

[Dark Shikari]

My next goal in improving x264 is to find some, if marginal, improvements to existing options, as I did with --subme 7. We don't just need extra insane, CPU-eating options for people with OCD ;)

First test: slightly improved ME HEX. Test this on various different sources with these two plain non-pthreaded builds. I want to see if its improvement over the regular HEX is a fluke, or consistent. (Both speedwise and qualitywise)

Original build, with the same compiler/settings, for comparison. (http://tjhsst.edu/~jgarrett/x264_original.exe)

Modified build (http://tjhsst.edu/~jgarrett/x264_mod.exe)

Test with --me hex, of course.

[CruNcher]

finaly you realized were the real improvements are needed, also look @ the partitions distribution i think something can still be optimized their in terms of detail preservation, it seems to me as x264 is wasteing bits on places they don't need to be wasted on :)

[LoRd_MuldeR]

Dark Shikari, now that you are looking for new things to improve, are there any of your previous improvements (Improved multi-hexagon search, Hadamard motion estimation, Simulated annealing) completed yet? And most important: Will any of those make it into official SVN soon ???

[Dark Shikari]

Dark Shikari, now that you are looking for new things to improve, are there any of your previous improvements (Improved multi-hexagon search, Hadamard motion estimation, Simulated annealing) completed yet? And most important: Will any of those make it into official SVN soon ???They're already available for download on the official unofficial build site as a separate executable and source--they'll probably be merged soon but you can already get a completely finished executable.

Note that I tossed out Simulated Annealing and replaced it with --me IMH since it ended up being not nearly as good as I expected.

[Gabriel_Bouvigne]

Feeding the ME with additional candidates coming from the ME done on the scaled/subpicture. Might provide some interesting candidate vectors (after scaling them up, of course).

[LoRd_MuldeR]

They're already available for download on the official unofficial build site as a separate executable and source--they'll probably be merged soon but you can already get a completely finished executable.

Right, but I mainly use Avidemux (http://forum.doom9.org/showthread.php?t=126164) for my encoding tasks. So I would need an Avidemux build that includes a patched libx264.dll in order to be able to use your improvements. And most important: They will need to modify the x264 Configure Dialog in Avidemux, so you can actually enable those new options. I'm sure this won't happen as long as those patches are not in the official SVN...

BTW: What about the AQ patch, which is around for quite some time now and seems to be used by many people? Will it ever be committed to the official SVN or do the x264 developers dislike that patch for some reason?

[ToS_Maverick]

speaking of AQ:
what about other HVS improvements? i've seen enough streams from german HDTV, where the encoder keeps much of the dirty, dithered look, although high compression is used. i've played a lot with the x264 settings, but it's very difficult to replicate this behaviour.

[Sharktooth]

i doubt AQ will be committed since it's an empirical workaround for something that theoretically should work but practically doesnt work as it should.
the actual AQ approach is to help this situation but it assigns bits only to dark or blue areas regardless of the fact those places may need those bits or not.

[LoRd_MuldeR]

hmm, as long as the AQ patch does help in some situations (if you explicitly enable AQ) and doesn't do any harm if your keep AQ disabled (default), why not commit it? It could be removed as soon as it's not required any more... Does it too much interfere with existing features?

[Sharktooth]

akupenguin doesnt like empirical stuff (the same reason the hrd patch is not committed)

[Dark Shikari]

You know, it would actually help if a few of you tested what I put in the original post ;)

[Sharktooth]

i cant... i'm sadly rebuilding one of my boxes with Vista and it's making me swear very bad words at M$...

[Pomyk]

I tested on a 1000 frame anime sequence. The modified build used about 1% less bitrate at the same ssim/psnr (6 ref, 3 bframes, me 6, trellis 2, 1000kbit/s) and was marginally faster.

[talen9]

Well, it's time to stop lurking on this forum and try to be helpful .. :)

I did a simple test, on 1761 frames, source is "300" R2 PAL Dvd, with mixed slow/high motion, dark/light scenes; the script is not so lightweight, and neither is the x264 configuration used ... let me know if you think it's better to change something ;)

Ah, the CPU is a PIV @2.66Ghz, single core, no HT ...

Common encoder configuration:
--crf 18 --ref 4 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --bime --weightb --direct auto --filter -1,0 --analyse p8x8,b8x8,i4x4,p4x4 --vbv-maxrate 25000 --threads auto --thread-input --sar 832:583 --progress --no-dct-decimate

with x264_original:
avis [info]: 704x416 @ 25.00 fps (1761 frames)
x264 [info]: using SAR=832/583
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: slice I:18 Avg QP:17.00 size: 37254 PSNR Mean Y:47.57 U:51.97 V:51.96 Avg:48.58 Global:48.23
x264 [info]: slice P:812 Avg QP:19.70 size: 18561 PSNR Mean Y:44.92 U:51.24 V:51.09 Avg:46.14 Global:45.12
x264 [info]: slice B:931 Avg QP:21.36 size: 8022 PSNR Mean Y:44.20 U:51.62 V:51.45 Avg:45.53 Global:44.02
x264 [info]: mb I I16..4: 34.5% 0.0% 65.5%
x264 [info]: mb P I16..4: 8.8% 0.0% 8.9% P16..4: 35.3% 16.6% 12.6% 2.6% 2.6% skip:12.6%
x264 [info]: mb B I16..4: 0.7% 0.0% 2.8% B16..8: 26.0% 2.5% 6.1% direct: 18.7% skip:43.1%
x264 [info]: direct mvs spatial:76.6% temporal:23.4%
x264 [info]: ref P 67.3% 17.0% 10.0% 5.8%
x264 [info]: ref B 72.3% 17.8% 6.1% 3.7%
x264 [info]: SSIM Mean Y:0.9798280
x264 [info]: PSNR Mean Y:44.563 U:51.451 V:51.289 Avg:45.843 Global:44.523 kb/s:2636.12

encoded 1761 frames, 4.60 fps, 2636.28 kb/s


with x264_Dark_mod:

avis [info]: 704x416 @ 25.00 fps (1761 frames)
x264 [info]: using SAR=832/583
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [warning]: VBV maxrate specified, but no bufsize.
x264 [info]: slice I:18 Avg QP:17.11 size: 36920 PSNR Mean Y:47.49 U:51.95 V:51.94 Avg:48.51 Global:48.18
x264 [info]: slice P:785 Avg QP:19.77 size: 18767 PSNR Mean Y:44.82 U:51.21 V:51.04 Avg:46.05 Global:45.04
x264 [info]: slice B:958 Avg QP:21.34 size: 7961 PSNR Mean Y:44.24 U:51.62 V:51.46 Avg:45.57 Global:44.07
x264 [info]: mb I I16..4: 34.6% 0.0% 65.4%
x264 [info]: mb P I16..4: 8.9% 0.0% 8.9% P16..4: 35.5% 16.7% 12.7% 2.6% 2.6% skip:12.2%
x264 [info]: mb B I16..4: 0.7% 0.0% 2.7% B16..8: 26.1% 2.6% 6.1% direct:18.5% skip:43.2%
x264 [info]: direct mvs spatial:76.0% temporal:24.0%
x264 [info]: ref P 66.7% 17.2% 10.2% 5.9%
x264 [info]: ref B 72.5% 17.7% 6.1% 3.7%
x264 [info]: SSIM Mean Y:0.9797742
x264 [info]: PSNR Mean Y:44.531 U:51.442 V:51.278 Avg:45.813 Global:44.509 kb/s:2614.83

encoded 1761 frames, 4.52 fps, 2614.99 kb/s

So, at least for this short clip and for the chosen encoder parameters, new version seems slightly slower and the quality of the encoding pretty much the same (if I understand the log correctly, that is :p).

PS: on a second run of both tests, the resulting FPS is 4.62 for the original build and 4.53 for your modded one ...

[J_Darnley]

I've run a short test today, these are the results.

AviSynth source of a pre-processed (denoised, cropped and greyscaled, selected 6%) source, The Longest Day, black & white PAL DVD source.

AVISource("C:\DVD\Projects\The_Longest_Day\The_Longest_Day_lossless.avi", audio=false)
Spline36Resize(640, round((height*640*45.0)/(width*16*64.0))*16)

D:\AviSynth>x264_original --crf 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-r
do --mixed-refs --bime --trellis 1 --progress --output "The_Longest_Day_x264_test_original.264" "The_Longest_Day_x264_te
st.avs"
avis [info]: 640x272 @ 25.00 fps (15450 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:175 Avg QP:20.78 size: 18248 PSNR Mean Y:44.90 U:100.00 V:100.00 Avg:46.65 Global:45.84
x264 [info]: slice P:7276 Avg QP:22.91 size: 5068 PSNR Mean Y:42.62 U:100.00 V:100.00 Avg:44.37 Global:43.47
x264 [info]: slice B:7999 Avg QP:24.60 size: 1319 PSNR Mean Y:42.88 U:100.00 V:100.00 Avg:44.61 Global:43.27
x264 [info]: mb I I16..4: 27.1% 0.0% 72.9%
x264 [info]: mb P I16..4: 5.1% 0.0% 6.6% P16..4: 41.0% 16.3% 8.4% 0.9% 0.6% skip:21.0%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.7% B16..8: 30.4% 2.1% 4.6% direct: 1.9% skip:60.2%
x264 [info]: ref P 82.9% 10.4% 6.7%
x264 [info]: ref B 90.7% 6.6% 2.7%
x264 [info]: SSIM Mean Y:0.9809617
x264 [info]: PSNR Mean Y:42.779 U:100.000 V:100.000 Avg:44.524 Global:43.385 kb/s:655.19

encoded 15450 frames, 12.64 fps, 655.27 kb/s

D:\AviSynth>x264_mod --crf 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-rdo --
mixed-refs --bime --trellis 1 --progress --output "The_Longest_Day_x264_test_original.264" "The_Longest_Day_x264_test.av
s"
avis [info]: 640x272 @ 25.00 fps (15450 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:173 Avg QP:20.76 size: 18470 PSNR Mean Y:44.86 U:100.00 V:100.00 Avg:46.61 Global:45.83
x264 [info]: slice P:7264 Avg QP:22.91 size: 5064 PSNR Mean Y:42.62 U:100.00 V:100.00 Avg:44.37 Global:43.46
x264 [info]: slice B:8013 Avg QP:24.61 size: 1321 PSNR Mean Y:42.87 U:100.00 V:100.00 Avg:44.61 Global:43.27
x264 [info]: mb I I16..4: 26.6% 0.0% 73.4%
x264 [info]: mb P I16..4: 5.2% 0.0% 6.5% P16..4: 41.1% 16.5% 8.4% 0.9% 0.5% skip:20.8%
x264 [info]: mb B I16..4: 0.2% 0.0% 0.7% B16..8: 30.5% 2.1% 4.5% direct: 1.9% skip:60.1%
x264 [info]: ref P 82.7% 10.5% 6.8%
x264 [info]: ref B 90.7% 6.6% 2.7%
x264 [info]: SSIM Mean Y:0.9809429
x264 [info]: PSNR Mean Y:42.777 U:100.000 V:100.000 Avg:44.522 Global:43.383 kb/s:654.52

encoded 15450 frames, 12.40 fps, 654.61 kb/s

From what I can see in the log, there isn't much difference between the two. Negligable changes in SSIM, PSNR and bitrate. I will run another test tonight on a larger and longer source, that is if you are still wanting tests.

[Dark Shikari]

I guess that settles it then--the new algorithm isn't worth it, because while it does boost speed and quality/bitrate sometimes, it slows things down too much in other cases to be useful.

[ToS_Maverick]

wait a minute, here are my results:

C:\Dokumente und Einstellungen\Administrator\Desktop>x264_original.exe --crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --output "F:\Video\Black.Pearl.Sample.mkv" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.6 --aq-sensitivity 10 >log.txt
avis [info]: 768x320 @ 23.98 fps (3624 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [info]: slice I:109 Avg QP:19.83 size: 16257 PSNR Mean Y:43.08 U:46.40 V:47.48 Avg:43.99 Global:42.84
x264 [info]: slice P:1950 Avg QP:21.96 size: 12722 PSNR Mean Y:44.06 U:47.22 V:48.21 Avg:44.96 Global:44.72
x264 [info]: slice B:1565 Avg QP:23.61 size: 4638 PSNR Mean Y:42.71 U:46.43 V:47.29 Avg:43.69 Global:43.50
x264 [info]: mb I I16..4: 21.8% 48.3% 29.9%
x264 [info]: mb P I16..4: 9.4% 23.6% 9.7% P16..4: 29.4% 18.8% 6.8% 0.0% 0.0% skip: 2.3%
x264 [info]: mb B I16..4: 2.7% 5.3% 1.4% B16..8: 36.0% 2.5% 5.4% direct:10.5% skip:36.2%
x264 [info]: 8x8 transform intra:54.8% inter:29.8%
x264 [info]: ref P 74.7% 17.6% 7.7%
x264 [info]: ref B 78.1% 16.9% 5.0%
x264 [info]: SSIM Mean Y:0.9760868
x264 [info]: PSNR Mean Y:43.446 U:46.852 V:47.788 Avg:44.382 Global:44.090 kb/s: 1790.99

encoded 3624 frames, 14.56 fps, 1791.23 kb/s

C:\Dokumente und Einstellungen\Administrator\Desktop>x264_mod.exe --crf 20.0 --level 3 --keyint 100 --min-keyint 1 --ref 3 --mixed-refs --no-fast-pskip --bframes 2 --b-pyramid --bime --weightb --filter -2,-2 --analyse p8x8,b8x8,i4x4,i8x8 --8x8dct --vbv-bufsize 1835 --vbv-maxrate 10000 --threads auto --thread-input --progress --no-dct-decimate --output "F:\Video\Black.Pearl.Sample.mkv" "F:\Video\Black.Pearl.Sample.avs" --aq-strength 0.6 --aq-sensitivity 10 >log.txt
avis [info]: 768x320 @ 23.98 fps (3624 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2
x264 [info]: slice I:109 Avg QP:19.72 size: 24527 PSNR Mean Y:45.67 U:47.92 V:49.07 Avg:46.40 Global:45.93
x264 [info]: slice P:1947 Avg QP:21.95 size: 12655 PSNR Mean Y:44.08 U:47.25 V:48.25 Avg:44.98 Global:44.75
x264 [info]: slice B:1568 Avg QP:23.63 size: 4635 PSNR Mean Y:42.70 U:46.46 V:47.33 Avg:43.69 Global:43.50
x264 [info]: mb I I16..4: 16.4% 40.6% 43.0%
x264 [info]: mb P I16..4: 9.4% 23.6% 9.5% P16..4: 29.0% 19.2% 6.9% 0.0% 0.0% skip: 2.3%
x264 [info]: mb B I16..4: 2.6% 5.3% 1.4% B16..8: 36.1% 2.6% 5.4% direct:10.4% skip:36.3%
x264 [info]: 8x8 transform intra:54.1% inter:29.6%
x264 [info]: ref P 74.6% 17.7% 7.7%
x264 [info]: ref B 78.1% 16.8% 5.1%
x264 [info]: SSIM Mean Y:0.9764853
x264 [info]: PSNR Mean Y:43.530 U:46.928 V:47.874 Avg:44.464 Global:44.189 kb/s: 1830.17

encoded 3624 frames, 14.46 fps, 1830.41 kb/s


seems the quality improvement is from the higher bitrate :(

[Sharktooth]

do not use AQ for PSNR/SSIM quality comparison. it screws everything.
also, there is no quality improvement since the bitrate produced at the same CRF is higher... hence the higher metrics results...

[J_Darnley]

I know you said that it probably wasn't worth doing more tests but I did and here are the results:
x264_original --crf 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-rdo --mixed-refs
--bime --trellis 1 --output "x264_test_original_fotr.264" "fotr\testraw.avs"
avis [info]: 704x416 @ 25.00 fps (33000 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:510 Avg QP:19.69 size: 22169 PSNR Mean Y:46.73 U:52.16 V:52.32 Avg:47.77 Global:47.14
x264 [info]: slice P:13765 Avg QP:21.63 size: 7193 PSNR Mean Y:44.97 U:53.15 V:53.30 Avg:46.08 Global:44.93
x264 [info]: slice B:18725 Avg QP:23.24 size: 1792 PSNR Mean Y:44.18 U:53.98 V:54.18 Avg:45.31 Global:44.31
x264 [info]: mb I I16..4: 43.4% 0.0% 56.6%
x264 [info]: mb P I16..4: 12.1% 0.0% 8.9% P16..4: 39.7% 11.9% 4.1% 0.2% 0.1% skip:22.9%
x264 [info]: mb B I16..4: 0.3% 0.0% 0.5% B16..8: 28.8% 1.1% 3.3% direct: 2.7% skip:63.1%
x264 [info]: ref P 73.2% 19.1% 7.7%
x264 [info]: ref B 85.6% 9.3% 5.1%
x264 [info]: SSIM Mean Y:0.9816765
x264 [info]: PSNR Mean Y:44.552 U:53.604 V:53.783 Avg:45.671 Global:44.593 kb/s:872.03

encoded 33000 frames, 8.66 fps, 872.13 kb/s

x264_mod --crf 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-rdo --mixed-refs
--bime --trellis 1 --output "x264_test_mod_fotr.264" "fotr\testraw.avs"
avis [info]: 704x416 @ 25.00 fps (33000 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:504 Avg QP:19.72 size: 22214 PSNR Mean Y:46.70 U:52.17 V:52.34 Avg:47.75 Global:47.12
x264 [info]: slice P:13711 Avg QP:21.63 size: 7191 PSNR Mean Y:44.98 U:53.18 V:53.33 Avg:46.09 Global:44.94
x264 [info]: slice B:18785 Avg QP:23.25 size: 1794 PSNR Mean Y:44.18 U:53.97 V:54.13 Avg:45.31 Global:44.30
x264 [info]: mb I I16..4: 43.4% 0.0% 56.6%
x264 [info]: mb P I16..4: 12.1% 0.0% 8.8% P16..4: 39.9% 12.1% 4.2% 0.2% 0.1% skip:22.6%
x264 [info]: mb B I16..4: 0.3% 0.0% 0.5% B16..8: 28.9% 1.2% 3.3% direct: 2.7% skip:63.1%
x264 [info]: ref P 73.0% 19.2% 7.8%
x264 [info]: ref B 85.6% 9.4% 5.1%
x264 [info]: SSIM Mean Y:0.9816771
x264 [info]: PSNR Mean Y:44.551 U:53.618 V:53.771 Avg:45.668 Global:44.589 kb/s:869.69

encoded 33000 frames, 8.67 fps, 869.79 kb/s

x264_original --qp 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-rdo --mixed-refs
--bime --trellis 1 --output "x264_test_original_fotr.264" "fotr\testraw.avs"
avis [info]: 704x416 @ 25.00 fps (33000 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:520 Avg QP:18.00 size: 26395 PSNR Mean Y:47.68 U:52.77 V:52.93 Avg:48.69 Global:48.34
x264 [info]: slice P:13757 Avg QP:21.00 size: 9027 PSNR Mean Y:45.46 U:53.40 V:53.51 Avg:46.56 Global:45.94
x264 [info]: slice B:18723 Avg QP:22.70 size: 2423 PSNR Mean Y:44.53 U:54.14 V:54.27 Avg:45.65 Global:45.05
x264 [info]: mb I I16..4: 40.5% 0.0% 59.5%
x264 [info]: mb P I16..4: 11.4% 0.0% 10.1% P16..4: 38.0% 12.5% 4.8% 0.3% 0.1% skip:22.7%
x264 [info]: mb B I16..4: 0.3% 0.0% 0.6% B16..8: 28.9% 1.3% 4.2% direct: 3.9% skip:60.8%
x264 [info]: ref P 73.8% 18.6% 7.6%
x264 [info]: ref B 86.2% 9.1% 4.7%
x264 [info]: SSIM Mean Y:0.9837183
x264 [info]: PSNR Mean Y:44.967 U:53.811 V:53.934 Avg:46.079 Global:45.439 kb/s:1110.71

encoded 33000 frames, 8.91 fps, 1110.81 kb/s

x264_mod --qp 21 --ref 3 --partitions "all" --me hex --subme 6 --bframes 3 --weightb --b-pyramid --b-rdo --mixed-refs
--bime --trellis 1 --output "x264_test_mod_fotr.264" "fotr\testraw.avs"
avis [info]: 704x416 @ 25.00 fps (33000 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 3DNow!
x264 [info]: slice I:515 Avg QP:18.00 size: 26515 PSNR Mean Y:47.68 U:52.74 V:52.90 Avg:48.68 Global:48.33
x264 [info]: slice P:13705 Avg QP:21.00 size: 9029 PSNR Mean Y:45.47 U:53.41 V:53.54 Avg:46.57 Global:45.95
x264 [info]: slice B:18780 Avg QP:22.70 size: 2432 PSNR Mean Y:44.54 U:54.10 V:54.24 Avg:45.66 Global:45.05
x264 [info]: mb I I16..4: 40.4% 0.0% 59.6%
x264 [info]: mb P I16..4: 11.4% 0.0% 10.1% P16..4: 38.2% 12.6% 4.8% 0.3% 0.1% skip:22.4%
x264 [info]: mb B I16..4: 0.3% 0.0% 0.6% B16..8: 28.9% 1.3% 4.2% direct: 3.9% skip:60.8%
x264 [info]: ref P 73.6% 18.7% 7.7%
x264 [info]: ref B 86.1% 9.2% 4.7%
x264 [info]: SSIM Mean Y:0.9837243
x264 [info]: PSNR Mean Y:44.976 U:53.791 V:53.928 Avg:46.085 Global:45.441 kb/s:1109.52

encoded 33000 frames, 8.76 fps, 1109.62 kb/s

If you are expecting this to be faster, did you use any optimisations? I was reading your AQ thread and in that you said that there wasn't any assembly in the AQ code and that is why is was slow.

[Dark Shikari]

If you are expecting this to be faster, did you use any optimisations? I was reading your AQ thread and in that you said that there wasn't any assembly in the AQ code and that is why is was slow.
All I did was replace the radius-2 hexagon search with a radius-4 and then a radius-2. This helped with DIA (where I replaced a radius-1 search with a radius-2-then-radius-1), but hex already was efficient enough that it wasn't particularly useful.