Current Patches, Where to get them, How they affect speed/output

[rack04]

x264 core:72 r1232M x86

Download (http://www.mediafire.com/?sharekey=66628d7bf878e80e7432d3c9683f450ae04e75f6e8ebb871)

Built by rack04 on August 25, 2009, 9:20:51 AM CST
$ gcc --version
gcc.exe (GCC) 4.4.1 (x86.core2.Komisar)
$ ./configure --extra-cflags="-march=core2"
Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
debug: no
gprof: no
PIC: no
shared: no
visualize: no
$ make fprofiled VIDS="bigbuckbunny.avs LosslessTouhou.avs riverbed.1920x1080.yuv"

Patched with:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)

[JEEB]

x264 r1235 64bit unpatched:
download (http://jeeb.fiveforty.jp/x264/revision1235/x264.exe) ; hash (http://jeeb.fiveforty.jp/x264/revision1235/x264.md5)

built on Aug 27 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, otherwise defaults

________________________________________________________________________________

x264 r1235 32bit
download (http://jeeb.fiveforty.jp/x264/1235/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1235/relnotes.txt)

built on Aug 27 2009, gcc: 4.3.4 20090220 (prerelease) (x32.generic.Komisar)
fprofiled, -march=i686


x264 r1235 64bit
download (http://jeeb.fiveforty.jp/x264/1235_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1235_x64/relnotes.txt)

built on Aug 27 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


patched with:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)


A bit longer list this time since I just merged two changelogs >:3

[rack04]

x264 r1235M x86

Download (http://www.mediafire.com/?sharekey=3dd461e796995ebe1f8e0fff488e27e0e04e75f6e8ebb871)

Built by rack04 on August 27, 2009, 9:14:16 AM CST
gcc --version
gcc.exe (GCC) 4.3.4 20090526 (prerelease) (x86.core2.Komisar)
-march=core2
make fprofiled

Patched with:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)

[techouse]

x264_x64_r1235_unpatched (http://techouse.project357.com/builds/revision1235/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1235/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1235_techouse (http://techouse.project357.com/builds/x264_x86_r1235_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1235_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1235_techouse (http://techouse.project357.com/builds/x264_x64_r1235_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1235_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff

[woah!]

x264_x64_r1235_unpatched (http://techouse.project357.com/builds/revision1235/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1235/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1235_techouse (http://techouse.project357.com/builds/x264_x86_r1235_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1235_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1235_techouse (http://techouse.project357.com/builds/x264_x64_r1235_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1235_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff

i get this with you patched x86 version: unrecognised option `--nal-hrd'

rack04 version above you with same patches works ok...

[juGGaKNot]

32/64 bit ? are you sure you have the patched one ?

--longhelp

[woah!]

i said x86 patched version so 32bit patched...

rack04 has these options: --nal-hrd --pulldown

techouse's doesnt, and thats fine to know, so i will use another version which does.

[techouse]

I'm pretty sure I patched it, but I'll recheck my building scripts and report back later.

EDIT: You're right, I had a typo in my script and cause of that it didn't find the diff file. Thanx for noticing it :) ONLY my patched r1232 and r1235 builds are affected. I'm fixing/rebuilding r1235 now.

[woah!]

np ... i am guessing anyone who does bluray stuff would have said something soon enough too :)

[techouse]

Fixed. :)

[moviefan]

What I have been wondering about is:

--march=i686/core2/... - What is the difference?
--fprofiled: What is this for?

[kemuri-_9]

--march=i686/core2/... - What is the difference?

march=i686 is used by default to have gcc use the cmov instruction which speeds up some of the c code a bit here and there.
core2 does just about nothing; it's mostly used to have gcc schedule tasks in way that's optimal for core2 as the instructions barely differ from that of march=i686.

--fprofiled: What is this for?

fprofiled is a scheme that you compile a program with that capability, execute it, and then recompile it optimizing based on which code paths were executed the most/least from your executions.
other compilers can have different names for such a feature, but gcc's is called 'fprofile'.

[LoRd_MuldeR]

Profiling means that the application is analyzed while it's executing and processing "real" input data. The info collected during the profiling process can be used by the compiler to enable additional optimizations (or to enable the optimizations in parts of the program where they are really needed). Compiling x264 with fprofiled takes much longer than without, because the binary is compiled twice (once before the profiling and again after the profiling is completed). Also you need to provide a sample video clip that will be encoded several times (with different settings in order to cover all code paths) during the profiling process...

[moviefan]

OK, very interesting! Thanks for the explanations!

[JEEB]

x264 r1239 64bit unpatched:
download (http://jeeb.fiveforty.jp/x264/revision1239/x264.exe) ; hash (http://jeeb.fiveforty.jp/x264/revision1239/x264.md5)

built on Aug 29 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, otherwise defaults

________________________________________________________________________________

x264 r1239 32bit
download (http://jeeb.fiveforty.jp/x264/1239/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1239/relnotes.txt)

built on Aug 29 2009, gcc: 4.3.4 20090220 (prerelease) (x32.generic.Komisar)
fprofiled, -march=i686


x264 r1239 64bit
download (http://jeeb.fiveforty.jp/x264/1239_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1239_x64/relnotes.txt)

built on Aug 29 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


patched with:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)

[rack04]

x264 r1239M x86

Download (http://www.mediafire.com/?sharekey=23986c34d519238b2fb2ca15d7ea42d9e04e75f6e8ebb871)

Built by rack04 on August 28, 2009, 10:13:17 PM CST
GCC 4.3.4 20090526 (prerelease) (x86.core2.Komisar)
-march=core2
fprofiled

Patched with:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)

[buzzqw]

for any interest

http://www.64k.it/andres/data/x264/x264.1239.x86.tar.gz

gcc 4.4.1

$ ./configure
Platform: X86
System: LINUX
asm: yes
avis input: no
mp4 output: yes
pthread: yes
matroska: yes

Patched with: x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)

BHH

[rack04]

x264 r1240M x86

Download (http://www.megaupload.com/?d=TO4FKXQ9)

Built by rack04 on August 29, 2009, 1:39:08 PM CST
GCC 4.3.4 20090526 (prerelease) (x86.core2.Komisar)
-march=core2
fprofiled

Patched with:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)

[JEEB]

x264 r1240 64bit unpatched:
download (http://jeeb.fiveforty.jp/x264/revision1240/x264.exe) ; hash (http://jeeb.fiveforty.jp/x264/revision1240/x264.md5)

built on Aug 30 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, otherwise defaults

________________________________________________________________________________

x264 r1240 32bit
download (http://jeeb.fiveforty.jp/x264/1240/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1240/relnotes.txt)

built on Aug 30 2009, gcc: 4.3.4 20090220 (prerelease) (x32.generic.Komisar)
fprofiled, -march=i686


x264 r1240 64bit
download (http://jeeb.fiveforty.jp/x264/1240_x64/x264.exe) ; release notes (http://jeeb.fiveforty.jp/x264/1240_x64/relnotes.txt)

built on Aug 30 2009, gcc: 4.3.4 20090220 (prerelease) (x64.generic.Komisar)
fprofiled, -march=core2


patched with:

x264_win_zone_parse_fix_06.diff (http://kemuri9.net/dev/x264/patches/x264_win_zone_parse_fix_06.diff)
x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)


________________________________________________________________________________

And something deeply experimental.

x264 r1240 with MixAQ and OreAQ patches: download (http://jeeb.fiveforty.jp/x264/1240/x264_aqpatches.7z)

built on Aug 30 2009, gcc: 4.3.4 20090220 (prerelease)
fprofiled, default CPU flags, --extra-ldflags="-lz"


Patched with the patches:

The stuff applied on the patched builds
x264_OreAQ12_r1240.diff (http://jeeb.pastebin.ca/1547857)
x264_MixAQ_r1240.diff
(http://jeeb.pastebin.ca/1547858)


If you plan on building, the file 'AQDebugLog.h' has to be in the root of your x264 codebase:

for OreAQ (http://jeeb.pastebin.ca/1547807)
for MixAQ (http://jeeb.pastebin.ca/1547805)


These patches are made by Seraphy and Muken AKA VFR Maniac, each developing their own versions and VFR Maniac updating the standard patches for newer revisions. And yes, the patches seem to need zlib. Tobinaka has written some articles on these AQ modes on Doom9, so please search for his posts if you have any basic questions on the patches, and please do report any findings if you decide to test these builds.

Also, I took these patches in this time because of the large amount of interest they have gathered in Japan overall. Personally I didn't dislike what an older MixAQ build did with the blackpearl sequence on 500kbps, but otherwise I'm not saying any of these patches greatly increases quality or something like that. Isn't it fun to have something not-so-usual on your hands?

[VFR maniac]

Hi, JEEB.
I updated OreAQ qp_adj calculation according to rev1236.

Please check below.
http://seraphy.fam.cx/~seraphy/cgi-bin/cbbs.cgi?mode=one&namber=2269

Edit: Woops! I passed by you.

[JEEB]

Yes, thank you. Didn't see that before my builds completed and I had revisited your site after writing my post. I had gotten info on you not updating the patch for 1239 so I kind of skipped the pre-checking phase, putting my bad(?) edit into the first, pre-edit builds. Oh well, won't happen again.

[juGGaKNot]

http://www.esnips.com/nsdoc/7a9f4e8d-75cf-4258-89a6-0c5725bb5534/?action=forceDL

what about this ? usable ?

[JEEB]

Since that URL keeps redirecting me to the login screen, I can't tell for sure, but I guess that's a link for VFR_Maniac's own builds. They contain much more patches, I have only added the most basic ones into mine that were available. Also, if that's the OreAQ_frame_edge one I guess that's something even newer. Feel free to test whichever build works for you and how they work.

[juGGaKNot]

http://www.esnips.com/web/x264experimental

a weightp diff

[techouse]

x264_x64_r1240_unpatched (http://techouse.project357.com/builds/revision1240/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1240/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1240_techouse (http://techouse.project357.com/builds/x264_x86_r1240_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1240_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1240_techouse (http://techouse.project357.com/builds/x264_x64_r1240_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1240_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff

[JEEB]

For weightp I would recommend the usage of the git repository rather than random patches since I do not know of any official diff releases (no bad intent to VFR maniac, but since A) the weightp repository is up-to-date atm and because B) it has the newest changes I'd consider it a better option than any patches floating around).

Of course goes without saying that the weightp functionality is heavily under construction at the moment, so I'd rather not make any larger perceptions by looking at the current state of the repo / any patch that might be based on an older revision from the same repo. To have interest in that functionality is completely normal, though.

[G_M_C]

For weightp I would recommend the usage of the git repository rather than random patches since I do not know of any official diff releases (no bad intent to VFR maniac, but since A) the weightp repository is up-to-date atm and because B) it has the newest changes I'd consider it a better option than any patches floating around).

Of course goes without saying that the weightp functionality is heavily under construction at the moment, so I'd rather not make any larger perceptions by looking at the current state of the repo / any patch that might be based on an older revision from the same repo. To have interest in that functionality is completely normal, though.

Isn't is just better to wait untill Dark Shikari gives the OK on this ? He is the GSoC mentor on this project AFAIK, so he can judge the state of this best. He also has the better overview on the whole codebase, maybe the weightp patch breaks things we cannot oversee or or simply dont know about.

But speaking personally; I cant wait to see the results of weightp either ;)

[JEEB]

Yes, I was trying to hint at just that while remaining at the pose of "But if you wish as much as to apply something like that, at least use the newest revision officially available."

[kemuri-_9]

from the current progression of talk in the dev channel
the 3 patches mentioned below will seemingly be committed in the following order:

slicing support, threaded slicetype, weightp
each one breaks a small or decent bit of something for what's following it, so it's going to be a bit longer before weightp will see the main repository

[Razorholt]

x264_x64_r1240_unpatched (http://techouse.project357.com/builds/revision1240/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1240/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1240_techouse (http://techouse.project357.com/builds/x264_x86_r1240_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1240_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1240_techouse (http://techouse.project357.com/builds/x264_x64_r1240_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1240_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_hrd_pd_interlace.16.fix.1232.diff (http://pastebin.org/11898)
x264_win_zone_parse_fix_06.diff

Hi Techouse -

Just so you know, that build (x86) makes x264.exe crash on the second pass - at least for me. When I try earlier versions or any other builds from other folks they are running fine. :o

Thanks,
- Dan

UPDATE: It works now, but after I changed the SetMTmode() setting from "1,2" to "2". I don't know what happened here.

[G_M_C]

from the current progression of talk in the dev channel
the 3 patches mentioned below will seemingly be committed in the following order:

slicing support, threaded slicetype, weightp
each one breaks a small or decent bit of something for what's following it, so it's going to be a bit longer before weightp will see the main repository

I suspected as much, as i wrote before.

Isn't is just better to wait untill Dark Shikari gives the OK on this ? He is the GSoC mentor on this project AFAIK, so he can judge the state of this best. He also has the better overview on the whole codebase, maybe the weightp patch breaks things we cannot oversee or or simply dont know about.
[...]


But all three patches/additions you mention will provide big steps forward. x264 is taking great steps in growing towards it's full potential it seems. I think that these are exiting times to be around seeing this happening, and actually be able to take advantage of x264 while its becoming this good :)

[wyti]

sorry for the noob question, but what is threaded slicetype and what will it do quality / speed wise ?

[JEEB]

Threaded slicetype can be dumb'ified as "multithreaded b-adapt 2" I guess, although writing it like this _is_ cutting around corners and isn't the right way.

[shon3i]

can somebody provide new build since slices added? aslo with HRD patch for full BD compilancy :)

[imk]

r1243M built with ICC.

Windows:
x264-r1243M-imk-win.7z (http://imk.cx/pc/x264/x264-r1243M-imk-win.7z)
win_build_info.txt (http://imk.cx/pc/x264/win_build_info.txt)

Patches used:
x264_icc_08_win.diff
x264_win_zone_parse_fix_06.diff


The HRD patch will need updated. It fails to patch currently. I'll rebuild later when the patch gets updated.

[kemuri-_9]

Threaded slicetype can be dumb'ified as "multithreaded b-adapt 2" I guess, although writing it like this _is_ cutting around corners and isn't the right way.

no that 'dumbification' is actually inappropriate to how it works.

the patch adds a new thread that is persistent throughout the entire code that is dedicated to performing slicetype decisions only.
this does not multi-thread slicetype, but gives it its own HIGH priority dedicated thread just for it.
this would be a stepping stone to multi-threading it.

on average the speedup from this is 5-10%
but from testing some fairly bizarre combinations of settings, the highest i've seen the encoding rate increase was ~56%

[JEEB]

I gave it a slight try (http://pastebin.ca/1549778) (NAL HRD - x264_hrd_pd_interlace.16.fix.1243.test.diff)

It sure as hell builds, looks quite close to what it was and encodes the fprofile things, but I have no idea if it does its job correctly with the newer changes. Also I'll have to see how much modification the open gop patch will need... (once again I have no idea if it will work correctly even if I get to change it correctly).

[techouse]

Yea, I'd also rather wait for the author of the HRD patch to take a look at it before building anything...

[JEEB]

Indeed, I got the open gop patch to build as well with a modified patch, but since the internals have changed this (http://pastebin.ca/1549828) (done from a NAL_HRD patched one) as well can only be viewed as nothing more but a little plaything before the maker of these patches comes to the end of modifying them for the current revision with the added slices. :)

EDIT：

Here's the unpatched build for the time being (64bit, r1243, fprofiled, default configure):
download (http://jeeb.fiveforty.jp/x264/revision1243/x264.exe) ; md5 hash (http://jeeb.fiveforty.jp/x264/revision1243/x264.md5)

[techouse]

x264_x64_r1243_unpatched (http://techouse.project357.com/builds/revision1243/x264.exe) | MD5 (http://techouse.project357.com/builds/revision1243/x264.md5)
GCC 4.4.1 20090803 (x64.core2.Komisar), unpatched, generic, fprofiled

________________________________________________________________________________

x264_x86_r1243_techouse (http://techouse.project357.com/builds/x264_x86_r1243_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x86_r1243_techouse.txt)
GCC 4.4.1 20090803 (x86.core2.Komisar), fprofiled, -march=core2

x264_x64_r1243_techouse (http://techouse.project357.com/builds/x264_x64_r1243_techouse.7z) | INFO (http://techouse.project357.com/nfo/x264_x64_r1243_techouse.txt)
GCC 4.4.1 20090803 (x86_64.core2.Komisar), fprofiled, -march=core2

Patches used:

x264_win_zone_parse_fix_06.diff

[VFR maniac]

x264_x86_r1243_with_nal-hrd.rar (http://www.esnips.com/doc/bd9f9d67-2923-4b00-9343-966f04ee7297/x264_x86_r1243_with_nal-hrd)
Please use slices + nal-hrd at your own risk.

[shon3i]

why at risk?

[JEEB]

As the comment on the file says, he thinks he has fixed the patch in its current form for building and has built a binary, but he is not sure if it works correctly with slices on. VFR Maniac has much more knowledge than I do therefore if you're going to try, you could do it with his build or by building with his patch instead of what I quickly threw together last night.

In other words, so far every mod. patch is just something that makes the current revision of the patch to work with the current revision of x264. Actual computation and modification to work 100% correctly with slices hasn't been taken into account. For that we wait for the next revision that might come up sooner or later.

[G_M_C]

I vote for that we wait on Tharald fixing this (his ?) patch ;)

[kemuri-_9]

I vote for that we wait on Tharald fixing this (his ?) patch ;)

It is Trahald's, but it seems that him and Dark_Shikari are going about getting MMCO into the main repository to fix the issue with x264's b-pyramid.
so the nal-hrd patch is seemingly on the back burner for a bit.

[moviefan]

Can anyone tell, if the b-pyramid fix, that is being worked on, is going to be Blu-ray compliant afterwards? I ran about some comments about x264's b-pyramid being different from the Blu-ray spec's way of doing b-pyramid. (maybe a bit unprofessionally described, but I don't know about this stuff in detail)

[microchip8]

@moviefan

are you referring to the level of hierarchical b-pyramid implementations or something else? Also, as it now stands, x264 will in the future enforce strictdpb which will be builtin (no option to disable it) in order to comply as much as possible when using the b-pyramid option

[G_M_C]

It is Trahald's, but it seems that him and Dark_Shikari are going about getting MMCO into the main repository to fix the issue with x264's b-pyramid.
so the nal-hrd patch is seemingly on the back burner for a bit.

No matter, i'll wait for his fix rather than we (or anyone else) fixing it, and possibly unintentionally breaking something else :)

[LoRd_MuldeR]

Can anyone tell, if the b-pyramid fix, that is being worked on, is going to be Blu-ray compliant afterwards? I ran about some comments about x264's b-pyramid being different from the Blu-ray spec's way of doing b-pyramid. (maybe a bit unprofessionally described, but I don't know about this stuff in detail)

As far as I know, there's no such thing as "B-Pyramid" in the H.264 standard or in the BD specs.

B-Pyramid is just x264's implementation of "hierarchical B-Frames", but there is no limitation in the standard that says that the hierarchy must be a pyramid.

However there seems to be a problem with the way x264 manages the buffered reference frames, which can break Level compliance...

[moviefan]

Yep, something like that... As I said, I don't know details, but as you are talking about breaking the decoding buffer limit, this is something I can recall. So it would be good if b-pyramids do not break the buffer limits as I think they can be beneficial for compression sometimes.