Current Patches, Where to get them, How they affect speed/output

[Ranguvar]

its slower... because it has to use slower functions, or a whole other function to re-align the data

Slower to compile? Or the compiled app is slower?

[Shinigami-Sama]

Slower to compile? Or the compiled app is slower?

...srsly?...
slower to run...

[kemuri-_9]

the stack align problem causes some sse2 assembly functions to be unusable or crashes will occur:
namely the

x264_deblock_v_luma_sse2
x264_deblock_h_luma_sse2
x264_deblock_v_luma_intra_sse2
x264_deblock_h_luma_intra_sse2

x264_pixel_avg_weight_16x16_sse2
x264_pixel_avg_weight_16x8_sse2
x264_pixel_avg_weight_8x16_sse2
x264_pixel_avg_weight_8x8_sse2
x264_pixel_avg_weight_8x4_sse2

functions, so a good number of them become fubar when compiled in MSVC (and apparently ICC from above statements)....
so it will encode slower without these.

However, if your computer is so old it doesn't even have sse2, you won't notice a thing :cool:

[ajp_anton]

I've been wondering whether anyone tried to do speed comparisons between a build compiled by GCC 3.4.5 and 4.3.*, since I got the 3.4.5 one to encode faster and would like to know whether anyone else experienced the same, or something different.All 4.x.x builds I've tried crashes on Win2003 Server x64. They work on all 32-bit XPs I have. How do I find out why?

[kemuri-_9]

get/generate a debug build and see what the error is when it crashes.

running through gdb would help too

[tobinaka]

does anybody know if this is correct and if it has any negative effect on quality if zone option is used currently?

VFR maniac said that when using zones, fgo offsets pbratio twice. It would seem that with zones usage, the relatively-recent options of changing parameters cause problems.

ICC has (afaik) the same stack alignment problem that MSVC does.

Seraphy, a Japanese x264 builder, has noticed some stack alignment problems and tried to solve them. I'm sorry I can't understand the details of them even if I read readme.txt in his patched builds (http://seraphy.fam.cx/~seraphy/program/x264/%E3%81%8A%E3%81%BE%E3%81%91/) because of my no background. VFR maniac provides Seraphy's stack alignment correction patch "x264_aligned.diff" in his experimental builds (http://www.esnips.com/web/VFRmaniac-Softwares).

I told those two to join the official supports but I don't know if they will or not. Anyway, for now, this diff may help the discussion. Try it.

[skystrife]

I just did some testing, and a gcc 4.3.1 fprofiled build with --extra-cflags="-march=core2" was not any faster (in my case) than a gcc 3.4.5 fprofiled build with --extra-cflags="-march=pentium2". (In fact, in this particular case, the gcc 3.4.5 build ran faster by 0.17 fps for the first pass, and by 0.05 fps on the second pass... I'd call that pretty negligible though.)

Source was 720x480, ran with pretty maxed settings (16 ref, 16 bframes, subme 7, trellis 2, partitions all, 8x8dct, me umh, merange 24, 1.2 aq-strength).

[Sharktooth]

march=pentium2 ... it's all you need. using core2 or other fancy stuff is not needed at all for x264 since all the heavy stuff is already optimized by hand...
ICL and MSVS = crap.
is it that so difficult to :search: ?!?!?!? it has already been discussed to death...

[Ranguvar]

ICC doesn't seem crap at all, it consistently beats out GCC in all the tests I've seen so far. And there are ways to de-cripple the binaries so that they are not significantly slower on AMD.

That aside, I'll now be using -march=pentium2, thanks...
Just to make sure, as I can't find any definitive info (the gcc man isn't specific enough), will march=pentium2 binaries run on any i686-family CPU with MMX support? And if so, can I safely use it as the only build I provide, since x264 only supports non-MMX CPUs with no asm anyways?

[kemuri-_9]

pentium2
Intel Pentium2 CPU based on PentiumPro core with MMX instruction set support.

In other words it's for all computers with at least MMX support.

even the MMX asm can not be used by tossing --asm 0 at x264, disabling all asm usage completely, so for an -march=i686 build (yes it's different than pentium2), they could use --asm 0 and still use x264 on their stone age computers

Edit:
like some of the others, ran and logging tests comparing -march and a couple gcc versions, will be able to post them soon when they finish.
so everyone can see the cold hard facts for themselves.

[akupenguin]

even the MMX asm can not be used by tossing --asm 0 at x264, disabling all asm usage completely
Never use --asm unless you're me :). It can only disable things that are valid on your cpu, or enable things that are valid but slow (like shuffles on core2), since the autodetection already chooses optimally.
And neither --asm nor autodetection can entirely disable mmx, since some functions are inlined.

[kemuri-_9]

ah ha, i had forgotten about the inlining!
well i use --asm when i profile, so i can force some on/off compared to the autodetect, to get a more overall profile,
since x264.h gives away the formula for what is what pretty well

My current test run is done, i just have the fprofiling save to log, easiest way of showing the diffs:

GCC 3.4.5 series:
k8 (http://kemuri9.net/dev/x264/x264_profile.gcc-3.4.5.k8.log), pentium2 (http://kemuri9.net/dev/x264/x264_profile.gcc-3.4.5.pentium2.log), prescott (http://kemuri9.net/dev/x264/x264_profile.gcc-3.4.5.prescott.log)

GCC 4.3.1 series:
amdfam10 (http://kemuri9.net/dev/x264/x264_profile.gcc-4.3.1.amdfam10.log), core2 (http://kemuri9.net/dev/x264/x264_profile.gcc-4.3.1.core2.log), k8 (http://kemuri9.net/dev/x264/x264_profile.gcc-4.3.1.k8.log), pentium2 (http://kemuri9.net/dev/x264/x264_profile.gcc-4.3.1.pentium2.log), prescott (http://kemuri9.net/dev/x264/x264_profile.gcc-4.3.1.prescott.log)

all binaries available for dl:
http://kemuri9.net/dev/x264/

overall examination:
use 3.4.x; use pentium2 or k8.
since these are on my AMD Phenom (amdfam10), it's fully understandable to see k8 > pentium2;
now if someone on a core2 would compare a -march=k8 to a -march=pentium2 and see what the results are, we can settle this completely.

maybe i'll try a gcc 4.4.0 suite and see if my computer explodes while trying to do it :sly:

[Sharktooth]

ICC doesn't seem crap at all, it consistently beats out GCC in all the tests I've seen so far. And there are ways to de-cripple the binaries so that they are not significantly slower on AMD.
since we're talking about x264... ICC=crap for building x264...
the reason is: https://forum.doom9.org/showthread.php?p=1174147#post1174147 coz of this: https://forum.doom9.org/showthread.php?p=1174093#post1174093

[Ranguvar]

Okay, it's crap at this specific time, for one specific app. That doesn't mean the whole thing is crap. Every app has its bugs.

[Sharktooth]

exactly... since we're in the x264 thread, ICC is crap and also it's uber crap for non-intel x86 CPUs in any case...

[Ranguvar]

it's uber crap for non-intel x86 CPUs in any case

http://www.server01.org/patch-AuthenticAMD.html

[kemuri-_9]

ICC can be considered, but until the supposed (as i have no knowledge on this outside of this thread) stack alignment problem it should remain on the back burner from lack of complete ASM support.

[Sharktooth]

http://www.server01.org/patch-AuthenticAMD.html
........
But you are warned that modifying, disassembling or reverse engineering the Intel C++ compiler goes against the Intel EULA (End User License Agreement). So do at your own risk.
illegal... also ICC is payware... while gcc is free (and faster for x264). so i dont know why you're bothering acclaiming ICC when it is a total crap for x264.

[Disabled]

illegal...
Thats just for patching the ICC itself. Patching your own compiled binaries would be legal for sure.
also ICL is payware
ICC for Linux is free for non commercial projects:
http://www.intel.com/cd/software/products/asmo-na/eng/download/eval/219771.htm

Your other arguments still hold though.

[Sharktooth]

only for linux and for non commercial use.
also patching binaries for non-intel CPUs... every time... is a painfull process...

[Disabled]

I wrote Linux only even in my original post and I never knew x264 was a commercial product.
And patching binaries is painfull... like writing one extra line into the makefile?
If ICC would generate faster x264 binaries I'm sure these would be the mainly used binaries even on windows. But as it doesn't - I agree with you - this discussion is useless.

[bob0r]

x264.949.modified.01.exe (http://files.x264.nl/x264.949.modified.01.exe)
libx264-61.949.modified.01.dll (http://files.x264.nl/libx264-61.949.modified.01.dll)

x264-psyrd-0.6.diff (on by default, adjust with: --psy-rd) <-- patch using -l parameter (ignore whitespaces): patch -p1 -l < x264-psyrd-0.6.diff

x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

Not using x264.new.bframes.decision.04.diff (highly experimental, enabled with: --b-adapt 2) because it fails --no-b-adapt

[Sharktooth]

I wrote Linux only even in my original post and I never knew x264 was a commercial product.
And patching binaries is painfull... like writing one extra line into the makefile?
If ICC would generate faster x264 binaries I'm sure these would be the mainly used binaries even on windows. But as it doesn't - I agree with you - this discussion is useless.
x264 can be licensed and used as a commercial product...
however the point is quite different. IIRC ICC for linux wont produce binaries for windows... so it's quite useless for windows users also, icc for windows is a fully commercial product that will cost $$$. the evaluation version is fully working but it's time limited...
on the other side we have gcc, that's free/OSS, multiplatform and produces faster x264 binaries.
It's quite obvious what to choose...

[kemuri-_9]

Not using x264.new.bframes.decision.04.diff (highly experimental, enabled with: --b-adapt 2) because it fails --no-b-adapt

i worked on it a bit and got it to where it will accept --no-b-adapt properly
x264_new_bframe_decision_04.5.diff (http://kemuri9.net/dev/x264/patches/x264_new_bframe_decision_04.5.diff)
Would like to have DS and/or akupenguin look at it as it could probably be simplified
(I just couldn't manage to simplify it as i had tried in a few ways)

[skystrife]

x264.949.modified.exe (http://www.mediafire.com/?mknq4m3wtm9) - Alternate Download (http://skystrife.com/x264/x264.949.modified.exe)
libx264-61.949.modified.dll (http://www.mediafire.com/?aqaauaambww) - Alternate Download (http://skystrife.com/x264/libx264-61.949.modified.dll)

Patches used:

x264_psy_rdo_0.6.diff
x264_new_bframe_decision_04.5.diff <-- This patch is highly experimental, only enabled with --b-adapt 2. The --no-b-adapt parameter now works.
x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

gcc 3.4.5 fprofiled build.

[elguaxo]

thank you both! :)

[tobinaka]

Thank you Kemuri_09! I've just failed my encoding test through inattention of using --no-b-adapt with new B-frame decision patch.
And, thank you skystrife! Your continual posts help me follow the specific changes of patches in the past.
Of course, all the developers, too!

[Sagekilla]

Just my 2 cents, I don't know if it's done already but --no-b-aapt should be aliased to --b-adapt 0 ;)

[J_Darnley]

i worked on it a bit and got it to where it will accept --no-b-adapt properly
x264_new_bframe_decision_04.5.diff (http://kemuri9.net/dev/x264/patches/x264_new_bframe_decision_04.5.diff)
Would like to have DS and/or akupenguin look at it as it could probably be simplified
(I just couldn't manage to simplify it as i had tried in a few ways)

Another way to fix that --no-b-adapt problem is with the following changes to x264.c (pruned):
#define OPT_PROGRESS 261
#define OPT_VISUALIZE 262
#define OPT_LONGHELP 263
+#define OPT_NO_B_ADAPT 264
Add a one-of-these so that you can get a special case when parsing arguments/options

+ { "b-adapt", required_argument, NULL, 0 },
+ { "no-b-adapt", no_argument, NULL, OPT_NO_B_ADAPT },
Add the case into the option structure

param->i_scenecut_threshold = -1;
- param->b_bframe_adaptive = 0;
+ param->i_bframe_adaptive = X264_B_ADAPT_NONE;
break;
+ case OPT_NO_B_ADAPT:
+ param->i_bframe_adaptive = X264_B_ADAPT_NONE;
+ break;
Add the case and what to do.

Full diff based on the 03 patch (the one that has the X264_BUILD change): http://pastebin.com/m290ab8ae
I called it x264_newBframe_0.31.diff here but you guys can argue over that.

Since the --no-b-adapt option is removed from the help, there is always the option of just removing that option and having x264 error with "unknown option -- no-b-adapt". I guess that is not desired because that is not really different to the current behavior except that it now has the scary words "invalid argument: no-b-adapt = (null)" to scare people away.

Oh, I also have a white space corrected psyRDO patch that avoids using the --ignore-whitespace (-l for you terse people) if anyone feels there is a need for it.

[Ranguvar]

x264 can be licensed and used as a commercial product...
however the point is quite different. IIRC ICC for linux wont produce binaries for windows... so it's quite useless for windows users also, icc for windows is a fully commercial product that will cost $$$. the evaluation version is fully working but it's time limited...
on the other side we have gcc, that's free/OSS, multiplatform and produces faster x264 binaries.
It's quite obvious what to choose...

I like supporting open source projects, and GCC is no exception. I never said ICC was better than GCC -- just faster, usually. I also never said ICC should replace GCC. I just wanted to note that ICC is far from crap. Everything has its limitations. GCC tends to be slower, ICC gets the short end of the stick in almost everything else. The entire idea was just an odd thought anyways. If, somehow, x264 started working with ICC at several times the speed (obviously just an analogy) then I think even you would be anxious to test it out... a 'free' speedup for as much as ICC has done with other apps makes me very anxious to check it out with x264.

Again, this was all just a whim :rolleyes:

@kemuri-_9: Thanks very much for the patch! :)

[kemuri-_9]

Since the --no-b-adapt option is removed from the help, there is always the option of just removing that option and having x264 error with "unknown option -- no-b-adapt". I guess that is not desired because that is not really different to the current behavior except that it now has the scary words "invalid argument: no-b-adapt = (null)" to scare people away.

It's wanted for backwards compatibility for those who are using it with --no-b-adapt still
a patch can provide new features but should not ruin compatibility with unpatched builds - removing still usable options.

I figured out what i was trying to do after looking over it a bit more, updated it on my site - super simplified now, even more than how you were doing it.

the critical change is now
- p->b_bframe_adaptive = atobool(value);
+ {
+ p->i_bframe_adaptive = atobool(value);
+ if( !b_error ) return 0;
+ b_error = 0;
+ p->i_bframe_adaptive = atoi(value);
+ }

basically says
1. try and accept this as a boolean (as it originally was - maintaining backwards compatibility)
2. if it works accept the value and leave/stop.
3. if it breaks, reset the error (so x264 doesn't cry and /wrists on us)
4. accept as an int (gaining the new --b-adapt <int> functionality)

Edit: I've been editing the pissfire out of this post :<

[martino]

now if someone on a core2 would compare a -march=k8 to a -march=pentium2 and see what the results are, we can settle this completely.
Well, I was hoping that someone other than the lazy me would have done it, but seems like my hopes were short-lived, so here goes one test that I did with all the binaries that you put up.

http://martino.pastebin.com/f7caae7e8
(the last one, x264_mine, was just my own build, without any flags specified)

This was on a C2D E6750 @ 3.2 GHz.


UPDATE: http://martino.pastebin.com/f55e48a4a (5000 frames of the same clip)

[Adub]

That's interesting, considering yours is the fastest on the 1st pass. Not quite so, on the second, but close enough.

So, no flags beats flags? o_0

[akupenguin]

http://martino.pastebin.com/f55e48a4a
Fail: one run of each is never statistically significant. And I seriously doubt your times are repeatable to within 0.1%, which is what they'd need to be to get the same ordering after averaging a bunch of trials.
Also, x264_mine produced different results (different frametype even), so it's not the same code. Either that or gcc screwed up, but given the number of patches around I think the former is more likely.

[martino]

Oh yes indeed, I forgot that my one didn't have the bframe patch... :/

[Sharktooth]

That's interesting, considering yours is the fastest on the 1st pass. Not quite so, on the second, but close enough.

So, no flags beats flags? o_0
not properly... his build had no b-frame patch...
however, with x264 optimization flags are almost useless, so are the "core2" or "k8" optimized builds... so are the ICC builds... etc.
all the hard stuff is optimized by and in asm, so the compiler wont make any noticeable difference.

[Adub]

Ah, that's right. I had forgotten that he excluded the b-frames patch.

[Ranguvar]

bob0r, skystrife, may I ask how you are compiling the DLL? I know how to do it, but x264 seems to need some modifications (http://avidemux.org/admForum/viewtopic.php?pid=28804) before the DLL will work in AviDemux. Is that exactly what you have done?

[LoRd_MuldeR]

bob0r, skystrife, may I ask how you are compiling the DLL? I know how to do it, but x264 seems to need some modifications (http://avidemux.org/admForum/viewtopic.php?pid=28804) before the DLL will work in AviDemux. Is that exactly what you have done?

gcc 3.4.5 fprofiled build.

They use GCC 3.4.5/3.4.6, hence it can not work with Avidemux. You need to use GCC 4.x.x plus the "DLL Aligment-Fix" patch in order to get a working x264 DLL.
Also any patches other than "Psy RDO" will break compatibility to Avidemux, except for patches that don't change the API. Without "Psy RDO" it won't work either.
Finally the new B-Frames patch causes weird things in Avidemux: The last few frames are lost in CRF and 2-Pass will crash at the end of the second pass...

I provide Avidemux compatible builds here regularly:
http://forum.doom9.org/showthread.php?p=1175312#post1175312

[Ranguvar]

Alright, thanks for the info.

Is there a diff for the alignment fix available? That way, I can at least create working DLLs for other apps that use it.

[LoRd_MuldeR]

Alright, thanks for the info.

Is there a diff for the alignment fix available? That way, I can at least create working DLLs for other apps that use it.

In fact all you need to do is adding one single line with __attribute__((force_align_arg_pointer)) right before the "x264_encoder_encode" function :)

Here is the diff, which I currently use:
diff --git a/encoder/encoder.c b/encoder/encoder.c
index cffaeeb..34cbe8c 100644
--- a/encoder/encoder.c
+++ b/encoder/encoder.c
@@ -1222,6 +1222,7 @@ static int x264_slices_write( x264_t *h )
* B 5 2*4
* B 6 2*5
****************************************************************************/
+__attribute__((force_align_arg_pointer))
int x264_encoder_encode( x264_t *h,
x264_nal_t **pp_nal, int *pi_nal,
x264_picture_t *pic_in,

It shows a warning while patching, but does work anyway ^^

[Ranguvar]

Thanks :) I knew what line to add, but I was hoping to get a diff, since I have a script set up to compile x264, and just doing a diff patch is simple :)

[Ranguvar]

http://sites.google.com/site/ranguvar13/x264-builds

Direct download (http://sites.google.com/site/ranguvar13/x264-builds/rang_x264_r0949.7z?attredirects=0), Mirrors (http://www.uploadjockey.com/download/2880965/rang_x264_r0949_1.7z)

x264 r949 from Git (patched).

Open this archive with the free, multi-platform tool 7-Zip or p7zip. Compressed with LZMA.
The src folder contains the patched source code.
The bin folder contains a binary executable for all i686-compatible Windows PCs with MMX support,
and a DLL for usage in some applications (with the notable exception of AviDemux).

Official source and vanilla builds: http://x264.nl/
Changelog: http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog
Patches and discussion at Doom9's forum: http://forum.doom9.org/


Applied patches (included, unchanged, in the patches folder):

patch -p1 < ../x264diffs/x264_dll_alignment_fix.01.diff
patch < ../x264diffs/x264.progress.indication.01.diff
patch -p1 < ../x264diffs/x264_hrd_pulldown.09_interlace.diff
patch -p1 -l < ../x264diffs/x264-psyrd-0.6.diff
patch -p1 < ../x264diffs/x264_new_bframe_decision_04.5.diff


Compiled by Ranguvar on August 26th, 2008, with GCC 4.3.1 on Windows XP Professional x64 SP2.

CLI used: ./configure --enable-shared --extra-cflags="-march=pentium2 -pipe" && make fprofiled VIDS="../enctests/deadline_cif.y4m"

Platform: X86
System: MINGW
asm: yes
avis input: yes
mp4 output: yes
pthread: yes
gtk: no
debug: no
gprof: no
PIC: no
shared: yes
visualize: no

[kemuri-_9]

however, with x264 optimization flags are almost useless, so are the "core2" or "k8" optimized builds... so are the ICC builds... etc.

actually if you looked at the logs i generated from the profiling, the k8 was rather consistently faster than the pentium2 on my AMD cpu.
so it looks like
intel line -> -march=pentium
amd line -> -march=k8
(boy wouldn't that normally be obvious?)

but the core2 side of things seems to need a bit more testing between the k8 and pentium2.

I don't have a core2, so would be asking someone else these tasks.
and my logs also consistently showed that 3.4.x was faster than 4.3.1,
so only if you need that dll would you choose 4.3.1 for compiling it, should still use 3.4.x for the binary.

[Sharktooth]

but was the speed gain worth it?
i mean if you were to release a generic build, what would you choose?

[kemuri-_9]

yeah for an overall generic build i would probably do a pentium2, but if you can offer both, then do that too...

couldn't you write some C# (in megui... it's C# right?) to check the Windows environment vars to see what the processor is and give them one matching their CPU?

i mean, there's gotta be something that can grab and check the PROCESSOR_IDENTIFIER variable and see if it contains 'AMD' or 'Intel'

should try a -march=k6 build and see how it does, since it's practically the AMD version of the p2. guess i'll get on that later.

[Ranguvar]

I'm gonna guess that if march=k8 is faster on AMD CPU's, it's because of the 3dNOW! and 3dNOW!+.

So, march=athlon should be the best choice for AMD, as it works with both of those but doesn't require much else :)

[LoRd_MuldeR]

couldn't you write some C# (in megui... it's C# right?) to check the Windows environment vars to see what the processor is and give them one matching their CPU?

You could simply run "cpuinfo.exe" (download (http://oss.netfarm.it/mplayer-win32.php)) and parse the console output...

[skystrife]

Just a note for all the patchers:

.:8:44.04pm:. ( skystrife ) psyrd haet commit
.:8:44.12pm:. ( Dark_Shikari ) skystrife: it does.
.:8:44.27pm:. ( Dark_Shikari ) I will have to do a significant but trivial rewrite of psy-RD

x264_psyrdo_0.6.diff won't patch with r950.

[skystrife]

x264.950.modified.exe (http://www.mediafire.com/?vecakajaaqy) - Alternate Download (http://skystrife.com/x264/x264.950.modified.exe)
(discontinuing dll unless someone was using it for something other than avidemux and had it working as I was building it; I'm not shifting from 3.4.5)

Patches used:

x264_psy_rdo_0.6_r950.diff <-- Updated to patch with r950 (check psyRDO thread).
x264_new_bframe_decision_04.5.diff <-- This patch is highly experimental, only enabled with --b-adapt 2. The --no-b-adapt parameter now works. Updated version.
x264_hrd_pulldown.09_interlace.diff
x264.progress.indication.01.diff

gcc 3.4.5 fprofiled build.