x264 asm issues when using with ffmpeg or mencoder

[pspvideo9]

When we try to encode x264 video using Celtic Druid’s binaries, we get the following results:

Celtic Druid’s x264 builds (standalone)
Intel Core Duo: Works fine
Athlon XP: Works fine

Celtic Druid’s FFmpeg builds
Intel Core Duo: Causes FFmpeg to Crash
Athlon XP: Works Fine

Celtic Druid’s Mencoder builds
Intel Core Duo: Causes Mencoder Crash
Athlon XP: Works Fine

Celtic Druid’s Mencoder builds (with noasm)
Intel Core Duo: Works Fine
Athlon XP: Works Fine

These results seem to indicate the following:
SSE2 optimizations are working fine in x264
SSE2 optimizations are not working when encoding x264 through FFmpeg or Mencoder

Questions
1. Does anyone have any ideas of what could cause such contradictory results?
2. Since Celtic Druid is using the same yasm to assemble all three versions, how could there be such differences?
3. Does gcc play any when compiling/assembling these optimizations?
4. Could using different gcc versions to compile each of these binaries explain these results?

[LoRd_MuldeR]

There already is a thread about that issue:
http://forum.doom9.org/showthread.php?t=129123

[pspvideo9]

Well it looks like someone may have found the problem:
http://celticdruid.no-ip.com/phpBB3/viewtopic.php?f=9&t=285#p894

The jist: For some reason the asm in libxvid and libx264 don't want to coexist when linked it the same binary on win32 + minGW + SSE2. This explains why x264.exe worked, as it never links to libxvid.

Solutions: Don't link to libxvid or add 4 bytes of padding to x264's pixel-sse2.asm as shown in the above link.

[squid_80]

The better fix would be to specify SECTION_RODATA as 16-byte aligned, if it isn't already.

[akupenguin]

It is already.
%macro SECTION_RODATA 0
%ifidn __OUTPUT_FORMAT__,macho
SECTION .text align=16
fakegot:
%else
SECTION .rodata align=16
%endif
%endmacro

[squid_80]

So is the linker messing up?

[pspvideo9]

Well according to the thread on:
http://celticdruid.no-ip.com/phpBB3/viewtopic.php?f=9&t=285&st=0&sk=t&sd=a

They think its a binutils problem on win32.
Also, it looks like that 4 byte padding hack no longer fixes things.

This low level stuff is a bit over our heads so we have to wait and see what all those smart asm guys come up with over there.

Another oddity is why this crash isn't occurring on some AMD machines that have SSE2 in their CPU's.
Users that didn't crash: Athlon 64 2800+, Athlong 64 X2, another Athlong 64 X2
Since mingw doesn't support 64-bit, the amd64 optimizations shouldn't be in use in these 32-bit binaries and it should be using the standard SSE2 optimizations.

[akupenguin]

Another oddity is why this crash isn't occurring on some AMD machines that have SSE2 in their CPU's.
Certain SSE2 functions are disabled on AMD cpus because they're slower than MMX.

[clsid]

We used to have SSE2 alignment problems in libavcodec. I don't know if the situation is similar here, but if the alignment of function arguments is the problem, then adding "__attribute__((force_align_arg_pointer))" to the function declarations could fix it. This requires GCC 4.2.0 or newer.

[akupenguin]

x264 doesn't ever load xmm registers from function arguments.
ffmpeg has problems with unaligned stack, but x264 fixes that with an asm function that aligns the stack.