Issue introduced in libx264 r1077

[gruntster]

Since r1077 of libx264, Avidemux crashes when encoding if b_transform_8x8 (--8x8dct) is turned off. Unfortunately the issue doesn't occur in debug builds or when run through gdb. I have only be able to test the issue on Windows 32-bit so not sure if it occurs on other OSes or CPUs.

The only error I can capture is logged by Avidemux's error handler:

*********** EXCEPTION **************
Registers:
EAX: 000010C2  EBX: 000001AD  ECX: 10000000  EDX: 00000000  ESI: 00000000
EDI: 00000002  ESP: 0021C260  EBP: 000001AD  EIP: 6F7CF14D  EFlags: 00010246

Exception Code: EXCEPTION_INT_DIVIDE_BY_ZERO (C0000094)
Exception Flags: 00000000

Origin:
C:\Program Files\Avidemux 2.4\libx264-66.dll(x264_macroblock_analyse+0x4F14D) [0x6F7CF14D]
*********** EXCEPTION **************

Anyone have any ideas what's causing this? Perhaps another Windows alignment issue?

[J_Darnley]

Are you only getting this with the library? If you can reproduce it with the binary, try a debug build with full optimisations: http://users.telenet.be/darnley/x264/x264-debug+opts.7z
Another bug, which was just fixed, only appeared for one guy when the opts were on.

[gruntster]

I'm only getting the problem with the library. I can't get the CLI to fail. Another user is getting the same problem on Windows. Most users probably don't turn off the 8x8dct option (which Avidemux 2.4 turns on by default) so the reports will be low.

Is there an easy way to compile x264.exe to use libx264.dll rather than compiliing everything into the exe?

[LoRd_MuldeR]

Here are fresh builds, please test:
http://forum.doom9.org/showpost.php?...postcount=1041

I included a debug build plus a debug build with optimizations

[Dark Shikari]

Well, we already know where that crash is, because there's only one IDIV in macroblock_analyse.

if( i_intra_cost == COST_MAX )
                i_intra_cost = i_cost * i_satd_intra / i_satd_inter + 1;

Er, is this supposed to be

if( i_intra_cost == COST_MAX )
                i_intra_cost = i_cost * i_satd_intra / (i_satd_inter + 1);

Or do I have my C precedence rules wrong again?

[LoRd_MuldeR]

So the cause of the problem was found, right? But wouldn't that effect the CLI encoder too?

[Dark Shikari]

Originally Posted by LoRd_MuldeR

So the cause of the problem was found, right? But wouldn't that effect the CLI encoder too?

Nor would revision 1077 have anything to do with it...

[LoRd_MuldeR]

Are you going to fix it anyway?

[gruntster]

I can reproduce a crash with basically every revision from r1077 to r1114. Debug builds or running the process through gdb stops it from crashing so perhaps it may not have been introduced in r1077 but it's the first revision I can reliably reproduce the fault.

[LoRd_MuldeR]

Originally Posted by gruntster

Debug builds or running the process through gdb stops it from crashing

Did you try a debug build with optimizations yet? Normally debug builds don't have optimizations enabled.

[gruntster]

Yes I have.

[gruntster]

Originally Posted by Dark Shikari

Er, is this supposed to be

if( i_intra_cost == COST_MAX )
                i_intra_cost = i_cost * i_satd_intra / (i_satd_inter + 1);

If it helps, this seems to make the problem disappear.

[Dark Shikari]

If that makes the problem disappear, there is something more subtle wrong; i_satd_inter should never, ever, ever be zero. A quick trace of the good shows that it being zero is in fact physically impossible if the code works as intended.

More precisely:

a->i_lambda * (log2f(i+1)*2 + 0.718f + !!i) + .5f;

i_satd_inter cannot equal zero unless this line, for i >= 0 and a->i_lambda >= 1, can equal zero.

[gruntster]

After adding logging, i_satd_inter equals 0 after about 35 frames. Any debug code you'd like me to add to pinpoint it further?

[Dark Shikari]

Originally Posted by gruntster

After adding logging, i_satd_inter equals 0 after about 35 frames. Any debug code you'd like me to add to pinpoint it further?

See my edit in the above post...

[LoRd_MuldeR]

So is the fix (maybe better "workaround") proposed in post #5 still valid?

I mean, is it okay to apply that change until the actual cause of the problem is resolved ???

[gruntster]

Originally Posted by Dark Shikari

a->i_lambda * (log2f(i+1)*2 + 0.718f + !!i) + .5f;

i_satd_inter cannot equal zero unless this line, for i >= 0 and a->i_lambda >= 1, can equal zero.

If I add logging around this line the problem goes away. Since this is one of the lines introduced in r1077 it tends to suggest something is going awry in this area.

[Dark Shikari]

Originally Posted by LoRd_MuldeR

So is the fix (maybe better "workaround") proposed in post #5 still valid?

I mean, is it okay to apply that change until the actual cause of the problem is resolved ???

Theoretically, but it's clear that this bug can only be caused if something else is failing, so that change just covers it up.

[gruntster]

Perhaps MinGW's implementation of log2f is faulty?

[LoRd_MuldeR]

Originally Posted by Dark Shikari

Theoretically, but it's clear that this bug can only be caused if something else is failing, so that change just covers it up.

Thanks. A temporary workaround is better than nothing. Better than a potential crash for sure