x264, multiple threads & determinism

[mroz]

I've just built my first quad core box & am experimenting with lovely fast threaded encoding, via a benchmark (http://www.techarp.com/showarticle.aspx?artno=442&pgno=1) being reported here (http://forums.hexus.net/showthread.php?p=1211891&posted=1#post1211891).

The non deterministic aspect of x264's (0.55.663) multi threaded output has me puzzled in one respect. I could understand if the output is non-deterministic as a side effect of theading & also the support for an experimental option which intentionally makes output non-deterministic as a result of an algorithm to boost quality. What I don't get is why that option is called --non-deterministic, since in my mind that implies without it, output will be deterministic, which it clearly isn't.

Can anyone explain?

Also, is the --non-deterministic option currently regarded as stable?

Can anyone point me at some info on how this works or at least what it does - aside from the brief description of 'improves smp output quality at expense of determinism'?

Thanks.

[Gabriel_Bouvigne]

I think that this is just a legacy option. But you are right about the fact that the presence of this option is a bit puzzling.

[Manao]

AFAIK, without non-deterministic, x264 is deterministic. That is, with a fixed number of threads, you'll always get the same results ( but if you change the number of threads, you'll get different results ).

[mroz]

I think that this is just a legacy option.
Ah, right. Thanks.

[mroz]

AFAIK, without non-deterministic, x264 is deterministic. That is, with a fixed number of threads, you'll always get the same results ( but if you change the number of threads, you'll get different results ).
The benchmark I linked to runs the same encode five times. When threads is greater than one, each of these five encodes will have a different file length. It's even mentioned in the benchmark's faq which I also linked to. So x264 seems not to be deterministic.

Can anyone comment on the official position? IOW is this known & expected behaviour or does it reflect a bug/problem with either x264, some other part of the benchmark or (unlikely) all the hardware involved in the benchmark on which this has been observed? Thanks for the feedback.

[Unearthly]

The benchmark I linked to runs the same encode five times. When threads is greater than one, each of these five encodes will have a different file length. It's even mentioned in the benchmark's faq which I also linked to. So x264 seems not to be deterministic.

Can anyone comment on the official position? IOW is this known & expected behaviour or does it reflect a bug/problem with either x264, some other part of the benchmark or (unlikely) all the hardware involved in the benchmark on which this has been observed? Thanks for the feedback.

I did come across a bug last weekend that shows up in multi-threaded 2 pass (what this test does). The problem occured in the second pass, where as you increased the number of threads (I was testing 100), it would both decrease in quality and miss the bitrate by a larger amount (bitrate increased). I'm not sure if it only triggers at a certain number of threads or if the problem is gradual after 1 thread, I haven't done extensive testing there. It was just -really- obvious at 100 threads.

I told akupenguin about it (he helped me pinpoint what situation caused the bug) and he said he found the problem, though I don't know if he has a fix for it yet.

I'm not sure if this will fix the determinism or not, as I do remember my CRF test encodes having different sizes (trivially, but different).

[Gabriel_Bouvigne]

AFAIK, without non-deterministic, x264 is deterministic. That is, with a fixed number of threads, you'll always get the same results ( but if you change the number of threads, you'll get different results ).
Actually that's not the case. (and that's why we are a bit puzzled first about the --non-deterministic parameter)
Only deterministic with 1 single encoding thread.

[RaynQuist]

From this thread (http://forum.doom9.org/showthread.php?t=128331), I believe encodes with the same number of threads are supposed to be the same. Looks to me the problem is caused by using subme 6 and 8x8dct at the same time. The benchmark also uses both of those options.

[mroz]

So it seems this is an unintended side effect of certain combinations of options. Interesting. Thanks.