Thread related doubt

[NaturalThoughts]

If threading is enabled.....

Then is it possible that every time I encode the input file with the same configuration (only changing no. of threads like 1, 2, 4, 8 ...), the output file may have different PSNR or may fail in Bit exactness test.

Let me clear it more...
(same configuration)
Thread 1 -> encode1.264
Thread 2 -> encode2.264
Thread 4 -> encode4.264
Thread 8 -> encode8.264

All output file may fail for bit exactness and the same PSNR.
If it is true?

Then is it also possible that
Thread 8 -> encode8_0.264
Thread 8 -> encode8_1.264
Thread 8 -> encode8_2.264

(I am just encoding the same input , with same configuration and 3 times)
encode8_0.264, encode8_1.264 and encode8_2.264 may fail for bit exactness and the same PSNR.

Thanks in advance.........

[LoRd_MuldeR]

If you encode with a different number of threads, the output is expected to be different.

However the loss in quality (compression efficiency) for using more threads should be negligible for any reasonable number of threads.

But if you encode with the same number of threads, you should get identical output for identical input (and identical settings), unless "b_deterministic" is disabled.

Now the questions are: How do you decode/process your source video and is that method guaranteed to be deterministic ???

Also how do you measure bit exactness? Do you look at the "raw" h.264 bit-stream -or- at the complete file including container overhead and meta data?

[kemuri-_9]

with the way that threading works in x264, an encode with threads X when compared to one with threads Y will generally be very very slightly different.

however, two encode passes with both threads=X, should have bit-exactness (granted everything else is still the same)

[NaturalThoughts]

First of all, Thanks for kind reply.


But if you encode with the same number of threads, you should get identical output for identical input (and identical settings),

Well, I agree with that, but in most of the time according to my observation, encoded raw .264 files are passing the bit exactness test, except when I enable 8 threads. (1, 2, 4 threads output .264 files are identical)

unless "b_deterministic" is disabled.

Now the questions are: How do you decode/process your source video and is that method guaranteed to be deterministic ???


Well yes, in my configuration b_deterministic is set to 0. I was not sure what exactly this setting does. Could you just brief me about it....and how it may affect the PSNR or bit exactness test even with the same configuration, including same no. of threads.... :stupid:


Also how do you measure bit exactness? Do you look at the "raw" h.264 bit-stream -or- at the complete file including container overhead and meta data?
As Already mentioned earlier I matching raw .264 files output.


with the way that threading works in x264, an encode with threads X when compared to one with threads Y will generally be very very slightly different.

however, two encode passes with both threads=X, should have bit-exactness (granted everything else is still the same)

Even I expected the same thing and for many streams it is true also, but I got that for some my test the encoded .264 files are failing for bit matched test...

Although LoRd_MuldeR told it is expected if "b_deterministic" is disabled. I have no idea how does it affect ........... lets here from him first or may be Dark Shikari have some input on it.

[akupenguin]

in my configuration b_deterministic is set to 0. I was not sure what exactly this setting does.
"deterministic" means "do the same thing every time". If you disabled it, then you're telling x264 that it doesn't have to be consistent.

[NaturalThoughts]

"deterministic" means "do the same thing every time". If you disabled it, then you're telling x264 that it doesn't have to be consistent.

ahhh ok... many thanks. :thanks:

what n how does it exactly do?
What Algo. it performs?
Is there any thread(forum thread..... :)) available related to it so that I can understand it more.

[kemuri-_9]

Is there any thread(forum thread..... :)) available related to it so that I can understand it more.

you can do a simple search in the source for the variable
h->param.b_deterministic
it's only used in two places currently.

[LoRd_MuldeR]

Some basic info on how multi-threading works in x264 is available here:
http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=doc/threads.txt;hb=HEAD

[benwaggoner]

"deterministic" means "do the same thing every time". If you disabled it, then you're telling x264 that it doesn't have to be consistent.
Is there any advantage to running without deterministic? I could imagine it'd be slightly faster if there's more leeway in floating-point precision or something.

[akupenguin]

Floating-point rounding could be different on different architectures, but it would still be deterministic. x264's nondeterminism has to do with threads. Mostly using as much data from other threads as happens to be currently available, rather than only what is guaranteed to have been available at the last sync point.

[LoRd_MuldeR]

Floating-point rounding could be different on different architectures, but it would still be deterministic. x264's nondeterminism has to do with threads. Mostly using as much data from other threads as happens to be currently available, rather than only what is guaranteed to have been available at the last sync point.

Any particular reason why "b_deterministic" is enabled by default?

Using as much data as available from other threads/frames instead of just what's guaranteed should improve the result, at least slightly, right?

Also getting non-deterministic results should be perfectly fine for "regular" encodes, as we usually encode each clip only one single time.

Well, testing/debugging is a different story, as it requires reproducible results. But for that purpose we could re-enabled the deterministic mode easily...

[akupenguin]

All programs, period, should be deterministic by default (except insofar as they're intentionally generating random numbers).

[LoRd_MuldeR]

All programs, period, should be deterministic by default (except insofar as they're intentionally generating random numbers).

I agree that programs generally should never be non-deterministic, unless explicitly stated. But I don't see how it would hurt in this case.

Or are there any other negative side-effects of settings "b_deterministic" to 0 :confused:

[NaturalThoughts]

Thanks for all reply.......

Overall moral of this discussion I got is "Dont judge the technical concept by counting no. of lines written for it" :p

I checked the code, what I understood is ...
In the multithreaded scenario

The motion vector search range changes by

for( j=0; j<i_ref; j++ )
{
x264_frame_cond_wait( fref[j], thresh );
thread_mvy_range = X264_MIN( thread_mvy_range, fref[j]->i_lines_completed - pix_y );
}

but if "h->param.b_deterministic"' is enabled, the motion vector search range will not change on the fly while encoding, which is already calculated based on no. of threads in x264_validate_parameters() as following lines of code says (as per my understanding).

if( h->param.b_deterministic )
thread_mvy_range = h->param.analyse.i_mv_range_thread;

[NaturalThoughts]

I got one doubt related to threading........ :confused: :p

I will put my understanding first, then will proceed to doubt.
as in common.c
param->analyse.i_me_range = 16; i.e. search window size is 16 for motion estimation.

And in encoder.c

h->param.analyse.i_mv_range_thread = r2;
where r2 = (h->param.i_height + X264_THREAD_HEIGHT) / h->param.i_threads (approximately)

And according to this doc http://akuvian.org/src/x264/sliceless_threads.txt (http://akuvian.org/src/x264/sliceless_threads.txt) with given example of 2 threads.

Start encoding frame #0. When it's half done, start encoding frame #1. Thread #1 now only has access to the top half of its reference frame, since the rest hasn't been encoded yet. So it has to restrict the motion search range.

So my query is :
Anyhow search window for motion vector is 16, and agree for X264_THREAD_HEIGHT(16 for the macroblock in progress + 3 for deblocking + 3 for motion compensation filter + 2 for extra safety) which is 24, why not start encoding of frame#1 when pixel rows equal to X264_THREAD_HEIGHT of frame #0 is encoded, why we are restricting it untill half of the frame# 0 encoded??? I am expecting delay in starting frame #1 encoding is more than that it required ?

Thanks in advance.

[Dark Shikari]

The motion search window is the range from the best predictor, not from the (0,0) vector, to search.

[NaturalThoughts]

The motion search window is the range from the best predictor, not from the (0,0) vector, to search.

What would be the minimum and maximum search range for best predictor........ (ideally in case of without threading or say in threads --1.)