Threaded lookahead issue

[Chengbin]

First of all, I would like to say my thanks to the x264 developers for this feature. It really made encoding quite a bit faster in CRF mode.

I'm encountering some issues.

1) With this patch, wouldn't the CPU usage of my quad core be 100%? It is not always 100%. Half the time it is around 75%

2) In the changelog, it said threaded lookahead is a separate thread with a higher priority. Well, I think the priority is too high. My computer lags considerably when the CPU is at 100%, to the point my mouse movement is not smooth, and the words I'm typing right now does not appear instantly as I press the key.

[thewebchat]

Re 1):

Note also that this does not split lookahead itself into multiple threads yet; this may be added in the future.

[the6thday]

I've seen something similar to issue Nr. 2:

Running Windows 7 X64 RTM @ Intel core2quad Q9550@3,4ghz

When using r1246 x86 the complete windows gui freezes for ~3 seconds every ~25 seconds during the first encoding pass @~65% cpu usage, the second pass runs fine @~100% cpu usage... (this doesn't happen with r1243 x86)

And on the good side: The first pass is ~5fps faster and the second pass is ~2fps faster with r1246

commandline used: --profile high --level 4.1 --pass x --bitrate 7000 --stats ".stats" --thread-input --deblock -1:-1 --b-adapt 2 --direct auto --ref 2 --vbv-bufsize 50000 --vbv-maxrate 50000 --rc-lookahead 30 --me umh --trellis 2 --output "output" "input"

[Selur]

While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?

about the freezes: got no windows freezes here during 1st pass (Running Windows 7 X64 RTM @ Intel core2quad Q9250@2,6ghz)

Cu Selur

[burfadel]

rev 1246 should be faster by the sounds of it than rev 1243, so do some comparative tests :)

[kemuri-_9]

While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?

No, sync-lookahead is completely independent of rc-lookahead (as it says in the commit message).

due to some possible deadlock problems at very low values of sync-lookahead, it's clipped to the range of [bframes+threads,250] to prevent that possibility
generally you should never need to to change the value of the buffer.

the thread is also disabled during the 2nd pass since it's useless there.

the thread it creates is created with the maximum possible thread priority:
on windows for 'normal' class priority for the process,
the main program has a priority of '8' or '9', while the lookahead thread has a priority of '15' or so.

This was intended as the processing time needed by the lookahead thread is significant in a number of situations and all the encoder threads will block for it to complete some frames.

if you don't want x264 using all of your processor power with the change, decrease the number of encoding threads by 1 or 2.

[Selur]

to my defense: the independence of sync-lookahead and rc-lookahead doesn't mean it couldn't have been limited by keyint as maximum. :)

[bframes+threads,250]
didn't think which defaults to auto (threads+bframes buffer size) from the change log would directly translate to thread count + number of b-frames -> thanks for that clarification

[detmek]

I did some small test. I used 3000 frames clip, Diablo 3 trailer, 1280x544 resized to 720x304 using spline36. Command line is --preset medium --crf 22 --me umh --b-adapt 2 --tune ssim.
It seams that v1246 is 7,5% faster then v1243 on Athlon X2 4000+. With --bframes 16 it is 20-25% faster. But, this is only one test clip.
Final bitrate is the same.

[simps]

Looks like you guys don't have your windows tunned up, and your cpu / overclocks.
There is A LOT you need to do on windows reg and configuration to make it perfect. Anyway, I might spend some time to do a guide for you guys.
This is what I get even on 1st pass, using avisynth script too. Nothing is laggy, I can even play games while encoding... default windows settings IS BAD.

http://img25.imageshack.us/img25/7379/x264.png

[foxyshadis]

simps, that isn't a problem caused by overclocking, tuning, or anything else, it really is exactly what the first post described: When a thread priority is higher than the windows desktop and every other application running, the system will freeze up. The slower your first-pass settings, the less likely you are to see it. Try fast-first-pass instead and watch the system crawl.

[Dark Shikari]

There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.

[kemuri-_9]

There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.

you said to do it in the first place.

[Dark Shikari]

you said to do it in the first place.No I didn't, I said that all threads should be lowered in priority except for the lookahead thread.

[kemuri-_9]

the issue with x264 being over-prioritized on windows is now fixed with r1247.

[LoRd_MuldeR]

the issue with x264 being over-prioritized on windows is now fixed with r1247.

:thanks:

[Selur]

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

[Chengbin]

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

1: It is pointless
2: It will eat way too much RAM.

[kemuri-_9]

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

and what if keyint is 250? (the default)
and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.

either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.

[Selur]

and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.
since the thread is about the sync-lookahead, one could assume I speak of the sync-lookahead ;)

either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.
okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])

[kemuri-_9]

h->param.rc.i_lookahead = x264_clip3( h->param.rc.i_lookahead, 0, X264_LOOKAHEAD_MAX );
{
int maxrate = X264_MAX( h->param.rc.i_vbv_max_bitrate, h->param.rc.i_bitrate );
float bufsize = maxrate ? (float)h->param.rc.i_vbv_buffer_size / maxrate : 0;
float fps = h->param.i_fps_num > 0 && h->param.i_fps_den > 0 ? (float) h->param.i_fps_num / h->param.i_fps_den : 25.0;
h->param.rc.i_lookahead = X264_MIN( h->param.rc.i_lookahead, X264_MAX( h->param.i_keyint_max, bufsize*fps ) );
}
...
#ifdef HAVE_PTHREAD
if( h->param.i_sync_lookahead )
h->param.i_sync_lookahead = x264_clip3( h->param.i_sync_lookahead, h->param.i_threads + h->param.i_bframe, X264_LOOKAHEAD_MAX );
if( h->param.rc.b_stat_read || h->param.i_threads == 1 )
h->param.i_sync_lookahead = 0;
#else
h->param.i_sync_lookahead = 0;
#endif



okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])

A. rc-lookahead is not necessarily capped to keyint, the mb-tree lookahead is
- rc-lookahead is used with both mb-tree and vbv-lookahead
B. sync-lookahead is a different buffer that has absolutely nothing to do with keyint.
- I said it clips to [bframes+threads,250] because that's what it does!

[Selur]

okay, sync-lookahead is limited to [bframes+threads,250], that's now clear even for me. :)
but now I'm a bit confused about rc-lookahead:
Is rc-lookahead is only limited to keyint if mb-tree is enabled, or is the rc-lookahead one can set via the command line always [1-250] and x264 automatically only uses the first keyint frames of the rc-lookahead ?

Cu Selur

Ps.: thanks for bearing with me :)

[LoRd_MuldeR]

I assume that the lookahead for MB-Tree is "naturally" limited to Max-Keyint, because MB-Tree keeps track on how information is propagated.
But no information will ever be propagated across an IDR/Key frame, because an IDR frame clears all buffered frames...

[Selur]

so mb-tree uses a subset of rc-lookahead ?

[Dark Shikari]

But no information will ever be propagated across an IDR/Key frame, because an IDR frame clears all buffered frames...Wrong again...

[LoRd_MuldeR]

Wrong again...

Again? And would you be so kind and explain? :D

How can any information from frames before an IDR frame propagate to frames after the IDR frame, when "all reference pictures" are marked as "unused for reference" immediately ??? :confused:

[Dark Shikari]

Again? And would you be so kind and explain? :D

How can any information from frames before an IDR frame propagate to frames after the IDR frame, when "all reference pictures" are marked as "unused for reference" immediately ??? :confused:It doesn't in the bitstream, but it does in macroblock tree. See lines 731-734 of encoder/slicetype.c.

[Selur]

still a bit confused :)
here's how I understood it:
- rc-lookahead and sync-lookahead are two different lookaheads (so there values are independent)
- sync-lookahead is limited to [threads+bframes, 250]
- rc-lookahead when used for mb-tree, this is what was earlies always called mbtree-lookahead is limited to [1, min(keyint,250)]
- rc-lookahead itself is limited to [1-250]
if the above is right so far, what do I specify with --rc-lookahead when I enter it in the command line?
a. always the rc-lookahead (and mb-tree-lookahead is automatically limited internally )
b. normally rc-lookahead and when mb-tree is enabled the mb-tree lookahead (so when mb-tree is enabled mbtree equals rc-lookahead)
c. something I missed

Cu Selur

[kemuri-_9]

- rc-lookahead and sync-lookahead are two different lookaheads (so there values are independent)
this line is correct.
- sync-lookahead is limited to [threads+bframes, 250]
This is mostly correct: 0 is an accepted value to disable it.

- rc-lookahead when used for mb-tree, this is what was earlies always called mbtree-lookahead is limited to [1, min(keyint,250)]
- rc-lookahead itself is limited to [1-250]

this is mostly correct: 0 is an accepted value to disable mb-tree and vbv-lookahead

if the above is right so far, what do I specify with --rc-lookahead when I enter it in the command line?


you're specifying the absolute maximum frame buffer size that can be used by both vbv-lookahead (when vbv is used as well) and mb-tree.

i.e.
--keyint 200 --rc-lookahead 60
though mb-tree and vbv-lookahead can technically use up to a 200 frame buffer (vbv-lookahead can actually be even higher depending on other settings),
you're limiting them to use only 60 frames.

[Selur]

though mb-tree and vbv-lookahead can technically use up to a 200 frame buffer
shouldn't mbtree-lookahead then be limited to [0, min(keyint,200)] instead of [0, min(keyint,250)] ?

And vbv-lookahead equals rc-lookahead when vbv restrictions are enabled, right?

Cu Selur

[kemuri-_9]

shouldn't mbtree-lookahead then be limited to [0, min(keyint,200)] instead of [0, min(keyint,250)] ?

....
keyint is 200 in the example i was trying to get you to understand with, so MIN(250,200) simplifies to 200.

I'm not going to continue this as
A. I feel like I've done all I could to have you understand
B. this is no longer related to threaded lookahead
but you continue to pose questions contrary to what i say as if you're not reading at all.

Edit:
the mewiki may help as it has things worded slightly differently, give it a read: rc-lookahead (http://mewiki.project357.com/wiki/X264_Settings#rc-lookahead)

[Selur]

I jsut misinterpreted the 'technically' but thanks for the help :)