Threaded lookahead issue

[Chengbin]

First of all, I would like to say my thanks to the x264 developers for this feature. It really made encoding quite a bit faster in CRF mode.

I'm encountering some issues.

1) With this patch, wouldn't the CPU usage of my quad core be 100%? It is not always 100%. Half the time it is around 75%

2) In the changelog, it said threaded lookahead is a separate thread with a higher priority. Well, I think the priority is too high. My computer lags considerably when the CPU is at 100%, to the point my mouse movement is not smooth, and the words I'm typing right now does not appear instantly as I press the key.

[thewebchat]

Re 1):

Note also that this does not split lookahead itself into multiple threads yet; this may be added in the future.

[the6thday]

I've seen something similar to issue Nr. 2:

Running Windows 7 X64 RTM @ Intel core2quad Q9550@3,4ghz

When using r1246 x86 the complete windows gui freezes for ~3 seconds every ~25 seconds during the first encoding pass @~65% cpu usage, the second pass runs fine @~100% cpu usage... (this doesn't happen with r1243 x86)

And on the good side: The first pass is ~5fps faster and the second pass is ~2fps faster with r1246

commandline used: --profile high --level 4.1 --pass x --bitrate 7000 --stats ".stats" --thread-input --deblock -1:-1 --b-adapt 2 --direct auto --ref 2 --vbv-bufsize 50000 --vbv-maxrate 50000 --rc-lookahead 30 --me umh --trellis 2 --output "output" "input"

[Selur]

While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?

about the freezes: got no windows freezes here during 1st pass (Running Windows 7 X64 RTM @ Intel core2quad Q9250@2,6ghz)

Cu Selur

[burfadel]

rev 1246 should be faster by the sounds of it than rev 1243, so do some comparative tests

[kemuri-_9]

Originally Posted by Selur

While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?

No, sync-lookahead is completely independent of rc-lookahead (as it says in the commit message).

due to some possible deadlock problems at very low values of sync-lookahead, it's clipped to the range of [bframes+threads,250] to prevent that possibility
generally you should never need to to change the value of the buffer.

the thread is also disabled during the 2nd pass since it's useless there.

the thread it creates is created with the maximum possible thread priority:
on windows for 'normal' class priority for the process,
the main program has a priority of '8' or '9', while the lookahead thread has a priority of '15' or so.

This was intended as the processing time needed by the lookahead thread is significant in a number of situations and all the encoder threads will block for it to complete some frames.

if you don't want x264 using all of your processor power with the change, decrease the number of encoding threads by 1 or 2.

[Selur]

to my defense: the independence of sync-lookahead and rc-lookahead doesn't mean it couldn't have been limited by keyint as maximum.

[bframes+threads,250]

didn't think which defaults to auto (threads+bframes buffer size) from the change log would directly translate to thread count + number of b-frames -> thanks for that clarification

[detmek]

I did some small test. I used 3000 frames clip, Diablo 3 trailer, 1280x544 resized to 720x304 using spline36. Command line is --preset medium --crf 22 --me umh --b-adapt 2 --tune ssim.
It seams that v1246 is 7,5% faster then v1243 on Athlon X2 4000+. With --bframes 16 it is 20-25% faster. But, this is only one test clip.
Final bitrate is the same.

[simps]

Looks like you guys don't have your windows tunned up, and your cpu / overclocks.
There is A LOT you need to do on windows reg and configuration to make it perfect. Anyway, I might spend some time to do a guide for you guys.
This is what I get even on 1st pass, using avisynth script too. Nothing is laggy, I can even play games while encoding... default windows settings IS BAD.

[foxyshadis]

simps, that isn't a problem caused by overclocking, tuning, or anything else, it really is exactly what the first post described: When a thread priority is higher than the windows desktop and every other application running, the system will freeze up. The slower your first-pass settings, the less likely you are to see it. Try fast-first-pass instead and watch the system crawl.

[Dark Shikari]

There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.

[kemuri-_9]

Originally Posted by Dark Shikari

There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.

you said to do it in the first place.

[Dark Shikari]

Originally Posted by kemuri-_9

you said to do it in the first place.

No I didn't, I said that all threads should be lowered in priority except for the lookahead thread.

[kemuri-_9]

the issue with x264 being over-prioritized on windows is now fixed with r1247.

[LoRd_MuldeR]

Originally Posted by kemuri-_9

the issue with x264 being over-prioritized on windows is now fixed with r1247.

[Selur]

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

[Chengbin]

Originally Posted by Selur

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

1: It is pointless
2: It will eat way too much RAM.

[kemuri-_9]

Originally Posted by Selur

Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)

and what if keyint is 250? (the default)
and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.

either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.

[Selur]

and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.

since the thread is about the sync-lookahead, one could assume I speak of the sync-lookahead

either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.

okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])

[kemuri-_9]

    h->param.rc.i_lookahead = x264_clip3( h->param.rc.i_lookahead, 0, X264_LOOKAHEAD_MAX );
    {
        int maxrate = X264_MAX( h->param.rc.i_vbv_max_bitrate, h->param.rc.i_bitrate );
        float bufsize = maxrate ? (float)h->param.rc.i_vbv_buffer_size / maxrate : 0;
        float fps = h->param.i_fps_num > 0 && h->param.i_fps_den > 0 ? (float) h->param.i_fps_num / h->param.i_fps_den : 25.0;
        h->param.rc.i_lookahead = X264_MIN( h->param.rc.i_lookahead, X264_MAX( h->param.i_keyint_max, bufsize*fps ) );
    }
...
#ifdef HAVE_PTHREAD
    if( h->param.i_sync_lookahead )
        h->param.i_sync_lookahead = x264_clip3( h->param.i_sync_lookahead, h->param.i_threads + h->param.i_bframe, X264_LOOKAHEAD_MAX );
    if( h->param.rc.b_stat_read || h->param.i_threads == 1 )
        h->param.i_sync_lookahead = 0;
#else
    h->param.i_sync_lookahead = 0;
#endif

Originally Posted by Selur

okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])

A. rc-lookahead is not necessarily capped to keyint, the mb-tree lookahead is
- rc-lookahead is used with both mb-tree and vbv-lookahead
B. sync-lookahead is a different buffer that has absolutely nothing to do with keyint.
- I said it clips to [bframes+threads,250] because that's what it does!