Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd September 2009, 13:13   #1  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Threaded lookahead issue

First of all, I would like to say my thanks to the x264 developers for this feature. It really made encoding quite a bit faster in CRF mode.

I'm encountering some issues.

1) With this patch, wouldn't the CPU usage of my quad core be 100%? It is not always 100%. Half the time it is around 75%

2) In the changelog, it said threaded lookahead is a separate thread with a higher priority. Well, I think the priority is too high. My computer lags considerably when the CPU is at 100%, to the point my mouse movement is not smooth, and the words I'm typing right now does not appear instantly as I press the key.
Chengbin is offline   Reply With Quote
Old 2nd September 2009, 13:30   #2  |  Link
thewebchat
Advanced Blogging
 
Join Date: May 2009
Posts: 483
Re 1):

Quote:
Note also that this does not split lookahead itself into multiple threads yet; this may be added in the future.
thewebchat is offline   Reply With Quote
Old 2nd September 2009, 13:38   #3  |  Link
the6thday
Registered User
 
Join Date: Dec 2006
Posts: 6
I've seen something similar to issue Nr. 2:

Running Windows 7 X64 RTM @ Intel core2quad Q9550@3,4ghz

When using r1246 x86 the complete windows gui freezes for ~3 seconds every ~25 seconds during the first encoding pass @~65% cpu usage, the second pass runs fine @~100% cpu usage... (this doesn't happen with r1243 x86)

And on the good side: The first pass is ~5fps faster and the second pass is ~2fps faster with r1246

commandline used: --profile high --level 4.1 --pass x --bitrate 7000 --stats ".stats" --thread-input --deblock -1:-1 --b-adapt 2 --direct auto --ref 2 --vbv-bufsize 50000 --vbv-maxrate 50000 --rc-lookahead 30 --me umh --trellis 2 --output "output" "input"

Last edited by the6thday; 2nd September 2009 at 13:41.
the6thday is offline   Reply With Quote
Old 2nd September 2009, 15:08   #4  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?

about the freezes: got no windows freezes here during 1st pass (Running Windows 7 X64 RTM @ Intel core2quad Q9250@2,6ghz)

Cu Selur
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 2nd September 2009, 15:55   #5  |  Link
burfadel
Registered User
 
Join Date: Aug 2006
Posts: 2,235
rev 1246 should be faster by the sounds of it than rev 1243, so do some comparative tests
burfadel is offline   Reply With Quote
Old 2nd September 2009, 15:58   #6  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
Quote:
Originally Posted by Selur View Post
While at just to be sure: sync-lookahead is like rc-lookahead limited to keyint as max, right?
No, sync-lookahead is completely independent of rc-lookahead (as it says in the commit message).

due to some possible deadlock problems at very low values of sync-lookahead, it's clipped to the range of [bframes+threads,250] to prevent that possibility
generally you should never need to to change the value of the buffer.

the thread is also disabled during the 2nd pass since it's useless there.

the thread it creates is created with the maximum possible thread priority:
on windows for 'normal' class priority for the process,
the main program has a priority of '8' or '9', while the lookahead thread has a priority of '15' or so.

This was intended as the processing time needed by the lookahead thread is significant in a number of situations and all the encoder threads will block for it to complete some frames.

if you don't want x264 using all of your processor power with the change, decrease the number of encoding threads by 1 or 2.
__________________
custom x264 builds & patches | F@H | My Specs
kemuri-_9 is offline   Reply With Quote
Old 2nd September 2009, 16:14   #7  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
to my defense: the independence of sync-lookahead and rc-lookahead doesn't mean it couldn't have been limited by keyint as maximum.

Quote:
[bframes+threads,250]
didn't think which defaults to auto (threads+bframes buffer size) from the change log would directly translate to thread count + number of b-frames -> thanks for that clarification
__________________
Hybrid here in the forum, homepage

Last edited by Selur; 2nd September 2009 at 16:19.
Selur is offline   Reply With Quote
Old 2nd September 2009, 18:31   #8  |  Link
detmek
Registered User
 
Join Date: Aug 2009
Posts: 475
I did some small test. I used 3000 frames clip, Diablo 3 trailer, 1280x544 resized to 720x304 using spline36. Command line is --preset medium --crf 22 --me umh --b-adapt 2 --tune ssim.
It seams that v1246 is 7,5% faster then v1243 on Athlon X2 4000+. With --bframes 16 it is 20-25% faster. But, this is only one test clip.
Final bitrate is the same.
detmek is offline   Reply With Quote
Old 2nd September 2009, 18:57   #9  |  Link
simps
Registered User
 
Join Date: Feb 2005
Posts: 149
Looks like you guys don't have your windows tunned up, and your cpu / overclocks.
There is A LOT you need to do on windows reg and configuration to make it perfect. Anyway, I might spend some time to do a guide for you guys.
This is what I get even on 1st pass, using avisynth script too. Nothing is laggy, I can even play games while encoding... default windows settings IS BAD.

simps is offline   Reply With Quote
Old 2nd September 2009, 19:56   #10  |  Link
foxyshadis
ангел смерти
 
foxyshadis's Avatar
 
Join Date: Nov 2004
Location: Lost
Posts: 9,380
simps, that isn't a problem caused by overclocking, tuning, or anything else, it really is exactly what the first post described: When a thread priority is higher than the windows desktop and every other application running, the system will freeze up. The slower your first-pass settings, the less likely you are to see it. Try fast-first-pass instead and watch the system crawl.
__________________
There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. ~ Ed Howdershelt
foxyshadis is offline   Reply With Quote
Old 2nd September 2009, 19:57   #11  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,690
There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.
Dark Shikari is offline   Reply With Quote
Old 2nd September 2009, 20:08   #12  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
Quote:
Originally Posted by Dark Shikari View Post
There is a known issue with the fact that Kemuri decided to force the lookahead thread to REALTIME priority on Windows... which is a rather bad idea.

I'm waiting on a fix from him.
you said to do it in the first place.
__________________
custom x264 builds & patches | F@H | My Specs
kemuri-_9 is offline   Reply With Quote
Old 2nd September 2009, 20:16   #13  |  Link
Dark Shikari
x264 developer
 
Dark Shikari's Avatar
 
Join Date: Sep 2005
Posts: 8,690
Quote:
Originally Posted by kemuri-_9 View Post
you said to do it in the first place.
No I didn't, I said that all threads should be lowered in priority except for the lookahead thread.
Dark Shikari is offline   Reply With Quote
Old 3rd September 2009, 01:36   #14  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
the issue with x264 being over-prioritized on windows is now fixed with r1247.
__________________
custom x264 builds & patches | F@H | My Specs
kemuri-_9 is offline   Reply With Quote
Old 3rd September 2009, 01:38   #15  |  Link
LoRd_MuldeR
Software Developer
 
LoRd_MuldeR's Avatar
 
Join Date: Jun 2005
Location: Last House on Slunk Street
Posts: 12,921
Quote:
Originally Posted by kemuri-_9 View Post
the issue with x264 being over-prioritized on windows is now fixed with r1247.
__________________
There was of course no way of knowing whether you were being watched at any given moment.
How often, or on what system, the Thought Police plugged in on any individual wire was guesswork.


LoRd_MuldeR is offline   Reply With Quote
Old 3rd September 2009, 10:50   #16  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 3rd September 2009, 12:32   #17  |  Link
Chengbin
Registered User
 
Join Date: Oct 2007
Posts: 1,059
Quote:
Originally Posted by Selur View Post
Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)
1: It is pointless
2: It will eat way too much RAM.
Chengbin is offline   Reply With Quote
Old 3rd September 2009, 14:30   #18  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
Quote:
Originally Posted by Selur View Post
Why is lookahead limited to 250? (I mean does it make any sense to lookahead more than keyint (or 2xkeyint) ?)
and what if keyint is 250? (the default)
and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.

either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.
__________________
custom x264 builds & patches | F@H | My Specs

Last edited by kemuri-_9; 3rd September 2009 at 14:34.
kemuri-_9 is offline   Reply With Quote
Old 3rd September 2009, 17:00   #19  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,627
Quote:
and you plainly said 'lookahead' which can now refer to either sync-lookahead or rc-lookahead.
since the thread is about the sync-lookahead, one could assume I speak of the sync-lookahead

Quote:
either way, 250 was used for sync-lookahead because that's the cap that was already in place for rc-lookahead.
okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])
__________________
Hybrid here in the forum, homepage
Selur is offline   Reply With Quote
Old 3rd September 2009, 18:13   #20  |  Link
kemuri-_9
Compiling Encoder
 
kemuri-_9's Avatar
 
Join Date: Jan 2007
Posts: 1,348
Code:
    h->param.rc.i_lookahead = x264_clip3( h->param.rc.i_lookahead, 0, X264_LOOKAHEAD_MAX );
    {
        int maxrate = X264_MAX( h->param.rc.i_vbv_max_bitrate, h->param.rc.i_bitrate );
        float bufsize = maxrate ? (float)h->param.rc.i_vbv_buffer_size / maxrate : 0;
        float fps = h->param.i_fps_num > 0 && h->param.i_fps_den > 0 ? (float) h->param.i_fps_num / h->param.i_fps_den : 25.0;
        h->param.rc.i_lookahead = X264_MIN( h->param.rc.i_lookahead, X264_MAX( h->param.i_keyint_max, bufsize*fps ) );
    }
...
#ifdef HAVE_PTHREAD
    if( h->param.i_sync_lookahead )
        h->param.i_sync_lookahead = x264_clip3( h->param.i_sync_lookahead, h->param.i_threads + h->param.i_bframe, X264_LOOKAHEAD_MAX );
    if( h->param.rc.b_stat_read || h->param.i_threads == 1 )
        h->param.i_sync_lookahead = 0;
#else
    h->param.i_sync_lookahead = 0;
#endif
Quote:
Originally Posted by Selur View Post
okay, so is it like rc-lookahead also capped to keyint ? (which would give us a range of [threads+bframes, min(keyint, 259)])
A. rc-lookahead is not necessarily capped to keyint, the mb-tree lookahead is
- rc-lookahead is used with both mb-tree and vbv-lookahead
B. sync-lookahead is a different buffer that has absolutely nothing to do with keyint.
- I said it clips to [bframes+threads,250] because that's what it does!
__________________
custom x264 builds & patches | F@H | My Specs

Last edited by kemuri-_9; 3rd September 2009 at 18:18.
kemuri-_9 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 20:32.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.