x264 multi-core (4+) threading optimization

[morph166955]

EDIT: To make it easier for people to find the thread_pool04 patch is available at http://www.benswebs.com/public/x264/patches/x264_thread_pool.04c.r680.diff


I've made the disturbing realization that as of right now every method I have tested on sending video to x264 is faster on xp then it is in freebsd...which really should not be! I'm looking for some options (or what other people are doing) to try out on this system to get x264 to run at its peak speed wize. This machine is an octacore xeon which is pulling well over 100fps in xp when doing the megui benchmark using avisynth/mt.dll yet it barely hits 100 if at all in bsd...even if i pipe it raw y4m files! I really want to get this machine running as fast as humanly possible (which for this thing should be obsurdly fast) and I'd rather not run it in XP. x264 is not running anywhere near 100% (infact its closer to 35% on all of the cores) when in first pass and only maybe 75% in the second pass so I know it can definitely speed up. So please...any ideas on how to pipe the video into x264 to run this thing to its max would be greatly appreciated.

(note for mods: im not looking for "bests" here...im looking for options to test and opinions on what may or may not be the ideal settings on this setup)

[Dark Shikari]

Tried using more threads if its not using all the CPU power?

[AGDenton]

Raw Y4M == HDD bottleneck. It is imperative that you decode the input in another program, then pipe it to x264 (using | or a fifo). However, x264 tends to assume its standard input to be raw YUV, not Y4M, so you'll have to apply the patch that adds a --y4m-input option (or use a fifo with name **.y4m).

Have you tried ffmpeg -f yuv4mpegpipe | ...?

Anyhow, there's always the option of using Avisynth under wine with avs2yuv, which was done for that exact purpose. It will be slower than under Windows, but not by much.

AG

[morph166955]

forgot to mention...the source was loaded in a ramdisk. the ram is 4 gig worth of 240Pin DDR2 667 FB-DIMM's...so shall we say...no slow down on the source in terms of accessing it :D

[akupenguin]

No problems here. x264 reading from a rawyuv file uses all the cpu I can give it. (not a ramdisk, just relying on Linux's file-chache to keep it in ram)

[morph166955]

At this point I've found that (at least freebsd) has some issues with scheduling for more then 4 cores (it wont even consider letting x264, ffmpeg or mencoder have more then 9 threads which i assume is 1 for input and 8 for x264).

I'm (saddly) going to end up running xp as the primary os on this box for now and possibly run some form of linux in a vmware workstation to do testing even though this will slow down the operation of the vmware slightly (although with that many cores its probably negligable). I'm moving things around on the drives now so that I can change my partition types w/o having to export my files off of the system onto externals or w/e. I already have windows loaded on this box on another parition so I'm going to use acronis to redo the paritions and just make it one huge one on this disk. Once I get something working stably in vmware I'll probably migrate over to that OS but I at least want something to tinker with until then seeing as it may take a good while to do that.

I'll let ya all know how it goes in a few hours once i do some more tweaking.

[giandrea]

At this point I've found that (at least freebsd) has some issues with scheduling for more then 4 cores (it wont even consider letting x264, ffmpeg or mencoder have more then 9 threads which i assume is 1 for input and 8 for x264).

I'm (saddly) going to end up running xp as the primary os on this box for now and possibly run some form of linux in a vmware workstation to do testing even though this will slow down the operation of the vmware slightly (although with that many cores its probably negligable). I'm moving things around on the drives now so that I can change my partition types w/o having to export my files off of the system onto externals or w/e. I already have windows loaded on this box on another parition so I'm going to use acronis to redo the paritions and just make it one huge one on this disk. Once I get something working stably in vmware I'll probably migrate over to that OS but I at least want something to tinker with until then seeing as it may take a good while to do that.

I'll let ya all know how it goes in a few hours once i do some more tweaking.

That's what I tought. Try running Linux, it's better at multithreading. Try Ubuntu server and build x264, MPlayer, ffmpeg and what else with optimizations for your processor.

[morph166955]

I'm going to run vmware on the xp install to test different distro's for speed (assuming that vmware degrades each speedwize approximately equally) as well as stability and ease of use. I was goign to try debian, fedora and ubuntu first...then maybe a few of the other ones depending how those do. I think its going to be a lot of trial and error to get the most optimal setup for me.

[akupenguin]

test different distro's for speed
What kind of speed could you test?

Responsiveness of their default desktop environment? (choice of window manager doesn't have much to do with distro)
Speed at which they run x264? (should depend only on the kernel's smp scheduler and the amount of crap running in the background, not anything distro-specific)
Boot time? (the amount of time I spend waiting for a reboot over the lifespan of a computer is probably less than the time it would take to install one alternate distro)

[morph166955]

yea i was going to shoot for encoding speed basically...most of it was going to be ease of installing/compiling software and such (and just how i liked it overall). speedwize the only thing i can think of is if the os has any added software running that can take up cpu time and/or different libraries (such as the threads library) that may affect how it all runs. its mostly going to come down to how i like the system overall and im probably just going to end up playing with it over the course of a few weeks...ill just run everything in xp until then it should do for now.

[morph166955]

and xp bites the dust...it wont let me access more then 2 gig of my ram w/o going to xp64...fedora core 6 here i come!

[morph166955]

ok i loaded up fedora core6, compiled x264, and piped the megui test through mencoder using the x264 options in their job1-2.xml file for a first pass. i'm still only getting 35-40% cpu utilization on the first pass putting me at ~177fps (yea i know im complaining about 177fps on x264...sounds almost too good to be true right).

@akpenguin
what would be the most optimal way for me to create a file from the test mpeg into something that a non-y4m patched x264 can read the fastest? ffmpeg? mplayer/mencoder?

[akupenguin]

ffmpeg, mplayer, and mencoder will all produce identical yuv files.
shortest commandline is `ffmpeg -i test.mpg -f rawvideo -y test.yuv`

I might have reproduced your problem (though not as great a magnitude given only 2 cores). I have two core2 duo systems here, and one gets 105% smp efficiency (i.e. 100% cpu use and 5% less total cpu-time than 1 thread needed) while the other gets 90% efficiency (i.e. 85% cpu use and 5% less total cpu-time). This is true even with intra-only, which should have no synchronization overhead at all. So I don't know why it happens, but at least I can test alternatives.

[morph166955]

Heres some CPU Utilization numbers for you. Source is generated from using the megui test .avs file thru mencoder into a raw yuv file. x264 runs were done using the megui test options (see below). You must divide the cpu percentage by 8 i believe to get the average. Pass 1 is still only around 45% and pass 2 is around 80%. These final #'s were generated after the passes were all run (they were obviously run pass1 then pass2 not all the 1's then all the 2's). I did 10 runs with these settings.


PASS1:
encoded 1488 frames, 189.16 fps, 989.84 kb/s
27.75user 0.42system 0:07.87elapsed 357%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 182.12 fps, 989.84 kb/s
28.00user 0.41system 0:08.17elapsed 347%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 187.77 fps, 989.84 kb/s
27.84user 0.41system 0:07.93elapsed 356%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 188.89 fps, 989.84 kb/s
27.83user 0.46system 0:07.88elapsed 358%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 188.22 fps, 989.84 kb/s
27.87user 0.42system 0:07.91elapsed 357%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 187.57 fps, 989.84 kb/s
27.79user 0.40system 0:07.94elapsed 355%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 185.17 fps, 989.84 kb/s
27.85user 0.41system 0:08.04elapsed 351%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 187.65 fps, 989.84 kb/s
27.78user 0.46system 0:07.93elapsed 356%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 185.20 fps, 989.84 kb/s
27.72user 0.42system 0:08.04elapsed 350%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 188.29 fps, 989.84 kb/s
27.77user 0.44system 0:07.91elapsed 356%CPU (0avgtext+0avgdata 0maxresident)k

PASS2:
encoded 1488 frames, 116.38 fps, 1009.85 kb/s
84.39user 0.90system 0:12.87elapsed 662%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.82 fps, 1009.74 kb/s
84.26user 0.91system 0:12.82elapsed 664%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.19 fps, 1009.75 kb/s
84.25user 0.90system 0:12.88elapsed 660%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.74 fps, 1009.39 kb/s
84.41user 0.91system 0:12.83elapsed 665%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.34 fps, 1009.38 kb/s
84.42user 0.84system 0:12.87elapsed 662%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.08 fps, 1009.44 kb/s
84.27user 0.93system 0:12.90elapsed 660%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 116.15 fps, 1009.78 kb/s
84.28user 0.89system 0:12.89elapsed 660%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 115.93 fps, 1009.79 kb/s
84.22user 0.92system 0:12.91elapsed 659%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 115.99 fps, 1009.38 kb/s
84.24user 0.87system 0:12.91elapsed 659%CPU (0avgtext+0avgdata 0maxresident)k
encoded 1488 frames, 115.59 fps, 1009.48 kb/s
84.33user 0.87system 0:12.95elapsed 657%CPU (0avgtext+0avgdata 0maxresident)k


x264 options:

x264 --pass 1 --bitrate 1000 --stats test-NEW.stats --bframes 3 --b-pyramid --direct auto --subme 1 --analyse none --vbv-maxrate 25000 --me dia --merange 12 --threads auto --thread-input --progress --no-psnr --no-ssim -o /dev/null stream.yuv 640x480

x264 --pass 2 --bitrate 1000 --stats test-NEW.stats --ref 3 --bframes 3 --b-pyramid --weightb --direct auto --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --me umh --merange 12 --threads auto --thread-input --progress --no-psnr --no-ssim -o /dev/null stream.yuv 640x480

[AGDenton]

What happens if you lower mvrange-thread manually and increase the number of threads?

[akupenguin]

The extreme case is --keyint 1. I-frames can all be encoded independently without any mvrange-thread.

[morph166955]

@akupenguin
how is mvrange-thread calculated in x264 when its set to auto (i assume thats the default?)

I'm a little confused about the whole mvrange-thread thing all together as far as what it exactly does and there isnt a whole lot of stuff on here minus the random post saying "yes use it its good". Whats a good way for me to calculate it to optimize speed w/o it being overly crazy where it will affect my bitrate?

[AGDenton]

The extreme case is --keyint 1. I-frames can all be encoded independently without any mvrange-thread.

That's the case here? I thought default keyint was 250 or something...

[akupenguin]

That's the case here? I thought default keyint was 250 or something...
Right. But if you're going to change options, keyint 1 is the obvious one to try.

how is mvrange-thread calculated in x264 when its set to auto
Divide the frame height by the number of threads, subtract the buffer needed for encoding and filtering. That's the amount of space available between threads.
Then take half, round up to a multiple of the macroblock height. That's the space allocated to mvrange-thread.
The other half is scheduling leeway, so that threads don't have to wait for eachother after every row.

The derived value of mvrange-thread is printed at the beginning if you run `x264 -v ...`

[morph166955]

ah...cool. thanks for the explanation! its comming up now default at 24. I'm moving some files around right now on those drives so once thats done (should be soon) I'll run a few passes with lower numbers for that (im going to try it first at threads=12 first for different values to see what that does, then ill play around with different #'s of threads)

[akupenguin]

I think thread synchronization benefits from an illustration.
http://akuvian.org/images/x264_threads_pic.png
Assume 3 threads and no B-frames, so the threads are encoding consecutive P-frames. Assume mvrange-thread = 40 (2.5 macroblocks). In the illustration I ignore the temporal offsets and pretend it's one frame, to better show the spatial relationship of the threads.
black = macroblock boundaries.
red = macroblock currently being encoded by the numbered thread.
blue = pixels that have been encoded, but have not been deblocked and hpel filtered yet.
dark green = pixels that have been fully encoded and filtered by the thread below, and are avilable for use (motion search) by the thread above. The height of this band is equal to mvrange-thread.
light green = pixels that have been fully encoded and filtered by the thread below, but the thread above hasn't synchronized yet so it doesn't know they're available.

At the time of this snapshot, threads 1 and 2 are happily encoding, but thread 3 is waiting on thread 2. 3 won't continue until it has another row's worth of pixels available, because if it continued now it would only have a 24 pixel range to search.

mvrange-thread shouldn't directly affect speed, rather it controls the number of threads that can run concurrently. (Which is why there is no single default value of mvrange-thread, and instead it's derived from the number of threads requested.) Each thread takes up some amount of vertical space (mvrange-thread + 1.5 macroblocks (the .5 comes from deblock and hpel, which each require some border)), and all the threads that are running in referenced frames at once must be able to stack vertically within the height of one frame. Non-referenced B-frames are exempt, since nothing is waiting for them.

[morph166955]

im still running tests but I had to post this up...even though keyint=1 is insane I ran it anyway...x264 set @ threads=12, mvrange-thread=24, keyint=1

encoded 1488 frames, 289.30 fps, 1005.11 kb/s
17.49user 0.27system 0:05.14elapsed 345%CPU (0avgtext+0avgdata 0maxresident)k

encoded 1488 frames, 219.21 fps, 1015.43 kb/s
44.20user 0.80system 0:06.86elapsed 656%CPU (0avgtext+0avgdata 0maxresident)k


yea...thats 289fps!!! and it still is only at 43% cpu!

Minus altering keyint, setting mvrange-thread is only affecting me by 3-5fps on each pass and that can be contributed to other processes using the drive for a ms here and there. 24 was the default mvrange-thread i got from -v so i included it here.

[morph166955]

@ak i just saw your post after i posted my last one...that was a very good explanation of it. THANK YOU!

[morph166955]

ok im getting a little stumped here. ive just tested all sort of values of mvrange_thread and mvrange from 2 up thru 48 (all different values since I thought I was going nuts and/or had something really backwards). the most that this does is a 3-5fps difference here or there (186-190 is basically the range, close to 188 most times). other then keyint=1, nothing has gotten this thing to speed up at all. even at keyint=1 the thing cpu ran at like 45%. I tried to access the source off of the 10K RPM to see if there were any speed differences...none.

[morph166955]

holy multiple posts batman...but i think i have a reasoning for the speed cap. Instead of just running on one small video clip like the megui test, i decided to test it on something longer so i picked the dvd of die another day. i imaged my dvd down to an iso so that there was no lag from that. i then used mkfifo to create a pipe. i had mencoder do the rawvideo output and i resized it using scale=854:480,crop=848:352:4:62,pullup. after letting it sit i noticed that when watching it in top (in thread view as opposed to an overall view) the cpus are all running somewhere between 20 and 60 percent however on each refresh theres only one active thread of x264 (occasionally theres a second one running at 2% on one core). shouldn't i see multiple threads running at the same time? is it possible that x264 is either not requesting enough data or that im unable to send the data to it fast enough in some form or something like that?

[akupenguin]

That's just a limitation of top. It only displays threads that live longer than top's refresh rate (default: 1 sec). But x264 spawns a new thread for each frame and kills it when the frame is done, which is much less than 1 sec at 289 fps, so you're only seeing the master thread. The total cpu usage (in the header lines) should still be correct. Another case this limitation shows up is if I'm compiling lots of small c files, each instance of gcc goes by quickly enough that I can be at 100% cpu usage without any individual processes showing as taking any cpu.

[morph166955]

ah...ok. well pass 1 just finished on it using the identical settings of the megui test (while probably not deseriable in all cases i wanted to keep as much as i could the same to avoid speculation on other issues). it finished at 112fps and through out the entire encode top read between 35 and 40% cpu. pass2 is running now and its between 65 and 85 percent hovering in the mid 70's and occasionally peaking to 89 but thats rare.

[morph166955]

aku let me ask you this then since im still kinda new with the whole x264 threading thing (this is really my first threaded box that actually had the power to do anything). if you were running x264 on my box with this source (die another day from dvd/iso @ 848x352) what would your command lines be for these two passes?

[morph166955]

bump...still havin the bottleneck...any other ideas guys?

[AGDenton]

You know, apart from the thing akupenguin's trying to identify with >100% SMP efficiency, the issue here really is that your job (a not-really intensive processig of a DVD source) is just not easily parallelizable. You can already see it here : the second pass, which is more intensive than the first, parallelizes more than the first (>650% CPU vs ~350% CPU). In other words, you're hitting the ceiling of the "sliceless threading" algorithm.

This ceiling will be higher (and you want it to be >800% for your Xeons) if you process higher-def content, or process it in a more intensive way. So try doing HQ-Insane on a 1080p source, and you'll probably find that all your cores are all at work :).

If you want to process this particular job faster, you'll have to ditch x264's threading and use x264farm instead. Try setting up a controller and an agent under your computer in XP...

AG

[morph166955]

I think that inevitably what im going to do is write a script/program similar to x264farm but for a local encode setup to remove any overhead from the network stuff and avoid the issue that x264farm requires a windows controller which I wont have in this setup.

[akupenguin]

But there's no good reason for sliceless threading to have a ceiling. If intra-only doesn't get perfect scaling on an arbitrarily high number of cpus, that's a bug. And it's probably the same bug that's limiting scaling in the real use cases too.

[morph166955]

Well you are the x264 master...and I am ready willing and able to run what ever kind of tests you want me to run on this to assist in diagnosing and repairing any bug or what ever you want to call this issue. I'm still going to attempt to work on some software similar to x264farm but for local use. If nothing else, it should let me pause encodes for large stuff and for now let me peg my cpu. What I figure may work "best" is running 4 x264 processes, each with threads set to 3. That "should" give me the 12 threads that I would have originally run, and alleviate the issue I'm seeing here. I'll let you know how it goes.

@akupenguin
In regards to my above software, I want to run a pass0 similar to x264farm (at least from what I have heard about it). What would be a good way to run a very fast pass over the source to find my scenecuts so that I can setup the pass1 and pass2 files to make the most efficient use of x264's encodes

[akupenguin]

This has probably been discussed in the x264farm threads.
Short answer: overlap the 1st pass chunks by about 1 gop length. Pick any I-frame in the overlap region, and make that the split point.

[morph166955]

ok so let me make sure im 100% right with this before i go coding something. lets for arguments sake say that there is an I frame every 250 frames (so 0, 250, 500, etc). I could do the encodes so that lets say the first chunk is 0-999, the second chunk so its 750-1999, the third chunk at 1750-2999, the fourth at 2750-3999, and so on and so forth (obviously those lengths are arbitrary and involves a whole lot of unnecessary encoding but lets just go with it for the theory/understanding of it and I'll tinker with good values for distance of the chunks).

[akupenguin]

yes.
You can reduce the amount of redundant work if you have some monitoring process to terminate each encoder when it finds a split point, but the simple way is as you said.

[morph166955]

awesome. what options would you recommend to do a super fast pass to generate an accurate stats file that I can parse to get those I files? (im still kinda getting used to using x264 straight as opposed to through mencoder so please bare with me until i get all the syntax down)

[akupenguin]

While I suppose I do have some favourite settings, I try to leave recommendations to 3rd parties :) See whatever megui uses for "fast 1st pass".

[morph166955]

fair enough...ill check it out. I'm also running some passes using mencoder to pipe a raw yuv stream and then doing a SAD comparison of the Y parts (thanks omion for the idea!). assuming I can tweak that to settings comparable to x264 that should allow me to compare the frames w/o ever encoding them. Once I have that data then deciding where I want to make my splits should be a matter of personal pref (ill prob put a cli in so i or anyone else using this can run it at what they want)

[akupenguin]

I added some timing constructs to find out where the idleness was happening. afaict most of it is due to a delay between when one frame finishes and when the master thread notices it. If the frame is already done when the master calls pthread_join then it returns immediately, but if it has to wait then there's an additional delay (2.5ms) beyond the time needed to finish the frame.
This might be fixed by switching to a thread-pool. It's a shame, I liked the "call a subroutine in a new thread" model better, but I guess there's a reason most threaded apps use thread-pool.

[morph166955]

ah...makes sense. I'm going to be running some tests using my memory pipe software and x264 later on to see where it maxes out. as of last night, it looked like it was between 3-6 threads. more info to come later!

oh and just to throw something else in the pot...when doing a straight SAD comparison of the frames when the source is in the ram...it was working at close to 950fps (that was using 8 sad comparison pthreads). while i realize that an sad comparison is no where near the amount of work encoding takes, it definitely shows that the system is capable of doing it.

[Mutant_Fruit]

If the frame is already done when the master calls pthread_join then it returns immediately, but if it has to wait then there's an additional delay (2.5ms) beyond the time needed to finish the frame.
2.5ms is a fairly steep cost if it's a fixed delay. The faster your machine is, the worse that delay is as compared to the time needed to encode a frame. At 100fps, that'd represent a 25% increase in time needed to encode a frame (if the thread was forced to wait).

[akupenguin]

I'm sure it's not a fixed cost, and it's not even constant (2.5 is just the average), but the settings I measured that on were running at 140fps.

[morph166955]

@aku...do you want me to run that code on this system to see what I average time wize?

[akupenguin]

apply patch (http://akuvian.org/src/x264/x264_thread_delay.diff).
run: x264 $options --quiet &>log
run: x264_thread_delay.pl (http://akuvian.org/src/x264/x264_thread_delay.pl) log
and/or send me log.

x264_thread_delay.pl should print something like
avg delay (join first): 2459 us (1535 samples)
avg delay (stop first): 11 us (693 samples)
avg idle before join: 591 us (693 samples)

"avg delay (join first)" is what I referred to above.
"avg idle before join" is only valid if # threads <= # cpus. Otherwise, it's just the time that a given thread spends idle, which doesn't necessarily mean the cpu was idle.

[morph166955]

OK here are your results. CPU was running ~35-40%, pass averaged 137fps. run using:
mencoder dvd://1 -dvd-device DIEANOTHERDAY_DISC1.ISO -ovc raw -of rawvideo -nosound -o stream.yuv -vf harddup,scale=854:480,crop=848:352:4:62,format=i420 -fps 24000/1001 -mc 0 -noskip -frames 5000

x264 --threads 8 --progress --bitrate 1025 --stats DIEANOTHERDAY_DISC1.stats --bframes 3 --b-pyramid --subme 1 --analyse none --me dia --pass 1 --fps 23.976 --quiet -o /dev/null stream.yuv 848x352 > joinlog.txt 2>&1

Your pearl script gave me:

avg wallclock per frame: 7271 us (5000 samples)
avg delay (join first): 297 us (4 samples)
avg delay (stop first): 23 us (4996 samples)
avg idle before join: 34955 us (4996 samples)


If you want the log itself let me know and ill send it to you. That average idle before join seems abnormally high...

That was the first 5000 frames, I'm going to run it full through now to see what that does.

[akupenguin]

updated patch (http://akuvian.org/src/x264/x264_thread_delay.01.diff).
This time I want the following option sets, with the log itself:
--subme 1 --analyse none --qp 24 --bframes 3 --b-pyramid
--subme 1 --analyse none --qp 24 --scenecut -1 --bframes 3 --no-b-adapt
--subme 1 --analyse none --qp 24 --scenecut -1 --keyint 1

5000 frames each is fine, unless your longer run gives wildly different numbers. (And of course still make sure the input is on a ramdisk or equivalent)

[morph166955]

The full length run just ran and it spit this out:

avg wallclock per frame: 7428 us (190375 samples)
avg delay (join first): 182 us (7 samples)
avg delay (stop first): 25 us (190368 samples)
avg idle before join: 35815 us (190368 samples)

I'm applying your patch and running it now. I'll let you know shortly.

[morph166955]

aku am i applying this new patch in addition to the other or should i put it on a clean x264?

EDIT: nevermind, i see that the new on includes the old one...

[morph166955]

ok heres the output & cli's from the runs. I ran this from my ram buffer that i wrote. 5000 frames each. the last run managed to get the cpu up to around 60%.


./x264_thread_delay.pl job1.txt
avg wallclock per frame: 5705 us (5000 samples)
avg delay (join first): 995 us (4 samples)
avg delay (stop first): 20 us (4996 samples)
avg idle before join: 22907 us (4996 samples)
FPS: 175.67

./x264_thread_delay.pl job2.txt
avg wallclock per frame: 3615 us (5000 samples)
avg delay (join first): 879 us (127 samples)
avg delay (stop first): 13 us (4873 samples)
avg idle before join: 10603 us (4873 samples)
FPS: 273.70

./x264_thread_delay.pl job3.txt
avg wallclock per frame: 3808 us (5000 samples)
avg delay (join first): 369 us (57 samples)
avg delay (stop first): 10 us (4943 samples)
avg idle before join: 9930 us (4943 samples)
FPS:259.60


run with:

x264 --subme 1 --analyse none --qp 24 --bframes 3 --b-pyramid --threads 8 -o /dev/null stream2.yuv 848x352 > job1.txt 2>&1

x264 --subme 1 --analyse none --qp 24 --scenecut -1 --bframes 3 --no-b-adapt --threads 8 -o /dev/null stream2.yuv 848x352 > job2.txt 2>&1

x264 --subme 1 --analyse none --qp 24 --scenecut -1 --keyint 1 --threads 8 -o /dev/null stream2.yuv 848x352 > job3.txt 2>&1


as far as the ram cache thing goes...the way that I've been running it is:

start mencoder & set to write to stream1.yuv
start x264 & set to read from stream2.yuv
start ram cache...it reads all 5000 frames into memory from mencoder & then writes them out to stream2 as fast as it can. I think the only way that I can make it faster is to not freeup the ram until its written all the frames out but I dont really think thats causing any sort of a bottle neck since I can dump it to /dev/null

I assume thats the fastest way to run it.

EDIT: oops forgot to tack on your logs...here you go... had to 7z them so they would be small enough

[morph166955]

so i was running a few more benchmarks. if i can figure out some way to run mencoder's x264 in two threads that would optimize how fast I can pull data off the drive and decode it. while i realize this isnt a foolproof method for benchmarking, its the best i could do for now. i forked 2 identical mencoder threads and wrote a small program that would read frames from each. when i ran it on more then 2 threads, the mencoder threads seemed choked (couldnt get over 65-75% cpu) and ran slower. at 2 threads, the two mencoders ran near 100% and it gave me a hair under 400fps (~396fps).

I'm trying to figure out if I can run framestep=2 or something like that and get even frames out of one and odd from the other or something to that effect. assuming i can then that should theoretically maintain a stable enough memory cache for the speeds im looking for since i doubt ill surpass that (im averaging like 200fps when im trying to simultaniously fill the back of the cache and work on the frames in the front of it.) So if that works...then that hopefully removes the last bottleneck that i'm should find when writing my x264 parallel encoder.

p.s. ill just note now before anyone says "well theres your problem" that the 5000 frame tests above were not done with that program, it was done with one that does all the reading into memory and then dumps that out.

EDIT:
I love how i come up with an idea, then the moment after i type it all out and post it i have a brain storm on how to do it. I'm officially decoding H264 in linux multithreaded. it was rather simple and all thanks to the tinterlace filter in mencoder. i have one set to tinterlace=1 and the other to 2. i wrote a small program to read both and then spit both back in order and if you weren told that It was running that way you wouldnt be able to see a difference. Its late and i have a test tomorrow morning then class till late tomorrow night so I doubt that I'll be able to implement this how I want to tomorrow but hopefully i'll be able to get this working on friday and if that happens then I shouldnt have any issues writing some parallelizing software over the weekend/next week/and on and on and on since thats how software is.

[akupenguin]

updated patch (http://akuvian.org/src/x264/x264_thread_delay.04.diff) and updated parser (http://akuvian.org/src/x264/x264_thread_delay.04.pl). same option sets.

[morph166955]

All three had the following first 3 lines from the parser:


Use of uninitialized value in subtraction (-) at ./x264_thread_delay.pl line 22, <> line 8.
Use of uninitialized value in subtraction (-) at ./x264_thread_delay.pl line 22, <> line 13.
Use of uninitialized value in subtraction (-) at ./x264_thread_delay.pl line 22, <> line 29.



Job1:

avg wallclock per frame: 5670 us (5000 samples)
avg cpu per slice: 15130 us (5000 samples)
avg prolog_cp: 1531 us (5000 samples)
avg prolog_rc: 2645 us (5000 samples)
avg prolog_etc: 706225484933 us (5000 samples)
avg prolog: 21188 us (5000 samples)
avg thread create: 1815 us (5000 samples)
avg delay (join first): 58 us (4 samples)
avg delay (stop first): 9 us (4996 samples)
avg idle before join: 22775 us (4996 samples)
avg epilog: 914 us (5000 samples)
avg between frames: 497 us (4989 samples)



Job2:

avg wallclock per frame: 3651 us (5000 samples)
avg cpu per slice: 14124 us (5000 samples)
avg prolog_cp: 766 us (5000 samples)
avg prolog_rc: 6 us (5000 samples)
avg prolog_etc: 706225510474 us (5000 samples)
avg prolog: 11716 us (5000 samples)
avg thread create: 948 us (5000 samples)
avg delay (join first): 710 us (118 samples)
avg delay (stop first): 8 us (4882 samples)
avg idle before join: 10912 us (4882 samples)
avg epilog: 948 us (5000 samples)
avg between frames: 1690 us (4989 samples)



Job3:

avg wallclock per frame: 5894 us (5000 samples)
avg cpu per slice: 36904 us (5000 samples)
avg prolog_cp: 1524 us (5000 samples)
avg prolog_rc: 1605 us (5000 samples)
avg prolog_etc: 12 us (5000 samples)
avg prolog: 3142 us (5000 samples)
avg thread create: 1181 us (5000 samples)
avg delay (join first): 607 us (2132 samples)
avg delay (stop first): 10 us (2868 samples)
avg idle before join: 7445 us (2868 samples)
avg epilog: 818 us (5000 samples)
avg between frames: 511 us (4992 samples)


you can pull the logs from http://www.benswebs.com/x264/ they were too big to upload even when zipped so i just put them on my webspace.

i should be around all day tomorrow and the whole weekend to run as much code as you want so just lemme know.

[akupenguin]

thread pool patch (http://akuvian.org/src/x264/x264_thread_pool.00.diff). benchmark patch (http://akuvian.org/src/x264/x264_thread_pool.00.bench.diff) to be applied on top of thread pool. parser (http://akuvian.org/src/x264/x264_thread_delay.pl). I'm interested in both the logs (as before) and in the performance of thread pool compared to svn (without the benchmark part of the patch, since that might affect performance).

thread pool should eliminate "avg thread create", and might or might not affect "avg delay (join first)". I also know how to eliminate epilog, though I didn't do it in this patch.

weirdness: a quick test (on a core2 duo) says that this patch is 3% faster than svn when using 2 threads, but 3% slower than svn when using 3 threads.

[morph166955]

im having a heat issue right now with the box...prob wont be run until tomorrow morning. i just got some zalman cpu grease but im sure you can imagine how fun its going to be to remove 2 xeon heatsyncs, clean them and the cpu's, reapply, and reassemble. ill let ya know.

[morph166955]

aku...im getting ready to run your test. do you want me to apply that patch to r650 or r651?

[akupenguin]

Shouldn't matter, as long as you use the same base version for all tests. So 651, just in case I later add something that does interact with 651.

[morph166955]

ok ran it against 650 just now (didnt see your post till it was done). I'll run it against 651 shortly.

run 1 got the cpu between 39 and 48 percent...occasionally hitting 50-52 but that was rare. it averaged 196fps

run 2 got the cpu in the low 60's. it averaged 348fps (niiiiiiice :D)

run 3 got the cpu between 65 and 75 percent. it averaged 174 fps.

you can get your log files from http://www.benswebs.com/x264/jobs-r650-threadpool.00-bench.00.tar.gz

Here is the output from your benchmark program:


avg wallclock per frame: 5085 us (5000 samples)
avg cpu per slice: 14274 us (5000 samples)
avg prolog_cp: 1410 us (5000 samples)
avg prolog_rc: 2561 us (5000 samples)
avg prolog_etc: 706305303995 us (5000 samples)
avg prolog: 19226 us (5000 samples)
avg thread create: 150 us (5000 samples)
avg delay (join first): 30 us (4 samples)
avg delay (stop first): 4 us (4996 samples)
avg idle before join: 21174 us (4996 samples)
avg epilog: 839 us (5000 samples)
avg between frames: 220 us (4989 samples)

avg wallclock per frame: 2875 us (5000 samples)
avg cpu per slice: 14160 us (5000 samples)
avg prolog_cp: 773 us (5000 samples)
avg prolog_rc: 8 us (5000 samples)
avg prolog_etc: 706305327110 us (5000 samples)
avg prolog: 9400 us (5000 samples)
avg thread create: 444 us (5000 samples)
avg delay (join first): 391 us (587 samples)
avg delay (stop first): 9 us (4413 samples)
avg idle before join: 6888 us (4413 samples)
avg epilog: 866 us (5000 samples)
avg between frames: 580 us (4989 samples)

avg wallclock per frame: 5733 us (5000 samples)
avg cpu per slice: 36137 us (5000 samples)
avg prolog_cp: 1432 us (5000 samples)
avg prolog_rc: 1638 us (5000 samples)
avg prolog_etc: 11 us (5000 samples)
avg prolog: 3083 us (5000 samples)
avg thread create: 519 us (5000 samples)
avg delay (join first): 202 us (2131 samples)
avg delay (stop first): 7 us (2869 samples)
avg idle before join: 8261 us (2869 samples)
avg epilog: 871 us (5000 samples)
avg between frames: 360 us (4992 samples)

[morph166955]

oh crap...i need to appologize for something. the script that I had running was fubaring the third job (there was a typo in one of the variables so in reality it was being called w/o any options). this only effects the last like 2-3 tests since i wrote it recently. I'm rerunning the r651 tests now with the updated script ill post up the results shortly. SORRY!

[morph166955]

this is r651 patched (and now a proper job 3). I'm going to run a test on the performance of the raw svn and then the svn + thread_pool w/o benchmark now.

run 1 is averaging between 40 and 50 percent cpu @ 199fps
run 2 is averaging between 55 and 65 percent cpu @ 360fps
run 3 is averaging between 55 and 65 percent cpu @ 327fps

Heres the output of your perl script:

Starting Run @ Sat Apr 21 13:53:29 EDT 2007

================================================================================

JOB 1: --subme 1 --analyse none --qp 24 --bframes 3 --b-pyramid X264BIN:x264-r651-patched
avg wallclock per frame: 5033 us (5000 samples)
avg cpu per slice: 13556 us (5000 samples)
avg prolog_cp: 1407 us (5000 samples)
avg prolog_rc: 2526 us (5000 samples)
avg prolog_etc: 706306835056 us (5000 samples)
avg prolog: 19033 us (5000 samples)
avg thread create: 108 us (5000 samples)
avg delay (join first): 47 us (3 samples)
avg delay (stop first): 5 us (4997 samples)
avg idle before join: 21553 us (4997 samples)
avg epilog: 847 us (5000 samples)
avg between frames: 193 us (4989 samples)
COMPLETE: DID 5000 FRAMES in 25.042000 seconds @ 199.664564 FPS

================================================================================

JOB 2: --subme 1 --analyse none --qp 24 --scenecut -1 --bframes 3 --no-b-adapt X264BIN:x264-r651-patched
avg wallclock per frame: 2793 us (5000 samples)
avg cpu per slice: 13309 us (5000 samples)
avg prolog_cp: 764 us (5000 samples)
avg prolog_rc: 7 us (5000 samples)
avg prolog_etc: 706306857886 us (5000 samples)
avg prolog: 9151 us (5000 samples)
avg thread create: 484 us (5000 samples)
avg delay (join first): 278 us (525 samples)
avg delay (stop first): 8 us (4475 samples)
avg idle before join: 6996 us (4475 samples)
avg epilog: 858 us (5000 samples)
avg between frames: 568 us (4989 samples)
COMPLETE: DID 5000 FRAMES in 13.855000 seconds @ 360.880549 FPS

================================================================================

JOB 3: --subme 1 --analyse none --qp 24 --scenecut -1 --keyint 1 X264BIN:x264-r651-patched
avg wallclock per frame: 3074 us (5000 samples)
avg cpu per slice: 13261 us (5000 samples)
avg prolog_cp: 703 us (5000 samples)
avg prolog_rc: 5 us (5000 samples)
avg prolog_etc: 11 us (5000 samples)
avg prolog: 721 us (5000 samples)
avg thread create: 233 us (5000 samples)
avg delay (join first): 562 us (152 samples)
avg delay (stop first): 9 us (4848 samples)
avg idle before join: 9222 us (4848 samples)
avg epilog: 757 us (5000 samples)
avg between frames: 615 us (4992 samples)
COMPLETE: DID 5000 FRAMES in 15.264000 seconds @ 327.568134 FPS

================================================================================

End Of Run. Sat Apr 21 13:55:36 EDT 2007


logs @ http://www.benswebs.com/x264/jobs-r651-threadpool.00-bench.00.tar.gz

[morph166955]

this is a raw r651 run:

run1 was between 35 and 40 percent cpu @ 178fps
run2 was between 45 and 55 percent cpu @ 282fps
run3 was between 45 and 50 percent cpu spiking to 56 a few times @ 274fps

this is a run with r651 patched ONLY with the thread_pool patch and not the benchmark patch:

run1 was between 40 and 45 percent cpu @ 201fps
run2 was between 60 and 70 percent cpu @ 368fps (now thats a jump!)
run3 was between 55 and 65 percent cpu @ 331fps

to me thats no appreciable difference in framerate with the benchmark vs non-benchmark runs...its like 5 maybe 10 fps difference and that could be caused by almost anything. it is however a very noticable difference with the thread_pool patch on runs 2 and 3 and a decently noticed one on run1.

[akupenguin]

Nice. Now can you try varying the number of threads (maybe between 8 and 16)? since both should benefit from threads>cpus, but there's no guarantee that the optimal numbers are the same.
Ideally the result of that would be a table of fps for each of (3 option sets) x (2 versions) x (9 thread numbers).

Radon sidenote: instead of looking at `top` and saying "35 to 40 percent", you can run `time x264 ...` and get an exact number. Less work too. Though an exact number for cpu usage doesn't really matter, since it doesn't disinguish between real work and synchronization overhead, so it can't compare two versions, it can only serve as a rough estimate of how much potential optimization is left.

[morph166955]

those were all run at threads=8 (as well as everything else i've been running since you said running at threads=8 earlier was good to get proper numbers. ill run 12 and 16 now...give me a few.

[morph166955]

just when doing some initial runs (im getting time to work properly with the forking and everything its being not nice) im noticing that one of the cores is only what i would call a "dead" core. its user cpu usage is litterally 0.0 for most of the time (it does hop around occasionally to another core but thats maybe once in the entire run). thats on threads 12 btw...

[morph166955]

ok...a table you want a table yee shall have... :D

just to make sure I'm right with this before i have it do a bagilion runs, confirm for me that these are in fact the options you want:


JOBS:
#1: --subme 1 --analyse none --qp 24 --bframes 3 --b-pyramid
#2: --subme 1 --analyse none --qp 24 --scenecut -1 --bframes 3 --no-b-adapt
#3: --subme 1 --analyse none --qp 24 --scenecut -1 --keyint 1



Versions:
#1: raw/vanilla r651
#2: r651 w/ thread_pool_patch.00.diff & thread_pool_patch.00.bench.diff


and threads set at 8,9,10,11,12,13,14,15,16

assuming thats right...ill write a script to run exactly that and have it generate a table (probably a csv file since thats easy) of the encoding time, fps and cpu. shouldnt take too long for the whole thing to run, writing that script is probably going to be the longest part of the whole thing lol.

[morph166955]

r651 + threads_pool + benchmark @ 12 threads:


Starting Run @ Sat Apr 21 15:22:41 EDT 2007

================================================================================

JOB 1: --subme 1 --analyse none --qp 24 --bframes 3 --b-pyramid X264BIN:x264-r651-patched-bench
avg wallclock per frame: 5082 us (5000 samples)
avg cpu per slice: 13578 us (5000 samples)
avg prolog_cp: 1407 us (5000 samples)
avg prolog_rc: 2542 us (5000 samples)
avg prolog_etc: 706310046150 us (5000 samples)
avg prolog: 19197 us (5000 samples)
avg thread create: 105 us (5000 samples)
avg delay (join first): 33 us (3 samples)
avg delay (stop first): 5 us (4997 samples)
avg idle before join: 42185 us (4997 samples)
avg epilog: 848 us (5000 samples)
avg between frames: 210 us (4985 samples)

264BUFFER: COMPLETE: DID 5000 FRAMES in 25.283000 seconds @ 197.761342 FPS
x264: encoded 5000 frames, 196.75 fps, 887.69 kb/s
86.65user 1.12system 0:25.42elapsed 345%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+20687minor)pagefaults 0swaps

================================================================================

JOB 2: --subme 1 --analyse none --qp 24 --scenecut -1 --bframes 3 --no-b-adapt X264BIN:x264-r651-patched-bench
avg wallclock per frame: 2669 us (5000 samples)
avg cpu per slice: 16373 us (5000 samples)
avg prolog_cp: 768 us (5000 samples)
avg prolog_rc: 9 us (5000 samples)
avg prolog_etc: 706310068257 us (5000 samples)
avg prolog: 8784 us (5000 samples)
avg thread create: 518 us (5000 samples)
avg delay (join first): 687 us (234 samples)
avg delay (stop first): 7 us (4766 samples)
avg idle before join: 13456 us (4766 samples)
avg epilog: 889 us (5000 samples)
avg between frames: 534 us (4985 samples)

264BUFFER: COMPLETE: DID 5000 FRAMES in 13.230000 seconds @ 377.928949 FPS
x264: encoded 5000 frames, 374.57 fps, 853.20 kb/s
67.63user 1.55system 0:13.35elapsed 518%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+15838minor)pagefaults 0swaps

================================================================================

JOB 3: --subme 1 --analyse none --qp 24 --scenecut -1 --keyint 1 X264BIN:x264-r651-patched-bench
avg wallclock per frame: 2678 us (5000 samples)
avg cpu per slice: 13459 us (5000 samples)
avg prolog_cp: 691 us (5000 samples)
avg prolog_rc: 6 us (5000 samples)
avg prolog_etc: 10 us (5000 samples)
avg prolog: 708 us (5000 samples)
avg thread create: 777 us (5000 samples)
avg delay (join first): 1264 us (38 samples)
avg delay (stop first): 10 us (4962 samples)
avg idle before join: 15898 us (4962 samples)
avg epilog: 777 us (5000 samples)
avg between frames: 573 us (4988 samples)

264BUFFER: COMPLETE: DID 5000 FRAMES in 13.282000 seconds @ 376.449330 FPS
x264: encoded 5000 frames, 373.29 fps, 3585.97 kb/s
72.64user 1.13system 0:13.39elapsed 550%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+6206minor)pagefaults 0swaps

================================================================================

End Of Run. Sat Apr 21 15:24:43 EDT 2007

[morph166955]

I'm about half way through the tests you wanted run and im finding something interesting. its peaking around threads=10 and then as im increasing over that its falling off. im also getting very weird and non linear results on some of these (increasing threads on one doesnt necessarily mean it will do better then a previous run however it also doesnt mean it will do worse...its kinda a crap shoot). any ideas why?

[morph166955]

you wanted a table...so i give you a table! :D

http://www.benswebs.com/x264/x264%20Runs.jpg

it should be noted...the two red columns are merely the numerical difference of the two data columns.

other then the first test threads set at 9 or 10 seems to make the most gain over the vanilla image.

hope this helps! sorry for the delay i'm still having thermal issues on this setup and the rising temperature outside surely isnt helping at all. im working on getting my landlord to install my a/c asap so i can get this place regulated a little better to prevent me from hitting thermal cut.

also note im changing the topic for this thread since its definitely taken a turn towards another focus.

EDIT: I've uploaded the benchmark outputs for all threads 1-16 to http://www.benswebs.com/x264/ in case you want those

EDIT2: There was something funky with threads=9 especially in job 1 so I reran it and it gave much more linear results to the rest of the test so I have updated the table to reflect that. I also added all of the results for threads 1-16. ENJOY!

[Adub]

It looks to be that your best performance comes from 9 to 11 threads, which is very interesting. I am always interested in maximizing performance and these tests are broadening my knowledge, while also forcing me to reconcile the fact that the best number varies with each application.

[akupenguin]

x264_thread_pool.01.diff (http://akuvian.org/src/x264/x264_thread_pool.01.diff)
Now it no longer explicitly assigns one frame to one thread, but keeps a queue of frames and lets each thread grab the next frame whenever it's done with the previous. Queue size is --thread-queue and defaults to 2*threads.
No benchmark patch this time, as most of the numbers wouldn't mean the same thing in this algorithm.

As before, I'm interested in the table of fps vs threads.

[morph166955]

sounds good...im setting up the job to run now. i'm only going to run it by itself since we already have the numbers for the vanilla 651 build. gimme a few hours ill have some shiny results for you.

do you want me to modify --thread-queue or just leave it to default?

[morph166955]

I'm getting a hell of a lot of errors from x264:


x264 [warning]: specified frame type (5) is not compatible with keyframe interval
x264 [warning]: specified frame type (5) is not compatible with keyframe interval
x264 [warning]: specified frame type (3) is not compatible with keyframe interval
x264 [warning]: specified frame type (3) is not compatible with keyframe interval


also...its not stoping when its running out of frames to process. the buffer shows that its dumped all 5000 frames (its final output before exiting is "COMPLETE: DID 5000 FRAMES in 25.137000 seconds @ 198.909973 FPS") but x264 is frozen at 4996 frames (run with --progress -v). I've uploaded the log to http://www.benswebs.com/x264/x264-threadpool.01-threads8.err.txt so you can see whats going on.

[akupenguin]

x264_thread_pool.02.diff (http://akuvian.org/src/x264/x264_thread_pool.02.diff)

x264 [warning]: specified frame type (5) is not compatible with keyframe intervalThat was a simple typo. (didn't notice because my benchmark clip didn't have any scenes longer than keyint)

And the hang was a race condition whereby I locked the wrong mutex so a variable that changed wasn't protected by anything.

Random note: --thread-input (which is implied by --threads) still uses the create/join method. But that shouldn't matter for rawyuv input.

[morph166955]

I havent gotten completely through yet but there seems to be a very steady drop off of fps & cpu usage above threads=7. However, it should be noted that on job 3 @ threads=7 i hit 741% cpu (800 being the max) and a record of 498.72 fps.

i'm still having my thermal problems so it seems that these tests wont be finished until tonight when I can get this place a little cooler.

[morph166955]

aku im sorry to do this but its unavoidable. i cant run this system in here until the a/c gets installed. the machine is hitting thermal cut before i can even get through 1 full set of jobs for 1 thread setting. i talked to my landlord and there supposed to get the a/c in either tomorrow or tuesdsay at the latest so once that goes in were good to crank this thing 24/7. i just dont want to risk burning this thing out now after all that I have spent to get it going in the first place. I'll let ya know once I run the tests which I hope is tomorrow but no promises. Thanks again for all your help on working this out and don't think for a second that this speed bump is going to stop me once I'm past it.

[Hellworm]

akupenguin: You propably know that, but the patch also slightly increases performance (by 2-3%) on not so extreme situations (dual core): cpu usage increased to around 190% (before:185%) and fps to 37 from 36. Tested with mencoder, x264 revision 654, 3 threads, 5 bframes. With 5 threads it reaches to 195%/38fps.

[Mutant_Fruit]

While i *really* doubt this will have much performance impact, i as wondering if popping more than one chunk from the queue would make an appreciable difference in performance. I suspect the cost of acquiring/releasing the mutex as compared to processing a chunk from the processing queue is relatively small, so this would be a bit pointless, but i don't know the exact cost of synchronisation.

It'd only really make a difference if lock contention was high and there was a large delay in getting the lock as compared to the time needed to process the chunk.

EDIT: The reason i mentioned this is because morph166955 mentioned that there appears to be a steady dropoff in FPS for larger numbers of threads. If increased contention is really the cause of the performance dropoff, it might make the above change worthwhile.

EDIT 2: A good way to check this would be to put some logging code that measures the time between trying to get the lock and actually getting it and running that with a large number of threads (4-5) on a single core. If someone could supply me with a win32 build that does that, i'll run the benchmarks.

[morph166955]

ok the guy is supposedly coming tomorrow with the a/c so I should be able to generate all sorta fun results for you by then. just fyi im not going to be around thursday & friday i'm flying out of town for a job interview I should be back early saturday I can run more tests once i'm home. im kinda crossing my fingers right now that its the temperature of this room and not the computer itself causing the heat problems. i should know more in a few days.

[burfadel]

What kind of room temperature are you talking about? If its above 30C (86F) then lowering it to 22C (72F -standard room temperature) should definately at least help. If the temperature is around 30C and its overheating then better case circulation should help... If you're talking 35C (95F) or above then I say your problem is the room temp... at idle on a non-overclocked system you can run them in 45C heat, but they're really not designed for that! Aren't computers designed to be operated at standard international room temperature (22C)?...

[akupenguin]

i as wondering if popping more than one chunk from the queue would make an appreciable difference in performance.
Not possible. A thread can't take 2 consecutive frames from the queue, because the threading method relies on encoding N consecutive frames at once. If any are skipped, then the following frames can't start either. A thread can't take 2 non-consecutive frames, because it would have to know in advance exactly how long it will take to encode the first frame. If frames don't finish in the planned order, then again some threads are waiting for others.

The reason i mentioned this is because morph166955 mentioned that there appears to be a steady dropoff in FPS for larger numbers of threads. If increased contention is really the cause of the performance dropoff, it might make the above change worthwhile.
It could be lock contention, or it could be scheduling contention (extra context switches).

[morph166955]

the room is around 80F give or take but its a small room and their is barely any circulation so its stagnant. when the machine is not doing anything it runs around 45C...it hovers around 55-60C normally when running however if there really going for it one hits 62-63C and its game over. i'll have a definite answer tomorrow when the a/c goes in. i may yank out the 7 in 1 card reader in front so that theres more inlet space up there (not too much up front but there are ALOT of slits along the side to let heat in and out. I'll figure it out I have a few good ideas its just a matter of eliminating the room as a cause.

[morph166955]

It could be lock contention, or it could be scheduling contention (extra context switches).

have you seen the article (i think i saw it on slashdot yesterday) about the scheduling updates that are due to go into the linux kernel soon? Think that will help with any of that?

[akupenguin]

I read that article, but I can't tell if it affects x264. Our ideal situation is one thread per cpu, no context switches except to grab a new frame. If we need more threads than cpus (to fill in gaps from synchronization within a frame, since the gaps between frames are handled by the queue), then it's scheduling between identical tasks. In the case of identical tasks, the only possible scheduler parameter is the tick size, which is already configurable.

edit: ok, I benchmarked the scheduler patch. it significantly improved (+5%) the speed of x264.thread_pool.0 with >= 3 threads (on 2 cores). It had no significant effect on <= 2 threads nor on x264.thread_pool.2 nor x264.svn.

[morph166955]

yay for it being cool again in here! im running the thread_pool.02 passes now, I'll post an updated table soon. as it stands now though from the runs that are done, threads >= cores slows down as threads increases. threads = 7 seems to be the fastest actually on all 3 runs (wonder why not threads = 8)?

EDIT: I stand corrected, it seems that the plateau is at threads = 5 not threads = 7. I'm putting the rest into the table now but thats what it seems like is going on. I only ran up to threads=12 because of the noticed drop off as i got up that high.

EDIT2: Updated table to show thread_pool.02. Somethings going funky with my webspace at this moment and its timing out for some reason so if its not fixed shortly ill figure something else out. long story short, threads between 5 & 7 depending on the job is the fastest on this patch.

EDIT3: My webspace is running ungodly slow right now for reasons unknown to me, http://www.benswebs.com/x264/x264%20Runs.jpg in case the image fails to load on my other post theres a link directly to it. you may have to hit refresh a few times for it to go but it will go eventually it just did for me. sorry for the annoyance I dont know why its doing it at all.

[burfadel]

EDIT3: My webspace is running ungodly slow right now for reasons unknown to me, http://www.benswebs.com/x264/x264%20Runs.jpg in case the image fails to load on my other post theres a link directly to it. you may have to hit refresh a few times for it to go but it will go eventually it just did for me. sorry for the annoyance I dont know why its doing it at all.

Seems to be fine now... but of course now that I've said that!

[morph166955]

yea the host was probably running something that was bogging the box...possibly a backup or something im not sure its hosted elsewhere.

[legoman666]

I would like to do some testing with the updated x264 build with the thread pool, but have no idea how to apply the patch to the source and compile it for win32. Because as it is, x264 can only keep my 2.6ghz AMD X2 @ ~75-90% utilization with 3 threads and I really would like to get the full potential out of this thing :D

[morph166955]

just cause i'm a nice guy...

http://www.benswebs.com/x264/x264-r654-thread_pool.02-win32.exe

and hers a non patched version of r654 to use as a vanialla build to compare too:

http://www.benswebs.com/x264/x264-r654-vanillasvn-win32.exe

compiled with mingw32. its r654 w/ thread_pool.02 patch applied (benchmark patch NOT applied). compile done using my own build of gcc 4.3.0, no mp4 support (i dont have gpac loaded and most people use raw264 or mkv output anyway), and I only used make (not fprofiled, ill make some source up eventually to do a fprofiled build but I dont think it will do very much here). also note this does NOT have the aq patch loaded into it. If you really really really want one with aq loaded i'll consider making a build with it however I think for the purposes of this test putting anything additional onto the vanilla svn is a bad idea since it may skew the results and give inaccurate data.

EDIT: just remember, while the threadpool patch will probably show you something on a dual core, its much more apparent when you have 4 (or in my case 8) cores. also run tests, do so with a raw vanilla build (posted above) and with the patched build at threads 1 thru 3 if your going to run tests on this so we have a good basis to compare. I'm going to work on compiling my ram buffer for windows so that we eliminate the hard drive as a bottle neck (which i've found it can be).

[legoman666]

Most of the encoding I do is super high quality at 720p, and at that high of a bitrate and high quality, it goes so slow that i could read aloud the 1's and 0's for the encoder and it still wouldn't bottleneck. :P

The tests I'll do will be tomorrow (current encode I'm doing will be done in 25 hours so I have to wait until that is done). To do the comparisons, I'll most likely try converting blu ray -> 720p at various bitrates and quality settings.

Thanks for the binaries :D

[morph166955]

I think I may end up doing the same to see if a larger frame into x264 will fill in the gaps. I'll load up a 1080p video and pipe it into x264 to see if the larger video does anything for me speedwize on this. i'm running a test now to measure how fast i can decode a dvd off of a drive and get that into memory to see what my upper limit is.

also note...i think i was wrong about the cpu heat thermal cut thing...i am now under the belief that its my ram thats reaching a thermal limit. fbdimms get HOT! i'm probably going to order out one of those ram coolers to see if that will make this play nice.

[morph166955]

ok so I just ran completely through x264 both passes without hitting thermal cut so I think im good with the whole heat thing assuming that i can keep this place cool finally so I think were good for running tests so aku...bring it on!

[legoman666]

Hmmmm I'm running into a problem when I try to use more than 1 thread on the thread pool build. Once it encodes the 3rd frame, it just stops.

avis [info]: 1280x688 @ 23.98 fps (501 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [warning]: VBV maxrate specified, but no bufsize.
encoded frames: 3/501 (0.5%), 3.10 fps, eta 0:02:40
I:\hdtv\megui>

no error message or anything. Is there some setting or command I;m doing incorrectly? :(

[morph166955]

whats the exact command your running and how are you passing the source to x264?

[legoman666]

--pass 2 --bitrate 10654 --stats "C:\innocence.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "C:\innocence.mkv" "C:\innocence.avs"

innocence.avs is a fairly simple job:

LoadPlugin("I:\hdtv\megui\tools\dgindex\DGDecode.dll")
LoadPlugin("C:\PROGRA~2\GORDIA~1\AviSynthPlugins\UnDot.dll")

SetMemoryMax(64)

mpeg2source("C:\innocence.d2v")

trim(1000,1500)
crop(0,24,1920,1032)
LanczosResize(1280,688)
Undot()

The same command works just fine with the vanilla x264.

[morph166955]

the only thing i can think of is that i didnt put the avis stuff in properly when i compiled it but i was under the impression that was something built into x264 and not an additional library i had to add. i assume you tested it with both of my builds as well as someone elses build?

[legoman666]

Yea, it worked fine in the vanilla x264 you compiled and it works fine with the r654 x264.exe I grabbed from x264.nl.

[morph166955]

intersting. whats your source?

[legoman666]

its an mpeg2 blu ray file (which the newest version of DGIndex supports, w00t)

[morph166955]

i just used the threadpool.02 on a hddvd source (scaled 1080p -> 720p) and used some really insane settings and i was able to get it to 90-95% cpu usage! wahoo!

[bob0r]

http://x264.nl/x264_654.exe
http://x264.nl/x264_654_thread_pool_02.exe

(difference between this and x264.nl version, this is not compiled with make fprofiled)


$ patch -p0 < x264_thread_pool.02.diff
patching file `encoder/slicetype.c'
patching file `encoder/encoder.c'
Hunk #5 succeeded at 952 (offset -6 lines).
Hunk #7 succeeded at 1310 (offset -6 lines).
Hunk #9 succeeded at 1526 (offset -6 lines).
Hunk #11 succeeded at 1819 (offset -6 lines).
patching file `encoder/ratecontrol.c'
patching file `encoder/analyse.c'
patching file `x264.c'
patching file `common/cpu.c'
patching file `common/cpu.h'
patching file `common/pixel.c'
patching file `common/common.c'
patching file `common/amd64/deblock-a.asm'
patching file `common/common.h'
patching file `x264.h'

[legoman666]

http://x264.nl/x264_654.exe
http://x264.nl/x264_654_thread_pool_02.exe

(difference between this and x264.nl version, this is not compiled with make fprofiled)

both yours and morph's threadpool exe quit after encoding the 3rd frame when using more than 1 thread. Works fine with 1 thread. i've tried it with multiple sources and they all do the same thing.

I:\hdtv\megui>x264-r654-thread_pool.02-win32.exe --pass 1 --bitrate 3500 --stats
"C:\houselol.stats" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subm
e 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --threads 2 --thre
ad-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "C:\_Hous
e 320.avs"
avis [info]: 1280x720 @ 59.94 fps (101 frames)
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!
x264 [warning]: VBV maxrate specified, but no bufsize.
encoded frames: 3/101 (2.9%), 10.68 fps, eta 0:00:09
I:\hdtv\megui>

it just stops.

[bob0r]

Roger that, lets wait for the penguin to raise from his sleep!

[akupenguin]

x264_thread_pool.03.diff (http://akuvian.org/src/x264/x264_thread_pool.03.diff)
pthread_mutex_init() doesn't actually do anything on linux, so there are no symptoms when I omit one.

[bob0r]

http://x264.nl/x264_654.exe
http://x264.nl/x264_654_thread_pool_03.exe

[legoman666]

yay, that one works for threads >1!

expect some benchmarks/test for dual core soon. Just have to wait for this encode I'm doing right now to finish.

[legoman666]

As promised, here are the results. Job 1 and 2 are 2 pass, the fps I compared are for the second pass only.
Setup:
Abit AT8 32X mobo
AMD X2 3800+ @ 2.6ghz (600mhz over stock)
2gb PC3200
36gb x2 10k rpm raptors in RAID 0
Job 1 first pass settings:
--pass 1 --bitrate 2846 --stats "C:\firefly ep3 - The Train Job1.stats" --bframes 3 --b-pyramid --direct auto --filter -2,-1 --subme 1 --analyse none --vbv-maxrate 25000 --ratetol 2.0 --me dia --threads 2 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output NUL "C:\firefly ep3 - The Train Job2.avs"
Job 1 second pass settings:
--pass 2 --bitrate 2846 --stats "C:\firefly ep3 - The Train Job1.stats" --ref 10 --mixed-refs --no-fast-pskip --bframes 3 --b-pyramid --b-rdo --bime --weightb --direct auto --filter -2,-1 --subme 6 --trellis 2 --analyse all --8x8dct --vbv-maxrate 25000 --ratetol 2.0 --me umh --threads 1 --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "C:\firefly ep3 - The Train Job1.mkv" "C:\firefly ep3 - The Train Job2.avs"
Job 2 first pass settings:
--pass 1 --bitrate 2846 --stats "C:\firefly ep3 - The Train Job2.stats" --no-cabac --subme 1 --analyse none --vbv-maxrate 25000 --me dia --threads 2 --thread-input --progress --no-psnr --no-ssim --output NUL "C:\firefly ep3 - The Train Job2.avs"
Job 2 second pass settings:
--pass 2 --bitrate 2846 --stats "C:\firefly ep3 - The Train Job2.stats" --no-cabac --subme 6 --analyse p8x8,b8x8,i4x4,p4x4 --vbv-maxrate 25000 --threads 2 --thread-input --progress --no-psnr --no-ssim --output "C:\firefly ep3 - The Train Job2.mkv" "C:\firefly ep3 - The Train Job2.avs"

Job 3 single pass settings:
--bitrate 2846 --ref 3 --bframes 3 --b-pyramid --b-rdo --bime --weightb --subme 6 --trellis 1 --analyse all --8x8dct --vbv-maxrate 25000 --threads 2 --thread-input --progress --no-psnr --no-ssim --output "C:\firefly ep3 - The Train Job3.mkv" "C:\firefly ep3 - The Train Job2.avs"


Heres a link to the .xls spreadsheet for those who want to take a look: ftp://ftp:roflcopter666@72.49.176.178:47024/benchmark.xls
Anyway, here are the results, they're pretty self explanatory. First column is number of threads, second column is the vanilla.exe, the third column is for the thread pool.03.exe, and the last column is the thread pool's % gain over the vanilla.exe.

Job 1
threads: x264-r654-vanillasvn-win32.exe x264_654_thread_pool_03.exe thread pool % gain over vanilla
1 1.13 1.14 0.8850
2 1.89 1.89 0.0000
3 2.09 2.08 -0.4785
4 2.09 2.07 -0.9569
5 2.09 2.07 -0.9569
6 2.09 2.06 -1.4354
7 2.09 2.05 -1.9139


Job 2

threads: x264-r654-vanillasvn-win32.exe x264_654_thread_pool_03.exe thread pool % gain over vanilla
1 6.09 6.12 0.4926
2 9.02 9.12 1.1086
3 9.09 9.05 -0.4400
4 9.07 9.03 -0.4410
5 9.05 8.98 -0.7735
6 9.04 8.95 -0.9956
7 8.99 8.88 -1.2236


Job 3

threads: x264-r654-vanillasvn-win32.exe x264_654_thread_pool_03.exe thread pool % gain over vanilla
1 3.37 3.36 -0.2967
2 5.22 5.40 3.4483
3 5.72 5.74 0.3497
4 5.73 5.74 0.1745
5 5.71 5.76 0.8757
6 5.75 5.72 -0.5217
7 5.76 5.74 -0.3472


Kind of disappointing, thought I might see more gain from the thread pool modification, but as per the title of the thread, its mainly for >2 cores. So I wasn't expecting too much.

[delacroixp]

i just used the threadpool.02 on a hddvd source (scaled 1080p -> 720p) and used some really insane settings and i was able to get it to 90-95% cpu usage! wahoo!
Pretty miraculous... what kind of fps do you get at 720p given 18-CQ-CRF ?


:):D:eek:
Pascal


*** NOTE ***
Have you tried any multiple encodes... like 2 or 3 simultaneously....

[Mutant_Fruit]

Kind of disappointing, thought I might see more gain from the thread pool modification, but as per the title of the thread, its mainly for >2 cores. So I wasn't expecting too much.
Could you try encoding samples with both a higher and lower resolution? I'm interested in how much that affects the relative performances of unpatched versus patched. For example, if Firefly was 720p->720p, would the relative difference in performance be higher?

Doing a benchmark using 2 threads should be enough, but if you want to run it using more than 2, feel free ;)

[legoman666]

will do. the Firefly encodes I was doing were all 1080i -> 720p. I'll try some other encodes as well for reference.

[legoman666]

This job is 1080i ->1080p using the megui HQ-slowest profile. Its 101 frames of Firefly.
threads: x264-r654-vanillasvn-win32.exe x264_654_thread_pool_03.exe thread pool % gain over vanilla
2 1.09 1.1 0.9174
3 1.26 1.25 -0.7937


And this second job is 624x352 using the megui HQ-slow profile. It's 1001 frames of a House episode.

threads: x264-r654-vanillasvn-win32.exe x264_654_thread_pool_03.exe thread pool % gain over vanilla
2 17.67 18.16 2.7731
3 21.19 21.23 0.1888


The lower resolution encode actually showed a decent increase with 2 threads. With 3 threads the difference is less pronounced and quite negligable (.04fps, w00t).

Any other tests you want me to run? I'd be happy to oblige :D

[morph166955]

Pretty miraculous... what kind of fps do you get at 720p given 18-CQ-CRF ?


:):D:eek:
Pascal


*** NOTE ***
Have you tried any multiple encodes... like 2 or 3 simultaneously....

that was on crf=18 and megui's insane profile. I'll be back in town tomorrow so I should be able to run the threadpool03 tests once I get in. I'm waiting on a new power supply and some new fans but I should be good for now assuming I dont peg the cpu's for like 25 mins...

[delacroixp]

I'll be back in town tomorrow so I should be able to run the threadpool03 tests once I get in. I'm waiting on a new power supply and some new fans but I should be good for now assuming I dont peg the cpu's for like 25 mins...
I recommend Stealth 120's (http://www.vantecusa.com/product-cooling.html)... they're cool and quite...

I have 4 on my system... 2 underneath the case, which is raised 1 inch off the ground with rubber, high-rise feet... 1 out the front and 1 out the back...
What happens is that the heat from the HDD's blows out the front and the heat from the CPU and Graphics card blows out the back... the 2 fans blowing in, underneath the case, cause slight internal pressure which helps carry out the heat...
All ground level vents have been taped-up to prevent cool air blowing in and directly out again...

Actually the rear fan is a 92mm stealth since the case wasn't happy about a 120...


:):D:eek:
pascal

[morph166955]

I recommend Stealth 120's (http://www.vantecusa.com/product-cooling.html)... they're cool and quite...

I have 4 on my system... 2 underneath the case, which is raised 1 inch off the ground with rubber, high-rise feet... 1 out the front and 1 out the back...
What happens is that the heat from the HDD's blows out the front and the heat from the CPU and Graphics card blows out the back... the 2 fans blowing in, underneath the case, cause slight internal pressure which helps carry out the heat...
All ground level vents have been taped-up to prevent cool air blowing in and directly out again...

Actually the rear fan is a 92mm stealth since the case wasn't happy about a 120...


:):D:eek:
pascal

just got in...gunna run some tests soon...

fans are already ordered. 3x thermaltake A1357 80mm running in tandum blowing directly over the motherboard, 4x thermaltake cyclo CL-R0023 ram coolers, and 2x dynatron H46 cpu coolers (of the 6 socket 771 coolers, this one had the best user ratings, i hope they make more). psu should be here sooner then later it supposedly shipped however antec is kinda slow with posting tracking numbers or any of that fun stuff. i can still run the tests i just have to make sure that i give the cpu's some cooling off time every 4-5 thread runs or so.

[morph166955]

Heres threadpool.03 for threads 1-8...it leveled off after 7 on the high ones and 5 on the low ones so i didnt run any more. Charts updated, I'm going to repost it here so you dont have to go hunting for it. also fyi the old ones were r651, this was r654. the diff was built on r654 so i figured you wanted it to be against that.

http://www.benswebs.com/x264/x264%20Runs.jpg

[morph166955]

I had another thought in regards to this and a possible fix. I'm not sure if this is even possible based on how x264 is written and/or does its encode. as far as I know, x264 creates threads one at a time and each one encodes one frame. Would it be possible for x264 to do batches of frames per thread somehow? Maybe somehow read ahead several frames, buffer the frames into memory (possibly as raw yuv frames since i'd assume they are decoded by this point), do its scene cut detection on them, and then encode an entire scene instead of just one frame. i realize that could involve a HUGE memory buffer but I'd bet that those of us with higher end systems probably have the ram to support this option.

just to give you some numbers based on what I know using my memory buffer, 5000 frames @ 848x352 is ~2.08gig/2135meg of memory (this is based on 447744 bytes per frame). if were using a default of 250 frames max between keyframes, 8 threads are used and every gop hits the 250 limit, that puts us at 2000 frames which is around 854meg. I have tested my own system and know that i can output a max of around 2700 fps from my ram when a small c program that i wrote to read one frame at a time in a while loop as opposed to when I dump it into an encoder. 2700fps should peg my system out with no problem. :P

as an addendum to this thought, an idea to save memory would be maybe to do like 2 threads per gop (kinda like running 4 instances of x264 each with threads=2 instead of 8 with threads=1). you would need less memory to store the frames (half or more pending how many threads per gop). we also know that at threads=2-3 we can get close to maxing out the cpu usage/fps output. just a thought.

[Inventive Software]

You wouldn't even need that big a buffer if x264's encoding and re-filling the buffer that quick, 1000 frames I reckon would cut it, 2000 just to be sure. I'm surprised you're even getting 3 figure FPS rates TBH, good job getting those CPUs. :D

[morph166955]

the only way im getting the 3 figure fps rates is by first dumping the raw yuv frames into a ram disk of sorts and then feeding that to x264 directly. currently I can only decode at a max of 350fps at most with my current method, i am however attempting a new method now which may speed up the process significantly, only time will tell on that though.

[Mutant_Fruit]

Would it be possible for x264 to do batches of frames per thread somehow?
I already asked, and the answer was no.

In fairness, the overhead due to synchronisation should be pretty negligible and by pretty negligible i mean < 1%. If it is taking up a substantial amount of time, then there's something not right going on.

[lexor]

In fairness, the overhead due to synchronisation should be pretty negligible and by pretty negligible i mean < 1%. If it is taking up a substantial amount of time, then there's something not right going on.

Well pthreads may work that way, but I doubt it. Synchronization gets very expensive as threading goes up, even on trivially parallel applications there is a limit on how many threads you can get going before communication overhead overshadows multithreading gains. Take something like matrix-vector product, it is trivial to make parallel, but on a 32 CPU SGI MIPS box it tops out at 16-18 CPUs with a naive algorithms (unless you work with obscenely huge matrices where you can do smarter things, I'm sure SGI will pay you huge money if you find a way to keep the linear growth of performance past 16 CPUs for smaller matrices, i.e. under 100k by 100k entries) even though you have divided workload evenly between all of them, so to get performance of the rest you've got to feed them something else to do.

This is why neither data nor task parallel approach will get you top performance as the degree of parallelism goes up, you have to combine them. From what I see (and I could be wrong here) x264 task parallelizes and it can't data parallelize because once you have got to block level and have to do math that math is pretty simple and sequential. A plateau is expected.

the only way im getting the 3 figure fps rates is by first dumping the raw yuv frames into a ram disk
that and having a simple command line, try using command line from the HQ-Insane profile and feel the pain. :)

[Mutant_Fruit]

Synchronization gets very expensive as threading goes up, even on trivially parallel applications there is a limit on how many threads you can get going before communication overhead overshadows multithreading gains.
I'd disagree on that, but it's possible we're referring to different things here.

Synchronisation is cheap. Parallelism is hard. If you want to keep the cost of synchronisation down, you have to make sure you only lock when you have to. If you lock access while calculating something that can take several hundred microseconds/miliseconds which doesn't need thread safety, then you will not scale.

Each sync lock takes less than 100 nanoseconds to acquire (assuming another thread doesn't have it), so the few dozen that x264 would do a second are nothing, far less than 1% of processing time. If x264 is acquiring/releasing somewhere in the region of one hundred thousand locks a second, then there'd definitely be potential for optimisation.

[lexor]

Synchronisation is cheap. Parallelism is hard.
On second thought we are talking about different things (if I am right about x264 being task parallel), I remember it used to parallelize on slices, I think it does it on frames now, but that's not really important. In my matrix example, it was data parallel, once each cpu is finished with its own part, all results have to be sent to cpu0 to be combined into a final vector. This doesn't happen in x264 it should have very little communication between threads. The thing about performance wall for parallel algorithms still holds though, eventually more cores stops helping no matter how you parallelize.

And that's what we see happen in the table above, I'm actually surprised it got up to 5-7 cores before diminishing returns kicked in in full.

[burfadel]

There is a way thats impractical to implement or programme. Have a first pass which is basically a keyframe detection pass (very quick). The keyframes are then divided evenly among the cores, it can be out of order, and each scene is processed independently on each core. The scenes are then combined at the end. This method would however have some disk performance cost, but you would would not insert unnecessary key frames. The keyframe interval is a maximum of say 250 frames (PAL), so that means 6 per minute, 360 per hour minimum! In reality this would be much more. If there are several keyframes in each 250 slice then they are processed on the same core. Say if there are 1500 keyframes in the clip, and tallied up equals 700 roughly 250 frame blocks then that would allow 700 cores to encode with complete parallelism at 100 percent (unless used by other tasks) and even in each core there could be more than 1 thread operating.

Just an idea! - an impractical one but one that should work!

[akupenguin]

Each sync lock takes less than 100 nanoseconds to acquire (assuming another thread doesn't have it), so the few dozen that x264 would do a second are nothing, far less than 1% of processing time. If x264 is acquiring/releasing somewhere in the region of one hundred thousand locks a second, then there'd definitely be potential for optimisation.

The number of locks per frame for synchronization of input frames is: 2
The number of locks per frame for synchronization of motion estimation is: (# macroblock rows (30 for 480p)) * (# refs)

The latter don't strictly have to be locks, if I could find a more suitable synchronization primitive. What they really need is: memory barrier and wait until condition is true. There's no reason multiple threads can't be computing the condition at once.

@morph166955, burfadel:
Yes, and that's already implemented in ELDER and x264farm. You're just asking for a buffer on the front to make the API compatible with streaming.

@lexor:
Nothing in video compression is inherently serial. Everything can be paralellized to arbitrarily many cores. Some pieces of x264 are still serial because they aren't yet a bottleneck on ~8 cores. Or maybe they are, and that's what I'll have to fix next.

[lexor]

@lexor:
Nothing in video compression is inherently serial. Everything can be paralellized to arbitrarily many cores. Some pieces of x264 are still serial because they aren't yet a bottleneck on ~8 cores. Or maybe they are, and that's what I'll have to fix next.

Well that's the problem with me trying to give quick arguments for complex issues. I didn't mean x264 (or h264 standard) itself is inherently serial, all I meant is that when you get down to the block level, things are 8x8 at most, all the transforms are tiny matrices, there is definitely no point in parallelizing those, I'm pretty sure any modern CPU will compute the whole thing faster than you can scatter it over the cores.

[Mutant_Fruit]

I'm actually surprised it got up to 5-7 cores before diminishing returns kicked in in full.
I'd be surprised if the efficiency couldn't hit 80%+. The limitation is purely how you split the data up to be processed.

For example it might be possible for one thread to work from the bottom of a frame upwards and another could work from the top downwards. Every time a horizontal line is finished, you could update a variable thats shared between just those two threads so that they both know what lines are finished and what line is about to be processed next. This way neither thread accidently overwrites a different threads work.

If you process 5 frames at a time, that would be 10 threads running, but each thread is only contending with *one* other thread while the frame is being processed. When a full frame is finished, a master thread will have to be informed which would require a lock that all 5 pairs of threads would be contending for.

That may work out to be faster than trying to process 10 frames at the same time with each frame being processed by a single thread due to the interdependence between frames. Then again, it may not. I have no real understanding of what can be done in parallel and what can't when it comes to video encoding. So it's quite possible i'm talking out of my ass here ;)

I meant is that when you get down to the block level, things are 8x8 at most, all the transforms are tiny matrices, there is definitely no point in parallelizing those, I'm pretty sure any modern CPU will compute the whole thing faster than you can scatter it over the cores.
Aye, that'd be insanity. But you could split *groups* of transforms up to be processed seperately. Say the overhead in syncronising is 100 nanoseconds. If you split your transforms into groups that take 100 microseconds or more to calculate, you only lose 1% in performance due to synchronisation. But you gain substantially in overall performance because you can now scale to many more cores. Potentially enough to max out 8 cores.

[akupenguin]

Well that's the problem with me trying to give quick arguments for complex issues. I didn't mean x264 (or h264 standard) itself is inherently serial, all I meant is that when you get down to the block level, things are 8x8 at most, all the transforms are tiny matrices, there is definitely no point in parallelizing those, I'm pretty sure any modern CPU will compute the whole thing faster than you can scatter it over the cores.

h264 is inherently serial at the macroblock level. Though at DCT-block level it can be parallelized again, but this time with SIMD instead of threads.
No, what I meant is that frame-level and GOP-level threading can be extended up until the number of threads is a significant fraction of the number of frames in the movie. Though at some point you have to stop pretending that it fits in the current API.

[akupenguin]

For example it might be possible for one thread to work from the bottom of a frame upwards and another could work from the top downwards.
x264 used to do that ("slices"). I replaced it because frame-level threading is better.
* Slices are not compatible with frame-level threading. If you split one frame spatially between threads, then you can't also split multiple frames temporally without unduly restricting motion estimation and thus sacrificing lots of compression ratio.
* The encoder can't run backwards. Each slice has to start at a given macroblock and run forwards until it hits the next slice. So you have to decide in advance the distribution of macroblocks into slices, and any unevenness in the cpu-time per slice counts directly against scaling efficiency. (Even if you could encode backwards, you'd run into this with more than 2 threads per frame.) And you can't solve it just by adding more slices, because:
* Any spatial split sacrifices some compression ratio in proportion to the number of splits. (:search: if you care why)


But you could split *groups* of transforms up to be processed seperately. Say the overhead in syncronising is 100 nanoseconds. If you split your transforms into groups that take 100 microseconds or more to calculate, you only lose 1% in performance due to synchronisation. But you gain substantially in overall performance because you can now scale to many more cores. Potentially enough to max out 8 cores.

DCT for one macroblock costs 550 cycles. At 2.4 GHz, 100µs is 1/3 of a frame. And the same caveats for spatial splitting apply to anything that crosses more than 1 macroblock.

If you split a single macroblock per mode, then you're talking 15k cycles or 6µs in each mode. But then you lose any early termination thresholds, so there's a bunch of redundant work in addition to synchronization overhead.

[Mutant_Fruit]

So pretty much far beyond my comprehension ;) The way i look at it is that if there's a better way to do it, you could code it and test it before i even learnt my way around the codebase.

[morph166955]

ok maybe were thinking about this whole problem wrong way. i have some budding ideas but they are in need of some info. anyone got an idea thats really far outside of the box? something that sounds just absurd when you say it but might actually work?

@aku
i'm not sure if you can do this, but is it possible (and i mean in general not if the code can do it right now) for you to just handle the prediction part (scene cut, frame type decision) of the encode with out actually encoding the frame? i have an idea that im working on but it would rely on that being possible.

[akupenguin]

Yes. For the current threading method I already had to modify the frame type decision to not use any information from the encoding part.

[morph166955]

Yes. For the current threading method I already had to modify the frame type decision to not use any information from the encoding part.

good...then my theory has a potential to work. I realize you said that this is already in those other programs but I still think that this method may be a slightly better way of doing the threading then what we are trying here.

Heres what im proposing. create a thread that brings frames into a buffer and does the frame type & scene cut prediction on them. once we have that info, we know how large each scene will be. then we create a thread to process that scene as a whole. we can then create threads for the amount of scenes we want to process simultaneously. then just write out the results in order once the thread completes (may add some overhead storing the data if a preceding scene has not finished). we know that x264 can "peg" one thread with out any problem, so why not just run a lot of them? and as far as i can see there shouldn't be any bitrate penalty since all the frames are all based off of that initial I frame anyway.

I realize this would require quite a bit of changes to the way x264 does its work as of now but I think its worth looking into. as it stands now, using the current threading model only will get into the higher cpu usage as the frame dimensions get larger. this model would get into the higher cpu usage no matter what the size of the frame (theoretically assuming your not using an extremely tiny size) because you would only have one thread handling an entire scene. this also could theoretically eliminate thread creation and joining overheads since the threads stay running longer.

EDIT: aku, just out of curiosity and to save me the time digging through the source code, where abouts is the function that does the decision making? I'm sure I can figure it out once i look at it but given the amount of functions and files it could save me some time for having to dig through and find it. thanks!

[akupenguin]

slicetype.c:x264_slicetype_analyse()

[morph166955]

slicetype.c:x264_slicetype_analyse()

thanks!

[Mutant_Fruit]

then we create a thread to process that scene as a whole. we can then create threads for the amount of scenes we want to process simultaneously. then just write out the results in order once the thread completes (may add some overhead storing the data if a preceding scene has not finished)
So basically you want to process all the data between each IDR frame in a different thread. If i remember correctly (which i probably don't) IDR frames can only have backward references, not forward references. So if frame 1000 is an IDR, then nothing *before* frame 1000 can reference it, only frames 1001 and above can. This makes it the ideal point to split on for threading.

It'd be fairly trivial to get scaling by splitting the data based on IDR frames, assuming you have already run a first pass and know where they will be placed. However the problem with that is suppose you have 10 threads running. You can't really store all that data in memory, so you'll have to write a lot of stuff out to temporary files and then recombine it when the scenes are ready. For example, suppose you have 10 scenes that are 1000 frames long, do you really want to store 10,000 frames in memory before you can write them in the correct order to the disk?

[akupenguin]

Storing the compressed frames to be written in order isn't the problem. 10000 frames / 25fps * 2Mbit/s = 100 MB.
The problem is storing the uncompressed input frames after they've been read in order. Ok, so avisynth can read out of order (though depending on the filters that can add memory requirements and/or speed penalties of its own), but do you want to limit the new method to just x264cli w/ avisynth, no other applications? That would certainly count me out of implementing it.
(Though I have occasionally wondered what could be done if a codec were not limited to discrete passes and instead could request any sequence of input frames.)

[morph166955]

i dont think that storing the uncompressed frames is goign to require overhead thats out of the question. heres my math for why I say that:

for unscaled 16:9 dvd (cropped 854x480 -> 848x480)
848*480*1.5 = 610560 bytes per frame raw yuv
max 250 frames per gop = 250 * 610560 = 152640000 bytes -> 145.56 megabytes

assuming threads=cores
for dualcore systems thats ~291 meg
for quadcore systems thats ~582 meg
for octacore systems thats ~1164 meg

for a 1920x1088 HD recording (assuming were using mod16 and not regular 1080p)
1920 * 1088 * 1.5 = 3133440 bytes per frame
max 250 frames = 3133440 * 250 = ~747 meg

most dualcore systems now have at least 512meg if not 1 gig or more of ram. most quadcore systems have atleast a gig of ram. and i know my octacore system has 4 gig of ram but lets just say that anyone who has a system like this will have at least 2 gig. also these are worst case senarios of 250 frames per gop so its more likely that the systems are going to use less then that. now I understand that for 1080p thats more then the availible ram.

so heres an idea to quench that requirement, allow 2-4 threads of the current thread model per "gop thread". weve seen that we can run close to full cpu on 4 threads now with dvd source.

so you basically have two options in terms of the decision process.

1) you could create a required "0th pass" which simply does the frame type decisions and writes a x264_2pass.log file (or something similar). then using that file you could read in what you need, process the frames, and release them once your done.

or

2) do what your doing now in terms of passes and read in increments. you dont have to store the entire movie in the buffer, only what you need and maybe a frames more.

example of option 2 (assume for now were only using 1 thread per gop)...
1) x264 set to threads=4 & max_keyint=250 & 1 thread per gop

2) x264 reads 900 (arbitrary #, max 1000) frames and finds keyframes at 0,250,500,750,895.

3) x264 creates 4 threads, 0->249, 250->499, 500->749 and 750->894 and stores all the output as it finishes in ram and instantly releases the memory for the frames as it processes them.

4) assume the thread for 500->749 ends first. all 250 frames for this thread should be released from memory now thanks to the encoding thread and we never have to look at them again. we already have frames 894->899 in the buffer. x264 then reads up until a few frames after it finds the next keyframe and creates a new thread for that gop.

5) as the other first threads finish, they are dumped out to their output file in order and they are released from memory. more data is buffered and then encoded as needed until the end of the movie is reached.

While yes, x264 would become a memory hog, it would be able to do this. for non threaded and/or dualcore systems, you could simply set this to run 1 thread or 2 threads per gop and only do one gop at a time...basically doing what your doing now.

[morph166955]

@aku
just fyi, in case you dont want to go hacking up x264 right now, im working on a cli which uses libx264 that does this in a very limited set of parameters attempts to do this. im sure you have a faster and better method then how mine will do it, but it could give a good baseline to see if this will work or not at all and how fast it can do it.

[burfadel]

jthen we create a thread to process that scene as a whole. we can then create threads for the amount of scenes we want to process simultaneously. then just write out the results in order once the thread completes (may add some overhead storing the data if a preceding scene has not finished)

Thats exactly what I meant in my post a few before yours! Maybe a slightly different implementation but the general idea is the same. Therefore, its a good idea! The benefit of this increases as the number of cores increase. In Intel's future CPU concept of having 64 cores or more in a few years, this type of parallelism would become more important. Couple with their re-implementation of hyperthreading, as they have suggested they will do with the upcoming CPU's, that would technically allow 4 simulataneous excecuting threads on a 2 core machine, and on an 8 core machine 16 threads... on a 64 core machine 128 threads. Hypterthreading supposedly increases performance by ten percent, I have heard rumours ages ago of 4 thread hyperthreading that may add a couple more percent, so that would make an 8 core machine able to execute 32 parallel simultaneous threads.

[Mutant_Fruit]

Hypterthreading supposedly increases performance by ten percent, I have heard rumours ages ago of 4 thread hyperthreading that may add a couple more percent
And in some circumstances hyper threading destroyed performance and having it enabled made things far slower. It's a bit of a double edged knife. Hyper threading becomes less and less important as cores go up.

To be be honest, anything that is looking like it'll require more than 500+ megs of ram more than likely won't scale.

Take the exteme case of 16 refs. Do the 16 input frames have to be kept in memory? If so you'll also have the current frame you're working on and also the next frame (maybe two?) held in memory. That's 18 full frames per "gop". Multiply that by 8 threads, and you have ~580megs in memory (hi-def stuff).

Then, assume you're encoding at a high bitrate, ~8mbps and the gop that needs to be written to the disk is the longest and therefore slowest to encode, so the rest of them finish before that one does. That means you have about 10,000 / 25 * 8 megabits of information stored in memory, which equates to an additional 400 megs of ram.

Total ram: ~1GB. Thats enough to cause swapping on quite a lot of computers.

[akupenguin]

Take the exteme case of 16 refs. Do the 16 input frames have to be kept in memory?
Worse than that, x264 stores 4 copies of each of the 16 reference frames (4 halfpel filter offsets).

[rlandle]

I'm trying to use threads and I noticed that using CBR and increased number of threads reduces the quality (PSNR).

Is this due to limitations on the motion estimation or is it an artifact of the rate control?

[akupenguin]

Find out yourself: encode the same content with the same settings except not CBR.

[morph166955]

@aku
ive run into a small stumbling block with my code. ive gotten my prediction algs working close to what x264 uses (i decided that rather then use the ones in the library i was going to write my own for several reasons, nothing wrong with your code or anything just easier this way for me). My results are occasionally giving me a prediction for an I frame 2-4 frames before where I believe it should be. I noticed that in the x264 log (im running in pass1 mode) at those times the itex value is significantly higher then its surrounding frames (generally 30k+ as opposed to sub 20k).

I was looking through your code and I understand your methodology for deciding if a scene cut/I(DC)-Frame should be used with f_bias and the p & i costs (although I'm a little confused on how your generating the costs, i understand what they are doing and why you need them) however im not 100% sure how your generating the itex value (or what itex actually stands for) in these cases.

Since I only need to really know I(DC) or NOT-I(DC) and I dont care about the decision between P & B frames for my purposes, I figure that using the itex value can probably sufficiently handle this. any way you can give me a breakdown of the formula/method to calculate the itex value and/or another easier way to decide this that would mirror x264? I was finding that preparing the frames (eg creating the proper data structures, allocating their memory, setting the proper variables in them, etc) and trying to run the scenecut/slicetype_analyse functions was just to much work for what should be a simple process.

Thanks!

[akupenguin]

itex = bits spent on dct coefficients of intra macroblocks.

[morph166955]

so i suppose the next question is, "whats the next step from here?" should we continue to try the threadpool model? should we try to see if the thread-per-gop model is faster/more efficient w/o sacrificing abnormally high amounts of memory? should we try to come up with an option 3?

[legoman666]

You were hitting 90-95% utilization when you were doing a HD encode, obviously the threadpool model works. It's just with lower resolution sources x264 can't keep the cpu's pegged.

[morph166955]

while thats true, i'd think that we should be able to get x264 over 45% cpu usage when not using HD content. i'm still using x264 for my SD content and having that unused speed would be amazingly great. I can also peg x264 w/o threadpool if i throw a huge frame at it just like anyone else. its doing the smaller frames im more worried about at this point then the larger ones.

[check]

@morph, you should ensure your input can supply frames fast enough.

[morph166955]

@morph, you should ensure your input can supply frames fast enough.

input is stored as raw yuv in a ramdisk thats in 4gig of DDR2 667 FBDIMM's... id say thats fast enough :D.

[legoman666]

while thats true, i'd think that we should be able to get x264 over 45% cpu usage when not using HD content. i'm still using x264 for my SD content and having that unused speed would be amazingly great. I can also peg x264 w/o threadpool if i throw a huge frame at it just like anyone else. its doing the smaller frames im more worried about at this point then the larger ones.

just encode 2 things at the same time :P

[burfadel]

just encode 2 things at the same time :P

Or set the first encode from frames 0 to x, then the second iteration of x264 from frames x to finish! = 90 percent + CPU usage, and then it can be easily combined at the end!

Only a factional loss of bitrate if the x doesn't fall on a keyframe (a keyframe would be inserted here that may be unnecessary).

[DoctorEnsGabe]

I'm pretty sure that these are the exact things that morph is trying to avoid and is helping aku to solve with his monster 8-way box :)

[delacroixp]

while thats true, i'd think that we should be able to get x264 over 45% cpu usage when not using HD content. i'm still using x264 for my SD content and having that unused speed would be amazingly great. I can also peg x264 w/o threadpool if i throw a huge frame at it just like anyone else. its doing the smaller frames im more worried about at this point then the larger ones.
I doubt anybody will be using your kinda rig for sub-HD encodes...

However, having said that, given the release of quad-core sytems, it's not far fetched to imagine your average joe with a twin quad-core box crunching Mpeg-2 movies... either individually or, 2 and 3 in parallel...


:):D:eek:
Pascal

[morph166955]

I doubt anybody will be using your kinda rig for sub-HD encodes...

except me of course who still has dvd's that hes not done backing up... :D

just because i have a rig like this doesnt mean that i've locked myself solely into HD. SD is still a big part of media right now (especially with tv) so until thats gone I'm sure I'll be working on SD stuff. although i still giggle when i see my 720p source encoding faster then realtime.

oh and p.s. for those wondering, I think ive nipped my cooling problem in the butt. 3x thermaltake 80mm fans which can push 75CFM EACH are now positioned to blow air clear across my motherboard. although I think the cpu fans I got are junk...they are running HOTTER then the stock ones did however I believe that could be because the thermal grease hasnt had time to burn in yet so i'm going to leave it for a week to see what happens. if not, I found a different fan last night that will probably fix it and possibly give me some air flow for my ram as well (its a side mount type instead of top mount, so while ill be blowing warm air over the ram its better then having the heatsink like a wall between the ram and the fans)

[Inventive Software]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)

[morph166955]

im pretty confident that the thermal grease just has to work its way in. its been on there now less then 24 hours and the reviews on the cooler all say that the systems run 5-10 degrees or more cooler then the stocker so im hopeful. we shall see after i let the grease melt in. im leaving it just idle for now, ill run it a little later on to get a good heat variability. worst case scenario i replace them again with the ones i found last night which all in all i may do anyway. its a 2U case so a fan thats on top is kinda inefficient since it has almost no room to suck air in from, a side mount however has plenty of room plus the case fans will have already pressurized the air blowing at it so it will give even more air flow.

[Mutant_Fruit]

im pretty confident that the thermal grease just has to work its way in. its been on there now less then 24 hours and the reviews on the cooler all say that the systems run 5-10 degrees or more cooler then the stocker so im hopeful. we shall see after i let the grease melt in.
I'd say the odds are you've added too much thermal paste. There should be a very thin layer of thermal paste between the cooler block and CPU. If there's too much, it acts like insulation and the heat can't escape. Thermal paste doesn't "melt in" as such. Some thermal pads do "melt in" but that takes very little time to happen, a few hours at the most.

[delacroixp]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)

LoL


:):D:eek:
Pascal

[morph166955]

I'd say the odds are you've added too much thermal paste. There should be a very thin layer of thermal paste between the cooler block and CPU. If there's too much, it acts like insulation and the heat can't escape. Thermal paste doesn't "melt in" as such. Some thermal pads do "melt in" but that takes very little time to happen, a few hours at the most.

from everything ive read and seen over the past few years, thats a very inaccurate statement. higher end metalic thermal pastes require a minimum of 35 hours to kick in and in some cases almost 200 hours for them to completely set to optimal efficiency (most are around 100). im uzing the zalman thermal grease which is considered one of the good ones (no wars over zalmbn vs as5 vs w/e please) and zalman even says they take 35-40 hours for the paste to set to the point that its mostly efficient. and i definitely only put a very thin layer. one good thing with the zalman is that it comes with a nailpolish type applicator which makes it very easy to layout and smooth down to what a person wants. i do agree that too much can cause problems, but in this case i think my biggest problem is that the fan is on top and very close to the top of my case. more likely then not, im going to have to get that side mount cooler for this setup.

[Mutant_Fruit]

higher end metalic thermal pastes require a minimum of 35 hours to kick in and in some cases almost 200 hours for them to completely set to optimal efficiency (most are around 100).
Now that's news to me. I knew that some thermal pads needed to "melt in", but i never heard of a paste needing to melt in considering a paste can easily fill in all those tiny nooks and crannies on application whereas a pad can't.

Ah well, ya learn something new every day.

[delacroixp]

Ah well, ya learn something new every day.
I'll 2nd that...
I just wish I could unlearn some of the old stuff to make way for the new... before the RAM becomes too volatile and switches off completely...


:):D:eek:
Pascal

[burfadel]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

Put a digital thermometer sensor inside your case near the inlet of the cpu fans. Even if you have a case sensor its the position inside the case thats important. You'll probably find its several degrees higher than outside your case, so with fan ducts at least the air used for cooling is several degrees cooler.

The best CPU fan in the world is relatively pointless if cool airflow (and not just airflow) isn't available. A high flow helper fan may be beneficial at the inlet of the fan ducts as long as its pointing the correct way!

[morph166955]

ive figured out the air flow, im sucking cold air in right from behind the dvd drive, through the 3x 80mm's, past the two cpu's (parallel with the fans so both get equal flow), then over the ram and out the back. im working on it, ill figure it out eventually.

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)


ive figured out the air flow, im sucking cold air in right from behind the dvd drive, through the 3x 80mm's, past the two cpu's (parallel with the fans so both get equal flow), then over the ram and out the back. im working on it, ill figure it out eventually.

How about one of those 10 inch 'slinky' piping-ducts they use on air-con systems to hook-up your a/c directly into the 'Monster'...
We're rooting for you... this is more fun than having a quad-quad, water-cooled, vapour-chill, overclockers-extreme...

I still think the Abacus rocks...


:):D:eek:
Pascal

[morph166955]

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my frat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day hes working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until theres a loud thud and his computer sounds like its going to destroy itself. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

back on topic...the power supply just arrived as i was typing this so im putting that in and then going to do some high power tests to see if it was in fact a power issue not a heat issue causing the system to reboot. ill report soon.

[delacroixp]

It seams that the guys at Hardware & Software, MeGUI CPU Time Test - Compare different CPUs encoding the same file (http://forum.doom9.org/showthread.php?t=122318) have also got a similar test crunching thing going... though, unfortunatelly, without akupenguin's input to add-value to their research and developement...


:):D:eek:
Pascal

[morph166955]

so back on topic...whats the next step here?

[Inventive Software]

More tests! :D

[burfadel]

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my frat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day hes working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until theres a loud thud and his computer sounds like its going to destroy itself. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

back on topic...the power supply just arrived as i was typing this so im putting that in and then going to do some high power tests to see if it was in fact a power issue not a heat issue causing the system to reboot. ill report soon.

lol! putting the duct from outside in a cool/cold climate is a good idea, as long as the end is completely covered with secured flyscreen (a very fine wire mesh). It is durable, doesn't stop air flow, but stops everything else including small insects.

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

How about one of those 10 inch 'slinky' piping-ducts they use on air-con systems to hook-up your a/c directly into the 'Monster'...

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my fat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day he's working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until there's a loud thud and his computer sounds like its going to self-destruct. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

lol! putting the duct from outside in a cool/cold climate is a good idea, as long as the end is completely covered with secured flyscreen (a very fine wire mesh). It is durable, doesn't stop air flow, but stops everything else including small insects.
LoL... nothing beats a naturally free air-con... a regular 5 or 6 inch circular wire-grill in the fan itself wouldn't hurt airflow either...


:):D:eek:
Pascal

[delacroixp]

so back on topic...whats the next step here?

You were hitting 90-95% utilization when you were doing an HD encode, obviously the threadpool model works. It's just that with lower resolution sources x264 can't keep the cpu's pegged.
Perhaps, multiple SD encodes are the way to go... potentially, even 7 or 8... especially if you have tons of SD material...


:):D:eek:
Pascal

[delacroixp]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)
Funny enough I tried that just now with an Mpeg2 DAResolution (1024x576) encode of the BBC's Hiroshima...
I'm using an AMD Athlon dual-core which runs exceptionally cold anyway...
Between 20% and 40% the FPS were steady at 5.88 but after placing the 15 inch (37.5 cm) floor-fan facing the PC with both side-covers off... the FPS rose to 6.24 until 70%... when they fell off and steadied at 6.15 FPS.

I live near the coast at 34º South... roughly equivalent to Dallas-Texas, Phoenix-Arizona or the tip of Spain in northern latitudes... it never gets very cold and only rarely dips below freezing... but my encodes speed up significantly at night or when it's cooler...
Even Internet Explorer can affect your encodes and I'll often even switch off non-essentiall services on the PC to gain a little speed during some of those 'rough' encodes when... the fps keep on dipping all the way through the encode like some stricken aircraft...

I guess that we won't be moving to the Antarctic or the 'Darkside of the Moon' anytime soon but... cooling does make a difference...
Is there any app that maps FPS-to-time,percentage,frameNo in a simple line graph... during H264 encodes ???


:):D:eek:
Pascal

[legoman666]

you're a moron if you think you're encoding speed has anything to do with you placing a fan next to your machine. plain and simple.

[delacroixp]

you're a moron if you think you're encoding speed has anything to do with you placing a fan next to your machine. plain and simple.
I'm sure that you're right...


:):D:eek:
Pascal


btw
I never think... I leave that to the experts...

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

Not only case temparature... but motherboard too...
It's one of those neglected components when it comes to cooling... CPU, GPU and even HDD's are usually first in the queue...
However, using a desk-fan can even run your power-supply stone cold...

I've got 6 drives in 2 sets of RAID (stripped and mirrored) but you'll be amazed how many computer errors can be traced back to over-heating...
Sure, your system may be running within spec tolerances... but even then your components are sufferring from gradual degradation...

Heat causes electron-flow-resistance to increase, while extra cooling may induce super conductivity... and improved performance in general...


:):D:eek:
Pascal

[burfadel]

Whilst heat may increase the resistance, super conductivity, by definition of the term, would require that you chill it to around 2 kelvin or less (-271C or less), where 1 kelvin = 1C, except from absolute 0!... the only practical place for temperatures so cold is for radio astronomy.

The other thing to get hot is the motherboard chipset. Most of the time they put stupid little heatsinks on them that aren't sufficiently large, or poor quality fans that die sooner rather than later! Some now use heatpipes for the chipset, with the heatsink at the rear of the case.

[delacroixp]

the only practical place for temperatures so cold is for radio astronomy.

Possibly transmission cables and maglev trains (http://www.howstuffworks.com/question610.htm) though Vapo (http://www.extremeoverclocking.com/reviews/cases/VapoChillPE_5.html)Chill (http://www2.asetek.com/default.asp?showPage=startside.asp&contentSection=2) was quite popular with PC's at some stage...


:):D:eek:
Pascal

[burfadel]

Possibly transmission cables and maglev trains (http://www.howstuffworks.com/question610.htm) though Vapo (http://www.extremeoverclocking.com/reviews/cases/VapoChillPE_5.html)Chill (http://www2.asetek.com/default.asp?showPage=startside.asp&contentSection=2) was quite popular with PC's at some stage...


:):D:eek:
Pascal

Yes very true! It works but its not practical! Its very difficult to keep a very large area (which would be required for Maglev trains & transmission cables) at the temperature that allows super-conductivity (ideally below -271C). That why it wasn't initiated in current Maglev train trials.

For radioastronomy, the area to keep cold is relatively small and easy to manage. The cold is also used to reduce noise.

[delacroixp]

The cold is also used to reduce noise.
I suppose it (http://coolcosmos.ipac.caltech.edu/cosmic_kids/AskKids/arecibo.shtml) would include background star-clutter or big bang (http://deepspace.jpl.nasa.gov/dsn/history/dsn67.html) wave residue... a bit like the noise found on certain video encodes...


:):D:eek:
Pascal

[morph166955]

guys i dont mean to be a kill joy here but can we try to get this thread back on topic? the thread_pool.03 patch was the best of the 3 although it still bottlenecks below max cpu on non HD sources. Unfortunately I'm finishing up my work so I can graduate in a week (yes its scary) so I haven't had time to work on my thread model (doing multiple scenes at the same time in different threads). anyone else getting closer to getting this working?

[Mutant_Fruit]

hread model (doing multiple scenes at the same time in different threads). anyone else getting closer to getting this working?
From the sounds of it, that method won't scale. If the rough maths that i used earlier were anywhere near accurate, and my understanding of akupenguins comments were right, each thread would require an amount of ram so high, that you couldn't run more than 2 or maybe 3 with HD sources.

[morph166955]

2 or 3 with HD source "should" still peg an 8 core cpu if you have each gop-thread doing multi-threaded encoding of the current design (threads with in threads basically). HD source can peg 3-4 threads with out any problem, run two of them concurrently and there ya go! they may even be able to share some of that memory some how if there being run out of the same program instead of two x264 executables being run separately.

[Mutant_Fruit]

Well, multiply the numbers you see on this (http://forum.doom9.org/showthread.php?p=996360#post996360) post by 4 to get the amount of ram that will be actually required. That'd bring most machines to the ground.

This method on it's own would never scale, and even on current systems may perform hideously slowly. If x264 has to swap in and out from the harddisk to access frames performance will drop.

The memory isn't "shareable" as such because the memory is taken up by frames which have to be stored in memory. You can't reduce that memory without reducing ref's, threads or some other such thing. Thats the only bad thing i can see about your method.

[morph166955]

I agree, most computers will run into issues at that point...however...were not talking about most computers. were talking about systems that have 4 (or more realistically 8+) cores in them. id feel really really bad for anyone who built a machine with that much power and only a gig of ram. thats like building a tank and loading it with 9mm bullets instead of full size shells. i'd hope that any machine like that has a minimum of 2 gig if not 4 gig of ram in it.

however, i also realize x264 has to be able to handle all machines both big and small. I'd think that we could put a switch in x264 somewhere that can be set to how many gop blocks to thread out at a time, and default that to 1. at 1, it would run like x264 would now. above 1 it would do that many gop's simultaneously. i suppose it could also be set so that if its set to 1 then x264 does not buffer more then the amount of frames that it needs just like it does now.

I suppose that another option is needed on top of this. What about doing a scaled down version of my gop-thread idea and break each gop up into the # of threads were running. then we would only need to buffer in a max of what ever keyint is set to plus maybe 3-5 frames. heres a little breakdown of what im saying:

Say that the GOP has 250 frames (equal to keyint since thats a worst case senario). Lets also say you are running 4 threads. Have thread 1 doing frames 1-62, thread 2 doing 63-124, thread 3 doing 125-186, and then thread 4 doing 187-250. Now I realize that you cant break at just any point (like have a thread start at a b-frame), but isn't it possible to create the I frame first, and then break the image up at the closest p-frame to an even split? we would have the entire scene in the buffer to reference from so in theory we could have each thread look at a few frames before its part to make a decision about the first frame and then the frames after that, hopefully we wouldn't have to encode the frame just use it as a reference point.

I don't know if this is possible to do, but its yet another idea out of my jumbled brain.

[Inventive Software]

My initial thought is that you could split the whole encoding job into nearly identical jobs at intervals, and join the thing together right at the end, though with x264.

Take a 5000 frame encode. Split it into 4 jobs, 2 threads per job, that's 1250 frames per job. Of course, you could customise the number of jobs and threads at the command line. ;)

This may well need some more thought, and there probably will be slightly more IDR frames than necessary, but if you're a smart encoder, you'll use something like 250 anyway. What do you think?

[morph166955]

i just ran a test to see what would happen if i loaded 4 ram buffers with 500 frames each, and then simultaniously forked 4 x264 (vanilla r655 build) threads running @ threads=2 (i also tried @ threads=3 the effect was negligible) , i was able to run the system at >90% cpu usage since each x264 thread was using 185-190% cpu of the 200% max it should be able to do. each thread ran at around 105fps giving me a total of about 420fps. i ran these on the same settings as job1 uses in the earlier tests just to be uniform. in comparison, the highest fps that ive been able to get job1 was 202fps, thats more then double the speed that weve been able to accomplish. granted this is a very crude approach to how, but it at least shows that such an approach has the ability to succeed in reaching the 100% cpu usage/max fps goal

[akupenguin]

Heres threadpool.03 for threads 1-8...it leveled off after 7 on the high ones and 5 on the low ones so i didnt run any more.
I think I can explain the 7. In svn, there's no space between the serial setup and the slice encoding, so the setup essentially counts as part of one of the other threads. In thread_pool, the queue means they're not locked together, so the setup counts as a separate thread. (Setup was always in a different physical thread, but only now can it run concurrently.)

[morph166955]

oooooooooooooooooooooooooooooooooooooooooooooooooh...so 7 really equaled 8? that makes a whole lot more sense then!

[delacroixp]

This is a great project for a doctoral thesis...

:):D:eek:
Pascal

[morph166955]

i love this! i get the cooling issue fixed (and i mean FIXED...28C & 23C on the two chips right now!)...and one of my hard drives decides its time for it to die...and its only 18 months old! This whole thing is really getting annoying lol.

so back on topic...I found something interesting out about pthreads in the past few days that may help us (which you probably already know about aku), but ill explain it anyway. One of my final projects I'm working on for me to graduate is for some independent research that I'm doing with the physics department (im a comp sci major, physics minor...fun i know). Were using a series of formulas to generate images of planetary nebulas based on predefined values. The system were working on is a 17 node beowulf cluster each one housing dual hyper-threaded 3.2ghz xeons (nocona's i believe). The way the system is setup, is to have one machine act as the head and the others as slaves, the head does little to no work on the processing since its handling the data flows in both directions to all the slaves and compiling the final data they are sending back. to do this, we ended up running both the MPI libraries to handle the inter-machine communications and pthreads to allow us to doing threading on each machine and utilize both of the machines processors. since were generating one image the way we ended up doing it was to divide the image into almost equal segments (the last one is normally shorter) and use mpi to send that out to the machine (for example, slave1 would do lines 1-16 of the image). going with that example, the slave machine was then creating 1 thread per line (max of 4 threads at a time), joining the threads, doing all the pixels in that line, and then ending so another thread could be created. what we found was that there was a significant amount of time being spent creating and destroying the threads and that having it create threads only once to do multiple lines (eg 1-4,5-8...) gave us a noticable decrease in processing time.

I bring this up because i noticed in some of the debugs that we were running a while back that we were creating & joining threads quite often. Is there any way to leave a thread active longer and have it set to do multiple areas/frames/whatever before it exists instead of creating and destroying threads as often as we do now? say for example have it do 4 (arbitrary #) frames per thread instead of 1 or something like that? I would think that since this method removes several wait times for the creation & joining of threads that it opens up the possibility that more processing will actually happen and bring our fps/cpu% up.

I'm not sure what this does to the whole i/p/b frame thing but I figured I'd throw it out there with the hopes that it was a decent idea or that it would spark another idea in someone to aid in this discussion. On the other hand I could be 100% wrong and my idea may not work at all but hey if people didn't try to come up with different ideas for things we may still have squares instead of wheels!

[akupenguin]

Already discussed here.
It is not possible to assign multiple frames to a thread at once, but it is possible to never create new threads and only pass data to existing threads. thread_pool.03 does this. Much like x264 never allocates memory after startup.

[morph166955]

well there goes that idea then...

[delacroixp]

what we found was that there was a significant amount of time being spent creating and destroying the threads and that having it create threads only once to do multiple lines (eg 1-4,5-8...) gave us a noticable decrease in processing time.

it opens up the possibility that more processing will actually happen and bring our fps/cpu% up.

Good point... 100% CPU usage is useless without real-work... you really don't want a dig-hole, fill-hole kinda loop...
Getting maximum efficiency out of multiple-processors has always been the quest and unless you have non-serial jobs (4 guys painting wall vs putting in window then paint) it's difficult to harness the true potential power out there.
Also, as you say, lines of communication increase exponentially as any maths or stats 101 student will know...

Having said that, encoding and multi-processor, multi-core sytems seam to be great bedfellows.


:):D:eek:
Pascal

[check]

Much like x264 never allocates memory after startup.
Does this reduce the chance of caching inactive sections of x264's memory, or is it for some other obscure reason?

[Mutant_Fruit]

Does this reduce the chance of caching inactive sections of x264's memory, or is it for some other obscure reason?
The main reason is performance. Suppose you want to encode at 100 frames a second, and each frame is 2 megabytes when in memory then you'd be allocating and deallocating 200 megabytes of memory a second.

Alternatively, you allocate 100 two megabyte buffers and constantly reuse those buffers. You use exactly the same amount of memory as before, except you have zero allocations happening a second, therefore you get better performance.

If you can turn 500 allocations a second into zero allocations a second in performance critical areas, you'll get better performance.

EDIT: This strategy can also help if you have memory leaks. If you notice your pool of memory buffers is constantly getting smaller, you can narrow down the bits of code you have to check for the memory leak. If you're just looking at task manager and seeing memory increase all the time, that's not much use in tracking down the source of the issue.

[delacroixp]

well there goes that idea then...
You've had so many really solid ideas and you've achieved so much that, even if you left the project, as is, and enjoyed your graduation...
... it would still be a truly awesome success story...
I'm sure much of your research will be implemented into H264 and contribute to H264's improved abillity to handle 8+ cores... even 16 cores (quad-quad)...

I did my project on CRC (cyclic reduncy checksum) when I was at varsity... but we could only dream about this kind of power and hardware.
CRC is the same thing you get as an option in WinRAR and the system NASA uses to communicate with satellites from Mars...
In the first mission, 60% of the data was extraneous CRC checksum to overcome the severe corruption of coms.


:):D:eek:
Pascal

[morph166955]

You've had so many really solid ideas and you've achieved so much that, even if you left the project, as is, and enjoyed your graduation...
... it would still be a truly awesome success story...
I'm sure much of your research will be implemented into H264 and contribute to H264's improved abillity to handle 8+ cores... even 16 cores (quad-quad)...

Believe me, I don't see anything of what we have done here thus far a failure in any way. Even the tests that didn't work to our expectations were still a learning experience for the future. I'm working on compiling a custom linux distro right now for this box (see my post in the linux subsection on here for more info on it since yes I will be releasing it) that should eliminate any cpu overhead and lack of optimization that exists in the big distros. I'm hoping that this should help out a little bit more with speeding things up. If anyone wants more info about this distro, please post in that thread since I still want the focus of this thread to be optimizing x264 to work at its potential with lots of threads.

[morph166955]

so my semester is over, my college career is completed, and now I'm home for ~2 weeks before I start working with nothing but encoding to do. Anyone have any new ideas on how to do this? Now that I have some time I'm going to see if I can finish up my libx264 frontend that does the threading division that I was talking about earlier (the one thats basically like x264farm only local) to see if I can figure out a way that will work for us with out being the HUGE ram hog that it has the potential to easily become.

[Inventive Software]

What about some 1080p material, and see if that'll encode in realtime? There were some uncompressed AVI samples posted around here somewhere, so it shouldn't be that hard to dump the raw file to your RAMdisk.

BTW, congrats on completing your semester. :)

[burfadel]

How about if it encodes as it is now with say 4 cores or less, and with more than 4 cores (most likely 8 and above) a thread is run in the graphics card (or by itself in its own core) ahead of the rest of the video for scenecut decision, then just distribute the blocks of data to the individual cores. Each scenecut can be numbered and as each consecutive block is completed the output file is created. Such that in an 8 core system , if blocks 1,2,3,4,5,6 are completed, and 7,8 are longer, then 9,10,11,12,14,14 are started. The blocks 1 through 6 are written to the output folder, whilst the new blocks are in temporary status. Once 7 and 8 are completed they are written on the end etc. It sounds perfect in theory :)

Or as previously suggested, a very quick pass is completed for scenecut decision in crf mode, then the stats from that are used for the block distribution. If you are using a 2 pass encode then the scenecut decision blocks can be organised in the 1st pass. The amount of memory used shouldn't be as excessive as other methods as each block is distinctly separate from every other block and should use only the memory required to process that block.

[morph166955]

@aku
so seeing as thread_pool_03 is faster then (or at least equal too) whats in the svn now, any chance we can get that to be committed to the svn or do you still want to do more testing of some sort?

[ATT]

how does this translate to real world

2 hour 1080i/p HD movie MPEG2 to XH264 does it still get 120FPS or 200FPS im sure that's on SD right :)
with a quadcore/octcore system does it still take 2 days or more ?

right now i am sitting on a X24400 Noobish attempts doing some encoding a first pass on some anime: @ 6.20FPS

CLI: --pass 1 --stats .stats --bitrate 2000 --sar 1:1 -A all --aud --level 4.1 --ref 3 --mixed-refs --bframes 3 --direct auto --analyse none --me dia --subme 1 --threads 2 --no-ssim --no-psnr --progress
avis [info]: 1920x1088 @ 23.98 fps (150945 frames)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities MMX MMXEXT SSE SSE2 3DNow!

basically for PS3 the second pass going to take 21 hours last time i tried. ( i had an error and i lost everything just minutes before it was about to finish)

has anyone done before and after of 1-2 hour film an have before and after times ? dualcore/quadcore/octcore :sly:

[morph166955]

Hows this...I'll let ya know! The system has been "down" while I've been working on my custom linux distro on it for the past week or so. I've got a nice long 1080p movie that I'm planning on testing it on overnight tonight. While it doesn't run 200fps, it does run what one would expect it to run being that the size of the image is different. For example, I've been running the tests on a cropped dvd image thats 848x352. Lets assume for math purposes that it ran exactly 200fps. Lets also assume for a moment that I was maxing out the CPU's which I'm not currently able to do. For a 1080p image you wouldn't expect 200fps since the 1920x1080 image is ~7 times larger then the 848x352 is. However, you would expect it to be around 200/7 which is ~28.5. When I was running some very quick tests on it (5000 frames max), I was seeing frame rates in the low 20's. While its not perfect scaling, it was adequate enough for my tastes considering that most current systems doing full 1080p run below 5fps if even that. I do remember at one point saying "wow I'm encoding 1080p in insane quality in real time" so I must assume that since it was a 24000/1001 frame rate that I was getting around 24-25fps for that block.

I'm recompiling x264 tonight to work on this and I'll run some tests once thats all done and I've gotten a few other small things tweaked. So as I said before, I'll let ya know!

[708145]

i just ran a test to see what would happen if i loaded 4 ram buffers with 500 frames each, and then simultaniously forked 4 x264 (vanilla r655 build) threads running @ threads=2 (i also tried @ threads=3 the effect was negligible) , i was able to run the system at >90% cpu usage since each x264 thread was using 185-190% cpu of the 200% max it should be able to do. each thread ran at around 105fps giving me a total of about 420fps.

With just 8 cores I think 3 GOPs with 4threads will result in the same speedup with less memory usage. With more cores you'd want to find a better sweet spot (For 16 cores probably 5 GOPs with 5 threads each... I need to check this).

If you want to use GOP level encoding be aware that the main problem is not scaling performance but to get I frame positions right (as you already do) and also get the bit distribution right. Both x264farm and ELDER (my pet) come very close in that respect ;)

bis besser,
T0B1A5

[morph166955]

In case anyone is still wanting to play around with this I have created a new diff based on the current svn. There were some things moved from common.h to osdep.h so I have fixed that as well as altered version.sh to generate a version that has "_THREADPOOL04" at the end (eg mine is now 0.56.669_THREADPOOL04). I just ran a two pass on some 1080p stuff and it gave me a gooooooood boost in speed.

Just to note, there is nothing different functionality-wize between this and the tp03 patch, I just fixed it to work with the current svn (r670).

http://www.benswebs.com/x264_thread_pool.04a.r670.diff

EDIT: oops uploaded it to the wrong place, link works now

[morph166955]

Thread pool patch was updated to r678 and moved to http://www.benswebs.com/public/x264/x264_thread_pool.04b.r678.diff

Post #1 updated to have new link also

[delacroixp]

It's always nice to see some movement in the world of multi-core threading optimization... especially in light of AMD plans to release a triple core (http://seekingalpha.com/article/47473-amd-to-launch-triple-core-cpu-in-early-2008) in early 2008 and the need for a multi-core killer app (http://seekingalpha.com/article/47912-amd-needs-a-multi-core-killer-app?source=feed) ...

" Most of the software in the market cannot take advantage of multi-core chips, so why buy machines that boast to have them and pay the extra price ? " ( Sramana Mitra (http://seekingalpha.com/article/47912-amd-needs-a-multi-core-killer-app?source=feed)).


:):devil::D
Pascal

[708145]

especially in light of AMD plans to release a triple core (http://seekingalpha.com/article/47473-amd-to-launch-triple-core-cpu-in-early-2008) in early 2008.

Ah finally. A Tri-Athlon ;)
The fitting CPU for the best sport in the world :D

But on-topic: There are apps around AI that definitely benefit from more cores. Think "decision helpers", "memory expanders" and "information filters" that adjust to your preferences. Video encoding is already as parallel as it need to be.

[squid_80]

" Most of the software in the market cannot take advantage of multi-core chips, so why buy machines that boast to have them and pay the extra price ? " ( Sramana Mitra (http://seekingalpha.com/article/47912-amd-needs-a-multi-core-killer-app?source=feed)).
Doesn't this guy ever run more than one program at once?

[Sharktooth]

she is an idiot. she did a very superficial analysis and draw even some false conclusions.

[lexor]

quick question, I see that Cef's page lists this patch (and others), but does Cef's build actually have this patch? (or any of the ones in patch directory?)

[Adub]

Yeah, I believe Cef is fullfilling that hole that is Sharktooth in his absence.

Aka. He is building x264 with patches.

[morph166955]

well considering that the current revision of the patch wont apply cleanly to r680 i'm not sure how yes doing it. I'm going to update the patch this weekend to work with r680's naming conventions.

[Cef]

Well I just added "x264_" here and there to compile with r680. I'm not even sure what's this patch suppsoed to do though, so I don't know if it's actually done right. diff is here (http://mirror05.x264.nl/Cef/force.php?file=./patches/x264_thread_pool.r680.diff)

[burfadel]

Cef, separate issue, but will you be adding the --me-prepass patch in future builds? its a great option! Its good now, in another post it was talked that it will be modified to reduce its performance impact (which is not excessive at the moment).

[Dark Shikari]

Cef, separate issue, but will you be adding the --me-prepass patch in future builds? its a great option! Its good now, in another post it was talked that it will be modified to reduce its performance impact (which is not excessive at the moment).I need to fix a few things about it--Aku noticed a few problems with it that may both reduce its speed and its effectiveness below what it could be.

[burfadel]

I know :) maybe if the performance impact with the modifications is reduced enough it can be enabled by default, and then have an option of --nome-prepass or something?

Also, isn't imh supposed to replace umh? and also 'subme 8' replacing 'subme 7'?

[morph166955]

My patch is updated and uploaded to work with r680. Link on the main page has been updated.

[NaturalThoughts]

I have query regarding QP in case of multithreading enabled.

As in the case of single thread, each next frame QP is calculated based upon previous frame stats.

IPPPP.... case
e.g. 0th Frame (QP =26)
1st frame (QP modifies based on 0th frame stats),
2st frame (QP modifies based on 1th frame stats)
and so on......
But in the case of more than one thread, how QP is assigned for next frame as present frame or previous frame is being encoded.

0th Frame (QP =26)
1st frame (QP = ?),
2st frame (QP = ?)
and so on......

Thanks in advance.

[akupenguin]

1pass CRF/CQP: completely ignore previous frames' stats.
1pass ABR/2pass: QP adapts by stats from whichever frames have finished encoding. Any sane number of threads makes little difference, as the adaption time scale is much longer than the threading latency.
VBV: latency is important, so QP adapts by stats from whichever portions of each frame have finished encoding.

[NaturalThoughts]

1pass CRF/CQP: completely ignore previous frames' stats.
1pass ABR/2pass: QP adapts by stats from whichever frames have finished encoding. Any sane number of threads makes little difference, as the adaption time scale is much longer than the threading latency.


Ok I got it.
1) So Can I assume that for 1 pass CQP : Irrespective of number of threads (1,2,4,8,...) PSNR and output bit rate will be same, or that is a little chance of penalty with threads and PSNR and output bit rate may change....

2) 1 pass ABR : definitely there will be penalty in terms of PSNR and output bit rate by increasing number of threads.

[akupenguin]

With CRF/CQP there is a small threading penalty, due to the limitation on mvs.
With 1pass ABR there is an additional penalty due to the slightly less accurate distribution of bits.

[LoRd_MuldeR]

For any reasonable number of threads the penalty will be so small that it doesn't matter. In reality you probably won't be able to see the difference...

[NaturalThoughts]

With CRF/CQP there is a small threading penalty, due to the limitation on mvs.

you mean it will be because of this

h->param.analyse.i_mv_range_thread = r2;
of x264_validate_parameters() of encoder.c

With 1pass ABR there is an additional penalty due to the slightly less accurate distribution of bits.

Could u explain this a little bit......

[NaturalThoughts]

I feel that due to lines of code shown below........

h->param.analyse.i_mv_range_thread = r2;

and

h->mb.mv_max_spel[1] = X264_MIN( h->mb.mv_max_spel[1], thread_mvy_range*4 );

As motion vector search range will change, so penalty will there for both the case CQP/ABR . Am i right???

What will be the reason for additional penalty in ABR and how???

Thanks in advance.

[NaturalThoughts]

A dig a little bit more..... here is my understanding as of now.......

Ratecontrol module in the case of ABR and 2 threads.... (1pass only)
rce.blurred_complexity = f (current frame SATD)
q = f (rcc->wanted_bits_window, rcc->cplxr_sum) (e.g x= f(y))

So QP will be
0th frame:
rc->qp = Based on current frame complexity (based on SATD)

1st frame:
rc->qp = Based on current frame complexity (based on SATD) as 0th frame still not finished

2nd frame:
modify rcc->wanted_bits_window (number of frame done and fps), rcc->cplxr_sum (based on 0th frame bits consumed) .
rc->qp = Based on current frame complexity (based on SATD) with modified rcc->wanted_bits_window, rcc->cplxr_sum

3rd frame:
modify rcc->wanted_bits_window (number of frame done and fps), rcc->cplxr_sum (based on 1st frame bits consumed) .
rc->qp = Based on current frame complexity (based on SATD) with modified rcc->wanted_bits_window, rcc->cplxr_sum

4th frame:
modify rcc->wanted_bits_window (number of frame done and fps), rcc->cplxr_sum (based on 2nd frame bits consumed) .
rc->qp = Based on current frame complexity (based on SATD) with modified rcc->wanted_bits_window, rcc->cplxr_sum

and so on.........

Is it correct or am I missing something???

And if my understanding is correct then can I say in the case of threading enabled QP calculation is done without previous frame bits consumption data, so there will be chance of deviation from the desired average bit rate. So how x264 handle this to achieve ABR ???

And also can I say penalty increasing linearly with number of threads for ABR....

[akupenguin]

so there will be chance of deviation from the desired average bit rate. So how x264 handle this to achieve ABR?
x264's ABR accuracy is measured in percentage points of the total filesize anyway. A few extra frames of inaccuracy is nothing compared to even the nonthreaded tolerance. Which I said before: adaption time scale is much longer than the threading latency.

[NaturalThoughts]

Ok...
could u tell me about rcc->rate_tolerance....

how this affect final bit rate ???
I found this in common.c , function x264_param_default()
param->rc.f_rate_tolerance = 1.0;

And later in x264_ratecontrol_new() in ratecontrol.c

rc->rate_tolerance = h->param.rc.f_rate_tolerance;
and then
double abr_buffer = 2 * rcc->rate_tolerance * rcc->bitrate;

SO if rcc->rate_tolerance is 1.0 then abr_buffer is just double of the desired bitrate... :scared:

Can I vary it to 0.6 or 0.5 so it will be close to final bit rate.....???