x264 multi-core (4+) threading optimization

[burfadel]

just encode 2 things at the same time :P

Or set the first encode from frames 0 to x, then the second iteration of x264 from frames x to finish! = 90 percent + CPU usage, and then it can be easily combined at the end!

Only a factional loss of bitrate if the x doesn't fall on a keyframe (a keyframe would be inserted here that may be unnecessary).

[DoctorEnsGabe]

I'm pretty sure that these are the exact things that morph is trying to avoid and is helping aku to solve with his monster 8-way box :)

[delacroixp]

while thats true, i'd think that we should be able to get x264 over 45% cpu usage when not using HD content. i'm still using x264 for my SD content and having that unused speed would be amazingly great. I can also peg x264 w/o threadpool if i throw a huge frame at it just like anyone else. its doing the smaller frames im more worried about at this point then the larger ones.
I doubt anybody will be using your kinda rig for sub-HD encodes...

However, having said that, given the release of quad-core sytems, it's not far fetched to imagine your average joe with a twin quad-core box crunching Mpeg-2 movies... either individually or, 2 and 3 in parallel...


:):D:eek:
Pascal

[morph166955]

I doubt anybody will be using your kinda rig for sub-HD encodes...

except me of course who still has dvd's that hes not done backing up... :D

just because i have a rig like this doesnt mean that i've locked myself solely into HD. SD is still a big part of media right now (especially with tv) so until thats gone I'm sure I'll be working on SD stuff. although i still giggle when i see my 720p source encoding faster then realtime.

oh and p.s. for those wondering, I think ive nipped my cooling problem in the butt. 3x thermaltake 80mm fans which can push 75CFM EACH are now positioned to blow air clear across my motherboard. although I think the cpu fans I got are junk...they are running HOTTER then the stock ones did however I believe that could be because the thermal grease hasnt had time to burn in yet so i'm going to leave it for a week to see what happens. if not, I found a different fan last night that will probably fix it and possibly give me some air flow for my ram as well (its a side mount type instead of top mount, so while ill be blowing warm air over the ram its better then having the heatsink like a wall between the ram and the fans)

[Inventive Software]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)

[morph166955]

im pretty confident that the thermal grease just has to work its way in. its been on there now less then 24 hours and the reviews on the cooler all say that the systems run 5-10 degrees or more cooler then the stocker so im hopeful. we shall see after i let the grease melt in. im leaving it just idle for now, ill run it a little later on to get a good heat variability. worst case scenario i replace them again with the ones i found last night which all in all i may do anyway. its a 2U case so a fan thats on top is kinda inefficient since it has almost no room to suck air in from, a side mount however has plenty of room plus the case fans will have already pressurized the air blowing at it so it will give even more air flow.

[Mutant_Fruit]

im pretty confident that the thermal grease just has to work its way in. its been on there now less then 24 hours and the reviews on the cooler all say that the systems run 5-10 degrees or more cooler then the stocker so im hopeful. we shall see after i let the grease melt in.
I'd say the odds are you've added too much thermal paste. There should be a very thin layer of thermal paste between the cooler block and CPU. If there's too much, it acts like insulation and the heat can't escape. Thermal paste doesn't "melt in" as such. Some thermal pads do "melt in" but that takes very little time to happen, a few hours at the most.

[delacroixp]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)

LoL


:):D:eek:
Pascal

[morph166955]

I'd say the odds are you've added too much thermal paste. There should be a very thin layer of thermal paste between the cooler block and CPU. If there's too much, it acts like insulation and the heat can't escape. Thermal paste doesn't "melt in" as such. Some thermal pads do "melt in" but that takes very little time to happen, a few hours at the most.

from everything ive read and seen over the past few years, thats a very inaccurate statement. higher end metalic thermal pastes require a minimum of 35 hours to kick in and in some cases almost 200 hours for them to completely set to optimal efficiency (most are around 100). im uzing the zalman thermal grease which is considered one of the good ones (no wars over zalmbn vs as5 vs w/e please) and zalman even says they take 35-40 hours for the paste to set to the point that its mostly efficient. and i definitely only put a very thin layer. one good thing with the zalman is that it comes with a nailpolish type applicator which makes it very easy to layout and smooth down to what a person wants. i do agree that too much can cause problems, but in this case i think my biggest problem is that the fan is on top and very close to the top of my case. more likely then not, im going to have to get that side mount cooler for this setup.

[Mutant_Fruit]

higher end metalic thermal pastes require a minimum of 35 hours to kick in and in some cases almost 200 hours for them to completely set to optimal efficiency (most are around 100).
Now that's news to me. I knew that some thermal pads needed to "melt in", but i never heard of a paste needing to melt in considering a paste can easily fill in all those tiny nooks and crannies on application whereas a pad can't.

Ah well, ya learn something new every day.

[delacroixp]

Ah well, ya learn something new every day.
I'll 2nd that...
I just wish I could unlearn some of the old stuff to make way for the new... before the RAM becomes too volatile and switches off completely...


:):D:eek:
Pascal

[burfadel]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

Put a digital thermometer sensor inside your case near the inlet of the cpu fans. Even if you have a case sensor its the position inside the case thats important. You'll probably find its several degrees higher than outside your case, so with fan ducts at least the air used for cooling is several degrees cooler.

The best CPU fan in the world is relatively pointless if cool airflow (and not just airflow) isn't available. A high flow helper fan may be beneficial at the inlet of the fan ducts as long as its pointing the correct way!

[morph166955]

ive figured out the air flow, im sucking cold air in right from behind the dvd drive, through the 3x 80mm's, past the two cpu's (parallel with the fans so both get equal flow), then over the ram and out the back. im working on it, ill figure it out eventually.

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)


ive figured out the air flow, im sucking cold air in right from behind the dvd drive, through the 3x 80mm's, past the two cpu's (parallel with the fans so both get equal flow), then over the ram and out the back. im working on it, ill figure it out eventually.

How about one of those 10 inch 'slinky' piping-ducts they use on air-con systems to hook-up your a/c directly into the 'Monster'...
We're rooting for you... this is more fun than having a quad-quad, water-cooled, vapour-chill, overclockers-extreme...

I still think the Abacus rocks...


:):D:eek:
Pascal

[morph166955]

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my frat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day hes working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until theres a loud thud and his computer sounds like its going to destroy itself. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

back on topic...the power supply just arrived as i was typing this so im putting that in and then going to do some high power tests to see if it was in fact a power issue not a heat issue causing the system to reboot. ill report soon.

[delacroixp]

It seams that the guys at Hardware & Software, MeGUI CPU Time Test - Compare different CPUs encoding the same file (http://forum.doom9.org/showthread.php?t=122318) have also got a similar test crunching thing going... though, unfortunatelly, without akupenguin's input to add-value to their research and developement...


:):D:eek:
Pascal

[morph166955]

so back on topic...whats the next step here?

[Inventive Software]

More tests! :D

[burfadel]

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my frat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day hes working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until theres a loud thud and his computer sounds like its going to destroy itself. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

back on topic...the power supply just arrived as i was typing this so im putting that in and then going to do some high power tests to see if it was in fact a power issue not a heat issue causing the system to reboot. ill report soon.

lol! putting the duct from outside in a cool/cold climate is a good idea, as long as the end is completely covered with secured flyscreen (a very fine wire mesh). It is durable, doesn't stop air flow, but stops everything else including small insects.

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

How about one of those 10 inch 'slinky' piping-ducts they use on air-con systems to hook-up your a/c directly into the 'Monster'...

funny off topic story...i used to live up in troy, ny...it gets COLD in the winters. one of my fat brothers had that same genius idea. he ran a 10 inch ac duct from the window into his computer. one day he's working and he sees the pipe start to move. thinking that its wind he doesnt think anything of it until there's a loud thud and his computer sounds like its going to self-destruct. turns out a squirrel thought that it was a nice warm place to put itself out of the cold and it took a flying leap into the cpu fan. lets just say that was the last time he did that...and he loved cleaning squirrel parts out of his computer...cause when i say flying leap i mean this thing got ginsued and tossed by the fan.

lol! putting the duct from outside in a cool/cold climate is a good idea, as long as the end is completely covered with secured flyscreen (a very fine wire mesh). It is durable, doesn't stop air flow, but stops everything else including small insects.
LoL... nothing beats a naturally free air-con... a regular 5 or 6 inch circular wire-grill in the fan itself wouldn't hurt airflow either...


:):D:eek:
Pascal

[delacroixp]

so back on topic...whats the next step here?

You were hitting 90-95% utilization when you were doing an HD encode, obviously the threadpool model works. It's just that with lower resolution sources x264 can't keep the cpu's pegged.
Perhaps, multiple SD encodes are the way to go... potentially, even 7 or 8... especially if you have tons of SD material...


:):D:eek:
Pascal

[delacroixp]

Your other option if you're still having cooling problems is to get a desktop fan and point it full blast at the computer, preferably motherboard side, case open. Then see if you're getting issues with temperatures. :)
Funny enough I tried that just now with an Mpeg2 DAResolution (1024x576) encode of the BBC's Hiroshima...
I'm using an AMD Athlon dual-core which runs exceptionally cold anyway...
Between 20% and 40% the FPS were steady at 5.88 but after placing the 15 inch (37.5 cm) floor-fan facing the PC with both side-covers off... the FPS rose to 6.24 until 70%... when they fell off and steadied at 6.15 FPS.

I live near the coast at 34º South... roughly equivalent to Dallas-Texas, Phoenix-Arizona or the tip of Spain in northern latitudes... it never gets very cold and only rarely dips below freezing... but my encodes speed up significantly at night or when it's cooler...
Even Internet Explorer can affect your encodes and I'll often even switch off non-essentiall services on the PC to gain a little speed during some of those 'rough' encodes when... the fps keep on dipping all the way through the encode like some stricken aircraft...

I guess that we won't be moving to the Antarctic or the 'Darkside of the Moon' anytime soon but... cooling does make a difference...
Is there any app that maps FPS-to-time,percentage,frameNo in a simple line graph... during H264 encodes ???


:):D:eek:
Pascal

[legoman666]

you're a moron if you think you're encoding speed has anything to do with you placing a fan next to your machine. plain and simple.

[delacroixp]

you're a moron if you think you're encoding speed has anything to do with you placing a fan next to your machine. plain and simple.
I'm sure that you're right...


:):D:eek:
Pascal


btw
I never think... I leave that to the experts...

[delacroixp]

It is possible with your case to have fan ducts? Instead of sucking the already warmed air back through the cooler, cool air from outside is always flowing onto the cpu cooler. The output air from the cooler should also be cooler!, so it may decrease case temperature as well :)

Not only case temparature... but motherboard too...
It's one of those neglected components when it comes to cooling... CPU, GPU and even HDD's are usually first in the queue...
However, using a desk-fan can even run your power-supply stone cold...

I've got 6 drives in 2 sets of RAID (stripped and mirrored) but you'll be amazed how many computer errors can be traced back to over-heating...
Sure, your system may be running within spec tolerances... but even then your components are sufferring from gradual degradation...

Heat causes electron-flow-resistance to increase, while extra cooling may induce super conductivity... and improved performance in general...


:):D:eek:
Pascal

[burfadel]

Whilst heat may increase the resistance, super conductivity, by definition of the term, would require that you chill it to around 2 kelvin or less (-271C or less), where 1 kelvin = 1C, except from absolute 0!... the only practical place for temperatures so cold is for radio astronomy.

The other thing to get hot is the motherboard chipset. Most of the time they put stupid little heatsinks on them that aren't sufficiently large, or poor quality fans that die sooner rather than later! Some now use heatpipes for the chipset, with the heatsink at the rear of the case.

[delacroixp]

the only practical place for temperatures so cold is for radio astronomy.

Possibly transmission cables and maglev trains (http://www.howstuffworks.com/question610.htm) though Vapo (http://www.extremeoverclocking.com/reviews/cases/VapoChillPE_5.html)Chill (http://www2.asetek.com/default.asp?showPage=startside.asp&contentSection=2) was quite popular with PC's at some stage...


:):D:eek:
Pascal

[burfadel]

Possibly transmission cables and maglev trains (http://www.howstuffworks.com/question610.htm) though Vapo (http://www.extremeoverclocking.com/reviews/cases/VapoChillPE_5.html)Chill (http://www2.asetek.com/default.asp?showPage=startside.asp&contentSection=2) was quite popular with PC's at some stage...


:):D:eek:
Pascal

Yes very true! It works but its not practical! Its very difficult to keep a very large area (which would be required for Maglev trains & transmission cables) at the temperature that allows super-conductivity (ideally below -271C). That why it wasn't initiated in current Maglev train trials.

For radioastronomy, the area to keep cold is relatively small and easy to manage. The cold is also used to reduce noise.

[delacroixp]

The cold is also used to reduce noise.
I suppose it (http://coolcosmos.ipac.caltech.edu/cosmic_kids/AskKids/arecibo.shtml) would include background star-clutter or big bang (http://deepspace.jpl.nasa.gov/dsn/history/dsn67.html) wave residue... a bit like the noise found on certain video encodes...


:):D:eek:
Pascal

[morph166955]

guys i dont mean to be a kill joy here but can we try to get this thread back on topic? the thread_pool.03 patch was the best of the 3 although it still bottlenecks below max cpu on non HD sources. Unfortunately I'm finishing up my work so I can graduate in a week (yes its scary) so I haven't had time to work on my thread model (doing multiple scenes at the same time in different threads). anyone else getting closer to getting this working?

[Mutant_Fruit]

hread model (doing multiple scenes at the same time in different threads). anyone else getting closer to getting this working?
From the sounds of it, that method won't scale. If the rough maths that i used earlier were anywhere near accurate, and my understanding of akupenguins comments were right, each thread would require an amount of ram so high, that you couldn't run more than 2 or maybe 3 with HD sources.

[morph166955]

2 or 3 with HD source "should" still peg an 8 core cpu if you have each gop-thread doing multi-threaded encoding of the current design (threads with in threads basically). HD source can peg 3-4 threads with out any problem, run two of them concurrently and there ya go! they may even be able to share some of that memory some how if there being run out of the same program instead of two x264 executables being run separately.

[Mutant_Fruit]

Well, multiply the numbers you see on this (http://forum.doom9.org/showthread.php?p=996360#post996360) post by 4 to get the amount of ram that will be actually required. That'd bring most machines to the ground.

This method on it's own would never scale, and even on current systems may perform hideously slowly. If x264 has to swap in and out from the harddisk to access frames performance will drop.

The memory isn't "shareable" as such because the memory is taken up by frames which have to be stored in memory. You can't reduce that memory without reducing ref's, threads or some other such thing. Thats the only bad thing i can see about your method.

[morph166955]

I agree, most computers will run into issues at that point...however...were not talking about most computers. were talking about systems that have 4 (or more realistically 8+) cores in them. id feel really really bad for anyone who built a machine with that much power and only a gig of ram. thats like building a tank and loading it with 9mm bullets instead of full size shells. i'd hope that any machine like that has a minimum of 2 gig if not 4 gig of ram in it.

however, i also realize x264 has to be able to handle all machines both big and small. I'd think that we could put a switch in x264 somewhere that can be set to how many gop blocks to thread out at a time, and default that to 1. at 1, it would run like x264 would now. above 1 it would do that many gop's simultaneously. i suppose it could also be set so that if its set to 1 then x264 does not buffer more then the amount of frames that it needs just like it does now.

I suppose that another option is needed on top of this. What about doing a scaled down version of my gop-thread idea and break each gop up into the # of threads were running. then we would only need to buffer in a max of what ever keyint is set to plus maybe 3-5 frames. heres a little breakdown of what im saying:

Say that the GOP has 250 frames (equal to keyint since thats a worst case senario). Lets also say you are running 4 threads. Have thread 1 doing frames 1-62, thread 2 doing 63-124, thread 3 doing 125-186, and then thread 4 doing 187-250. Now I realize that you cant break at just any point (like have a thread start at a b-frame), but isn't it possible to create the I frame first, and then break the image up at the closest p-frame to an even split? we would have the entire scene in the buffer to reference from so in theory we could have each thread look at a few frames before its part to make a decision about the first frame and then the frames after that, hopefully we wouldn't have to encode the frame just use it as a reference point.

I don't know if this is possible to do, but its yet another idea out of my jumbled brain.

[Inventive Software]

My initial thought is that you could split the whole encoding job into nearly identical jobs at intervals, and join the thing together right at the end, though with x264.

Take a 5000 frame encode. Split it into 4 jobs, 2 threads per job, that's 1250 frames per job. Of course, you could customise the number of jobs and threads at the command line. ;)

This may well need some more thought, and there probably will be slightly more IDR frames than necessary, but if you're a smart encoder, you'll use something like 250 anyway. What do you think?

[morph166955]

i just ran a test to see what would happen if i loaded 4 ram buffers with 500 frames each, and then simultaniously forked 4 x264 (vanilla r655 build) threads running @ threads=2 (i also tried @ threads=3 the effect was negligible) , i was able to run the system at >90% cpu usage since each x264 thread was using 185-190% cpu of the 200% max it should be able to do. each thread ran at around 105fps giving me a total of about 420fps. i ran these on the same settings as job1 uses in the earlier tests just to be uniform. in comparison, the highest fps that ive been able to get job1 was 202fps, thats more then double the speed that weve been able to accomplish. granted this is a very crude approach to how, but it at least shows that such an approach has the ability to succeed in reaching the 100% cpu usage/max fps goal

[akupenguin]

Heres threadpool.03 for threads 1-8...it leveled off after 7 on the high ones and 5 on the low ones so i didnt run any more.
I think I can explain the 7. In svn, there's no space between the serial setup and the slice encoding, so the setup essentially counts as part of one of the other threads. In thread_pool, the queue means they're not locked together, so the setup counts as a separate thread. (Setup was always in a different physical thread, but only now can it run concurrently.)

[morph166955]

oooooooooooooooooooooooooooooooooooooooooooooooooh...so 7 really equaled 8? that makes a whole lot more sense then!

[delacroixp]

This is a great project for a doctoral thesis...

:):D:eek:
Pascal

[morph166955]

i love this! i get the cooling issue fixed (and i mean FIXED...28C & 23C on the two chips right now!)...and one of my hard drives decides its time for it to die...and its only 18 months old! This whole thing is really getting annoying lol.

so back on topic...I found something interesting out about pthreads in the past few days that may help us (which you probably already know about aku), but ill explain it anyway. One of my final projects I'm working on for me to graduate is for some independent research that I'm doing with the physics department (im a comp sci major, physics minor...fun i know). Were using a series of formulas to generate images of planetary nebulas based on predefined values. The system were working on is a 17 node beowulf cluster each one housing dual hyper-threaded 3.2ghz xeons (nocona's i believe). The way the system is setup, is to have one machine act as the head and the others as slaves, the head does little to no work on the processing since its handling the data flows in both directions to all the slaves and compiling the final data they are sending back. to do this, we ended up running both the MPI libraries to handle the inter-machine communications and pthreads to allow us to doing threading on each machine and utilize both of the machines processors. since were generating one image the way we ended up doing it was to divide the image into almost equal segments (the last one is normally shorter) and use mpi to send that out to the machine (for example, slave1 would do lines 1-16 of the image). going with that example, the slave machine was then creating 1 thread per line (max of 4 threads at a time), joining the threads, doing all the pixels in that line, and then ending so another thread could be created. what we found was that there was a significant amount of time being spent creating and destroying the threads and that having it create threads only once to do multiple lines (eg 1-4,5-8...) gave us a noticable decrease in processing time.

I bring this up because i noticed in some of the debugs that we were running a while back that we were creating & joining threads quite often. Is there any way to leave a thread active longer and have it set to do multiple areas/frames/whatever before it exists instead of creating and destroying threads as often as we do now? say for example have it do 4 (arbitrary #) frames per thread instead of 1 or something like that? I would think that since this method removes several wait times for the creation & joining of threads that it opens up the possibility that more processing will actually happen and bring our fps/cpu% up.

I'm not sure what this does to the whole i/p/b frame thing but I figured I'd throw it out there with the hopes that it was a decent idea or that it would spark another idea in someone to aid in this discussion. On the other hand I could be 100% wrong and my idea may not work at all but hey if people didn't try to come up with different ideas for things we may still have squares instead of wheels!

[akupenguin]

Already discussed here.
It is not possible to assign multiple frames to a thread at once, but it is possible to never create new threads and only pass data to existing threads. thread_pool.03 does this. Much like x264 never allocates memory after startup.

[morph166955]

well there goes that idea then...

[delacroixp]

what we found was that there was a significant amount of time being spent creating and destroying the threads and that having it create threads only once to do multiple lines (eg 1-4,5-8...) gave us a noticable decrease in processing time.

it opens up the possibility that more processing will actually happen and bring our fps/cpu% up.

Good point... 100% CPU usage is useless without real-work... you really don't want a dig-hole, fill-hole kinda loop...
Getting maximum efficiency out of multiple-processors has always been the quest and unless you have non-serial jobs (4 guys painting wall vs putting in window then paint) it's difficult to harness the true potential power out there.
Also, as you say, lines of communication increase exponentially as any maths or stats 101 student will know...

Having said that, encoding and multi-processor, multi-core sytems seam to be great bedfellows.


:):D:eek:
Pascal

[check]

Much like x264 never allocates memory after startup.
Does this reduce the chance of caching inactive sections of x264's memory, or is it for some other obscure reason?

[Mutant_Fruit]

Does this reduce the chance of caching inactive sections of x264's memory, or is it for some other obscure reason?
The main reason is performance. Suppose you want to encode at 100 frames a second, and each frame is 2 megabytes when in memory then you'd be allocating and deallocating 200 megabytes of memory a second.

Alternatively, you allocate 100 two megabyte buffers and constantly reuse those buffers. You use exactly the same amount of memory as before, except you have zero allocations happening a second, therefore you get better performance.

If you can turn 500 allocations a second into zero allocations a second in performance critical areas, you'll get better performance.

EDIT: This strategy can also help if you have memory leaks. If you notice your pool of memory buffers is constantly getting smaller, you can narrow down the bits of code you have to check for the memory leak. If you're just looking at task manager and seeing memory increase all the time, that's not much use in tracking down the source of the issue.

[delacroixp]

well there goes that idea then...
You've had so many really solid ideas and you've achieved so much that, even if you left the project, as is, and enjoyed your graduation...
... it would still be a truly awesome success story...
I'm sure much of your research will be implemented into H264 and contribute to H264's improved abillity to handle 8+ cores... even 16 cores (quad-quad)...

I did my project on CRC (cyclic reduncy checksum) when I was at varsity... but we could only dream about this kind of power and hardware.
CRC is the same thing you get as an option in WinRAR and the system NASA uses to communicate with satellites from Mars...
In the first mission, 60% of the data was extraneous CRC checksum to overcome the severe corruption of coms.


:):D:eek:
Pascal

[morph166955]

You've had so many really solid ideas and you've achieved so much that, even if you left the project, as is, and enjoyed your graduation...
... it would still be a truly awesome success story...
I'm sure much of your research will be implemented into H264 and contribute to H264's improved abillity to handle 8+ cores... even 16 cores (quad-quad)...

Believe me, I don't see anything of what we have done here thus far a failure in any way. Even the tests that didn't work to our expectations were still a learning experience for the future. I'm working on compiling a custom linux distro right now for this box (see my post in the linux subsection on here for more info on it since yes I will be releasing it) that should eliminate any cpu overhead and lack of optimization that exists in the big distros. I'm hoping that this should help out a little bit more with speeding things up. If anyone wants more info about this distro, please post in that thread since I still want the focus of this thread to be optimizing x264 to work at its potential with lots of threads.

[morph166955]

so my semester is over, my college career is completed, and now I'm home for ~2 weeks before I start working with nothing but encoding to do. Anyone have any new ideas on how to do this? Now that I have some time I'm going to see if I can finish up my libx264 frontend that does the threading division that I was talking about earlier (the one thats basically like x264farm only local) to see if I can figure out a way that will work for us with out being the HUGE ram hog that it has the potential to easily become.

[Inventive Software]

What about some 1080p material, and see if that'll encode in realtime? There were some uncompressed AVI samples posted around here somewhere, so it shouldn't be that hard to dump the raw file to your RAMdisk.

BTW, congrats on completing your semester. :)

[burfadel]

How about if it encodes as it is now with say 4 cores or less, and with more than 4 cores (most likely 8 and above) a thread is run in the graphics card (or by itself in its own core) ahead of the rest of the video for scenecut decision, then just distribute the blocks of data to the individual cores. Each scenecut can be numbered and as each consecutive block is completed the output file is created. Such that in an 8 core system , if blocks 1,2,3,4,5,6 are completed, and 7,8 are longer, then 9,10,11,12,14,14 are started. The blocks 1 through 6 are written to the output folder, whilst the new blocks are in temporary status. Once 7 and 8 are completed they are written on the end etc. It sounds perfect in theory :)

Or as previously suggested, a very quick pass is completed for scenecut decision in crf mode, then the stats from that are used for the block distribution. If you are using a 2 pass encode then the scenecut decision blocks can be organised in the 1st pass. The amount of memory used shouldn't be as excessive as other methods as each block is distinctly separate from every other block and should use only the memory required to process that block.