open letter to any x264 developer

[deadrats]

hello everyone,

this is my first post to this forum, i usually frequent the forums over at video help. regardless, i know that there is at least one x264 developer that frequents this forum and there are a number of questions i have had concerning x264 and i'm hoping he/she will be nice enough to address my questions.

1) any time the issue of gpu acceleration has been brought up with regards to x264, the response has ranged from the absurd to the nonsensical, as to why it either can't be done or why it wouldn't be beneficial to x264.

the most common objections i have seen take on the form of one, or more, of the following:

a) gpu's are great for code that is easily parallelized but video encoding can't be easily parallelized. this stance is usually mated to the assertion that gpu's only really start to show any benefit when the thread count is greater than 24 (at times the minimum thread number had been claimed as 32). this strikes me as a rather odd position to take as video streams are composed of tens of thousands of frames and each 1920x1080 frame is composed of over 2 million pixels, i find it hard to believe that one couldn't think of any method to launch more than 24 (or 32) processing threads to encode said stream.

i can think of any number of ways to launch hundreds of threads, perhaps the most obvious is to assign each gop to a single thread. basically you would have a read thread that analyses the video stream, each group of pictures it detects would be assigned to a separate array for processing, and you would have a thread that continuously concatenates the finished results to a final file.

b) another commonly asserted claim is that gpu's are great for floating point intensive code but that video encoding is "inherently" heavily integer based thus gpu acceleration would really be of little benefit except for a few functions like motion estimation.

this however shows a basic and fundamental mis-understanding of computer architectures. code is not inherently anything, it's up to the programmer to decide whether the math is performed via the fpu or the alu (or integer unit, if you prefer). the notion that a particular operation is inherently integer based has it's origins back in the days when the x87 co-processor existed as a separate processor.

back in those days, i remember my comp sci professors instructing us to structure our code to use integer math wherever possible, because not all computers had a co-processor and x87 processors were slower than the integrated alu on the cpu. this was done whether you were programming with basic, fortran or pascal and the same lessons were carried over when you started c/c++ programming.

the reality is that any integer operation can be evaluated by the fpu by simply re-declaring int as float. for those that don't know in most high level programming languages you need to declare a data type, such as int, long, long long, float, double, character and so on.

when you declare a data type you are telling the compiler to instruct the cpu (by replacing your commands with assembler equivalents) to carry out said operation either on the integer unit (alu, for you old school coders) or the fpu. by re-declaring int as float you have now made your code floating point intensive and thus you could benefit from the significantly better floating point capabilities of modern gpu's.

c) the last claim that i have seen made, and this was done in one of the entries in the "diary of an x264 developer" is that the various gpu related api's are not well documented. this one just boggles the mind, the cuda sdk is rife with code examples (including a functional h264 encoder and decoder), there is ample documentation, hell there are even graduate level college courses offered in cuda programming, the claim that there is insufficient documentation is a flat out lie. similarly open cl is well documented, microsoft more than documents dx10 and gpu powered encoding is even possible using dx9 (the original avivio was dx9 based and gpeg2 is an ffmpeg based, dx9 powered, mpeg-2 encoder that runs on any dx9 and later gpu, with excellent quality i might add).

thus the claim that any particular api isn't well documented comes across as disingenuous.

there are a couple of reasons i can think of as to why the x264 developers may be loathe to port x264 to run on a gpu, and i would like to hear from one, or more, of you guys if these are the primary reasons you seem reluctant to add gpu acceleration to x264:

1) you guys obviously believe in open source, gpl'd code and porting your encoder to a proprietary technology would fly in the face of what you guys stand for. if this is one of the, or the primary reason, i can respect that, and in fact support it, but it would be nice if you guys just said as much.

2) perhaps you guys don't know enough about coding for the gpu and this has been the biggest road block. again i can respect that, after all coding for the x86 architecture has been around for 20+ years and most people wouldn't know where to begin porting their code for a new architecture.

3) one of the biggest technical road blocks that i can see is that it appears that you guys have used function pointers with x264, am i correct? this would effectively prevent you from porting x264 to any gpu prior to fermi as fermi is (and ati's newest dx11 gpu's) are the first gpu's to support function pointers.

ignoring x264 for a second, have you guys ever considered perhaps coding a gpu accelerated, high quality decoder that you could integrate with x264? this would go a long way towards improving performance and would make x264 a true codec, rather than "just" an encoder.

have you guys ever considered taking the basic cuda encoder that comes with the sdk and simply using everything you guys have learn creating x264 and improving the quality of that encoder? areas where improvement could be made would be to B frames, adding 2 pass encoding and perhaps a bit of tweaking here and there to make it a decent h264 encoder.

while i'm at it, do you guys have any plans of making use of "sandy bridge's" integrated video transcoding engine or will you simply choose to focus on avx? in all honesty it does seem that the new 256 bit avx simd's will effectively render any discussion of gpu powered encoding mute, but the door is still open for gpu powered decoding.

lastly, considering bulldozer is slated to have an incredibly beefy 256 bit fpu, do you think it would behoove you make an bulldozer specific x264 variant, by swapping out all the int operations for floating point operations? bulldozer will feature two 128 bit integer units per core but that 256 bit fpu that's backed by two 128 bit mmx units looks to be a monster and considering it appears that x264 makes use of 32 bit math operations, doesn't it seem likely that changing all your ints to floats would allow bulldozer to just tear through an x264 encode?

sorry for the long post, any thoughts/comments are appreciated.

'rats.

[Dark Shikari]

hello everyone,

this is my first post to this forum, i usually frequent the forums over at video help. regardless, i know that there is at least one x264 developer that frequents this forum and there are a number of questions i have had concerning x264 and i'm hoping he/she will be nice enough to address my questions.

1) any time the issue of gpu acceleration has been brought up with regards to x264, the response has ranged from the absurd to the nonsensical, as to why it either can't be done or why it wouldn't be beneficial to x264.That's because people don't use "search". We've answered the question so many times that our fingers are sore.

a) gpu's are great for code that is easily parallelized but video encoding can't be easily parallelized. this stance is usually mated to the assertion that gpu's only really start to show any benefit when the thread count is greater than 24 (at times the minimum thread number had been claimed as 32). this strikes me as a rather odd position to take as video streams are composed of tens of thousands of frames and each 1920x1080 frame is composed of over 2 million pixels, i find it hard to believe that one couldn't think of any method to launch more than 24 (or 32) processing threads to encode said stream.24? No, more like 20,000. The problem is that your threads need to be exactly in sync with each other, performing identical operations, or else the loads can't be coalesced. Even multiple simultaneous motion searches (by any of x264's definitions) aren't sufficient to do this.

I'm not going to explain this again, go read a CUDA manual or something.

b) another commonly asserted claim is that gpu's are great for floating point intensive code but that video encoding is "inherently" heavily integer based thus gpu acceleration would really be of little benefit except for a few functions like motion estimation.

this however shows a basic and fundamental mis-understanding of computer architectures. code is not inherently anything, it's up to the programmer to decide whether the math is performed via the fpu or the alu (or integer unit, if you prefer). the notion that a particular operation is inherently integer based has it's origins back in the days when the x87 co-processor existed as a separate processor.

back in those days, i remember my comp sci professors instructing us to structure our code to use integer math wherever possible, because not all computers had a co-processor and x87 processors were slower than the integrated alu on the cpu. this was done whether you were programming with basic, fortran or pascal and the same lessons were carried over when you started c/c++ programming.

the reality is that any integer operation can be evaluated by the fpu by simply re-declaring int as float. for those that don't know in most high level programming languages you need to declare a data type, such as int, long, long long, float, double, character and so on.

when you declare a data type you are telling the compiler to instruct the cpu (by replacing your commands with assembler equivalents) to carry out said operation either on the integer unit (alu, for you old school coders) or the fpu. by re-declaring int as float you have now made your code floating point intensive and thus you could benefit from the significantly better floating point capabilities of modern gpu's.The point isn't that float is faster than int, it's that, compared to a CPU, it's relatively faster than int. Individual 8-bit integer operations are still at least as fast as float operations.

I don't need a 3rd-grade primer on how a computer works, by the way.

c) the last claim that i have seen made, and this was done in one of the entries in the "diary of an x264 developer" is that the various gpu related api's are not well documented. this one just boggles the mind, the cuda sdk is rife with code examples (including a functional h264 encoder and decoder), there is ample documentation, hell there are even graduate level college courses offered in cuda programming, the claim that there is insufficient documentation is a flat out lie. similarly open cl is well documented, microsoft more than documents dx10 and gpu powered encoding is even possible using dx9 (the original avivio was dx9 based and gpeg2 is an ffmpeg based, dx9 powered, mpeg-2 encoder that runs on any dx9 and later gpu, with excellent quality i might add).

thus the claim that any particular api isn't well documented comes across as disingenuous.So in other words, you haven't actually tried it yet and have no idea what you're talking about. I'm not going to go through everything you mentioned, but on a particularly idiotic note, the "functional encoder and decoder" just call included libraries, they don't include any real code.

1) you guys obviously believe in open source, gpl'd code and porting your encoder to a proprietary technology would fly in the face of what you guys stand for. if this is one of the, or the primary reason, i can respect that, and in fact support it, but it would be nice if you guys just said as much. Unrelated. I'd pick CUDA over OpenCL if I had to, because OpenCL is absolutely awful and CUDA is merely almost as bad.

2) perhaps you guys don't know enough about coding for the gpu and this has been the biggest road block. again i can respect that, after all coding for the x86 architecture has been around for 20+ years and most people wouldn't know where to begin porting their code for a new architecture.This has nothing to do with x86. I'd rather program for ARM.

3) one of the biggest technical road blocks that i can see is that it appears that you guys have used function pointers with x264, am i correct? this would effectively prevent you from porting x264 to any gpu prior to fermi as fermi is (and ati's newest dx11 gpu's) are the first gpu's to support function pointers.Again, you have no idea what you're talking about. You don't "port code" by "running it on a GPU". You write new code for the GPU.

ignoring x264 for a second, have you guys ever considered perhaps coding a gpu accelerated, high quality decoder that you could integrate with x264? this would go a long way towards improving performance and would make x264 a true codec, rather than "just" an encoder.There are no GPU decoders and will never be any GPU decoders, because the primary work done by a decoder is completely linear and non-parallelizable (and extremely branchy). All GPU decoders are actually ASICs on the chip, because the GPU itself is far too slow to do it.

Note: GPUs can do some parts of decoding, like iDCT and motion compensation. But this has been doable for quite some time merely using shaders; it's nothing new. And it's hardly even half of the decoding process, CPU-wise.

have you guys ever considered taking the basic cuda encoder that comes with the sdk and simply using everything you guys have learn creating x264 and improving the quality of that encoder? areas where improvement could be made would be to B frames, adding 2 pass encoding and perhaps a bit of tweaking here and there to make it a decent h264 encoder.It isn't open-source. Good luck REing it.

while i'm at it, do you guys have any plans of making use of "sandy bridge's" integrated video transcoding engine or will you simply choose to focus on avx? in all honesty it does seem that the new 256 bit avx simd's will effectively render any discussion of gpu powered encoding mute, but the door is still open for gpu powered decoding.Maybe; the SB so far seems to be vastly better designed than any GPU in terms of encoding, as it's an actual programmable encoder ASIC instead of some retarded hunk of floating-point ALUs slapped onto a giant register file with no cache. I can't really say much more than this, yell at Intel if you want them to tell you more.

lastly, considering bulldozer is slated to have an incredibly beefy 256 bit fpu, do you think it would behoove you make an bulldozer specific x264 variant, by swapping out all the int operations for floating point operations? bulldozer will feature two 128 bit integer units per core but that 256 bit fpu that's backed by two 128 bit mmx units looks to be a monster and considering it appears that x264 makes use of 32 bit math operations, doesn't it seem likely that changing all your ints to floats would allow bulldozer to just tear through an x264 encode?So you'd rather have 8 floats every cycle instead of 48 ints per cycle?

Your entire post suggests that you have very vague academic knowledge of these topics, but have never actually done any of it yourself and so doesn't actually know what most of the words mean.

[popper]

hmm , If i were a x264 developer i might say "patches welcome" to You, after all you did take computer science classes back in the day and you are a Gfx developer apparently.

an ordinary end user that didn't take any computer science classes with professor's might have had an excuse for not taking whats freely available code right now ,and porting it to their preferred Gfx API of choice, you cant say the same, if you want it, make the patches, get on IRC, post up the patch and have the x264/ffmpeg devs review it , done.... it gets put in the next available slot for anyone to compile and test.

[Dark Shikari]

hmm , If i were a x264 developer i might say "patches welcome" to You, after all you did take computer science classes back in the day and you are a Gfx developer apparently.

an ordinary end user that didn't take any computer science classes wit professor's might have have an excuse for not taking whats freely available and porting it to their preferred Gfx API of choice, you cant say the same, if you want it, make the patches, get on IRC, post up the patch and have the x264/ffmpeg devs review it , done....Thanks for the reminder, I forgot to mention that--patches welcome.

We've had over a dozen people claim they were going to work on this, and I have spent easily 100-200 hours of my life working with them and explaining things to them.

Every single one, including an official nVidia developer, has disappeared without completing anything worthwhile.

Of course, I'm still willing to waste more hours if someone else thinks they can do it.

[popper]

http://xtreview.com/addcomment-id-13790-view-Sandy-bridge-cache-system.html
SANDY BRIDGE CACHE SYSTEM

DATE:2010-09-19
"....Caches for each processor core now are called Last level cache (LLC). In Sandy bridge the cache of last level (LLC) works at a constant fixed frequency. In this case the latency with the work of cache are reduced and the second port for the data load is introduced...."

and more SB linked bits there BTW if your interested deadrats.

http://xtreview.com/images/1sandy%20bridge%20processors%20121214.jpg

whats a Uop cache

[deadrats]

24? No, more like 20,000. The problem is that your threads need to be exactly in sync with each other, performing identical operations, or else the loads can't be coalesced. Even multiple simultaneous motion searches (by any of x264's definitions) aren't sufficient to do this.

this is exactly the type of disingenuous claim i was talking about. it was in a post you made where the claim that cuda doesn't start to show any real speed advantage over cpu until the thread count hits 32 was made and as soon as i have the time and go back into the archives and find it i will be more than happy to provide the link where you made the claim.

So in other words, you haven't actually tried it yet and have no idea what you're talking about. I'm not going to go through everything you mentioned, but on a particularly idiotic note, the "functional encoder and decoder" just call included libraries, they don't include any real code.

this is what i am talking about, this is a flat out lie, the full source code is included and i have posted it in it's entirety at the video help forums.

Again, you have no idea what you're talking about. You don't "port code" by "running it on a GPU". You write new code for the GPU.

wow! first i never said to port x264 by running it on a gpu, i said to port it to run on a gpu. you don't write new code, you take your existing code and modify it so it runs on a new platform.

There are no GPU decoders and will never be any GPU decoders, because the primary work done by a decoder is completely linear and non-parallelizable (and extremely branchy). All GPU decoders are actually ASICs on the chip, because the GPU itself is far too slow to do it.

it amazes me that you have managed to code anything that works, let alone that people use. there are no gpu decoders and never will be?!? the work done by a decoder is linear and non-parallelizable?!?

this claim is so ridiculous it's downright laughable!!! i guarantee you that anyone that has ever done any coding is laughing himself senseless right now at the claim that the work done by a decoder is simultaneously "extremely branchy", "completely linear" and "non-parallelizable", you can't be for real right? come one, just admit you're pulling my leg and don't actually believe what you just wrote.

Maybe; the SB so far seems to be vastly better designed than any GPU in terms of encoding, as it's an actual programmable encoder ASIC instead of some retarded hunk of floating-point ALUs slapped onto a giant register file with no cache.

OMFG!!! just so i'm not mistaken, you just said that a gpu is nothing more than a "hunk of floating-point alu's slapped onto a register file with no cache"? that's what you said right? ROTFLMAO!!!

has someone highjacked the dark shikari account and decided to impersonate an x264 developer?

there's just no way you can be the same guy that coded x264, there's no way.

the claim that it's a hunk of "floating-point alu's (arithmetic logic units)" is absurd, an alu, by definition is an integer unit, you just said that a gpu is a hunk of "floating-point integer units...", that's like saying you're a right wing leftist, LOL.

just too funny.

So you'd rather have 8 floats every cycle instead of 48 ints per cycle?

how exactly would you have 48 ints per cycle? as i said bulldozer will feature two 128 bit integer units per core and one 256 bit fpu per core, on a per cycle basis it will be able, in theory, to fetch, execute and retire eight 32 bit floating point math operations or four 64 bit floating point operations (assuming you decided to use double instead of float).

likewise each 128 bit integer unit should, in theory, be able to fetch, execute and retire four 32 bit integer math operations and two 64 bit integer operations, thus both integer units should be able to perform the same number of ints per cycle as that one fpu, only difference is that you will be able to use half the threads with floating point math as compared to integer math.

the only way to get 48 ints per cycle, with two 128 bit integer units is if each int was a 5.3 bit operation and as i'm sure you know, every time you declared int in your code you are instructing the cpu to perform a 32 bit operation.

note the above calculations refer strictly to a cpu with one computational module (i.e. a "core" with 2 integer units and 1 floating point unit), thus all calculations need to be scaled by a factor of 4 when talking about a retail bulldozer chip.

it would appear that you do indeed need a 3rd grade level refresher in computer architecture after all.

Your entire post suggests that you have very vague academic knowledge of these topics, but have never actually done any of it yourself and so doesn't actually know what most of the words mean.

and your responses tell me that you know squat about fundamental computer architectures or how instructions are fetched, executed and retired.

thanks for the entertaining read, i got some good laughs out of your responses.

on a side note you did manage to convince me that perhaps i should never, ever use anything coded by you.

[deadrats]

hmm , If i were a x264 developer i might say "patches welcome" to You, after all you did take computer science classes back in the day and you are a Gfx developer apparently.

i am definitely not a software developer, i am an exterminator for the past 10 years (hence the screen name).

prior to getting into the pest control field i majored in physics and computer science and i am a certified unix system administrator. i was also working on a sse optimized linux distro for a while.

i did start programming before apple 2e's were considered state of the art and know quite a bit about programming in basic, dark basic (was working on a video game for a while), fortran, pascal and c/c++.

an ordinary end user that didn't take any computer science classes with professor's might have had an excuse for not taking whats freely available code right now ,and porting it to their preferred Gfx API of choice, you cant say the same, if you want it, make the patches, get on IRC, post up the patch and have the x264/ffmpeg devs review it , done.... it gets put in the next available slot for anyone to compile and test.

actually i did start toying with the idea of attempting to port x264 to cuda but the x264 developers have used some rather odd coding methods that makes it difficult to modify the code without breaking it. at times it seemed to me like they had coded themselves into a corner and rather than going back and restructuring the code they simply decided to create a kludge, practically a chinese fire drill of a coding mess. honestly, i have remarked numerous times that i am shocked that it works at all, let alone as well as it does (and i will give them that, it certainly does work well).

additionally, though dark shikari didn't confirm it, looking through the code i see coding structures that appear to be function pointers and prior to dx11 gpu's said structures weren't supported, thus it's impossible to port the code to run on older gpu's.

the reality is that with the pending release of sandy bridge and bulldozer (and avx), i don't think that there will be a place for gpu accelerated encoders, 256 bit simds should effectively render gpu encoding obsolete.

of course considering some of dark shikari's claims, i'm sure he/she will claim that intel hasn't adequately documented avx and make up a few silly excuses as to why it wouldn't really benefit x264 anyway.:D

[deadrats]

whats a Uop cache

a "uop" refers to a "micro-operation", specifically a decoded x86 instruction and the cache part refers to the portion of cache where said instruction is stored.

basically what happens is that sandy bridge will fetch an instruction and after it decodes it said instruction will be stored in a special cache:

http://arstechnica.com/business/news/2010/09/intels-next-must-have-upgrade-a-look-at-sandy-bridge.ars

[deadrats]

Of course, I'm still willing to waste more hours if someone else thinks they can do it.

no, i honestly don't think i can do it, primarily because you guys (i don't know how many of you there are) have made some odd programming decisions that effectively make it impossible to port the code without breaking things.

this however is not a shortcoming of gpu's, it's a shortcoming of x264. you do thinks in the code that quite frankly leave me scratching my head, it works, it works quite well, but it certainly isn't the cleanest code i have ever seen, not by a long shot.

[kemuri-_9]

ok, this thread has gotten to the point of flaming and return flaming.

the bottom line is x264 was designed to be a cpu encoder.
you can't just make a quick adjustment and it starts using a gpu.

sure, x264 does have a long history and as such it shows in the code.
if you think x264 is clunky whether in code or use, here's a simple reply to that: Don't look at it. Don't use it. Ignore that it exists

anyways, the point is, there have been people who have tried to get x264 to utilize a gpu,
they have either given up (iirc some guys trying opencl said that opencl in its current state can't even be used)
or ran off into oblivion to never be heard from again.

now then, you may take this as
'why is it left to the random/non-primary x264 developers to do this?
why aren't the primary developers trying it?'
that answer is a rather simple one,
they don't know the gpu languages/apis that are there and don't see a point in learning.
there has been nothing but disappointment in all attempts to date to do so and most of all there hasn't even been any realistic data that indicates using a gpu will help speed up anything at all.
(any data coming from a manufacturer about their own or competing products is likely to have been tampered with by marketing and as such can not be taken seriously)

which is why at the current time, gpus are good for decoding (see some avisynth source plugins and ffmpeg) leaving the cpu to encode.

[Dark Shikari]

i am definitely not a software developer

http://i54.tinypic.com/1z5rioi.jpg

<huge block of trolling>It's so obvious you're trolling at this point that even a blind Ebaumsworld user could see it. I mean seriously, someone who has never developed software telling me that I don't know how to develop software? Come on, that won't work for more than a few posts. I give it a 3/10, at best. Maybe 2/10.

If at any point you decide you actually have some real questions, feel free to ask them in #x264dev on Freenode! I do not hold grudges; if a troll is willing to reform his ways, I'm happy to answer his questions.

Of course, I'm happy to give you advice on trolling too: if you need suggestions on how to improve your technique, I can share some of my own trolling advice. Maybe then you'll be able to troll people for more than 2 posts. Here, I'll give you a few for starters!

1. Claim to be female.
2. Claim to develop a proprietary application which is better (but refuse to show any encoded streams because of an "NDA").
3. Claim we are funded by the evil Intel conspiracy to oppose GPU computing.

Stop by IRC and I can give you some more!

[Audionut]

Troll. End of story.

edit: Oh, DS beat me,

[Usedocne]

3. Claim we are funded by the evil Intel conspiracy to oppose GPU computing.

Hmm... I still have my slight suspicions. :D

[Audionut]

This threads reminds me that I have to visit NASA and tell them how to get the human race to the next galaxy.

[Puncakes]

Oh god, not this idiot again :(
Troll trolling troll: http://forum.videohelp.com/threads/325748-x264-soon-to-be-gpu-accelerated

(btw I still have no idea what I'm talking about.)

[aegisofrime]

I like how he claims to have majored in Physics and Computer Science but ended up in pest control anyway. He probably wasn't very good at whatever he majored in then?

[LoRd_MuldeR]

I think we can stop this pointless debate here ;)