x264 development

[Atak_Snajpera]

or use Avs2YUV to pipe from a 32-bit AviSynth process to x264 (this may be difficult if you aren't familiar with command-line interfaces and shell scripts).
Ripbot264 will automatically use this method on x64 OS. Just my 3 cents if you are to lazy to do it manually :)

[LoRd_MuldeR]

or use Avs2YUV to pipe from a 32-bit AviSynth process to x264 (this may be difficult if you aren't familiar with command-line interfaces and shell scripts).

I even put together a simple tool especially for that purpose:
http://forum.doom9.org/showthread.php?t=144140

[JarrettH]

It's getting super fast :devil:

Approximately 5hrs to encode a 2hr movie using rev1125 with Unrestricted 1-Pass Constant Quality HQ. Only using a Core 2 Duo E6600 (2.40 GHz) here. Swell job! :cool:
I bet you could knock that down to 3hrs with say an E8500 (3.16 GHz). No SSE4 on the E6600 model.

[Sagekilla]

Grab a Core i7 920 (Or better yet, 965) and that would be done in real time or faster ;)

[puffpio]

yes the latest batch of commits has me salivating...
dark shikari: any change of updating your x264 developer blog w/ insights into the latest round of commits that speed things up so? of course, i understand you have your own priorities and time constraints

[Dark Shikari]

yes the latest batch of commits has me salivating...
dark shikari: any change of updating your x264 developer blog w/ insights into the latest round of commits that speed things up so? of course, i understand you have your own priorities and time constraintsI probably should, and while I have a lot of stuff in the realm of midterms and such at the moment, I have no excuse for not updating it for so long.

... on the other hand, why should I write a simple, readable public explanation of how all our competitors could fix their assembly to not suck? We did a little objdump trip the other day and discovered that most of our proprietary competitors' asm code is still utterly atrociously terrible... ;)

[Chengbin]

Maybe you should update your blog when the 5th codec comparison finishes, and you can explain the comparison.

BTW, what revision of x264 was used for the codec comparison? The recent few days brought massive speed up, and considering the test machine was a Q6600, it will give x264 even more of an advantage.

I still can't believe they didn't use an i7 rig.

[Dark Shikari]

Maybe you should update your blog when the 5th codec comparison finishes, and you can explain the comparison.

BTW, what revision of x264 was used for the codec comparison? The recent few days brought massive speed up, and considering the test machine was a Q6600, it will give x264 even more of an advantage.

I still can't believe they didn't use an i7 rig.I'm not expecting too much from the comparison; the speed targets they used were unbelievably low for the test system, which may hurt x264 (as x264 is primarily optimized for somewhat-sane speed levels, not ultra-slow cases). I added my own "subme 11" algorithm for the purpose of wasting more time to bring x264 down to their speed targets (and getting a tiny bit more quality). It was literally written in about 2 minutes though, so if I actually took a few weeks to sit down and find ways to optimize more for the absurdly-slow speed case, I could probably do much better, especially since MSU is measuring nothing but SSIM (so I wouldn't have to worry about visual quality...)

[LoRd_MuldeR]

Hmmm, subme 11 http://www.soadfans.de/forum/images/armageddon/smilies/sabber.gif

[burfadel]

Hmmm subme 10? lol

I realise its probably meant to indicate an 'all frames' case, but if subme 11 features really do slow it down that much a targeted subme 10 may be of benefit?...! (hypothetically)

[Dark Shikari]

Hmmm subme 10? lol

I realise its probably meant to indicate an 'all frames' case, but if subme 11 features really do slow it down that much a targeted subme 10 may be of benefit?...! (hypothetically)If you can't figure out why I called it subme 11 (http://www.youtube.com/watch?v=d54UU-fPIsY) instead of subme 10... ;)

[Sagekilla]

x264: These go to 11 ;)

Any chance of seeing subme 11 in public builds?

[Dark Shikari]

x264: These go to 11 ;)

Any chance of seeing subme 11 in public builds?Only if someone comes up with an algorithm more reasonable than an 11x11 exhaustive qpel-RD search... :devil:

[Chengbin]

so if I actually took a few weeks to sit down and find ways to optimize more for the absurdly-slow speed case, I could probably do much better

Do you plan doing that?

subme11 sounds interesting. How much time does it waste?

[Dark Shikari]

Do you plan doing that?No.subme11 sounds interesting. How much time does it waste?A lot.

[LoRd_MuldeR]

commit 682b54d6175f98dfa14fec4d951f4b3b6e686b95 r1128
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Mar 17 11:01:57 2009 -0700

SSE2 zigzag_interleave
Replace PHADD with FastShuffle (more accurate naming).
This flag represents asm functions that rely on fast SSE2 shuffle units, and thus are only faster on Phenom, Nehalem, and Penryn CPUs.

I wonder, does this apply to "Kentsfield" Core 2 Quads (e.g Q6600) too?

(Note: x264 does display the "SSE2Fast" capability for my Q6600, although it doesn't equal any of the CPU types mentioned in the changelog)

[Dark Shikari]

commit 682b54d6175f98dfa14fec4d951f4b3b6e686b95 r1128
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Mar 17 11:01:57 2009 -0700

SSE2 zigzag_interleave
Replace PHADD with FastShuffle (more accurate naming).
This flag represents asm functions that rely on fast SSE2 shuffle units, and thus are only faster on Phenom, Nehalem, and Penryn CPUs.

I wonder, does this apply to "Kentsfield" Core 2 Quads (e.g Q6600) too?

(Note: x264 does display the "SSE2Fast" capability for my Q6600, although it doesn't equal any of the CPU types mentioned in the changelog)I said "FastShuffle" not "SSE2Fast". The entire point of "FastShuffle" is to explicitly leave out the Conroe.

[LoRd_MuldeR]

I said "FastShuffle" not "SSE2Fast". The entire point of "FastShuffle" is to explicitly leave out the Conroe.

I understand, thanks.

[burfadel]

Penryn is a type of Conroe, but we know what you mean :) (and yes it shows on a Q9400). Glad I got the Q9400 instead of the Q6600 (which many people seem to favour)!...

[Dark Shikari]

Penryn is a type of Conroe, but we know what you mean :) (and yes it shows on a Q9400). Glad I got the Q9400 instead of the Q6600 (which many people seem to favour)!...Conroe is the generic term for 65nm, Penryn for 45nm.

Perhaps you meant "Woodcrest is a type of Penryn" or something of the sort?

[burfadel]

Ah sorry! my mistake :)

Just realised I was probably thinking of Core 2, not Conroe :S

[burfadel]

Although this isn't entirely the right place to ask, but didn't think it was worth its own thread in itself. I have a very valid (in my opinion) question about the inbuilt noise reduction function in x264. The function I am referring to is the --nr xxx (where xxx is the noise reduction amount). I have noticed that is can result in quite a decent bitrate drop in CRF mode (10 percent) without visually impacting the output. Now, the question relates to the actual quality difference and whether it is perceptible to people. I have found --nr 500 to be a good amount.

The problem with working out the quality is its purely, and can only be, personal preference! I realised SSIM and PSNR are completely inaccurate means of determining the output as its a comparison to the avisynth output, comparing it to other denoisers in terms of SSIM etc can't be done as they occur before the avisynth output.

In terms of bitrate mode, I made a comparison a while ago, and for lower bitrate I thought the --nr 500 output one looked better, but this is simply because more bitrate can be given to the encode in general and not spent on noise. I'm pretty sure the noise reduction works slightly differently to avisynth based noise reduction (DS may be able to clarify that statement).

I'd love to know others opinions on this. x264 nr does reduce encoding speed, the higher the nr amount the slower (but smaller the resultant file). It does work on a diminishing return basis, but still returns a bitrate reduction in CRF mode at higher NR nonetheless.

Again, SSIM is a pointless comparison for this, the comparison can be visually only :) I think this was covered at some point but can't find it on search. It was a while ago so new comparisons may be a good thing anyway!

[Ranguvar]

IMHO using a light AviSynth denoiser would be better -- I don't know exactly how x264's denoiser works, but unless a lot of overhead is eliminated by doing the denoising in the encoder, the sheer number of denoisers available for AviSynth and their configurability means that there's always something better for any particular job.

If you have a capable GPU, FFT3DGPU can do light or heavy denoising FAR faster than the competition :)

[LoRd_MuldeR]

I think the advantage of x264's built-in denoiser is that the motion vectors, which are calculated anyway, can be re-used for the denoising...

[burfadel]

The difference is I saw a noticeable bitrate difference without a noticeable picture difference, something that in avisynth you just don't see... Good point about the motion vectors though, if its the case they're not re-used it would be be good if it were changed so they were! The NR in x264 could probably be advantageous if its code were optimised too, I don't believe it has been optimised much since its inclusion. The main point was to see if other people found it useful, even on top of avisynth noise reduction, to the extent that it provides benefits without affecting the picture noticeably. If it did then maybe it could be an area of focus for improvement.

[akupenguin]

The difference is I saw a noticeable bitrate difference without a noticeable picture difference, something that in avisynth you just don't see...
x264's nr operates directly on the inter residual, and quantizes dct coefficients more strongly when it estimates that they're probably noise. So it directly reduces bitrate in a way that you don't get if you just remove noise any old way and then ask the encoder to encode the resulting image at full precision.
The reuse of mvs is not just a speed optimization, it's part of the reason the denoiser works. Same for the use of dct, and in particular the same dct as x264 will choose for the encoding of that block.
The NR in x264 could probably be advantageous if its code were optimised too, I don't believe it has been optimised much since its inclusion.
You mean besides the ssse3 asm?

[burfadel]

Ah ok! I wasn' taking in to account things that indirectly affect its speed! I knew it worked different to avisynth based noise reduction, and it does seem advantageous even if there is Avisynth NR! It doesn't seem a well used function though, so I was wondering how others found it. As I said before, its something you can really take the SSIM of because you are changing the output, avisynth based NR changes the input the x264. You can have a filter that completely changes the input, but still have an almost perfect SSIM because you are only comparing the output of avisynth (input of x264) to the output of x264 :) hope that makes sense!

[LoRd_MuldeR]

I wonder how NR and Psy-RDO work together. Don't they work into opposite directions somehow?

[Dark Shikari]

I wonder how NR and Psy-RDO work together. Don't they work into opposite directions somehow?NR isn't used in mode decision where psy-RD is used, so they have no real effect on each other.

[PelPix]

Working well so far on my processor

[burfadel]

Thats kind of what I'm curious about, the subjective results people have about using it. I think in the past people were simple comparing the SSIM, which you can't do with using this NR. Its not a replacement for avisynth NR, it can be used in conjunction with. Actually using lighter NR in avisynth and strong NR in x264 seems to be more beneficial in many cases bitrate and picture quality wise than being heavy handed with avisynth NR. I'd love to see others perspectives of this. Again it is a visual test, SSIM is invalid because yoy are removing noise on the x264 output which ultimately lowers its 'sameness' for SSIM calculation. In certain circumstances such as bitrate mode, I can imagine it could actuallly raise SSIM as you are removing noise, reducing the encoding complexity and therefore the bitrate can be spent on improving the output of actual picture detail! NR scales up to 1000, an NR of 2 or 3 would do almost nothing! I use 500.

I take it this works on only Luma, is it possible and would it be beneficial to also work on chroma?...

[Selur]

little question: Does trellis (in x264) require cabac?

[J_Darnley]

little question: Does trellis (in x264) require cabac?
x264 --help
-t, --trellis <integer> Trellis RD quantization. Requires CABAC.

[Selur]

okay, thanks. this is more a practical problem (as in nobody wrote it) isn't? it iirc trellis+cavlc is possible but a pain to implement, right?

[Dark Shikari]

Thats kind of what I'm curious about, the subjective results people have about using it. I think in the past people were simple comparing the SSIM, which you can't do with using this NR. Its not a replacement for avisynth NR, it can be used in conjunction with. Actually using lighter NR in avisynth and strong NR in x264 seems to be more beneficial in many cases bitrate and picture quality wise than being heavy handed with avisynth NR. I'd love to see others perspectives of this. Again it is a visual test, SSIM is invalid because yoy are removing noise on the x264 output which ultimately lowers its 'sameness' for SSIM calculation. In certain circumstances such as bitrate mode, I can imagine it could actuallly raise SSIM as you are removing noise, reducing the encoding complexity and therefore the bitrate can be spent on improving the output of actual picture detail! NR scales up to 1000, an NR of 2 or 3 would do almost nothing! I use 500.I have seen very significant artifacts in anime encoding with nr=1000. IMO high values can be risky on some types of content.I take it this works on only Luma, is it possible and would it be beneficial to also work on chroma?...Probably not, and yes, it only works on luma.okay, thanks. this is more a practical problem (as in nobody wrote it) isn't? it iirc trellis+cavlc is possible but a pain to implement, right?Yes, it's possible, and there are some papers implementing it. It would likely be harder than CABAC. Patches implementing it are quite welcome.

When I got QNS working a while back it showed a significant improvement on CAVLC (but was exponentially too slow to be practical), so I suspect CAVLC trellis would help similarly.

[burfadel]

I wouldn't select 1000 either! I have been using 500, I was curious about the consensus of other people. Of course, they could select 273 if they so wish, even 200 has benefit bitrate wise. I guess its about balance, afterall an aq of 1 is more benefiical than 0, but an aq of 2 may be the exact opposite. I think its the same with NR, is just about finding that 'AQ 1' spot (comparison to AQ was purely for example, I know its not related - but the reasoning is!).

[jult]

The order in mewiki is the same as in x264 --longhelp. It also looks pretty logical to meNot to anyone I know. Alphabetical would be smarter. Or at least an option to change the order that way.

The writers assume that people who use the command-line interface have a decent set of tools to view and search such listings: grep and less for instance. If you don't have them, just direct the output to a file and open it in some text editor.Resting my case: Takes a lot of extra unnecessary time. Isn't as easy or logical as it can be.

I don't think any amount of documentation on the command-line parameters would help with this issue.With what issue? Of course it will, and I have already given the reasons and pointers towards that. By the way, I've been writing about it for years. It seems to become just as bad as linux dev in general has become: Messy. Badly documented. Badly linked online. Assuming users like to do things as if they still live in 1980; Typing and typing only. We copy paste and (right-)click these days.
I have no problems whatsoever using commandline interfaces, but having to mess in CLI more than is needed is tedious, overly complex and time-consuming. Denying that is like saying we don't like a good GUI.

[nm]

Not to anyone I know. Alphabetical would be smarter. Or at least an option to change the order that way.
So you want an alphabetical reference for options that you already know by name? It doesn't take that much work to contribute an ordered list of links to the Wiki page. Still, it would be easier to just press CTRL-F and type in the phrase you want to find.

The writers assume that people who use the command-line interface have a decent set of tools to view and search such listings: grep and less for instance. If you don't have them, just direct the output to a file and open it in some text editor.Resting my case: Takes a lot of extra unnecessary time. Isn't as easy or logical as it can be.
That is the standard way -h/--help works in command-line interfaces. I don't see any reason to change that behaviour.

don't think any amount of documentation on the command-line parameters would help with this issue.With what issue? Of course it will, and I have already given the reasons and pointers towards that.
You complained about failing to use 32-bit AviSynth with 64-bit x264. I don't think a command-line reference is the right place to list external dependencies (such as 64-bit x264 builds requiring 64-bit AviSynth). So you would never find an answer from --longhelp or any alphabetical option reference. A traditional place for such information would be a README or INSTALL file in the x264 source code tree, but since most people wouldn't notice them, a Wiki-based user's manual would be in order.

By the way, I've been writing about it for years. It seems to become just as bad as linux dev in general has become: Messy. Badly documented. Badly linked online.
I think the main reason for that is the lack of a high-profile official site (x264.org or x264.com). The devs were working on that (http://forum.doom9.org/showthread.php?t=139634), but apparently that project hasn't moved forward lately.

Assuming users like to do things as if they still live in 1980; Typing and typing only. We copy paste and (right-)click these days.
I like typing and copy&paste works much better with command lines than GUIs. x264 is a command-line tool. It does not have an official GUI and it may never get one.

I have no problems whatsoever using commandline interfaces, but having to mess in CLI more than is needed is tedious, overly complex and time-consuming. Denying that is like saying we don't like a good GUI.
How about using one of the numerous nice GUIs people have created and contribute documentation and tooltip texts to it if they are missing. Once the frontend gets better, other people start using it and it will be maintained actively along with x264.

[burfadel]

I was wondering whether gradfun2db could essentially be added to x264? I know it can be done through avisynth but I was thinking that there may be some way to incorporate it in to x264 and use some results of the x264 functions?

I know people say its pointless doing it before encoding, as the encoding itself will produce banding, but I think the exact opposite. The last thing you want to do is encode chroma banding and small artifacts and waste bitrate doing it. Remember, if there is banding and other colour abberrations, x264 has to spend bitrate actually encoding the banding, and it does a good job of it with VAQ! My idea would be to have a chroma smoother that has edge detection on the input source (negating the need for gradfun2db), and also maybe include the chroma optimiser that was trialled, if it can be tweaked and shown to be beneficial (they may work together). The chroma smoother with edge detection in my opinion, should be enabled by default as the most common sources for video is already compressed video, or raw video derived from compressed video (I know the decoding part is mostly done through avisynth). Even if you have a chroma smoother, it wouldn't hurt having the proposed method as well as edge detection will negate chroma blurring where its not needed (such as transition of colours and 'significant' chroma variation), and yet provide a good effect. This would be more beneficial to visual quality than bitrate (I presume). Hope that makes sense, the effect would be at most subtle and yet visually pleasing :)

[gizzin]

I just wanted to say thanks for this amazing encoder.

[Sagekilla]

@burfadel: Nice idea but that goes beyond x264's job, since that's pre-processing. If you want to fix your source so it's not screwed up, do it in avisynth. x264's job isn't crap in -> good out, it's to compress your input and keep it as close to the input as possible.

[lexor]

@burfadel: Nice idea but that goes beyond x264's job, since that's pre-processing. If you want to fix your source so it's not screwed up, do it in avisynth. x264's job isn't crap in -> good out, it's to compress your input and keep it as close to the input as possible.

I think he was mostly aiming at preventing x264 from creating banding when bitrate starved (or whatever other reason it has). Though I don't think gradfun2db's algorithm is applicable in that capacity either.

[Sagekilla]

x264 already has something to prevent banding: AQ.

Note: If you're bit starved you should either downscale or increase your bitrate. Either way, there's very little that could be done to stop artifacting once you're bitrate starved.

[burfadel]

x264 already has something to prevent banding: AQ.

Note: If you're bit starved you should either downscale or increase your bitrate. Either way, there's very little that could be done to stop artifacting once you're bitrate starved.

AQ works very well too! The main thing I was pointing out is any distortions (such as banding) on the input source is seen by x264 as detail, and hence the very good AQ encodes it! My point was to just provide an option (maybe as default) for subtle chroma smoothing and optimisation. This was already partly covered with the chroma optimiser tested earlier as an avisynth plugin (from XVID?) but extends on the concept. The main reason why I suggested it is most people don't use a deband filter etc before x264.

Like the comment above, 'x264 already has something to prevent banding: AQ', that reduces banding due to x264, but there's the banding in the original source that x264 actually spend bitrate to encode, as its detail! <--AQ does NOT remove source banding, it would actually help to encode it! There would also be other instances where AQ etc would use bitrate on chroma information (non visible in general unlike banding) where it wouldn't otherwise be needed.

[Dark Shikari]

AQ works very well too! The main thing I was pointing out is any distortions (such as banding) on the input source is seen by x264 as detail, and hence the very good AQ encodes it! My point was to just provide an option (maybe as default) for subtle chroma smoothing and optimisation. This was already partly covered with the chroma optimiser tested earlier as an avisynth plugin (from XVID?) but extends on the concept. The main reason why I suggested it is most people don't use a deband filter etc before x264.

Like the comment above, 'x264 already has something to prevent banding: AQ', that reduces banding due to x264, but there's the banding in the original source that x264 actually spend bitrate to encode, as its detail! <--AQ does NOT remove source banding, it would actually help to encode it! There would also be other instances where AQ etc would use bitrate on chroma information (non visible in general unlike banding) where it wouldn't otherwise be needed."Chroma smoothing" and "banding" have nothing to do with each other.

Banding is primarily a luma problem, not chroma. And smoothing creates banding, you need to dither to get rid of banding.

[Sagekilla]

Plus, what you're describing is nothing more than preprocessing. As I said before, x264's job is not preprocessing. That's for avisynth.

[burfadel]

Ah ok! that makes sense about the banding, I apologise for my ignorance! (hey, I saved someone else looking stupid) :D

[b66pak]

hi,

it is possible to output the x264 final statistic to a file? (.log)

thanks for this great tool...
_

[LoRd_MuldeR]

hi,

it is possible to output the x264 final statistic to a file? (.log)

thanks for this great tool...
_

x264.exe options > logfile.txt

And of course you can write your own front-end, capture STDOUT/STDERR and extract whatever information you need...

[kemuri-_9]

x264.exe options > logfile.txt

And of course you can write your own front-end, capture STDOUT/STDERR and extract whatever information you need...

as x264 outputs to stderr and not stdout the above would be 2> and not >

also there's a patch flying around from komisar that adds in a log file capability iirc.
though that would require building your own version with it....