Critique my x264 options

[Dark Shikari]

I use the following to encode my videos. I don't care about speed, though --me esa doesn't seem to give enough of a quality boost (its roughly 0.0003 more SSIM or so) to be worthwhile considering the fact that I can't use --threads with it.

script.avs is the avisynth file to convert the input file (FRAPS video) to YV12.

x264 --trellis 2 --me umh --bime --merange 64 --partitions all --8x8dct --pass 1 --bitrate 4000 --subme 7 --bframes 4 --b-pyramid --b-bias 2 --ref 3 --b-rdo --mixed-refs --direct auto --progress --threads 4 -o output.mkv script.avs

x264 --trellis 2 --me umh --bime --merange 64 --partitions all --8x8dct --pass 2 --bitrate 4000 --subme 7 --bframes 4 --b-pyramid --b-bias 2 --ref 3 --b-rdo --mixed-refs --direct auto --progress --threads 4 -o output.mkv script.avs

What options can I add to improve quality while keeping the same bitrate? Also note that I obviously don't use 4Mbit on all my videos; its just an example.

A sample video encoded with my settings can be found here (http://go-dl.eve-files.com/media/corp/shikari/h264_test.mkv) to give you an idea of what I'm encoding.

Another quick question: what kind of system requirements are needed to decode this kind of video at 1680x1050? My laptop with a Core 2 Duo 2Ghz does it fine but my desktop (2.88Ghz P4) stutters horribly. My desktop uses FFDShow, and my laptop uses VLC. Are there any particular options I'm using that are causing a large boost in decoder CPU requirement, or is it just the large video resolution?

[Terranigma]

What's the point in using 4 b frames?

[Dark Shikari]

What's the point in using 4 b frames?
I don't know, I'm not much of an x264 expert so I don't know what the best settings are and why. I haven't really managed to find any FAQs that explain what the best settings are to use and why. Like, I know what a B-frame is, but not why one would want to use a specific number of them.

[foxyshadis]

What's the point in using 4 b frames?

Same reason you might use more than 3 references: Trade more size/quality efficiency for some speed. x264's defaults handle b-frames much better than xvid's defaults in my experience, so using tons of them doesn't hurt.

More than 3 or 4 is very rarely useful in live-action, but some cartoons/anime will use tons of them in a row. Extra b-frames can make decoding harder if they're the nearly same size as p-frames though.

[CruNcher]

hmm it ends in a more complex stream (decode) and thus can be problematic for some restricted enviroments like mobile devices my last test showed that 1 Bframe more can endup in -5-10 fps @ decoding useing 720p, testing on 800 Mhz x86 (MMX).

[Blue_MiSfit]

That's a high resolution to decode, so you might want to step back from the B-Frames a bit. I'm not sure if that command line has CABAC enabled or not (I'm not a x264 CLI guru, <3 MeGUI..), but that's one of the biggest speed killers. It meant the difference between a slideshow and perfect decoding speed on my machine.

Also, you might want to try out CoreAVC for decoding, if you dont mind dropping a bit of cash. It's still the fastest decoder IIRC, though libavcodec (ffdshow/vlc) is catching up :)

~MiSfit

[Dark Shikari]

Is there a good way to test decoder speed like one can test encoder speed? I.e. something that will give you a number in FPS, so I can compare various numbers of b-frames and so forth for improved decoder speed.

In addition, which options in particular reduce decoder speed: I know that --bframes and CABAC do, but of the other high-quality items, what does? Does increasing --ref do so? And at what point do options like --ref cap out... 3 seems to work, as does 5; does it just keep increasing quality, or does it eventually get to the point where there is little point in increasing it further?

Also, what about in-loop deblocking? I see a lot of people using deblocking options; should I add it to my command line or not, and why?

[Mangix]

Is there a good way to test decoder speed like one can test encoder speed? I.e. something that will give you a number in FPS, so I can compare various numbers of b-frames and so forth for improved decoder speed.


you might want to take a look at timeCodec (http://haali.cs.msu.ru/mkv/timeCodec.exe)

[Blue_MiSfit]

And at what point do options like --ref cap out... 3 seems to work, as does 5; does it just keep increasing quality, or does it eventually get to the point where there is little point in increasing it further?


The answer to this is pretty subjective. However, I think the general consensus is that beyond 4 refs or so, there are diminishing returns. You can squeeze out a bit more quality with more refs, but at the cost of terribly slow encoding speed. I would assume that more refs also slows down decoding, not sure though.

~MiSfit

[Dark Shikari]

The answer to this is pretty subjective. However, I think the general consensus is that beyond 4 refs or so, there are diminishing returns. You can squeeze out a bit more quality with more refs, but at the cost of terribly slow encoding speed. I would assume that more refs also slows down decoding, not sure though.

~MiSfit

Using TimeCodec lowering B-frames to 2 from 4 sped it up by a very small amount; I'm guessing the encoder didn't actually use 4 B-frames most of the time. Bringing it from 2 to 0 boosted FPS by about 4%, but lowered quality a bit too.

--ref 3 and --ref 5 gave exactly the same FPS decoding.

Lowering the bitrate by 50% also only mildly improved decoding speed; only by about 4%.

[Dark Shikari]

I tried turning off -8 and it actually slowed down decoding. --no-cabac simply garbled the whole thing for some reason; the video stream was so screwed up that it wasn't only a mess, but it crashed FFDshow a few seconds in (though CoreAVC still worked).

So now that speeding things up is pretty much out, what other options can I change to improve quality? And as I said before, what about deblocking?

[burfadel]

Disabling CABAC for deconding speed kinda negates having such a high bitrate. CABAC is around 10 percent more efficient, so if your computer doesn't have any problems decoding it then leave it on (its on by default).

Another thing that should NOT BE TURNED OFF is the loop filter. This deblocking method is done at the encode stage and DOES NOT effect decode speed. It improves the quality of the encode so turning it off it pointless and not recommended!

The settings you chose doesn't give you a constant bitrate of 4000, it gives you an average bitrate of 4000. The first pass is used to determine where its best to use the bits, and the second pass uses this information to encode more efficiently. Scenes without much movement will then be given a lower bitrate and high motion scenes a higher bitrate, which gives you a higher quality encode. If the final size of the file doesn't matter in the sense of fitting it exactly to a CD/DVD then a constant quantiser mode would be better. You can try by experiment, a quantiser (lower is better) that equates to roughly your 4000 bitrate. A constant quantiser mode is advantageous if file size isn't an issue as its still constant quality (which your files are, but your files just conform to an average bitrate of 4000), but you don't need the first pass to work out where to put the bits.

Therefore, unless you have to have a specific file size, a constant quantiser is better. A lower quantiser gives a better picture, but its calculation is different to other codecs. Maybe try 21 (one pass encode remember) and compare it from there. You'll soon have a good idea of what a good quantiser is. Also remember that if the resolution changes you can use the same quantiser for the same quality! you don't have to try and work out what bit rate to use with what resolution.

If you really want to use an average bitrate (say 4000), then the first pass settings is kinda excessive... as its only used for data collection. The settings don't matter so much as the picture informaiton, and where to have the most bits stay relatively the same regardless of the compression settings used. Its a good idea to use the same number of b frames and reference frames, and motion range, but the ME and subme can be set lower.

b-frames using the settings you chose are adaptive. That means, only the number of b frames required is used up to the maximum which you selected. You could have the max b-frames set to 16! that would rarely be used but it lets the encoder decide. You've set a slightly positive bias for b-frames using the --b-bias option. A setting of 2 isn't that much, as 0 is for the encoder to completely decide, -100 is for no b-frames and 100 is for always use b-frmaes. You could probably remove the --b-bias setting altogether.

A very important setting you're missing is --weight-b (weighted prediction for b-frames)

Since it seems quality is your optimal goal you should use the --no-fast-pskip option. This may improve quality where there is a smooth gradient in colour (such as sky). This doesn't affect deconding speed, but may make the encoding speed slightly slower. Its a setting that was designed to give a nice encoding performance boost at the expense of a slightly lower picture quality, and is now the default. The performance return diminishes with higher settings, so you wouldn't notice too much difference using UMH motion seach, a subme of 7 and a merange of 64!

The optimal number of reference frames for speed vs quality is said to be 5 for normal encodes (that is, like movies encodes etc where the footage is real). For simple animation stuff (such as family guy, the simpsons etc), a reference frame setting of 16 is optimal for quality or file size and can significantly improve quality.reduce file size. Since speed isn't an issue for encoding 7 may be beneficial.

Adaptive quantiser is also a good thing! Its still experimental (apparently) but its an essential option for many people as you may find if you search. It improves quality in dark areas of the film. The option is --aq-strength. The options are between 0 (off=default) and 1.1. You would probably find 0.5 a good option for a high quality encode.

Now for deconding speed...
If you're using ffdshow make sure you're using the latest build. Its significantly faster than the 'official build' which is quite old. Its downloadable from:
http://ffdshow.info

Basically choose the one with the highest rev number (such as rev1092).

In the settings for ffdshow, under 'output' make sure 'use overlay mixer' and 'hw deinterlacing' is selected. Post processing is not required for h.264 and its a speed killer.

Pentium 4's, sorry to say, are not known for lightning fast performance. From what i've seen in real life scenerios in terms of encoding etc., it seems Intel understated the speed of the Core 2 Duo - possibly to make the P4 seem not so bad! They say the Core 2 Duo is 40 percent faster for the same processor speed, the actual figure seems to be much, much higher. A Core 2 Duo E6600 runs at only 2.4Ghz as default, but I can assure you its significantly faster than 40 percent faster than a P4 2.4Ghz.

So yes, your laptop can play it because its actually faster... if you put the latest ffdshow on your desktop, make sure there's enough free RAM, run the latest graphics drivers for video speed (they do effect the speed of the decode), and the latest sound drivers as that reduces CPU overhead you should be able to play it fine. If you have an antivirus make sure its not set to scan every file, this is a speed killer and it stops the av trying to scan the file whilst you're playing it. If there's still a problem, check the HD light, if its constantly on then there's a computer settings problem or not enough memory. The HDD should not be going flat out even if the CPU is!

have fun ;)

[Dark Shikari]

Disabling CABAC for deconding speed kinda negates having such a high bitrate. CABAC is around 10 percent more efficient, so if your computer doesn't have any problems decoding it then leave it on (its on by default).
I figured that.

Another thing that should NOT BE TURNED OFF is the loop filter. This deblocking method is done at the encode stage and DOES NOT effect decode speed. It improves the quality of the encode so turning it off it pointless and not recommended!
How do I turn it on? Or is it on by default?

The settings you chose doesn't give you a constant bitrate of 4000, it gives you an average bitrate of 4000. The first pass is used to determine where its best to use the bits, and the second pass uses this information to encode more efficiently. Scenes without much movement will then be given a lower bitrate and high motion scenes a higher bitrate, which gives you a higher quality encode. If the final size of the file doesn't matter in the sense of fitting it exactly to a CD/DVD then a constant quantiser mode would be better. You can try by experiment, a quantiser (lower is better) that equates to roughly your 4000 bitrate. A constant quantiser mode is advantageous if file size isn't an issue as its still constant quality (which your files are, but your files just conform to an average bitrate of 4000), but you don't need the first pass to work out where to put the bits.

Therefore, unless you have to have a specific file size, a constant quantiser is better. A lower quantiser gives a better picture, but its calculation is different to other codecs. Maybe try 21 (one pass encode remember) and compare it from there. You'll soon have a good idea of what a good quantiser is. Also remember that if the resolution changes you can use the same quantiser for the same quality! you don't have to try and work out what bit rate to use with what resolution.

I know about quantizers and the like as I've used Xvid for ages. The reason I want to give a specific bitrate is because I want to be able to decide "I want a file roughly this size" and make it that size. I don't want to have to fiddle with quality to get it near that size.

If you really want to use an average bitrate (say 4000), then the first pass settings is kinda excessive... as its only used for data collection. The settings don't matter so much as the picture informaiton, and where to have the most bits stay relatively the same regardless of the compression settings used. Its a good idea to use the same number of b frames and reference frames, and motion range, but the ME and subme can be set lower.
I figured, thanks.

b-frames using the settings you chose are adaptive. That means, only the number of b frames required is used up to the maximum which you selected. You could have the max b-frames set to 16! that would rarely be used but it lets the encoder decide. You've set a slightly positive bias for b-frames using the --b-bias option. A setting of 2 isn't that much, as 0 is for the encoder to completely decide, -100 is for no b-frames and 100 is for always use b-frmaes. You could probably remove the --b-bias setting altogether.
Ah k, that makes sense.

A very important setting you're missing is --weight-b (weighted prediction for b-frames)

Will add that now, thanks.

Since it seems quality is your optimal goal you should use the --no-fast-pskip option. This may improve quality where there is a smooth gradient in colour (such as sky).


The optimal number of reference frames for speed vs quality is said to be 5 for normal encodes (that is, like movies encodes etc where the footage is real). For simple animation stuff (such as family guy, the simpsons etc), a reference frame setting of 16 is optimal for quality or file size and can significantly improve quality.reduce file size. Since speed isn't an issue for encoding 7 may be beneficial.

Adaptive quantiser is also a good thing! Its still experimental (apparently) but its an essential option for many people as you may find if you search. It improves quality in dark areas of the film. The option is --aq-strength. The options are between 0 (off=default) and 1.1. You would probably find 0.5 a good option for a high quality encode.
Thanks for the info.

Pentium 4's, sorry to say, are not known for lightning fast performance. From what i've seen in real life scenerios in terms of encoding etc., it seems Intel understated the speed of the Core 2 Duo - possibly to make the P4 seem not so bad! They say the Core 2 Duo is 40 percent faster for the same processor speed, the actual figure seems to be much, much higher. A Core 2 Duo E6600 runs at only 2.4Ghz as default, but I can assure you its significantly faster than 40 percent faster than a P4 2.4Ghz.

So yes, your laptop can play it because its actually faster... if you put the latest ffdshow on your desktop, make sure there's enough free RAM, run the latest graphics drivers for video speed (they do effect the speed of the decode), and the latest sound drivers as that reduces CPU overhead you should be able to play it fine. If you have an antivirus make sure its not set to scan every file, this is a speed killer and it stops the av trying to scan the file whilst you're playing it. If there's still a problem, check the HD light, if its constantly on then there's a computer settings problem or not enough memory. The HDD should not be going flat out even if the CPU is!

have fun ;)
Well obviously the Core 2 Duo is faster, at nearly everything. It vastly outpowers my desktop; the only application I've found it equals it at is Prime95, which it can run one on each core anyways, each at the same speed as the 2.8Ghz P4.

Thanks for the info on the command line options, will add them now and compare.

[Dark Shikari]

Hmm, your suggestions actually lowered the SSIM, though that might be the adaptive quantization at work.

[burfadel]

The loop filter is on by default :) I hope you find the three extra options beneficial!

[Manao]

The loop filter is used both at encoding and decoding. Disabling it will speed up the decoding by 30% roughly, but you'll lose some quality.

burfadel : you're giving lot of advice, but lots of them are ill advised.

* Constant quantizer ? No, prefer either CRF ( without target size ) or 2 passes ( if you need a specific size )
* first pass with same number of reference ? No, no need for that. Only the bframes settings matter. Furthermore, the frame type are chosen during the first pass, so its quality mustn't be too low
* 16 references will never significantly improve quality nor size. Over 5 references, it's a useless overkill.
* adaptive quantization may or may not help. Beware though that it isn't enabled in most of the builds out there.
* post processing in FFDShow : adding some noise and a bit of sharpening always helps. Only the deblocking is useless since it's done during the actual decoding.

[burfadel]

Quite true! I was expecting someone to correct some things in there, I knew some of it wasn't correct but the main points I put in were :) I use crf myself! :D I just made a mistake when I was writing that part out!

The quality in the first past can't be too low, thats correct, I was pointing out that having a too high of a setting on the first past gives little improvement on the final encode and may not be speed effective! But you're right, having it too low is not beneficial either. I didn't know about the reference frame thing with that, but thinking about it now it kinda makes sense...

I agree with the 5 reference frames, but the settings he used are a bit overkill anyway (but would give a slightly better encode!) so 7 reference frames seems to fit the settings he already chose. The only time normally that 16 reference frames is beneficial is with encoding animation like 'Family Guy' etc, where the picture is very stable and not complex. I found that for a crf encode it results in a noticeably smaller file without loss of quality, or for a 2 pass encode a higher quality file for the same size.

It doesn't seem to effect encoder speed that much either, probably because animation is not a complex picture. But yeah, normally 5 reference frames! Only for animation is more beneficial.

As I said adaptive quantisation is still experimental, and isn't included normally. Some find that it does help and I like to use it! I should have added to that line that its only available in some builds people have done and is not part of the source code yet. If you look in the development thread for x264 you can find some up to date builds with the adaptive quantiser patch done by Cef.

Looking in to it, you're right deblocking is done at both the encoder side and decoder side. Either way its highly recommended that its not disabled especially at the encoder side. At the decoder side, you can set it so it skips deblocking if there are frame skips. That may be a beneficial setting on your P4. Its enabled by default on the latest ffdshow builds.

The loop filter is used both at encoding and decoding. Disabling it will speed up the decoding by 30% roughly, but you'll lose some quality.

burfadel : you're giving lot of advice, but lots of them are ill advised.

* Constant quantizer ? No, prefer either CRF ( without target size ) or 2 passes ( if you need a specific size )
* first pass with same number of reference ? No, no need for that. Only the bframes settings matter. Furthermore, the frame type are chosen during the first pass, so its quality mustn't be too low
* 16 references will never significantly improve quality nor size. Over 5 references, it's a useless overkill.
* adaptive quantization may or may not help. Beware though that it isn't enabled in most of the builds out there.
* post processing in FFDShow : adding some noise and a bit of sharpening always helps. Only the deblocking is useless since it's done during the actual decoding.

[Blue_MiSfit]

Another suggestion: reduce the strength of the inloop deblocker. I find that it smooths far too much for my tastes at 0,0. I usually run -2,0, or -2,1. Of course, reducing it too much means that you risk having blocks.

I always decode with CoreAVC, but have ffdshow process the YV12 output of CoreAVC. This lets me Lanczos4 Resize 2x (usually just under my CRT's 1280x960 resolution), with some luma sharpening. I also add noise (luma only), usually ~20 or so. This covers up any remaining blocks, hides any posterized gradients, and cheats my brain into seeing more detail than there actually is :D

Of course, since you're encoding for your LCDs native resolution, resizing will not be necessary. I suggest adding a bit of noise though..

~MiSfit

[akupenguin]

More than 3 or 4 is very rarely useful in live-action, but some cartoons/anime will use tons of them in a row. Extra b-frames can make decoding harder if they're the nearly same size as p-frames though.

x264 will only use tons of B-frames in a row in very low motion scenes. Those are the easiest to decode anyway, so adding B-frames there won't affect your ability to smoothly decode the movie as a whole.

[check]

My testing of reference frame efficiency showed a reasonably constant exponential dropoff in terms of size saved (from a crf encode) when increasing the number of refs. There's no real 'best' in such a situation, just pick the largest number you can bear :)

[Manao]

CRF != constant PSNR. You can't compare two CRF encoding only with their size.

[*.mp4 guy]

...and constant PSNR does not equate to constant SSIM, or VQM, or percieved quality. Good codecs do many things that reduce PSNR, and make it Less constant, but people percieve these as positive changes, your going to have to give a better reason then "psnr may be a little different" to invalidate such a simple observation.

[Manao]

Ok, a better reason ? Let's say I add an option --encode-as-black which, instead of encoding the video, encode a black video. The size will drop. So quality must be taken into account.

Now, step 2 : I'm ready to bet that nobody can tell the difference in quality between -r 10 and -r 16 with their own eyes. So, the size dropped by 0.5%, you can't notice the quality difference, so it's necessarily good ? No. Imagine repeating the process with lots of different options. Adding them together would then produce a noticable quality loss.

So, you need to be able to judge an option's quality, but you can't since your eyes won't tell the difference. You have to rely on an objective metric.

Finally, let's remember that x264 is optimized for PSNR. So, when testing an option :
- use constant quantizer, nothing else ( it happens that CRF is oblivous to all your settings - except bframes - concerning quantizer decision. Still it shouldn't be used )
- consider both size and PSNR

[*.mp4 guy]

I wasn't arguing that quality shouldn't be taken into account, that would be dumb, I was saying that a small drop in PSNR would not neccesarily mean that the results are actually worse, that was my only point. Obviously if you only optimize for one variable in a multi-variable problem you are going about things the wrong way, but PSNR is not a reliable way to measure visual quality, people often disagree with PSNR by very large margins, so making a decision based on a very small change in PSNR would be BAD, because PSNR is unreliable, unless the difference is very large.

[Manao]

PSNR is unreliableProperly used, it is reliable. PSNR can't judge a rate control, for example, but it can definitely tell the difference between -r 5 from -r 10.

[*.mp4 guy]

It can tell a difference, but it won't be the same difference that a person would see, any metric can tell you that a!=b, but a metric that can tell you how disturbing the difference between a and b would be to a human observer is very dificult to make.

[Sagittaire]

I wasn't arguing that quality shouldn't be taken into account, that would be dumb, I was saying that a small drop in PSNR would not neccesarily mean that the results are actually worse, that was my only point. Obviously if you only optimize for one variable in a multi-variable problem you are going about things the wrong way, but PSNR is not a reliable way to measure visual quality, people often disagree with PSNR by very large margins, so making a decision based on a very small change in PSNR would be BAD, because PSNR is unreliable, unless the difference is very large.

Well like say Manao, x264 is optimized for PSNR. And in practice x264 don't work so badly ... ???
if you don't believe PSNR, don't use RDO for example because it's typically a PSNR optimisation.

[*.mp4 guy]

This looks like it will turn into a full fledged debate over the validity of metrics, which I'm not interested in, let me reiterate one more time, PSNR is not a precise enough measure of quality to judge something like this. PSNR is a very easy way to guess the base efficiency of an encoder, so it makes sense that the X264 devs used it, it is a very commonly used metric in video encoding. However that isn't to say X264 wouldn't be better with some LAME quality psychovisual models.

[Sagittaire]

However that isn't to say X264 wouldn't be better with some LAME quality psychovisual models.

Video codec don't use advanced psycho model like audio codec simply because the eyes don't work like ears. PSNR work well in most case simply because PSNR model itself is a good psychovisual model. PSNR don't work for ears but work for eyes. It's a not perfect HVS model, and by far, but all the video codec in the world use PSNR.

[Manao]

There I want to distance myself from Sagittaire's opinion.

PSNR is fine to judge the raw efficiency of the codec. But it can't be used for anything else. PSNR is useless when it comes to judging a psychovisual model - and, no, getting the highest PSNR possible isn't a valid psychovisual model either.

However, when considering options that have no influence whatsoever on psychovisual matters - that is, in x264, every option except the rate control - PSNR can safely be used.

You all agree here that m6 > m3> m0, that -A all > -A none, that -r 5 > -r 1, that -8 is better, that -t 1/2 helps, that --me umh > --me dia. Well, guess what, PSNR says exactly the same thing.

And PSNR works because none of these options tries to consider the video from a psychovisual point of view.

Rate control, adaptive quantization fall into a different domain. Those are mostly governed by psychovisual considerations. That's why they don't make unanimity...

Back to the original case, check did consider only size, and both *.mp4 guy and me emphasized that quality had to be considered to.

[*.mp4 guy]

Well, thats where our opinions diverge I guess, I've seen way to many PSNR optimized "non-psychovisual" encoding options give horrible percieved quality in certain conditions, while still raising the PSNR/bits ratio to really beleive that psychovisuals don't play a part in every piece of a good encoder.

[Sagittaire]

Well, thats where our opinions diverge I guess, I've seen way to many PSNR optimized "non-psychovisual" encoding options give horrible percieved quality in certain conditions, while still raising the PSNR/bits ratio to really beleive that psychovisuals don't play a part in every piece of a good encoder.

Well if you know better way then make patch for x264. IMO if all the codec (XviD too) uses the PSNR it's because it done finally good visual result. IMO if all the codec use the PSNR it's perhaps because there are at this time no better HVS way.

[*.mp4 guy]

The codecs developers use PSNR because its time-eficient for them, SSIM would be an all around better metric, but PSNR is used instead because its fast. Keep in mind I'm not being critical of X264 or its developers, but that doesn't mean that I have to pretend they do everything in the best possible way, compromises have to be made.

[Manao]

PSNR is more widely used than SSIM not because it's faster ( SSIM is fast compared to the encoding time anyway ). It's because we are more used to it. We know how it behaves.

And since RD optimizations ( m6, m7, trellis in x264, VHQ 1-4, bVHQ, trellis in XviD ) are all about maximizing PSNR, judging the algorithm's efficiency with another metrics would be kinda pointless.

Ssim, sadly, is more difficult to optimize for, since it takes spatial dependency into account, which is complicated to consider while taking an encoding decision. Luckily, when PSNR rises, most often, SSIM does too, so...

[*.mp4 guy]

Ssim, sadly, is more difficult to optimize for, since it takes spatial dependency into account, which is complicated to consider while taking an encoding decision.
Well, thats nice to know, I stick by everything else I said, however.

[Dark Shikari]

This looks like it will turn into a full fledged debate over the validity of metrics, which I'm not interested in, let me reiterate one more time, PSNR is not a precise enough measure of quality to judge something like this. PSNR is a very easy way to guess the base efficiency of an encoder, so it makes sense that the X264 devs used it, it is a very commonly used metric in video encoding. However that isn't to say X264 wouldn't be better with some LAME quality psychovisual models.
Ugh, I don't want a debate in my help thread :rolleyes:

So far I've been optimizing for SSIM; adaptive quantization seems to hurt SSIM. Is this because AQ moves a bit of quality from the high-motion, high-detail areas to the background area to minimize blocking there?

[jethro]

Luckily, when PSNR rises, most often, SSIM does too, so...
Isn't SSIM better correlated with visual judgement of people?
...that an improvement in SSIM metric for the same codec & material closely mirrors the improvement in peoples' marks?
...and with PSNR curve people's marks are more spred and seem more "random"?
If yes, then why not use metrics better correlated with human vision?

[Manao]

Dark Shikari : adaptive quantization gives - relatively speaking - more bits to the macroblocks which have a low spatial complexity ( almost uniform areas ). It doesn't care about motion, nor about foreground/background. However, it happens that spatially uncomplex macroblocks are often those out of focus.

jethro : yes, SSIM is better correlated to visual judgement than PSNR. However, it's not significantly better for pictures, and especially not for movies, since both lack to take the temporal aspect into account, aspect which is the most important one imho.

And, once again, we come back to the way a codec works. SSIM considers a spatial neighborhood of the pixel in order to compute it's 'quality'. In order to optimize a decision ( motion vector, macroblock type ) for SSIM, we must be able to know what the neighborhood will be. However, we encode macroblocks one at a time, from the top left to the bottom right, which means that when we take the decision from the macroblock N, we don't know what the macroblock N+1 will be, so we can't really compute SSIM.

There have been attempts in x264 to replace the mean square error distorsion in the RD cost by a SSIM measure. I don't know where it went, but since nothing was commited, I don't think it worked as well as expected. Perhaps because PSNR is correlated enough to SSIM to do the job well enough.

In the end, SSIM is marginally better than PSNR, and much more complex to optimize for. So developpers choose the 'easy' way around, PSNR, which is easy and works anyway.

[akupenguin]

The attempts at optimizing for SSIM have so far been fruitless, but not due to any difficulties in computing SSIM. I tried restricting the area of SSIM to within the current macroblock, and SSIM-RD didn't produce measurably better SSIM than PSNR-RD did, even when the SSIM measured is the same restricted definition as was optimized for.
btw, SSIM (with the box-blur approximation) is not particularly slow. Sure, it's 4x slower than SSD, but that only adds about 3% to the total encode time. So it's only more complex in the sense of analytical manipulation (e.g. there's no SSIM-trellis), not in areas that were brute-forced anyway (MB mode decision).

[Dark Shikari]

I tried the following settings:

x264 --trellis 2 --me umh --bime --merange 64 --no-fast-pskip --partitions all --8x8dct --pass 3 --bitrate 4000 --subme 7 --bframes 16 --b-pyramid --ref 7 --b-rdo --mixed-refs --direct auto --weightb --progress --threads 4 --deblock 1:-1 -o output.mkv read_mkv.avs

And even with 3 passes (!) I can't get the bitrate down to 4000 kbps. It doesn't get lower than 4164. This is on a clip of the video in my original post.

What's going on?

[Manao]

How many frames is your clip made of ?

[Dark Shikari]

How many frames is your clip made of ?
503. I took about 1/4 of the main clip to reduce encoding time so I could test more options faster.

[Manao]

Then you got a 2.5% error on the bitrate ( 4000 <-> 4100 ), for 500 frames. That's only an error representing half a second of video. On a real movie - let's say two hours - the error will still be around half a second. It will represent less than 1 bps of error.

Two passes inaccuracies on short movie are to be expected.

[Dark Shikari]

Then you got a 2.5% error on the bitrate ( 4000 <-> 4100 ), for 500 frames. That's only an error representing half a second of video. On a real movie - let's say two hours - the error will still be around half a second. It will represent less than 1 bps of error.

Two passes inaccuracies on short movie are to be expected.
The reason I have a problem with the error is because I'm comparing the SSIM on different clips with a constant bitrate, and changing --bframes to 16 jumped up the bitrate by 100kbps, so its hard to tell whether the increased bitrate or the --bframes option actually improved the quality.

[Manao]

Then do a second test with another bitrate - lets say 3800 kbps - with the same settings. You'll then have two couples (SSIM,bitrate). You can then compare a third encoding, with different settings, to those two encodings, by looking whether the point is under or over the line formed by the first two couples.

That method works for most rate control, though, once again, I highly advise to use either CRF, two passes or constant quantizer - and to prefer the later actually, and on single scenes only.

The reason to use constant quantizer on single scene is that when you are encoding two scenes of different complexities, the scene with the lower PSNR is the scene influence most the (global) PSNR.

SSIM is yet another story, but the same problem arises with it. It's a measure made for pictures. It still has some sense for a scene, but not much across different scenes.

[Dark Shikari]

What PSNR should I use? There's Y, U, V, Average, and Global; which is most useful for comparing quality?

[Manao]

With a single scene, use global PSNR. With several scenes, you've got to keep an eye on both average and overall, but once again, you shouldn't really use PSNR in that case.

Edit : global = overall

[Dark Shikari]

With a single scene, use global PSNR. With several scenes, you've got to keep an eye on both average and overall, but once again, you shouldn't really use PSNR in that case.

Edit : global = overall
On this topic, PSNR never made sense to me; wouldn't it be more effective to measure the average, rather than peak signal to noise ratio? I would think that measuring peaks in the signal would be useless... why is PSNR the main measure? There must be a good reason.

[Manao]

PSNR = 10 log (255*255/average (squared error)). So it does take (only) the average error into account.

[Dark Shikari]

PSNR = 10 log (255*255/average (squared error)). So it does take (only) the average error into account.

Ah k, so its sort of a misnomer then.