What's the point of RDO?

[Terranigma]

What are the advantages of using RDO over multiple instances of QPEL (For the subpixel refinement)? RDO, for me sometimes seems to destroy the picture, maybe because I usually use 16 b frames with an increased biased setting, so maybe it has a conflict of interest or something, or perhaps this is a bug. This's all source dependent of course, but this bug seems to only occur with sources that's really grainy. After doing some tests, I not only found Multi QPEL faster, but it results in a more stable picture overall and the quality difference isn't noticeable, well not to my eyes, maybe only with a ssim/psnr test. So ultimately my question is, what's the logic behind RDO?

[burfadel]

Increasing the bias forces the use of more b-frames, above the recommended amount. b-frames are very low bitrate frames, so should be used wisely. I use 16 b-frames myself, but let the encoder decide when they are needed (so 16 would most likely hardly ever be used). I believe if you increase the bias too much you actually negate the benefit of it (but will end up with smaller file with crf, but probably a lower quality picture with cbr/abr). Only for animation may increasing the bias be beneficial, and even then only around 30, or 40 as a maximum.

So therefore due to the nature of b-frames, you should probably adjust your settings, then rdo may be beneficial and not detrimental. Thats not to say there's possibly not a bug, but its only showing due to your settings.

[Blue_MiSfit]

IIRC, RDO really helps in two pass mode, by optimizing the bitrate distribution.

I'm no expert when it comes to the inner workings of x264 though... :)

~MiSfit

[akupenguin]

RDO is not an alternative to qpel, it's an additional optimization to be applied after qpel.
The goal of the encoder is to maximize quality (psnr) per bitrate. In non-rdo mode it uses some heuristic (satd) to evaluate some decisions, and just hopes that's the best psnr-per-bitrate. In rdo mode it evaluates the exact psnr and bitrate of each alternative and picks the best. Now, psnr is not perceptual quality, so increasing psnr doesn't necessarily look better. In particular, grain can't be inter-predicted and takes lots of bits, so any decision that removes grain may improve psnr-per-bitrate. The only things you can do about that are: Come up with some quality metric (not psnr) which says that grain is visually important enough (compared to other types of distortion) to be worth its bit cost, or if your "instability" is not just loss of grain, the the metric should measure instability too. Or implement FGM so that the codec doesn't have to handle the grain.

IIRC, RDO really helps in two pass mode, by optimizing the bitrate distribution.
No. "rate distortion optimization" = "rate vs distortion" not "distorts the rate".

[Blue_MiSfit]

:D I stand corrected by the author (red faced)

[Terranigma]

Thanks for the info akupenguin and burfadel. After reading *.mp4 guy's Thread about AVC Matrices, and coming across this
B predict mode should be set to none if your source is very grainy(and you wan't it to stay that way), or has many low contrast areas, otherwise b-rdo should be disabled and b prediction should be set to auto. for the M4G HRM V2: Matrix. I found that the culprit was indeed, the rdo for b-frames option (And yes, like he said, B predict or b-rdo should be disabled for grainy sources). I'm neglecting to bother encoding my grainy clip, because I fear that disabling b-rdo will probably result in an overall lower quality picture in the end (for the same bitrate) since these b-frames won't get that extra boost from using rdo. :(