How do you check efficiency?

[Chengbin]

How do you compare the compression efficiency of x264 made encodes?

I've read many commits on x264.nl and the developer says "3-5% efficiency gain" or something.

We know that CRF doesn't work, so how do devs get these numbers?

[neuron2]

We know that CRF doesn't work I didn't know that. Can you elaborate, please?

[Chengbin]

I think Dark Shikari can explain that better than me.

I can give 3 examples though.

8x8dct vs no 8x8dct. We all know 8x8dct gives huge improvements, but video encoded without 8x8dct with the same CRF value will give a smaller file. The 8x8dct video will be larger, but the visual quality is significantly better. Same for psy-rd

Adding black border. If you add black borders to a video, even though the efficiency is lowered by about 1%, CRF changes drastically (up or down I forgot), because of image complexity (for example the non bordered video is 100% "complex", but the bordered video is only, say "80%" complex, because black borders are static)

Change of CRF systems, such as mb-tree and aq-mode 2.

[neuron2]

You say Dark Shikari acknowledges that CRF does not work? Can you give me a link, please?

[Chengbin]

You say Dark Shikari acknowledges that CRF does not work? Can you give me a link, please?

Here is a link

http://forum.doom9.org/showthread.php?t=148538&highlight=crf+quality

Second post

CRF is not a measure of quality; it is a command to ratecontrol. Different settings and sources at the same CRF are not guaranteed to give the same visual quality.

There are many like it.

This is the one for the border example.

http://forum.doom9.org/showthread.php?t=149403&highlight=crf+quality

The borders don't take any bits, the difference is that CRF's frame complexity metric takes into account the overall complexity of the frame--which is lower when you have large chunks of the frame that are completely black. Lower complexity ---> lower quantizer ---> higher bitrate.

[neuron2]

He didn't say it doesn't work. You are misrepresenting him!

And can't you get the gain percent by doing the same encode with and without the code change, keeping the source and settings constant? What does it have to do with CRF?

[Shinigami-Sama]

efficiency measurement -> crf doesn't work
seems perfectly valid to me



also pick 2pass bitrate, run on sample take PSNR/SIMM/etc
redo 2pass with new/modified build, take PSNR/SIMM/etc
compare
???
profit

[Dark Shikari]

5% efficiency gain means that we get +~0.25db PSNR at the same bitrate, or ~5% lower bitrate for the same PSNR.

Anything defined as an efficiency gain is independent of other aspects, such as psy optimizations: if the gain was only in SSIM, I would say "SSIM gain", not efficiency gain.

[Chengbin]

5% efficiency gain means that we get +~0.25db PSNR at the same bitrate, or ~5% lower bitrate for the same PSNR.

Anything defined as an efficiency gain is independent of other aspects, such as psy optimizations: if the gain was only in SSIM, I would say "SSIM gain", not efficiency gain.

I thought metrics isn't a good way to test video quality?

Or is it they're good for efficiency tests?

[LoRd_MuldeR]

Well, the proper way to compare the "efficiency" of different settings and/or different encoders would be to find the lowest possible bitrate for all encoders/settings that still delivers acceptable visual quality.

Of course that test would never be 100% objective and it will be a lot of work for sure.

Metrics aren't very good to judge the overall quality of a video, especially with Psy optimizations enabled. But at least you have some "hard" numbers ;)

If you "blindly" rely on metrics or PSNR-based efficiency measure, all you know is that the "most efficient encoder" is the most efficient one for retaining "high PSNR", which isn't necessarily equal to "good quality".

As we already know, Psy opts can hurt metrics while proving significant better visual quality. So when using a metric-based efficiency measure, the "better looking" encoder could actually be less efficient...

[Dark Shikari]

I thought metrics isn't a good way to test video quality?

Or is it they're good for efficiency tests?"Efficiency" is how effectively the encoder can optimize for a metric.

[Chengbin]

"Efficiency" is how effectively the encoder can optimize for a metric.

Ahhhh! I get it now.

[neuron2]

efficiency measurement -> crf doesn't work
seems perfectly valid to me Seems perfectly meaningless to me. Maybe if you used full competent English I could have a chance to parse it.