Which is a good image-quality metric?

[Forteen88]

Hi, I wonder, which metric is good for measuring image-quality?
I've seen some people often use "Avg QP" as image-quality metric (with low Avg QP as better quality), but isn't SSIM a better metric than that?

Yeah, I know the eyes are the best image-quality thing to use, if you got good eye-sight.

Thanks in advance.

[nevcairiel]

Avg QP is technically not a quality metric, but a compression metric. There is a relation to quality of course, but its not its primary purpose.

Metrics like SSIM are far better for that. Or Netflix' VMAF.

[alex1399]

For comparison the PSNR of luma component (Y) only is quite fast and not bad. The x-y plot of SSIM (Y) to PSNR (Y) doesn't have much outliers and they have some strong positive correlation. In addition, it's great on detecting minor image shift which eyes cannot differentiate.

[zub35]

If your purpose is to assess the quality, with minimal changes from the original (visually lossless (equivalent jpeg qualities 90+))

butteraugli - https://github.com/google/butteraugli

Other metrics of quality measurement will lie, because of the application of psychovisual algorithms in modern codecs.

[benwaggoner]

Avg QP is technically not a quality metric, but a compression metric. There is a relation to quality of course, but its not its primary purpose.

Metrics like SSIM are far better for that. Or Netflix' VMAF.
x265's Rate Factor is also more subjectively correlated than QP, since it is basically QP with an image complexity offset.

VMAF is the least-bad metric we have today, but it is still limited by being a per-frame metric, which then gets averaged across frames. So an encode with frames ranging from 70-80 and an encode with frames ranging from 50-100 would both average a VMAF of 75. But the one going from 50-100 is going to have a lot more poor looking frames, and would deliver a much poorer subjective experience.

VMAF is also calculated relative so source frame sizes, so the same VMAF score from a 720p source and a 1080p source will mean quite different things. The "right" way to do it would be to scale source and encodes up to 1080p to run the VMAF comparisons. But that really increases the calculation time for VMAF.

[Forteen88]

VMAF is the least-bad metric we have todayWhy isn't VMAF implemented in x265?

EDIT: I thought more about licensing, if it's any problem implementing it into x265 because of that.

[benwaggoner]

Why isn't VMAF implemented in x265?



Because no one has checked in code to do it :)?

VMAF elementary metric calculations should be MUCH faster to do in x265 than ffmpeg because the source and output frames are both right there in the CPU cache together.

Since the VMAF model itself changes, actually calculating VMAF itself wouldn’t be as valuable as calculating the elementary metrics, which would allow improving VMAF accuracy for existing encodes whenever the machine learning model is updated.

[Forteen88]

By the way, I read that VQM is also a good quality-metric,
This is also another argument against using simple metrics such as PSNR or SSIM to judge video quality: they cannot take into account perceptual effects like motion, since they only look at individual frames. More perceptually-based metrics like VQM or VMAF are better choices for evaluating video degradations.
https://slhck.info/video/2017/02/24/crf-guide.html