H.264 encoder evaluation tool

[ff58]

Hi,

I want to see the performance of our h.264 encoder which is running on an embedded system. For this purpose, I have encoded a raw YUV file and obtained a compressed h.264 file. I want to ask if it is possible to compare the YUV and h.264 files in terms of related measures such as PSNR? Is there any software tools than can be capable of doing these kind of comparison measurements?

Any other evaluation methods and tools will be very helpful?

Thanks&Best Regards,
Fikret

[LoRd_MuldeR]

x264 can calculate both, PSNR and SSIM, by using "--psnr" and/or "--ssim". If you use those, you should also set "--preset psnr" or "--preset ssim", respectively.

As for other H.264 encoders, I don't know if they have a similar feature, but you may always use the MSU tool:
http://compression.ru/video/quality_measure/video_measurement_tool_en.html

Having said that, you should be aware that objective quality metrics, such as PSNR or SSIM are not very good to judge the overall quality ;)

(We all know that the "Psy" optimizations in x264 can give a great perceived quality improvement, still they significantly hurt the PSNR and SSIM metrics)

[ff58]

x264 can calculate both, PSNR and SSIM, by using "--psnr" and/or "--ssim". If you use those, you should also set "--preset psnr" or "--preset ssim", respectively.

As for other H.264 encoders, I don't know if they have a similar feature, but you may always use the MSU tool:
http://compression.ru/video/quality_measure/video_measurement_tool_en.html

Having said that, you should be aware that objective quality metrics, such as PSNR or SSIM are not very good to judge the overall quality ;)

(We all know that the "Psy" optimizations in x264 can give a great perceived quality improvement, still they significantly hurt the PSNR and SSIM metrics)

Dear Lord_Mulder,

Thanks for your reply. If PSNR and SSIM are not very good, is there any other tools for evaluating the quality?

I want to see the effect of changing encoder parameters and determine the best configuration. So, you say that the best way to evaluate the quality is the subjective evaluation?

[pandy]

Thanks for your reply. If PSNR and SSIM are not very good, is there any other tools for evaluating the quality?


Yes, human brain + eye (simulator?) however such solutions are very expensive.

I think You misunderstood Lord Mulder - PSNR or SSIM are perfect tools to measure PSNR or SSIM however there is no direct translation for PSNR or SSIM to perceived - subjective quality - so PSNR or SSIM are good tools to compare PSNR or SSIM however best PSNR or SSIM encoder can be not even close to perceived, subjective quality.

[ff58]

Yes, human brain + eye (simulator?) however such solutions are very expensive.

I think You misunderstood Lord Mulder - PSNR or SSIM are perfect tools to measure PSNR or SSIM however there is no direct translation for PSNR or SSIM to perceived - subjective quality - so PSNR or SSIM are good tools to compare PSNR or SSIM however best PSNR or SSIM encoder can be not even close to perceived, subjective quality.

Dear Pandy,

Ok, I understand now. I have tried MSU software, it is really good for comparison. Thank you for your helps.

[LoRd_MuldeR]

I want to see the effect of changing encoder parameters and determine the best configuration. So, you say that the best way to evaluate the quality is the subjective evaluation?

Yes, definitely.

There are many examples where objective metrics (like PSNR or SSIM) can give a misleading results. As mentioned before, the "Psy" optimizations in x264 are known to give a huge perceived quality boost, but if you only trusted on PSNR or SSIM, you would think they reduce the quality. What you can learn from this is that objective metrics can be helpful for some tasks (sometimes you simply have to get "hard" numbers), but you should never blindly trust them. Basically, all commonly used metrics measure the "error" between the original and the encoded video and then express "quality" as the absence of error (assuming "perfect" quality for the original). That works up to some point, but it's not how the human eye works. Sometimes more "error" looks better than smoothing all the details out...

[Blue_MiSfit]

The problem with subjective comparison (in my experience) is that we're easily confused, and the differences are often quite minor - especially when making small changes across a large sample. Placebo effect also comes into play.

I've yet to find a foolproof way to compare videos.