How to automaticaly evaluate the quality of video?

[DavidXanatos]

How to automatically evaluate the quality of a video file?
Is there a tool for this?

David

[Ghitulescu]

yes, your eyes.

[DavidXanatos]

its not automatically, I need something for a few dozend files.

My eyes are to biased, thay only work relativly, i need something that makes an absolute measurement.

[Ghitulescu]

Quality is subjective.
Like music, car, women, wine ... you name it.

[LoRd_MuldeR]

its not automatically, I need something for a few dozend files.

My eyes are to biased, thay only work relativly, i need something that makes an absolute measurement.

What he wanted to say is that all the "objective" quality metrics existing today, such as MSE/PSNR or SSIM or VQM, only give a very rough estimate of the perceived quality. Moreover, the vast majority of all objective quality metrics that I'm aware of are so-called "full reference" metrics, which means that you need both, the "original" (perfect quality) video and the "test" video that you want to evaluate.

After all, the one and only way to judge video quality reliably is using your eyes. If you need an "unbiased" reference, you'll have to repeat the test with a sufficient number of persons and under well-defined viewing conditions. Then you can take the average score (or the median score) of all persons. And indeed, such a "subjective" quality assessment can't be automated...

[EDIT]

Anyway, if you want to play with objective quality metrics and do your own tests, you might want to check out the MSU tool:
http://compression.ru/video/quality_measure/video_measurement_tool_en.html

[jmac698]

Not that simply, no. Many filters need to do estimates of quality and there's mathematical formulas for this, for example to estimate focus, noise, blockiness, dot crawl, dropouts spots, etc. You can also guess based on bitrate, codec, dimensions, and colordepth. What aspect of quality are you trying to measure?

[LoRd_MuldeR]

Bitrate means absolutely nothing. Depending on the complexity of the source, the very same bitrate can look either great or horrible.

Some people still believe that "bits per pixel" (bpp) can serve as a quality estimate, but it is completely meaningless for the reasons that have been discussed over and over again.

And even video that has been encoded at an extraordinary high bitrate still might look horrible, e.g. if the original source was of crap quality or if the source was screwed up by inadequate filtering.

Finally the video's dimensions don't say anything either. For example video that has a "HD" resolution might have been upscaled from a low resolution source...

[jmac698]

I agree, I was making some assumptions there. It can be an indicator of quality; if similar camcorder home videos were encoded in different ways, you could always say more bits are better, correct? And that avc is generally half the filesize of mpg2 for the same quality? We're not talking cartoons or artificial test videos here.
It's like saying lagarith is better than gzip. There's no doubt of it.

[Ghitulescu]

Bitrate does mean something, its context is however decisive. Higher bitrates allow higher qualities (for identical conditions), but they don't provide as such higher qualities. HDTV allows for a better quality, but not all 1920x1080x24p images are high quality.
All the quality factors I know of are "degradation quality" parameters.
Quality per se is difficult to measure.

"Quality is me"

[DavidXanatos]

Well, I'm interested mainly in the 3 aspects:
1. blockiness
2. sharpness
2b. deinterlacing quality

[Ghitulescu]

I said it once and I repeat it:
The quality is in the eye of the watcher.
Unless you do your work for other people use your eyes and judge for yourself whether the image you have is ok or not. You'd be surprised to see how different see different people the same image.

[LoRd_MuldeR]

if similar camcorder home videos were encoded in different ways, you could always say more bits are better, correct?

Not, really. Even when recorded with the same equipment, if one of your "camcorder home videos" has higher complexity (e.g. more motion), then it will need more bitrate for the same quality compared to the others. Moreover the quality of your camcorder clips will be influenced highly by how well the clip was recorded (sharpness, illumination, shaking, etc), so the encoding is only one quality factor out of many.


And that avc is generally half the filesize of mpg2 for the same quality? We're not talking cartoons or artificial test videos here.

Well, given we encode from the identical source (with identical pre-processing) and we are using a good H.264 encoder, then indeed with H.264 you will be able to get away with a significant lower bitrate for the same quality, compared to MPEG-2. Still, when you are given two "unknown" encodes, one MPEG-2 and one H.264, you can't "blindly" judge quality from the pure numbers. That's because the MPEG-2 version might actually have been encoded from much higher quality source or the H.264 version might have been encoded with a crappy H.264 encoder. It's just impossible to know without looking at it...

[jmac698]

Anyhow,
here's an idea on how to detect blockiness. It is done in h.264's deblocking filter, but there's no variable exposed to say, this is how blocky it is. The x264 algorithm is used in dgdecode and fizick's deblock plugin. So looking at the script deblock qed, I see he builds a bitmap showing just the blocks edges and then deblocks them. I think if you intercept the script at that point you could come up with a measurement of blockiness, and it could be written to a file per frame.
It would take some work to come up with this but I think it could be done.
As for sharpness, that's pretty easy. I would say to take the fft of the video and look for the strongest high frequency that occurs over the video. That would give a good estimate of sharpness, provided something "sharp" appeared in the video at some point. I'm sure there's a plugin to do this somewhere.
For deinterlacing quality, I would just re-deinterlace it with the best available deinterlacer and compare the differences, the less different the better the original deinterlacing quality. This is a pretty rough measure as well.
I couldn't help you anymore than to say it's possible but would take an advanced scripter.

[LoRd_MuldeR]

Anyhow,
here's an idea on how to detect blockiness. It is done in h.264's deblocking filter, but there's no variable exposed to say, this is how blocky it is. The x264 algorithm is used in dgdecode and fizick's deblock plugin. So looking at the script deblock qed, I see he builds a bitmap showing just the blocks edges and then deblocks them. I think if you intercept the script at that point you could come up with a measurement of blockiness, and it could be written to a file per frame.

x264 doesn't have any "special" deblocking filter. It makes use of the "Inloop Deblocking" feature that is an integral part of the H.264 standard (as all the H.264 encoders do) and which therefore has to be implemented by any H.264 decoder. Moreover the deblocking feature of H.264 is not intended to remove blocking artifacts that are present in the original source, but to suppress the introduction of blocks by the H.264 compression. Therefore the deblocking filter used in H.264 doesn't have to "detect" the blocking artifacts. Simply put, the H.264 encoder "knows" at which places it introduced blocks and therefore the deblocking filter can be controlled based on that knowledge (this is done by adapting the filter strength based on the block's quantizer value). It can't serve as a general "blockiness" metric for arbitrary sources.

[jmac698]

Hi,
This is where I got the impression from:
Deblock_qed is fully based on Deblock() (implementation of H.264 deblocking, contained in DGDecode / older MVTools / Fizick's stand-alone plugin.)
From http://forum.doom9.org/showthread.php?p=944459
If there's a standalone plugin, obviously this is a "blind" deblocking algorithm.
"Perceptual Video Quality and Blockiness Metrics for Multimedia Streaming Applications" - S. Winkler, A. Sharma, D. McNally 2001 might help.
http://adsc.academia.edu/StefanWinkler/Papers/407491/Perceptual_Video_Quality_and_Blockiness_Metrics_for_Multimedia_Streaming_Applications

So yes, there is a pretty good measure of blockiness. But what is a quick and dirty way of getting that out of plugins? What if we just run a deblocker and compare the before and after at the 8x8 edges, and sum that into one number and write to a file?