VqmCalc 0.21 & SSIM 0.1: Some attempts to compare Video Quality

[Lefungus]

Hello

I've been playing a bit with xvid lately, and the compare function of avisynth. It calculates psnr from two clips.
I wasn't satisfied about psnr and searched different video quality metrics.
I found JNDMetrix-IQ, a full software that use DCT, and contrast masking/temporal masking. The concept was great.
http://www.sarnoff.com/products_services/video_vision/jndmetrix/documents/jndmetrix_overview.pdf

But it's patented/closed software

Then i found that
http://ise.stanford.edu/class/ee392j/projects/xiao_report.pdf

I found it feasible, so that's what i tried to implement in the following filter VqmCalc.

I'm still a newbie in c/c++. I learned coding for avisynth with this filter. So it's badly coded, slow and buggy. But i'll improve it as time pass.
I can't replicate results found on the previous white-paper as some vital informations are lacking like the matrix we need to use for spatial masking.
In this filter, i use MPEG Matrix with hope it reflects a bit Human Visual System.
No temporal masking is implemented in this method.

I've done a few experiments with a little clip encoded with xvid at constant quant2/quant4/quant6 and quant2 with VHQ1

Keep in mind these results may not reflect video quality

Mean VQM:
Quant2: 37.16
Quant4: 52.58
Quant6: 63.12
Quant2+VHQ1: 37.74

More points (may) means less quality. It seems to scale well with Quants but it doesn't like VHQ.

It will only work with YV12. And it seems to not works with B-Frames.
The clip must be mod8. If not it may crash :)
Another bugs i've encountered are:
-May crash when the log already exist
-May crash during the clip, try again until it works

I hope you're not afraid to test it :)

an example:

a=AVIsource("J:\Video\Source.avs")
b=AVIsource("J:\Video\Quant6.avi")
return VqmCalc(a,b,"results.log")

PS: I use fdct_mmx from xvid sources, so i guess i need to release it GPL, and include GPL headers. I'm not sure if i can, because it's inspired from a white paper. I'll be glad to receive any help on this subject.

[edited]: Corrected Links

VqmCalc-0.21 (http://perso.wanadoo.fr/reservoir/dl/VqmCalc-0.21.rar)

VqmCalcSrc-0.21 (http://perso.wanadoo.fr/reservoir/dl/VqmCalcSrc-0.21.rar)


SSIM-0.24 (http://perso.wanadoo.fr/reservoir/dl/SSIM-0.24.rar)

SSIMSrc-0.24 (http://perso.wanadoo.fr/reservoir/dl/SSIMSrc-0.24.rar)

[sh0dan]

Very interesting!

The links point to the same file (binary only) - you may need to correct that - then I'll have a look at your 'nasty' source. :)

[Lefungus]

Well, it seems it doesn't work anymore.

So i'll try to squash more bugs and make a proper release. Sorry

[Lefungus]

Ok, New version: 0.1

It is crashfree on my computer and maybe on others.
Next step, more speed, and more tuned results.
Now i use an excessively simple way to put more weight on low frequencies. I'll try to improve that

Here some results from a clip, 1 Pass,H263 Matrix, constant quantizer, Koepi build:

Quant 2: 40.1
Quant 3: 48.1
Quant 4: 57.9
Quant 5: 63.5
Quant 6: 70.1

Quant 4+Chroma: 57.3
Quant 4+Qpel: 59.7
Quant 4+MPEG: 56.8
Quant 4+Lumi: 58.2
Quant 4+VHQ1: 59.2
Quant 4+VHQ2: 59.2
Quant 4+VHQ3: 59.0
Quant 4+VHQ4: 58.5

[temporance]

I am very interested in this project and would like to help take it further by doing some coding myself!

First though, I want to understand the quality metric described in the paper (I'm not so happy with it as it works with 8x8 block DCTs - is it sensitive to blocking, for example?).

And I want to find out if there are any other popular and open metrics in the academic press that I could implement. Anyone got any ideas? What are VQEG using?

Thanks for giving me some good ideas and enthusiasm!

[Lefungus]

This implementation is too basic. And 8x8 DCT blocks, should effectively prevent block detection :/

I searched about interesting papers. There are some, but you'll find they don't give much informations on how to implement these algorithms.

An interesting video quality metric is PDM ( Perceptual Distorsion Metric), that looks really good on paper. From Stefan Winkler. He has many papers on the web.

You could search for Watson too, he created the VQM metric.

There was some results from VQEG, PSNR was still quite good, but PDM was easily above.

Some links
http://foulard.ee.cornell.edu/publications/masry_hvei03.pdf
http://dewww.epfl.ch/~nadenau/Research/Paper/HVS_Review_Final.pdf
http://stefan.winkler.net/Publications/chap2001.pdf

[temporance]

Originally posted by Lefungus
I searched about interesting papers. There are some, but you'll find they don't give much informations on how to implement these algorithms.I found that too. Nothing like an adequate description of the algorithms.
An interesting video quality metric is PDM ( Perceptual Distorsion Metric), that looks really good on paper. From Stefan Winkler. He has many papers on the web.Perhaps it would be possible to get someone like Stefan to help! I'm sure he has code in C or MatLab that we could integrate. If his metric became popular here then it could only help his career and reputation!

[temporance]

It seems only commercial projects publish their algorithms. LeFungus found the JND algo. Here's another one http://www.its.bldrdoc.gov/n3/video/pdf/vqm_techniques_v2.pdf

The companies also produce free evaluation software. I guess they want everyone to start using their metric.

LeFungus, you said "...but it's patented/closed software". To me, this isn't the biggest problem. A source-only release under GPL is generally allowed, even where there are patents.

The real problems are the complexity of making an independent implementation, and actually chosing which algo to implement. I want to know which algo will be chosen by VQEG, then I want to implenent that one :)

[MfA]

Ive always liked Zhou Wangs universal image quality index for its simplicity, only problem was that it broke down on uniformely colored surfaces ... he has recently published an adapted version which he calls the structure simularity index, which is still quite easy to compute.

[Lefungus]

http://www.cns.nyu.edu/~zwang/files/research/quality_index/demo.html
http://www.cns.nyu.edu/~zwang/

It's highly interesting Mfa !

And there's some code given :)

[MfA]

I posted a very basic C version of the universal image quality index to the xvid mailing list once, unoptimized apart from using a summed area table, adapting it for the SSIM should be pretty trivial.

[MfA]

The paper for SSIM (http://www.cns.nyu.edu/~zwang/files/papers/vssim.pdf) is now up at Zhou Wang's site (site is down, so for now a copy is here (http://home.student.utwente.nl/m.f.al/ssim/vssim.pdf)).

[MfA]

If anyone is interested, a decent C version of the SSIM measure from Zhou Wang is here (http://home.student.utwente.nl/m.f.al/ssim/ssim.cpp).

Vectorization should be easy to add. Basically all for loops before "here we calculate the sum over the 8x8 block of pixels" can be vectorized, prefetching should also help for older machines (Durons).

Another potential optimization is to use a circular buffer for the column sums to save on memory bandwith (on machines with 256KB+ of cache).

[Lefungus]

version updated to 0.2 for VQMCalc:
- Lots of bugs fixed.
- Calculations should be correct now. But motion has no influence yet on results.

version 0.1 of SSIM released:
- As for VQmCalc, motion isn't taken into account
- No vectorization yet, so it's quite slow.
- Thanks Zhou Wang and Mfa for their code !


Results given by these plugins are experimental, and shouldn't be trusted much until some serious tests are done.

[Lefungus]

When you compare two identical clips, you should have:
- VQM=0, the higher this number is, the lower the quality is.
- SSIM=1, quality decrease when this number decrease.

[Lefungus]

VqmCalc updated to 0.21:
-Two stupid bugs corrected. Previous results were false.

[arlsair]

First, thanks for these plugins. They really that, what I need to check my results.

SSIM works, but VqmCalc calculates as average "1.#QNAN0".
I compare two clips made from a original DVD and a copy. So there are b-frames. But I don`t think that is related to b-frames, or ?

An yes, it is slow (round about 0,5 RT with Athlon XP 1800+, 512MB, WinXP Pro). But speed doesn`t really matter.

regards
Arlsair

[sbeyer20]

Hi,

what is the exact meaning of average SSIM? Why is this value about 64 and not about 1????

What do you think is better for comparison: SSIm or VQM?

[MfA]

Ack, dont resurrect ancient threads please ... this still links a buggy version of SSIM.