Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 21st June 2003, 21:49   #1  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
VqmCalc 0.21 & SSIM 0.1: Some attempts to compare Video Quality

Hello

I've been playing a bit with xvid lately, and the compare function of avisynth. It calculates psnr from two clips.
I wasn't satisfied about psnr and searched different video quality metrics.
I found JNDMetrix-IQ, a full software that use DCT, and contrast masking/temporal masking. The concept was great.
http://www.sarnoff.com/products_serv...x_overview.pdf

But it's patented/closed software

Then i found that
http://ise.stanford.edu/class/ee392j...iao_report.pdf

I found it feasible, so that's what i tried to implement in the following filter VqmCalc.

I'm still a newbie in c/c++. I learned coding for avisynth with this filter. So it's badly coded, slow and buggy. But i'll improve it as time pass.
I can't replicate results found on the previous white-paper as some vital informations are lacking like the matrix we need to use for spatial masking.
In this filter, i use MPEG Matrix with hope it reflects a bit Human Visual System.
No temporal masking is implemented in this method.

I've done a few experiments with a little clip encoded with xvid at constant quant2/quant4/quant6 and quant2 with VHQ1

Keep in mind these results may not reflect video quality

Mean VQM:
Quant2: 37.16
Quant4: 52.58
Quant6: 63.12
Quant2+VHQ1: 37.74

More points (may) means less quality. It seems to scale well with Quants but it doesn't like VHQ.

It will only work with YV12. And it seems to not works with B-Frames.
The clip must be mod8. If not it may crash
Another bugs i've encountered are:
-May crash when the log already exist
-May crash during the clip, try again until it works

I hope you're not afraid to test it

an example:

Code:
a=AVIsource("J:\Video\Source.avs")
b=AVIsource("J:\Video\Quant6.avi")
return VqmCalc(a,b,"results.log")
PS: I use fdct_mmx from xvid sources, so i guess i need to release it GPL, and include GPL headers. I'm not sure if i can, because it's inspired from a white paper. I'll be glad to receive any help on this subject.

[edited]: Corrected Links

VqmCalc-0.21

VqmCalcSrc-0.21


SSIM-0.24

SSIMSrc-0.24

Last edited by Lefungus; 9th October 2005 at 14:52.
Lefungus is offline   Reply With Quote
Old 21st June 2003, 22:55   #2  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
Very interesting!

The links point to the same file (binary only) - you may need to correct that - then I'll have a look at your 'nasty' source.
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 21st June 2003, 23:06   #3  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Well, it seems it doesn't work anymore.

So i'll try to squash more bugs and make a proper release. Sorry
Lefungus is offline   Reply With Quote
Old 22nd June 2003, 19:22   #4  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Ok, New version: 0.1

It is crashfree on my computer and maybe on others.
Next step, more speed, and more tuned results.
Now i use an excessively simple way to put more weight on low frequencies. I'll try to improve that

Here some results from a clip, 1 Pass,H263 Matrix, constant quantizer, Koepi build:

Quant 2: 40.1
Quant 3: 48.1
Quant 4: 57.9
Quant 5: 63.5
Quant 6: 70.1

Quant 4+Chroma: 57.3
Quant 4+Qpel: 59.7
Quant 4+MPEG: 56.8
Quant 4+Lumi: 58.2
Quant 4+VHQ1: 59.2
Quant 4+VHQ2: 59.2
Quant 4+VHQ3: 59.0
Quant 4+VHQ4: 58.5
Lefungus is offline   Reply With Quote
Old 3rd July 2003, 17:25   #5  |  Link
temporance
Registered User
 
Join Date: Mar 2002
Posts: 486
I am very interested in this project and would like to help take it further by doing some coding myself!

First though, I want to understand the quality metric described in the paper (I'm not so happy with it as it works with 8x8 block DCTs - is it sensitive to blocking, for example?).

And I want to find out if there are any other popular and open metrics in the academic press that I could implement. Anyone got any ideas? What are VQEG using?

Thanks for giving me some good ideas and enthusiasm!
temporance is offline   Reply With Quote
Old 3rd July 2003, 18:07   #6  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
This implementation is too basic. And 8x8 DCT blocks, should effectively prevent block detection :/

I searched about interesting papers. There are some, but you'll find they don't give much informations on how to implement these algorithms.

An interesting video quality metric is PDM ( Perceptual Distorsion Metric), that looks really good on paper. From Stefan Winkler. He has many papers on the web.

You could search for Watson too, he created the VQM metric.

There was some results from VQEG, PSNR was still quite good, but PDM was easily above.

Some links
http://foulard.ee.cornell.edu/public...sry_hvei03.pdf
http://dewww.epfl.ch/~nadenau/Resear...view_Final.pdf
http://stefan.winkler.net/Publications/chap2001.pdf
Lefungus is offline   Reply With Quote
Old 3rd July 2003, 19:17   #7  |  Link
temporance
Registered User
 
Join Date: Mar 2002
Posts: 486
Quote:
Originally posted by Lefungus
I searched about interesting papers. There are some, but you'll find they don't give much informations on how to implement these algorithms.
I found that too. Nothing like an adequate description of the algorithms.
Quote:
An interesting video quality metric is PDM ( Perceptual Distorsion Metric), that looks really good on paper. From Stefan Winkler. He has many papers on the web.
Perhaps it would be possible to get someone like Stefan to help! I'm sure he has code in C or MatLab that we could integrate. If his metric became popular here then it could only help his career and reputation!
temporance is offline   Reply With Quote
Old 3rd July 2003, 21:31   #8  |  Link
temporance
Registered User
 
Join Date: Mar 2002
Posts: 486
It seems only commercial projects publish their algorithms. LeFungus found the JND algo. Here's another one http://www.its.bldrdoc.gov/n3/video/...hniques_v2.pdf

The companies also produce free evaluation software. I guess they want everyone to start using their metric.

LeFungus, you said "...but it's patented/closed software". To me, this isn't the biggest problem. A source-only release under GPL is generally allowed, even where there are patents.

The real problems are the complexity of making an independent implementation, and actually chosing which algo to implement. I want to know which algo will be chosen by VQEG, then I want to implenent that one
temporance is offline   Reply With Quote
Old 19th July 2003, 22:49   #9  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
Ive always liked Zhou Wangs universal image quality index for its simplicity, only problem was that it broke down on uniformely colored surfaces ... he has recently published an adapted version which he calls the structure simularity index, which is still quite easy to compute.
MfA is offline   Reply With Quote
Old 19th July 2003, 23:15   #10  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
http://www.cns.nyu.edu/~zwang/files/...ndex/demo.html
http://www.cns.nyu.edu/~zwang/

It's highly interesting Mfa !

And there's some code given
Lefungus is offline   Reply With Quote
Old 20th July 2003, 00:59   #11  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
I posted a very basic C version of the universal image quality index to the xvid mailing list once, unoptimized apart from using a summed area table, adapting it for the SSIM should be pretty trivial.
MfA is offline   Reply With Quote
Old 21st July 2003, 11:14   #12  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
The paper for SSIM is now up at Zhou Wang's site (site is down, so for now a copy is here).

Last edited by MfA; 15th August 2003 at 18:38.
MfA is offline   Reply With Quote
Old 26th July 2003, 20:16   #13  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
If anyone is interested, a decent C version of the SSIM measure from Zhou Wang is here.

Vectorization should be easy to add. Basically all for loops before "here we calculate the sum over the 8x8 block of pixels" can be vectorized, prefetching should also help for older machines (Durons).

Another potential optimization is to use a circular buffer for the column sums to save on memory bandwith (on machines with 256KB+ of cache).

Last edited by MfA; 15th August 2003 at 18:39.
MfA is offline   Reply With Quote
Old 29th July 2003, 15:22   #14  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
version updated to 0.2 for VQMCalc:
- Lots of bugs fixed.
- Calculations should be correct now. But motion has no influence yet on results.

version 0.1 of SSIM released:
- As for VQmCalc, motion isn't taken into account
- No vectorization yet, so it's quite slow.
- Thanks Zhou Wang and Mfa for their code !


Results given by these plugins are experimental, and shouldn't be trusted much until some serious tests are done.
Lefungus is offline   Reply With Quote
Old 29th July 2003, 15:57   #15  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
When you compare two identical clips, you should have:
- VQM=0, the higher this number is, the lower the quality is.
- SSIM=1, quality decrease when this number decrease.
Lefungus is offline   Reply With Quote
Old 10th August 2003, 15:29   #16  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
VqmCalc updated to 0.21:
-Two stupid bugs corrected. Previous results were false.
Lefungus is offline   Reply With Quote
Old 15th August 2003, 16:54   #17  |  Link
arlsair
Registered User
 
Join Date: Aug 2002
Location: Hamburg
Posts: 131
First, thanks for these plugins. They really that, what I need to check my results.

SSIM works, but VqmCalc calculates as average "1.#QNAN0".
I compare two clips made from a original DVD and a copy. So there are b-frames. But I don`t think that is related to b-frames, or ?

An yes, it is slow (round about 0,5 RT with Athlon XP 1800+, 512MB, WinXP Pro). But speed doesn`t really matter.

regards
Arlsair
arlsair is offline   Reply With Quote
Old 27th September 2005, 19:43   #18  |  Link
sbeyer20
Registered User
 
Join Date: Jul 2005
Posts: 25
Hi,

what is the exact meaning of average SSIM? Why is this value about 64 and not about 1????

What do you think is better for comparison: SSIm or VQM?
sbeyer20 is offline   Reply With Quote
Old 28th September 2005, 01:05   #19  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
Ack, dont resurrect ancient threads please ... this still links a buggy version of SSIM.
MfA is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 23:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.