Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 10th September 2003, 00:57   #1  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
SSIM 0.24, an objective video quality metric

This filter has been created following the ideas of Zhou Wang.
It has been coded with the great help of Mfa, who worked on the core functions.

For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision.
Some examples here
A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect.

This filter is designed to compute an SSIM value by two methods, the original one, and a "enhanced" one that weight these results by lumimasking. On the todo list is to include the motion weighting.

This filter has five parameters:
Code:
ssim(clip1,clip2,"results.csv","averageSSIM.txt",lumimask=true)
clip1 and clip2 are the reference clip and the compressed clip.
results.csv is the file where obtained SSIM values will be written (this can be easily read in excel or notepad for those unfamiliar with the comma separated variables format)
lumimasking switch between the two methods.

When the video is closed, the filter will write a file named "averageSSIM.txt" that will contain the global SSIM value.

An SSIM value is between 0 and 1, 1 meaning perfect quality.

To analyse locally the results, you could use the csv files, and manipulate data in any excel-clone. Examples:
codec A vs codec B
codec A with lumi option

In the csv file, when lumimasking is activated, both SSIM values and its weigth is written.

Nota:
-If you use B frames under xvid, trim the first dummy frame of the xvid clip, and the last frame of the original clip
-For those who used ssim 0.1, this version is also faster !

SSIM 0.24 Dll
SSIM 0.24 Sources
This filter is released under the condition of the gpl license.

[EDIT]
version updated to 0.21:
-flag for globalSSIM added
-some checks added

0.22:
-added "scaled" parameter

0.23:
-minor memory leak removed

0.24:
-various bugfixes
__________________

Some old avisynth filters

Last edited by Lefungus; 6th June 2004 at 10:47.
Lefungus is offline   Reply With Quote
Old 10th September 2003, 13:50   #2  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Some script examples:

Code:
a=mpeg2source("C:\Vobs\pacte.d2v").crop(14,92,692,416).lanczosresize(640,256)
b=AVISource("C:\Vobs\xvid.avi").trim(1,0)
return ssim(a,b,"results.csv","averageSSIM.txt",lumimask=true)
In this example, i compare the dvd source to the xvid compressed version. As bframes are used, i have to trim the first frame of the xvid version.

Code:
a=mpeg2source("C:\Vobs\pacte.d2v").crop(14,92,692,416).lanczosresize(640,256)
b=AVISource("C:\Vobs\divx.avi")
return ssim(a,b,"results.csv","averageSSIM.txt",lumimask=true)
There, i compare the same source to a divx compressed version. No need to cut any frames there.


In the two examples, i compare the dvd to mpeg-4 codecs. But the SSIM metric should works as better on any codec, video source.

Nota:
-Compared clips must have same width/height/length
-SSIM is computed in yv12 colorspace

[EDIT]
Updated scripts
__________________

Some old avisynth filters

Last edited by Lefungus; 10th September 2003 at 22:10.
Lefungus is offline   Reply With Quote
Old 10th September 2003, 13:55   #3  |  Link
Selur
Registered User
 
Selur's Avatar
 
Join Date: Oct 2001
Location: Germany
Posts: 5,265
seems to work fine,..

Thanks for this nice tool

It's especially cool if you do somethign like this:
Code:
A = Avisource("d:\Original.avs")
B = Avisource("d:\encode1.avi")
c = Avisource("d:\encode2.avi")

StackVertical(ssim(A,B,"Encode1results.csv",lumimask=false),ssim(A,C,"Encode2results.csv",lumimask=false))
since it allows you to see which frame get's what rating on which clip


Cu Selur

Ps.: would be cool to also have an alternative flag, to set the output for the average SSIM, so the script I used would produce two average SSIM values.

Last edited by Selur; 10th September 2003 at 15:15.
Selur is offline   Reply With Quote
Old 10th September 2003, 16:27   #4  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Lefungus, this is an answer to a prayer as a few of us have been trying to compare the effects of matrices and filter combos on low bitrate mpg produced by CCE 2.67 and other mpeg2 encoders. If you have time, I have a few questions, some of which are fairly basic, so be patient with me .

1> Could you give a short overview of the Y, U, V, SSIM, WEIGHT figures shown during the comparison. This may be a waste of your time as I realize that data is available later for comparison via the results.csv and averageSSIM.txt. I guess I am asking how I can "get a feel" for what is happening during the comparison pass. I am afraid this question may seem ambiguous to you. If so, no worry

2> Your method of seeking clip2. For clip2 I am presently using a d2v generated by dvd2avi which works great. However I was thinking of trying Nic's mpegsource as it automatically creates the d2v and would eliminate that step, but if I remember correctly it has trouble with multiple seeks. Are you doing an end to end on clip2 or multiple seeks?

/ADD
3> I was interested in comparing different encoded sample clips. To test that I used clip1 = "clip1.d2v" and clip2= "Copy of clip1.d2v" and indeed got an Average SSIM= 1.0. So far so good.

Next test was clip1=knowngood.d2v clip2=known_inferior.d2v and again it showed a lower Average SSIM= Average SSIM= 0.666261. Still looking good.

Next, and here is either the problem or my confusion showing, I reversed the order of the test above to clip1=known_inferior.d2v clip2=knowngood.d2v and this time got an Average SSIM= 0.657958 which was about the same value as the reverse test above. I guess I expected the result to be an "out of range" or >=1, yet it was not.

What are your thoughts? Is this type of encoded clip comparison practical?

Because most of the information was specific to DVD2SVCD, I placed a post here in the DVD2SVCD. I hope you do not mind. http://forum.doom9.org/showthread.php?s=&threadid=61172

Last edited by DDogg; 11th September 2003 at 03:49.
DDogg is offline   Reply With Quote
Old 10th September 2003, 18:44   #5  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Quote:
Originally posted by DDogg
I have a few questions, some of which are fairly basic, so be patient with me .
Do i look so rude ?

1)
Y, U, V values are SSIM values for each plane for the current frame. Later they are weighted and averaged. Y is luminance, and U,V colors. Human vision is much more sensible to luminance so this plane has much more influence than the color planes.
SSIM is the current quality index, scaled from 0 to 1.
Weight is the coefficient that will be applied to the current SSIM value when the global SSIM value will be computed.

2)
To use this filter i suggest you load the script into vdub, play it from beginning to end without skipping frames or seeking specific frames. Nic mpegsource behaviour is unknown when you seek into the video, so if you do it this way, no problem should arise. I don't use any special trick, if you call frame 1, i'll ask frame 1 from clip 1 and frame 1 from clip 2.


@Selur: it'll be added in next version
__________________

Some old avisynth filters
Lefungus is offline   Reply With Quote
Old 10th September 2003, 18:55   #6  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Theorically ssim(a,b) = ssim(b,a)

In your case it looks like it isn't

About comparisons:
a is a video clip
b is a compressed version of a
c is another compressed version a

You could use ssim(a,b) and ssim(a,c) and compare these results but i don't think you could get any information from doing ssim(b,c).

[EDIT]

On all my tests, ssim was perfectly symmetrical, so i can't reproduce your result DDogg :/


__________________

Some old avisynth filters

Last edited by Lefungus; 10th September 2003 at 22:03.
Lefungus is offline   Reply With Quote
Old 11th September 2003, 02:17   #7  |  Link
Prettz
easily bamboozled user
 
Prettz's Avatar
 
Join Date: Sep 2002
Location: Atlanta
Posts: 373
Quote:
Originally posted by DDogg
Next, and here is either the problem or my confusion showing, I reversed the order of the test above to clip1=known_inferior.d2v clip2=knowngood.d2v and this time got an Average SSIM= 0.657958 which was about the same value as the reverse test above. I guess I expected the result to be an "out of range" or >=1, yet it was not.
It sounds like the filter is set up assuming that the first clip is the "original", so putting the inferior clip as the original and comparing a better-looking clip to it doesn't mean anything in that context, and to me it seems sensible that doing that could produce any kind of wacky result.
Prettz is offline   Reply With Quote
Old 11th September 2003, 03:16   #8  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Quote:
and to me it seems sensible that doing that could produce any kind of wacky result.
Yes, it does, however it got stranger after that. Actually, when I took great care and exactly replicated the test B-C and C-B the csv yield *identical* results to six decimal places, which I thought was odd as I expected a whacky result of some kind, but not identical, although I guess you could say that was symetrical in a way . Anyway, we have been PMing all day and I sent up a set of files to help Lefungus work with mpeg2 as he mentioned most of his work had been with XVID and DIVX. I hope this tool ends up working with mpeg2 as it would be very useful.

Last edited by DDogg; 11th September 2003 at 03:18.
DDogg is offline   Reply With Quote
Old 11th September 2003, 04:28   #9  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
SSIM simply doesnt care on which input the reference is because it is entirely symmetrical, just like MSE/PSNR.

Last edited by MfA; 11th September 2003 at 04:31.
MfA is offline   Reply With Quote
Old 11th September 2003, 17:16   #10  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Well it is symmetrical if you don't screw up and get a framecount unbalance like I did

One thing for mpg2 users to be careful of is the the two samples must be the exact same frame count (obviously), however, when using a mpv of a verified 5001 framecount, I noticed when using dvd2avi to create the d2v used in mpeg2source("b.d2v") that the framecount became 4998. So it is best to use the trim statement of a slightly smaller value to actually insure the exact frame count and double check it in VDub using file/information. I used mpeg2source("b.d2v").trim(0,4499) to give me an exact 4500 frames.

I noticed lafungus has updated ssim to catch this framecount unbalance in the newer version he posted so maybe it will not slip past others like it did for me using the first version.

Sidenote:I was gently pointed to the heavy thread in the dvd2avi forum dealing with this very subject where DG is putting in an enormous amount of effort dealing with framecount errors between dvd2avi and mpeg2dec. I don't read that forum much so I was unaware. Thanks!


Last edited by DDogg; 11th September 2003 at 20:29.
DDogg is offline   Reply With Quote
Old 11th September 2003, 21:48   #11  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
In version 0.22, the global score is scaled from 0 to 100 instead of 0 to 1. it's just a simple conversion: newValue = 100 * pow( oldValue, 8)
With this, different values between two clips may be more visible.

example on a small xvid sample:

high bitrate clip: 82
low bitrate clip: 66

If you prefer it non-scaled, just add "scaled=false" as parameter.
For the same sample, it'll give:

high bitrate: 0.975
low bitrate: 0.949
__________________

Some old avisynth filters

Last edited by Lefungus; 11th September 2003 at 21:57.
Lefungus is offline   Reply With Quote
Old 11th September 2003, 23:15   #12  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Quote:
With this, different values between two clips may be more visible.
Very much so, with the new values it is a snap to quickly get an idea of the quality of the encode. Thanks very much for this new feature, it makes using SSIM much easier and straightforward.
DDogg is offline   Reply With Quote
Old 11th September 2003, 23:27   #13  |  Link
KpeX
Registered User
 
KpeX's Avatar
 
Join Date: Jun 2003
Location: Great Lakes, USA
Posts: 1,433
Great tool, Lefungus. I've also needed to adjust some framecounts to get the right frames when comparing D2V to AVI. I usually test the script first with a StackVertical and scan through it frame by frame in VDub to make sure the correct frames are corresponding.
__________________
KpeX
Audio FAQs: General | BeSweet | SVCD/MP2 | MP3 | Vorbis | AC3 | DTS | AAC
Linux Audio/Video FAQ
KpeX is offline   Reply With Quote
Old 13th September 2003, 21:17   #14  |  Link
deXtoRious
Registered User
 
Join Date: Feb 2003
Location: Riga, Latvia
Posts: 192
The tool is great, however it's approx. 5-6 times slower than PSNR or VQM. Is that just a lack of optimisation or is it inherent in the very nature of the algorithm?
deXtoRious is offline   Reply With Quote
Old 13th September 2003, 21:41   #15  |  Link
MfA
Registered User
 
Join Date: Mar 2002
Posts: 1,075
Could probably be sped up 2 times with assembly, more with subsampling ... but I prefer it slow over subsampled. Removing the memory leak I introduced might also help a bit.

PSNR sucks period, and VQM is poor in weighing blocking artifacts.

Last edited by MfA; 13th September 2003 at 21:53.
MfA is offline   Reply With Quote
Old 28th September 2003, 17:13   #16  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Sh0dan has added conditionnal functions, so the ssim metric could be used in conditionnal scripts.

For example:
Code:
a=avisource("c:\blah.avi")
b=a.filter()
return conditionalfilter(a,b,a,"SSIM_AVG(a,b)","<","0.98",true)
will return b, if ssim(a,b)>0.98, or a if b is bad ssim-wise (meaning it may filter too much). 0.98 is an arbitrary value, and should be replaced by your own adequate value.

It may also be used in a 3-pass encoding scenario, when you filter more encoded frames, that have a too low ssim value.

Code:
a=avisource("c:\blah.avi")
b=a.filter()
c=avisource("c:\blah-2pass-encoded.avi")
return conditionalfilter(a,b,b,"SSIM_AVG(a,c)","<","0.94",true)
I have no idea if it'll give good results, but the concept is interesting

Get the dll here
__________________

Some old avisynth filters
Lefungus is offline   Reply With Quote
Old 28th September 2003, 18:06   #17  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
I had a few concepts in mind for "extreme" compression optimizitation:

Case 1:
Do an ordinary two-pass encode. Compare the output to the source. If SSIM difference is big, do additional filtering on these frames for a third/fourth pass.

Case 2:
Compare original frame to a blurred version. This should give a fairly good idea on the amount of high-frequency (spatial) detail level. This value could be used for setting threshold for a filter.



PS: There is also conditional functions for retrieving Y,U & V-plane only SSIM difference.

SSIM_Y(clip a, clip b)
SSIM_U(clip a, clip b)
SSIM_V(clip a, clip b)
SSIM_AVG(clip a, clip b) = 0.8 * SSIM_Y(a,b) + 0.1 * SSIM_U(a,b) + 0.1 * SSIM_V(a,b)
__________________
Regards, sh0dan // VoxPod

Last edited by sh0dan; 28th September 2003 at 18:15.
sh0dan is offline   Reply With Quote
Old 28th September 2003, 18:14   #18  |  Link
Lefungus
Registered User
 
Lefungus's Avatar
 
Join Date: Apr 2002
Location: Bretagne
Posts: 192
Actually, i've followed the ssim way for average, so it's :

ssim_avg=0.8*ssim_y + 0.1*ssim_u +0.1*ssim_v

as Y plane has much more visual impact than UV planes
__________________

Some old avisynth filters
Lefungus is offline   Reply With Quote
Old 28th September 2003, 18:16   #19  |  Link
sh0dan
Retired AviSynth Dev ;)
 
sh0dan's Avatar
 
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
Didn't I write that?

(ok - I edited the post above to avoid misunderstandings)
__________________
Regards, sh0dan // VoxPod
sh0dan is offline   Reply With Quote
Old 30th September 2003, 01:04   #20  |  Link
DDogg
Retired, but still around
 
DDogg's Avatar
 
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
Wow...This sounds like a potential nugget of gold hidden away in this thread. I'm getting all excited. I think you are saying one could compare each frame back to the source and then apply filters to each frame based upon the quality index returned by ssim. Is this correct? I do have a difficult time understanding this stuff.

I wonder if one or both of you could flesh it out a little more and maybe post a more complete example of a test script so folks like me could do some testing. I can't quite get there by myself and could use the help.

In my case I would like to try it with CCE 1-pass vbr with two to three filter combos like none, deen, and heavy (maybe deen with light blurring?)

In re-reading the above I realize I may be confused as you both mention multipass as if this may be "pass based" conditional versus frame based. Any help you can give me towards understanding the above more fully and getting a working test script would be appreciated. I can do alright with this stuff once someone kickstarts me.

Btw, your "get the dll here" above is a new version or is it 0.23? Might be better to just point people to the top of the thread?

Last edited by DDogg; 30th September 2003 at 02:57.
DDogg is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2017, vBulletin Solutions Inc.