Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
10th September 2003, 00:57 | #1 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
SSIM 0.24, an objective video quality metric
This filter has been created following the ideas of Zhou Wang.
It has been coded with the great help of Mfa, who worked on the core functions. For a given reference video and a given compressed video, it is meant to compute a quality metric, based on perceived visual distortion. Unlike the well-known PSNR measure, it's not purely mathematical, and should correlate much better with human vision. Some examples here A higher MSE (and so lower PSNR) should mean that the compressed clip is a worse image but MSE and PSNR are flawed in this respect as numerous tests have shown. However with SSIM, according to tests carried out on the VQEG dataset, a higher Q (SSIM value) has a much better relation to the visual quality of the compressed clip. Despite this, bear in mind the SSIM metric still isn't perfect. This filter is designed to compute an SSIM value by two methods, the original one, and a "enhanced" one that weight these results by lumimasking. On the todo list is to include the motion weighting. This filter has five parameters: Code:
ssim(clip1,clip2,"results.csv","averageSSIM.txt",lumimask=true) results.csv is the file where obtained SSIM values will be written (this can be easily read in excel or notepad for those unfamiliar with the comma separated variables format) lumimasking switch between the two methods. When the video is closed, the filter will write a file named "averageSSIM.txt" that will contain the global SSIM value. An SSIM value is between 0 and 1, 1 meaning perfect quality. To analyse locally the results, you could use the csv files, and manipulate data in any excel-clone. Examples: codec A vs codec B codec A with lumi option In the csv file, when lumimasking is activated, both SSIM values and its weigth is written. Nota: -If you use B frames under xvid, trim the first dummy frame of the xvid clip, and the last frame of the original clip -For those who used ssim 0.1, this version is also faster ! SSIM 0.24 Dll SSIM 0.24 Sources This filter is released under the condition of the gpl license. [EDIT] version updated to 0.21: -flag for globalSSIM added -some checks added 0.22: -added "scaled" parameter 0.23: -minor memory leak removed 0.24: -various bugfixes Last edited by Lefungus; 6th June 2004 at 10:47. |
10th September 2003, 13:50 | #2 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
Some script examples:
Code:
a=mpeg2source("C:\Vobs\pacte.d2v").crop(14,92,692,416).lanczosresize(640,256) b=AVISource("C:\Vobs\xvid.avi").trim(1,0) return ssim(a,b,"results.csv","averageSSIM.txt",lumimask=true) Code:
a=mpeg2source("C:\Vobs\pacte.d2v").crop(14,92,692,416).lanczosresize(640,256) b=AVISource("C:\Vobs\divx.avi") return ssim(a,b,"results.csv","averageSSIM.txt",lumimask=true) In the two examples, i compare the dvd to mpeg-4 codecs. But the SSIM metric should works as better on any codec, video source. Nota: -Compared clips must have same width/height/length -SSIM is computed in yv12 colorspace [EDIT] Updated scripts Last edited by Lefungus; 10th September 2003 at 22:10. |
10th September 2003, 13:55 | #3 | Link |
Registered User
Join Date: Oct 2001
Location: Germany
Posts: 7,277
|
seems to work fine,..
Thanks for this nice tool It's especially cool if you do somethign like this: Code:
A = Avisource("d:\Original.avs") B = Avisource("d:\encode1.avi") c = Avisource("d:\encode2.avi") StackVertical(ssim(A,B,"Encode1results.csv",lumimask=false),ssim(A,C,"Encode2results.csv",lumimask=false)) Cu Selur Ps.: would be cool to also have an alternative flag, to set the output for the average SSIM, so the script I used would produce two average SSIM values. Last edited by Selur; 10th September 2003 at 15:15. |
10th September 2003, 16:27 | #4 | Link |
Retired, but still around
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
|
Lefungus, this is an answer to a prayer as a few of us have been trying to compare the effects of matrices and filter combos on low bitrate mpg produced by CCE 2.67 and other mpeg2 encoders. If you have time, I have a few questions, some of which are fairly basic, so be patient with me .
1> Could you give a short overview of the Y, U, V, SSIM, WEIGHT figures shown during the comparison. This may be a waste of your time as I realize that data is available later for comparison via the results.csv and averageSSIM.txt. I guess I am asking how I can "get a feel" for what is happening during the comparison pass. I am afraid this question may seem ambiguous to you. If so, no worry 2> Your method of seeking clip2. For clip2 I am presently using a d2v generated by dvd2avi which works great. However I was thinking of trying Nic's mpegsource as it automatically creates the d2v and would eliminate that step, but if I remember correctly it has trouble with multiple seeks. Are you doing an end to end on clip2 or multiple seeks? /ADD 3> I was interested in comparing different encoded sample clips. To test that I used clip1 = "clip1.d2v" and clip2= "Copy of clip1.d2v" and indeed got an Average SSIM= 1.0. So far so good. Next test was clip1=knowngood.d2v clip2=known_inferior.d2v and again it showed a lower Average SSIM= Average SSIM= 0.666261. Still looking good. Next, and here is either the problem or my confusion showing, I reversed the order of the test above to clip1=known_inferior.d2v clip2=knowngood.d2v and this time got an Average SSIM= 0.657958 which was about the same value as the reverse test above. I guess I expected the result to be an "out of range" or >=1, yet it was not. What are your thoughts? Is this type of encoded clip comparison practical? Because most of the information was specific to DVD2SVCD, I placed a post here in the DVD2SVCD. I hope you do not mind. http://forum.doom9.org/showthread.php?s=&threadid=61172 Last edited by DDogg; 11th September 2003 at 03:49. |
10th September 2003, 18:44 | #5 | Link | |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
Quote:
1) Y, U, V values are SSIM values for each plane for the current frame. Later they are weighted and averaged. Y is luminance, and U,V colors. Human vision is much more sensible to luminance so this plane has much more influence than the color planes. SSIM is the current quality index, scaled from 0 to 1. Weight is the coefficient that will be applied to the current SSIM value when the global SSIM value will be computed. 2) To use this filter i suggest you load the script into vdub, play it from beginning to end without skipping frames or seeking specific frames. Nic mpegsource behaviour is unknown when you seek into the video, so if you do it this way, no problem should arise. I don't use any special trick, if you call frame 1, i'll ask frame 1 from clip 1 and frame 1 from clip 2. @Selur: it'll be added in next version |
|
10th September 2003, 18:55 | #6 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
Theorically ssim(a,b) = ssim(b,a)
In your case it looks like it isn't About comparisons: a is a video clip b is a compressed version of a c is another compressed version a You could use ssim(a,b) and ssim(a,c) and compare these results but i don't think you could get any information from doing ssim(b,c). [EDIT] On all my tests, ssim was perfectly symmetrical, so i can't reproduce your result DDogg :/ Last edited by Lefungus; 10th September 2003 at 22:03. |
11th September 2003, 02:17 | #7 | Link | |
easily bamboozled user
Join Date: Sep 2002
Location: Atlanta
Posts: 373
|
Quote:
|
|
11th September 2003, 03:16 | #8 | Link | |
Retired, but still around
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
|
Quote:
Last edited by DDogg; 11th September 2003 at 03:18. |
|
11th September 2003, 17:16 | #10 | Link |
Retired, but still around
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
|
Well it is symmetrical if you don't screw up and get a framecount unbalance like I did
One thing for mpg2 users to be careful of is the the two samples must be the exact same frame count (obviously), however, when using a mpv of a verified 5001 framecount, I noticed when using dvd2avi to create the d2v used in mpeg2source("b.d2v") that the framecount became 4998. So it is best to use the trim statement of a slightly smaller value to actually insure the exact frame count and double check it in VDub using file/information. I used mpeg2source("b.d2v").trim(0,4499) to give me an exact 4500 frames. I noticed lafungus has updated ssim to catch this framecount unbalance in the newer version he posted so maybe it will not slip past others like it did for me using the first version. Sidenote:I was gently pointed to the heavy thread in the dvd2avi forum dealing with this very subject where DG is putting in an enormous amount of effort dealing with framecount errors between dvd2avi and mpeg2dec. I don't read that forum much so I was unaware. Thanks! Last edited by DDogg; 11th September 2003 at 20:29. |
11th September 2003, 21:48 | #11 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
In version 0.22, the global score is scaled from 0 to 100 instead of 0 to 1. it's just a simple conversion: newValue = 100 * pow( oldValue, 8)
With this, different values between two clips may be more visible. example on a small xvid sample: high bitrate clip: 82 low bitrate clip: 66 If you prefer it non-scaled, just add "scaled=false" as parameter. For the same sample, it'll give: high bitrate: 0.975 low bitrate: 0.949 Last edited by Lefungus; 11th September 2003 at 21:57. |
11th September 2003, 23:15 | #12 | Link | |
Retired, but still around
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
|
Quote:
|
|
11th September 2003, 23:27 | #13 | Link |
Registered User
Join Date: Jun 2003
Location: Great Lakes, USA
Posts: 1,433
|
Great tool, Lefungus. I've also needed to adjust some framecounts to get the right frames when comparing D2V to AVI. I usually test the script first with a StackVertical and scan through it frame by frame in VDub to make sure the correct frames are corresponding.
__________________
KpeX Audio FAQs: General | BeSweet | SVCD/MP2 | MP3 | Vorbis | AC3 | DTS | AAC Linux Audio/Video FAQ |
13th September 2003, 21:41 | #15 | Link |
Registered User
Join Date: Mar 2002
Posts: 1,075
|
Could probably be sped up 2 times with assembly, more with subsampling ... but I prefer it slow over subsampled. Removing the memory leak I introduced might also help a bit.
PSNR sucks period, and VQM is poor in weighing blocking artifacts. Last edited by MfA; 13th September 2003 at 21:53. |
28th September 2003, 17:13 | #16 | Link |
Registered User
Join Date: Apr 2002
Location: Bretagne
Posts: 192
|
Sh0dan has added conditionnal functions, so the ssim metric could be used in conditionnal scripts.
For example: Code:
a=avisource("c:\blah.avi") b=a.filter() return conditionalfilter(a,b,a,"SSIM_AVG(a,b)","<","0.98",true) It may also be used in a 3-pass encoding scenario, when you filter more encoded frames, that have a too low ssim value. Code:
a=avisource("c:\blah.avi") b=a.filter() c=avisource("c:\blah-2pass-encoded.avi") return conditionalfilter(a,b,b,"SSIM_AVG(a,c)","<","0.94",true) Get the dll here |
28th September 2003, 18:06 | #17 | Link |
Retired AviSynth Dev ;)
Join Date: Nov 2001
Location: Dark Side of the Moon
Posts: 3,480
|
I had a few concepts in mind for "extreme" compression optimizitation:
Case 1: Do an ordinary two-pass encode. Compare the output to the source. If SSIM difference is big, do additional filtering on these frames for a third/fourth pass. Case 2: Compare original frame to a blurred version. This should give a fairly good idea on the amount of high-frequency (spatial) detail level. This value could be used for setting threshold for a filter. PS: There is also conditional functions for retrieving Y,U & V-plane only SSIM difference. SSIM_Y(clip a, clip b) SSIM_U(clip a, clip b) SSIM_V(clip a, clip b) SSIM_AVG(clip a, clip b) = 0.8 * SSIM_Y(a,b) + 0.1 * SSIM_U(a,b) + 0.1 * SSIM_V(a,b)
__________________
Regards, sh0dan // VoxPod Last edited by sh0dan; 28th September 2003 at 18:15. |
30th September 2003, 01:04 | #20 | Link |
Retired, but still around
Join Date: Oct 2001
Location: Lone Star
Posts: 3,058
|
Wow...This sounds like a potential nugget of gold hidden away in this thread. I'm getting all excited. I think you are saying one could compare each frame back to the source and then apply filters to each frame based upon the quality index returned by ssim. Is this correct? I do have a difficult time understanding this stuff.
I wonder if one or both of you could flesh it out a little more and maybe post a more complete example of a test script so folks like me could do some testing. I can't quite get there by myself and could use the help. In my case I would like to try it with CCE 1-pass vbr with two to three filter combos like none, deen, and heavy (maybe deen with light blurring?) In re-reading the above I realize I may be confused as you both mention multipass as if this may be "pass based" conditional versus frame based. Any help you can give me towards understanding the above more fully and getting a working test script would be appreciated. I can do alright with this stuff once someone kickstarts me. Btw, your "get the dll here" above is a new version or is it 0.23? Might be better to just point people to the top of the thread? Last edited by DDogg; 30th September 2003 at 02:57. |
|
|