SSIM 0.24, an objective video quality metric

[sh0dan]

A "simple" script that chooses between three resizers.

mpeg2source("c:\temp\twinpeaks\pilot\pilot.d2v",idct=6)
crop(18,4,-10,-4)
pre_resize = last
lanczosresize(352,288)
# return last    #use this for first encode

v_org=last
v_comp=avisource("uncompressed.avi").converttoyv12()

soft = bilinearresize(pre_resize, 352, 288).subtitle("soft")
medium = bicubicresize(pre_resize, 352, 288).subtitle("medium")
sharp = lanczosresize(pre_resize, 352, 288).subtitle("sharp")

medium = conditionalfilter(medium, sharp, medium, "sim_diff", ">", "0.97")
conditionalfilter(medium, soft, medium, "sim_diff", "<", "0.96", false)
frameevaluate("sim_diff = ssim_avg(v_org, v_comp)")

First I encoded the file to MPEG1, using the return statement at the top .I didn't have any MPEG2 encoder on this machine, so I had to use MPEG1 and decompress the MPEG 1 -> AVI. MPEG2 is much easier, and doesn't require colorspace conversions.

Then the filter chooses between three different resizers based on the quality measurement. The numbers I've entered seemed to fit my source, but they will most likely have to be adjusted.

[DDogg]

hmmm, thanks, sometime I just think you just like to mess with my head and make me feel completely dumb. I'll have you know I am only partially dumb!

Let's assume we have a progressive source DVD and we have created our original.d2v. I was thinking v_comp=Avisource("original.d2v+Crop&resizingfilters.avs") would be the comparison part very much like a standard ssim comparison. I can't reconcile that with the example you gave (much to my shame). I don't even have a clue what you are doing with v_org=last

In reviewing the above posts on this topic from you and LaFungus it seems you are speaking of two different methods. One uses a comparison source which is an uncompressed AVI that I assume would have only had resizing applied. The other with an example by lafungus uses the original source.

The second one using original source from the d2v is where I am trying to get. I think it is obvious this stuff is a little above my skill set, however if I can understand it I think I could help some other partially dumb folks to do so also. More help would be appreciated.

[sh0dan]

Originally posted by DDogg
Let's assume we have a progressive source DVD and we have created our original.d2v. I was thinking v_comp=Avisource("original.d2v+Crop&resizingfilters.avs") would be the comparison part very much like a standard ssim comparison. I can't reconcile that with the example you gave (much to my shame). I don't even have a clue what you are doing with v_org=last

"pilot.d2v" is the original source. "uncompressed.avi" is the compressed result that has been decompressed to a huffyuv. Exchange the avisource with your compressed d2v. As I wrote I did this for a workaround since I didn't have an MPEG2 encoder.

For creating the first compressed result uncomment the "return last" and encode the result. You might want to adjust the script if you are doing SVCD/DVDR. If you need deinterlacing/ivtc insert it before "pre_resize = last".

The resize filters are just as an example - you can put additional filters inthere.

"v_org = last" just assigns the current video (after the lanczosresize) to v_org, as this has to be used for the SSIM comparison later.

[mf]

Your Subtitle()s are impairing your SSIM result .

[sh0dan]

No - because neither of these three sources are used for the compare (v_org is compared to v_comp).

[ronnylov]

I have following idea:

1) Encode without filter.

2) Encode the same source with a filter

3) Use SSIM and compare 1) with original and then compare 2) with original on each frame

4) Make a final third encoding and select filter or not filter on each frame of the original depending on if ssim on 2) was better than ssim on 1) or not.

I think this could be useful when making VCD (or any other CBR encoding format) where the bitrate is fixed and cannot vary with the quality of the input. If it's applied on a VBR encoding then the bitrate distribution makes the final result more unpredictable I think.

Do you think this could be a good method to improve the quality and can it be done with ssim and avisynth? Could it also be done with more than two encodings with serveral levels of filtering, always choosing the best filter combination on each frame depending on which encoded clip that had best ssim value?

I got the idea after reading shodan's rezising selecting script. But his script does not check the results after encoding when another resizer is applied and I want to select the best one on each frame based on results after encoding. But I don't exactly know how to code it in avisynth.

It would require several encoding passes (at least 3, two to compare with the original and one final) and is a little bit time consuming but if quality is very important maybe this does not matter.

[slk001]

How about attaching a "README.TXT" with the proper usage/syntax to the .DLL download?

[w2e3r4]

I have 3 questions:

1. What is the reason for scaling the result in the new version?
newValue=pow(oldValue,8)

2. How do I use ssim filter without actually doing the encoding and saving the avi file in VurtualDub? Is there an option to discard the output?

3. What colorspace should I use for the comparison?

Thanks,

[Manao]

Originally posted by w2e3r4
1. What is the reason for scaling the result in the new version?
newValue=pow(oldValue,8)

It's just to make the range of variation of the value larger. For example, if oldValue was varying between 0.94 and 0.96, newValue will vary between 0.61 and 0.72. It's just easier to read the value that way, because with the previous version, values were ranging roughly from 0.92 to 0.99.

2. How do I use ssim filter without actually doing the encoding and saving the avi file in VurtualDub? Is there an option to discard the output?

Open your script in VDub, and hit the 'play' button ( or space key ). When the processing of the clip is finished, close the clip ( ctrl - w )

3. What colorspace should I use for the comparison?

YV12

[HLW]

Lefungus, this seems to be a great tool! I am using it for my master thesis where I compare different codecs. But I need to document what is happening in the algoritm. Is it possible to get information of what weighing values you use in the algorithm? I`ve read the "Video Quality Assessment Based on Structural Distortion Measurment" by Zang, Lu and Bovik. Have you used the values suggested by them?

By the way, any plans of implement support for more colorspaces, for instance 4:2:2?

Best regards

Haakon

[MfA]

As they say ... the code is it's own documentation

But it uses the .8 .1 .1 weighting for the planes, luminance based weighting is implemented as suggested in the article (note that the SSIM results are the weighted average of SSIM values for blocks/frames though, so dark frames might have a low SSIM value anyway ... you have to look at the frame weight to see that that is not considered important). Motion based weighting is not performed at all.

4:2:2 would be easy to add, why dont you do it? (Be carefull though, this metric is really not designed for interlaced content ... although it will do in a pinch, but you should use seperatefields before applying it.)

Marco

[HLW]

I`m afraid thats outside my knowledge, but if anyone is capable of it and wants to do it, I would be happy! . However, I have another comment: I've tested SSIM with the same clip both as target and reference. And the strange thing is that I don't get SSIM value 100 as I would expect, but only something close (96-98, depending on the clip). Does anyone have comments on that?

[MfA]

A comment ... hmmm, I should test my code better?

It is a bug, Ill take a look.

[Manao]

if(lumimask)
{
  if (luminance)
  {
    frameSummedWeights = 0;
    for (y = 0; y < height / 2; y++)
	for (x = 0; x < width / 2; x++)
	  *(weights + y * strideUV + x) = 0.0f;
  }
}
else
  frameSummedWeights = (height - 7) * (width - 7);

If lumimask = true, and you're treating the chroma, you're not initializing properly the weights.

[MfA]

That's part of it, I just forgot for a moment that there are only (height-7)*(width-7) SSIM values in a luminance plane. So the divide coordinates by 2 trick will not map 4 weights in the luminance plane to 1 weight in the chroma plane.

[MfA]

Okay there is a slightly updated version here :

http://home.student.utwente.nl/m.f.al/ssim/

Ill verify if it gives the same results as the old one without lumimasking tomorrow ... and proof read it again. If anyone else wants to take a look Ill be glad for any help

[TorgoGuy]

Originally posted by MfA
Okay there is a slightly updated version here :

http://home.student.utwente.nl/m.f.al/ssim/

Ill verify if it gives the same results as the old one without lumimasking tomorrow ... and proof read it again. If anyone else wants to take a look Ill be glad for any help

I have only tried this test version and not the 0.23 version, but I think I have a bug to report.

AVS File:
=========

LoadPlugin("m:\proj\_plugins\ssimtest.dll")
a=AviSource("lotr1.avs")
b=AVISource("def.avi")
c=AviSource("GMCVHQ4.avi")
StackVertical(ssim(a,c,"ls_GMCVHQ4.csv","ls_GMCVHQ4.txt",lumimask=true),ssim(b,c,"ld_GMCVHQ4.csv","ld_GMCVHQ4.txt",lumimask=true))

Problem:
========
In the .csv files most lines look normal (with SSIM values and weight), but there are many blocks where the lines are just:

0.000000,0.000000
0.000000,0.000000
0.000000,0.000000
0.000000,0.000000

(that could be normal--not sure). The bigger problem seems to be that there are also many lines that look like this:

-1.#IND00,0.000009
-1.#IND00,0.000173
-1.#IND00,0.000559

where the SSIM column is always the same and the weight column varies somewhat.

Also, looking in the average SSIM file, it looks like this:

SSIM: Structural Similarity Index Metric 0.23
Average SSIM= -1.#J

Am I doing something wrong?

[MfA]

No, more bugs ... not dealing correctly with frames with near 0 weight on my part.

One more try, lets see if it works this time :

http://home.student.utwente.nl/m.f.al/ssim/

[TorgoGuy]

Thanks for the update. I'm rerunning the problem files. So far it looks like that bug is gone. All of the "0.000000" and "-1.#IND00" lines have been fixed so far.

I have a question regarding how the average value is calculated when using lumimasking. You explained that without lumimasking it is simply 100 * Pow(AvgSSIM, 8). With lumimasking, in place of "AvgSSIM" in the equation above do you use: ((the sum of (FrameSSIM * FrameWeight))/(the sum of the weights))?

Thanks a lot. To my eyes this (SSIM) does seem to be a very good quality measurement. Excellent work!

[TorgoGuy]

Originally posted by TorgoGuy
Thanks for the update. I'm rerunning the problem files. So far it looks like that bug is gone. All of the "0.000000" and "-1.#IND00" lines have been fixed so far.

Just a note to tell you that the problem files have been completed and the new version worked perfectly.