SSIM / Lumimask For Measuring Codec Quality

[On2Tech]

We've been reviewing the SSIM metric to see if we can use it to help us make our codec better.

Unfortunately, we've found that the lumimask feature as implemented and (as far as we can tell its implemented the way the inventor intended) has at least one serious flaw:

The frame and pixel weighting metrics use the average pixel intensities from 8x8 regions of both the raw and decompressed images to determine the frame / pixel weight.

In effect any 8x8 region in a clip whose raw and decompressed versions sum to an average of less than 40 is given a weight of 0. Unfortunately what this means is that if the source has pixel intensity of 80 and the compressed decompressed version has an intensity of 0 you get a weight of 0.

This is also true on an entire frame. EG if the entire frame has no 8x8 region whose pixel intensities exceed 80, and the decompressed version of the clip is black you will get a frame weight of 0.

you can see an example of this via anonymous ftp to our ftp site:

ssim funny (ftp://ftp.on2.com/ssim.zip)

The example we put up shows an extreme example of what can happen:

ssim.zip contains an avisynth script file and 2 raw clips

- one clip holds 19 black frames and 1 copy of a not so dark image with pixel intensities entirely less than 80

- the other clip holds 20 copies of same darkened image from mobile


Yet the 2 together come out as a perfect match with ssim of 100.0.

I think one way to correct for this problem and still follow the general spirit of lumimasking is to calculate pixel and frame weights from raw clip only.

( Of course this could lead to further problems. like now as long as the source frame is solidly black, the compressed frame could be anything you like )

[Sagittaire]

Pehabs you can patch SSIM plugin with your lumimask ...

lumimasking off
SSIM(source,video,"results.csv","out.txt",lumimask=0)

lumimasking type 1 (old methode)
SSIM(source,video,"results.csv","out.txt",lumimask=1)

lumimasking type 2 (your methode)
SSIM(source,video,"results.csv","out.txt",lumimask=2)

[temporance]

Maybe the designers of SSIM were trying to ensure that their algorithm is symmetric, that is SSIM(a, b) produces the same result as SSIM(b, a).

In my opinion, this is not a worthy aim for a psychovisual metric.

[Manao]

On2Tech : take max(luma(a), luma(b)), it will solve the black frame issue, and it'll still be symmetric

[temporance]

Aren't many of the worse blocking artefacts seen in dark areas / dark scenes? I would have thought sensitivity should have been increased in areas with (luma < 40/256). Instead it seems SSIM is completely ignoring the issue!

[On2Tech]

Originally posted by Manao
On2Tech : take max(luma(a), luma(b)), it will solve the black frame issue, and it'll still be symmetric

Thanks. This does seem to work to solve this problem. I'll try and get source code and dll up for people to try shortly. But first we want to attack another issue.

SSIM makes use of 3 separate metrics, a brightness value, a contrast value and a structural similarity.

The brightness metric uses an equation kind of like this:


2 * meanx * meany
-------------------------
meanx*meanx + meany*meany


and maybe this is working the way it was intended for luminance values ( changes in dark areas stand out more than changes in light) but its affect seems questionable on chroma.

Both u and v are offset by 128, and so there is little to say a low chroma value should be counted less than a high chroma value should be counted less than a high chroma difference. Yet that's what happens with the luma equation:


Taking a specific example of weirdness, the cases where
meanx=1, meany=6 and meanx=254, meany=249 are visually similar,
high saturation with opposite colors.

In the first case, the metric is (2*1*6)/(1+36) = 12/37 < 1/3.

In the second, we get (2*254*249)/(254*254 + 249*249) = 126492/126517
which is close to a perfect match.

This would of course lead to errors in specific colors counting much more than errors in other colors.

One suggested change to this is to simply offset the numbers for luminance errors by a peak value, ie instead of :


2 * meanx * meany
-------------------------
meanx*meanx + meany*meany


we could use :

1- (abs( meanx-meany) / 255)

or even the square of the above...


Any thoughts?

[Sagittaire]

Originally posted by On2Tech
Thanks. This does seem to work to solve this problem. I'll try and get source code and dll up for people to try shortly. But first we want to attack another issue.


yes ... :D

[tritical]

Sorry for digging up an old thread, but I was wondering if anyone had ever tried On2Tech's idea of using

1-(abs(meanx-meany)/255)

instead of

(2*meanx*meany)/(meanx*meanx+meany*meany)

for the luminance part of ssim when calculating ssim on chroma (u/v)? The original equation certainly doesn't make sense for the chroma planes (IMO), which I guess is to be expected since it is for luminance. Using an equation based on absolute difference instead of relative difference makes more sense for chroma (maybe not, I'm not an expert on the sensitivity of the HVS to different u/v value ranges), and is still fast to calculate. Taking the square, which would cause larger differences to have more of an affect, is probably a good idea as well.

Anyways, I was mainly interested in whether anyone had experiences and/or insights applying ssim to u/v? or in combining ssim (used only for luminance) with some other metric for u/v?