PSNR-SSIM volatility

[montython]

I have been playing around with various codecs and objective quality measures for a while. I noticed that samples encoded with particular codecs (namely vp7 and rv40) exhibit significantly higher psnr & ssim volatility. Especially for vp7, I have observed extreme cases...

It is true that volatility in such measures is something very natural. But the peculiar thing is that the jumps are in the form of periodic oscillations in large ranges. Probably "seasonality" is a better term to define this behavior. As far as I can see this has nothing to do with keyframe intervals or structure of I-P-B frame sequences. But for sure it is periodical/seasonal. Because it is not something random, I am not sure if this can be attributed to instability of a codec.

Maybe intentional? But, why?

I cannot think of a reason why such a codec behavior should be desired. Of course it doesn't ruin the overall metrics, but you end up with a strange distribution.

In terms of objective quality measures, is it reasonable to consider higher moments (variance for example) and penalize the variability in the distribution? Are there any quality measures implementing this? How does this behavior reflect to human visual perception?

[akupenguin]

As far as I can see this has nothing to do with keyframe intervals or structure of I-P-B frame sequences.
Are you sure? VP7 doesn't have B-frames, but it does have non-referenced P-frames whose quant can be increased for an effect similar to B-frames. I'm not sure those would show up with whatever method you used to examine I-P-B patterns.

[Sagittaire]

In fact it's a well know problem with Sharpenig setting: In fact there are a large local quant variability. This quant variability imply a big local quality flicking. If you don't want see this problem you must set "Sharpening strenght" at the lower value.

[montython]

To be more concrete on the subject, this is a demonstration based on a short sample.

DESCRIPTION:
This is not meant to be a comprehensive codec test, but rather an experiment on psnr volatility. The reference is a 512x288, 20-seconds sample trimmed from a movie trailer. Includes scene cuts, fades.

rv40 encoder: Helix Producer 11 (VBR, 2-pass, high encoder complexity)
vp70 encoder: VP70 VFW Codec - Personal Edition (2-pass, best quality, default settings for the rest)
x264 encoder: x264-r650 cli (2-pass, custom settings)
pass-1 settings:
--subme 1 --partitions "none" --bframes 3
--progress --no-psnr --no-ssim
pass-2 settings:
--ref 5 --bframes 3 --b-pyramid --weightb
--partitions "all" --8x8dct --direct "auto"
--me "umh" --subme 6 --b-rdo --mixed-refs --bime
--progress --no-psnr --no-ssim

For testing purposes, a low bitrate (approximately 265kbps) was chosen. The size of video streams in avi, rmvb and mp4 containers are approximately the same.

Second-pass encoding speeds were exactly the same for x264 and rv40 codecs. Encoding speed of vp70 with the best quality settings was about 2.75 times slower compared to the other two. For the first pass: Encoding times of x264 and vp70 were about the same, rv40 was slightly slower.

RESULTS:
http://i231.photobucket.com/albums/ee166/img_f/psnr.png
http://i231.photobucket.com/albums/ee166/img_f/psnr_ma10.png

It should be evident from the first graph that vp70 and rv40 series exhibit periodic fluctuations, which is a kind of "seasonality" effect. For the vp70 series, the range of fluctuations is larger and the peaks occur at every 8th frame. For rv40, the range is narrower and a common interval for peaks seems to be 4.
In the second graph, 10-period moving average series are used to compensate for the "seasonal" effect. This one should give a better idea of the trends throughout the sample.

For a closer examination of the fluctuation behavior, the last part of the sample (last 100 frames) is a good example: Peaks at every 8th frame (vp70) and peaks at every 4th frame (rv40) should be more clear in this subsample. I have no idea why this is the case, but 4 and 8 seem to be sort of magic numbers for these two codecs.

http://i231.photobucket.com/albums/ee166/img_f/psnr_subsamp.png

For those who are familiar with statistics/econometrics, the following is a rather simple regression analysis.

http://i231.photobucket.com/albums/ee166/img_f/x264.png
http://i231.photobucket.com/albums/ee166/img_f/rv40.png
http://i231.photobucket.com/albums/ee166/img_f/vp70.png

A brief explanation for reading the tables:
"AR" notation is used for auto-regressive terms. In econometrics, AR terms are used to examine periodic effects in a sample. For example, AR(2) term is used to check a periodic effect at every 2 periods. High t-statistic values (hence smaller probability) indicate significant effect. A probability value close to zero (lower than 0.05 for example) is considered to be a signal for the presence of a periodic effect for that term. The coefficient columns in the tables show the magnitude and the direction of the effect.

x264:
There is no number clearly standing out. Up to 6 frames, there seems to be evidence for blended relation.

rv40:
4 (with a large t-statistic) seems to be the magic number here by far. 1, 5 (and also 6) seem to be significant, but not as strong as 8.

vp70:
The extreme case is observed here. Every 8th frame has an important (??) role here. Also there seems to be a weaker effect for AR(1). This is probably due to references between consecutive frames, which is the type of behavior expected from almost all codecs.


p.s.
The peaks are not related to keyframes. The number of keyframes in the vp70, x264 and rv40 encoded material are 5, 4 and 3 respectively.

-------------------------------------------------------------------------
In fact it's a well know problem with Sharpenig setting
Sharpness setting of vp70 codec is zero (default value) in all my tests.

VP7 doesn't have B-frames, but it does have non-referenced P-frames whose quant can be increased for an effect similar to B-frames.
Do you think this is somehow related to the "every 8th frame" thing demonstrated above?

[Dark Shikari]

rv40 encoder: Helix Producer 11 (VBR, 2-pass, high encoder complexity)
vp70 encoder: VP70 VFW Codec - Personal Edition (2-pass, best quality, default settings for the rest)
x264 encoder: x264-r650 cli (2-pass, custom settings)
pass-1 settings:
--subme 1 --partitions "none" --bframes 3
--progress --no-psnr --no-ssim
pass-2 settings:
--ref 5 --bframes 3 --b-pyramid --weightb
--partitions "all" --8x8dct --direct "auto"
--me "umh" --subme 6 --b-rdo --mixed-refs --bime
--progress --no-psnr --no-ssim
This is somewhat unfair--you're using "best quality" settings on all the encoders except x264.

Use more refs and subme 7 at a minimum, and drop in --bframes 16 so the encoder can use however many it wants. And get a more recent build; 681 for example raised quality-per-bitrate by about 0.7% when subme 7 is used.

[montython]

Dark Shikari, this was not meant to be a standard codec comparison. (A 20-seconds sample is too short for this purpose anyway.) I just wanted to demonstrate the "peculiar" periodic oscillations I have been observing. I included x264 as a "well-behaved" reference.

I am aware of the more recent x264 builds and I know that even older builds can perform better. The settings I used for x264 are moderate settings achieving quite high quality.

By the way, for this sample in overall PSNR x264 beats the other two anyways:

x264: 39.9468
vp70: 39.7432
rv40: 38.6905

[Dark Shikari]

I just wanted to demonstrate the "peculiar" periodic ossillations I have been observing. I included x264 as a "well-behaved" reference.
x264: 39.9468
vp70: 39.7432
rv40: 38.6905
Yup, I noticed this also with VP7 when I tested it out. It overall is pretty close to x264 but its instability is somewhat unnerving. Its SSIM jumped all over, from as low as 0.8 to as high as 0.99, in a single clip I fed it.

Of course the real kicker is that in my decoding tests, FFDShow w/x264 was faster than VP7 by about 5%, ensuring that VP7 had absolutely no benefit over x264 at all.

RV40, of course, gets trounced.

[zambelli]

SSIM/PSNR have their "soft spots"... For example, they both tend to prefer soft images over sharp images with occasion artifacts. Also, they don't respond well to perceptual optimizations such as lumimasking or adaptive deadzone adjustment. But for all their flaws, they're still very useful metrics and serve as good tools for spotting anomalies.

Speaking of VP7... I don't envy On2 right now. H.264 momentum is huge and between H.264, MPEG-4, VC-1 and MPEG-2 the codec field looks pretty well saturated. I can't imagine a proprietary codec getting much of that pie.