Tips for comparing codecs with SSIm and PSNR

[sbeyer20]

Hi,

I want to do some quality test with Mpeg4 and AVC, but I have some problems with the quality tests. PSNR ist the problem with avisynth. But when I tried to do it the same way with SSIM I got some errors because the original and the compressed files are not of the same lenght.

Why do I get extra frames with AVC and ASP? And how to cut them?

I know that there are some of you which have allready done such test - could you please give me some advices?

Thanks

[Doom9]

First of all: don't cut, it's only going to give you grief. Second of all, which containers are you using? Putting AVC into AVI for instance introduces delay and drops some frames.. but that's a determinable constant, (which you can figure out even not knowing the specifics.. just open source and destination in a media player like MPC that allows frame accurate forward and backward jumps and figure out the offset), which you can then compensate for (wrap source and destination into an AviSynth script and use delay options to make frame X in the source match frame X in destination A and B).

[sbeyer20]

Thanks for this quick answer.

For quality tests I want to use the avi container (At the moment I use avisource for this ...or does the mp4 container work with directshowopen? I havent tested it)- for performance test I want to use mplayer and mp4 container.

I realy wondered why SSIM got this failure and why PSNR is infinitive at the first five or six frames. I also testesd the MSU Tool but I got some strange PSNR values in combination with HD content.

Do I have to look into every avi for the delay or dropped frames? I have planed to make very much encodes to test the features of x264 and XviD....

[Doom9]

Do I have to look into every avi for the delay or dropped frames?Only those that use b-frames, and it's different for every codec (unless you use the same decoding filter). And it's DirectShowSource - it handles everything DirectShow is able to open = everything a direct show based media player (like mpc) can open..

[stephanV]

For AVC it is really recommendable to use MP4 or Matroska with DirectShowSource(). There is not only a problem with a decoding delay and dropped frames (if you use VFW to encode), the last time i checked ffdshow also made a slight jump the first time it encounters a AVC b-frame in AVI. Apparently the decoder needs to know if b-frames are used.

If you want to use MPEG4 ASP with b-frames in AVI, use packed bit stream. This ensures you have no decoding delay. (so you basically have to worry about nothing).

why PSNR is infinitive at the first five or six frames.
Infinitive means there is no difference between the compared frames. Maybe the first few frames are black?

[sbeyer20]

Hi,

I have tested the MP4 Container with DirectShowSource and...it works fine. There is no offset or dummy frames. My Input is an ImageSource with 1080P uncompressed. Could it be that SSIM doesn´t work very well with this resolution?

Does Directshow manipulate the picture in any way? I wonder because the first pictures are black (as some of you already guess) and should be complete identical but PSNR is only around 40. Strange.

[stephanV]

Some renderers seem to increase the brightness just a little bit, this would definetly have an effect on PSNR.

[sbeyer20]

I use FFDSHOW as DirectShow Filter. Does anyone know if it affects something?

Could it be that Compare in combination with ConvertToYUY2 give different results than CompareYV12 ???

[sbeyer20]

Hi,

here I am again.

I now have tested about tenn clips and I wonder because they only differ in the numbers after the komma. (PSNR) Strange.

Because my original input is a series of tifs, I have to convert the colorroom.
But with different conversions I got different testresults. I tried to convert the Tif with ConvertToYuY2 and the MP4 too. But I have also tested ConvertYV12 - where I get different Results. Its the same with ConvertToRGB.

Why does this happen? Which Conversions would you suggest?

PS DirectShowSource works very well with MP4 Container as longs as you don´t shuffle.

[stephanV]

Hi,

here I am again.

I now have tested about tenn clips and I wonder because they only differ in the numbers after the komma. (PSNR) Strange.

Which codecs are you comparing (and what settings)? For testing between MPEG4 ASP codecs (XviD, DivX) this is perfectly normal.


Because my original input is a series of tifs, I have to convert the colorroom.
But with different conversions I got different testresults. I tried to convert the Tif with ConvertToYuY2 and the MP4 too. But I have also tested ConvertYV12 - where I get different Results. Its the same with ConvertToRGB.
How is this weird? Colorspace conversion are not lossless, certainly not in your case since YV12 ~ 12 bit, YUY2 ~ 16 bit and RGB = 24 bit. So all will hold different information and results will differ a small bit.

Which Conversions would you suggest?
All codecs normally use YV12 inside, so ConverttoYV12 seems the most logical one.

DirectShowSource works very well with MP4 Container as longs as you don´t shuffle.
Shuffle = seeking?

Why would you need to seek doing a PSNR test?

[sbeyer20]

Hi,

yes Shuffle means seeking. When you work with Professionel Equipment you dont say seeking...

I tested three different sources with stackhorizontal to look if theye are exact frameaccurate.

My source is an uncompressed 1080P Tiff Sequenze from an AVid HD DS, so my source colorspace is RGB. But H.264 uses YV12. So I have to convert anyway...

I tested different setings with h264 at 6000kbps. Numbers of B-frames , Ref-Frames, Cabac, etc....but all PSNR results are between 42,7 and 42,3 - So very little difference.

[stephanV]

Hi,

yes Shuffle means seeking. When you work with Professionel Equipment you dont say seeking...
Could be, shuffling doesnt exactly cover the meaning of what you do though... shuffling makes me think of cd tracks in random order... :D



I tested different setings with h264 at 6000kbps. Numbers of B-frames , Ref-Frames, Cabac, etc....but all PSNR results are between 42,7 and 42,3 - So very little difference.
Thats probably not abnormal. What happens if you change the bit rate to say, 3000 kbps? Note that a difference of 0.4 dB is considered to be quite big though.

[sbeyer20]

If I have the time I would also like to do some test with 3 or up to 2 Mbps. But this means having much time. Every Encode works with only 1fps and the PSNR test also are very slow. And this at an A64 3200.

It is very interesting to see the results which are very close - but the decoding times are very different.

Offtopic: Nearly every good Tape based System has an combined Shuttle/Jog-Wheel - so everybody says "shuttle" - so I also do say shuttle. But I think it´s my fault because I wrote shuffle and not shuttle. Sorry.