How to measure quality between source and final video?

[NoX1911]

Hi,
is it possible to measure the quality difference between source and reencoded video? Maybe by simply comparing Y,U,V values giving an average value at the end?

Any tools known?

[skystrife]

PSNR, SSIM, etc. (http://compression.ru/video/quality_measure/vqmt_manual_en.html has a good tool for comparing with a variety of metrics.)

But nothing is as valuable as your eyes, especially with VAQ and PsyRDO out in the wild. They may hurt metrics but improve quality.

The best method is to compare multiple frames by hand with your eyes, but a metric isn't ALWAYS a bad measurement, it's just sometimes not as visually pleasing as something of a lower metric but similar complexity.

[Zwitterion]

Your eyes >>> SSIM (http://forum.doom9.org/showthread.php?s=&threadid=61128) > PSNR (http://avisynth.org/mediawiki/Compare)

[Comatose]

I just do something like this:
avisource("lossless.avi")
interleave(subtitle("lossless"),dss2("h264-encode.mp4"))

and compare them in AvsP.

[Adub]

Use your eyes. If you don't think this is accurate enough, then you don't need to compare quality!

[NoX1911]

I'm aware of human perception but i'm also searching for the best objective, strict mathematical, fuzzy logic, world dominating solution possible. Sometimes i just need numbers.

SSIM and PSNR (avisynth 'compare' function) look promising.

Nice idea, Comatose. Will keep it as well.

[Sagekilla]

If you're looking from a purely mathematical point of view, neglecting the usage of any PSY-RD enhancements (since they make comparing objective metrics useless) your best bet is SSIM, followed by PSNR.

[Adub]

Yes, and this:
neglecting the usage of any PSY-RD enhancements (since they make comparing objective metrics useless) is key to remember, NoX1911.

[Sagekilla]

By neglecting I mean don't bother using psy-rd in any tests you want to make a objective comparison, since they throw things out of whack with PSNR/SSIM.

[Sharktooth]

metrics do not measure quality... only differencies.
one day i will show you why metrics are not good to measure quality...

[linyx]

I would also suggest using your eyes
I use something like this in MPC
Source=AviSource("Movie.avi")
Reencoded=AviSource("Encoded.avi")
StackHorizontal(Source,Reencoded)

Or maybe
Source=AviSource("Movie.avi").SelectEven()
Reencoded=AviSource("Encoded.avi").SelectOdd()
OverLay(Source,Reencoded,mode="blend",Opacity=0.5)

[J_Darnley]

Or maybe
Source=AviSource("Movie.avi").SelectEven()
Reencoded=AviSource("Encoded.avi").SelectOdd()
OverLay(Source,Reencoded,mode="blend",Opacity=0.5)

Now that's a strange thing to do. You how do you compare anything there? You take frame n and average it with frame n+1 and then the next output frame is the average of n+2 and n+3. Perhaps you mean just a simple Interleave() with the unmodified videos? Then you can easily switch between frame n in one video and frame n in another.

[Sagekilla]

@Sharktooth, I realize they are not good for measuring quality, I was merely pointing out the best possible situation that one should even consider using them ;) I never use them as a measurement of how well my videos come out, instead opting the good old visual analysis.


On that note, the following script works well too if you use AvsP:


a = Avisource("clip1.avi").Subtitle("Source")
b = Avisource("clip2.avi").Subtitle("Encoded")
Interleave(a,b)

[Sharktooth]

oh... mine was a general note. it was not directed to you.

[linyx]

Now that's a strange thing to do. You how do you compare anything there? You take frame n and average it with frame n+1 and then the next output frame is the average of n+2 and n+3. Perhaps you mean just a simple Interleave() with the unmodified videos? Then you can easily switch between frame n in one video and frame n in another.
:devil::stupid: I couldn't remember what the command was, so i just guessed with overlay. Yes, I meant a simple Interleave().

[Comatose]

On that note, the following script works well too if you use AvsP:


a = Avisource("clip1.avi").Subtitle("Source")
b = Avisource("clip2.avi").Subtitle("Encoded")
Interleave(a,b)

You are probably who inspired me to use this method :P

[Sagekilla]

Heh, unlikely ;) this method floats I believe. I just prefer it over StackX because I find those two hard to compare differences in the same area.

[Comatose]

Yup :p Switching between them this way makes even very small differences apparent >:3

[NoX1911]

I try to visualize per-pixel differences of all channels with subtract() but its not working very well at the moment since i cannot figure out the center of luma and chroma channels of subtract(). I tried several ranges (0..255, 16..235) but either pixels are clipped or too bright.

Subtract() outputs 50% gray if there is no difference between pixels.

There's maybe a ~1:2 ratio between diff values of subtract and real values. I will give it another shot later. If you can fix it the meantime feel free to post.

# Luma 16..235 // Chroma 16..240
# Subtract() 50% gray is 126 (by manual)

clip1=ffmpegsource("anchor.mpg")
clip2=ffmpegsource("anchor.mp4")
HiDiffY=Subtract(clip1, clip2).ColorYUV(off_y=129).invert ### 126+129=255 // 129 = too high (clipping)
LoDiffY=Subtract(clip1, clip2).ColorYUV(off_y=-126) ### 126-126=0
DiffY=Overlay(HiDiffY,LoDiffY,mode="Add")

HiDiffU=Subtract(clip1, clip2).ColorYUV(off_u=129).invert.UtoY
LoDiffU=Subtract(clip1, clip2).ColorYUV(off_u=-126).UtoY
DiffU=Overlay(HiDiffU,LoDiffU,mode="Add")

HiDiffV=Subtract(clip1, clip2).ColorYUV(off_v=129).invert.VtoY
LoDiffV=Subtract(clip1, clip2).ColorYUV(off_v=-126).VtoY
DiffV=Overlay(HiDiffV,LoDiffV,mode="Add")

clip3=Overlay(DiffU,DiffV,mode="Add").pointresize(DiffU.width*2,DiffU.height*2)
clip4=Overlay(clip3,DiffY,mode="Add")

#DiffV
clip4

Levels(0, 1, 64, 0, 255)

[708145]

Yup :p Switching between them this way makes even very small differences apparent >:3

Well you can find differences in stills very well but what about differences in motion and in your normal viewing conditions?

What I did back when I was active in video encoding *sigh@no_time_any_more* was to generate an avs script that displayed on the left a random input clip and on the right a random input clip. So you get all 4 combinations at random points in time. When I can't make out the events when a switch occurs when viewing at my normal viewing distance at normal lighting then I call it transparent and live with it.
I hope this was understandable.

bis besser,
T0B1A5

[Comatose]

The thing is that aiming for the best quality you can achieve in stills is sure to improve normal viewing quality as well, and usually by more since you set your goal higher.
If you have limitations such as bitrate or something, then you might have to just give up and live with what you have, but sometimes you can reach very high quality when aiming for only insignificant differences in stills.