Encoding quality check

[mparade]

I would like to check my encoded main video structures made by BD-RB using various CRF values to know which fulfill my quality requirements. I am not really on top of the topic, so I would need someone's help with some experience in quality checking procedure. I have already prepared the files for checking with CRF values of 16,17,18,19, 20. The source is my Dark Knight Rises BD. I have had two ideas for quality check so far: 1. make an avisynth script to load the original and and encoded source directly one under the other besides cropping the black borders of the frames inside the script as well and check them on a computer monitor. 2. Buy two identical BD player
and two identical display device which are big enough to see any quality differences between the sources. One of them would be displaying the source being not encoded while the other the ones to compare with.

Any help (any practical and reasonable quality comparison procedure) would be appreciated. Please sorry for my "laymanity". :)

Thanks for the efforts in advance.

[Sharc]

You could also make an avisynth script which interleaves 2 clips. Then you can step through the frames and compare the pairs sequentially fullscreen.

[mparade]

You could also make an avisynth script which interleaves 2 clips. Then you can step through the frames and compare the pairs sequentially fullscreen.

Yes, thank you I will do that on a relatively big display...I think it is even better than if I were to use Stackhorizontal or Stackvertical...My only problem with this method is if I am jumping around (not playing end-to-end) and looking at the frames the play goes out of sync and then it moves back and forth throughout..

If I use trim to restrict the framecount to play the same has happened in Virtualdub. In addition the play of the sources became disturbing (vibrating) too (due to the sync issue). Anyway, I am using virtualdub to handle the script. My current avs file is as follows:

LoadPlugin("E:\Users\X\Downloads\dgavcdec109\DGAVCDecode.dll")
a = DirectShowSource("M:\BD-50\THE DARK KNIGHT RISES\DISC1\DARK_KNIGHT_RISES\BDMV\STREAM\00020.m2ts")
b = DirectShowSource("F:\BDMV\STREAM\00000.m2ts")
Interleave(a,b)

[Guest]

No need to load DGAVCDecode.dll in that script.

[jdobbs]

Yes, thank you I will do that on a relatively big display...I think it is even better than if I were to use Stackhorizontal or Stackvertical...My only problem with this method is if I am jumping around (not playing end-to-end) and looking at the frames the play goes out of sync and then it moves back and forth throughout..

If I use trim to restrict the framecount to play the same has happened in Virtualdub. In addition the play of the sources became disturbing (vibrating) too (due to the sync issue). Anyway, I am using virtualdub to handle the script. My current avs file is as follows:

LoadPlugin("E:\Users\X\Downloads\dgavcdec109\DGAVCDecode.dll")
a = DirectShowSource("M:\BD-50\THE DARK KNIGHT RISES\DISC1\DARK_KNIGHT_RISES\BDMV\STREAM\00020.m2ts")
b = DirectShowSource("F:\BDMV\STREAM\00000.m2ts")
Interleave(a,b)Unfortunately visual inspection is too subjective and also very time consuming. But no matter how you do it there will be someone who explains why it isn't a good method.

I would suggest using the PSNR (peak signal-to-noise ratio) available in AVISYNTH's Compare() filter. You feed the newly encoded video and the original into AVISYNTH at the same time and it calculates the PSNR. You can even output to a file so you can collect the data for statistical analysis (if you're feeling ambitious). Like everything else it isn't perfect but it is probably about as close as you're going to get.

[raffriff42]

My only problem with this method is if I am jumping around (not playing end-to-end) and looking at the frames the play goes out of sync and then it moves back and forth throughout..You need to encode that script with Huffyuv, Lagarith, UTVideo or some other lossless, intra-only codec in order to get random access (jumping around). You will probably want to Trim out one or more short segments to save time.
Trim(0, 1000) + Trim(30000, 31000)

[jdobbs]

Here is an example of PSNR testing I did recently to compare two encoders using similar input settings:U=7
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5778 1.7554
Mean Deviation: -0.2865 -0.0560 +0.1374
PSNR: 38.3900 49.3840 113.0590
Overall PSNR: 47.2469
U=6
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5727 1.8617
Mean Deviation: -0.2824 -0.0558 +0.1501
PSNR: 37.8044 49.5623 113.0590
Overall PSNR: 47.3430
U=4
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5394 1.7126
Mean Deviation: -0.2688 -0.0562 +0.1314
PSNR: 38.7745 49.9945 113.0590
Overall PSNR: 47.9252
U=2
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5183 1.6034
Mean Deviation: -0.2520 -0.0521 +0.0933
PSNR: 39.3200 50.3035 113.0590
Overall PSNR: 48.3147
U=1
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4951 1.5673
Mean Deviation: -0.2369 -0.0396 +0.0829
PSNR: 39.5391 50.6595 113.0590
Overall PSNR: 48.6991
======================================================
Ultrafast
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.6671 1.6027
Mean Deviation: -0.2224 -0.0260 +0.2059
PSNR: 39.4357 48.1060 113.0590
Overall PSNR: 46.3559
Medium
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4568 0.9306
Mean Deviation: -0.2536 -0.0321 +0.0340
PSNR: 43.1644 50.5743 113.0590
Overall PSNR: 49.0122
Slow
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4498 0.9232
Mean Deviation: -0.2435 -0.0337 +0.1690
PSNR: 43.0575 50.6748 113.0590
Overall PSNR: 49.0893
Medium 2 pass
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4439 0.8791
Mean Deviation: -0.2291 -0.0320 +0.0435
PSNR: 44.3512 50.7754 113.0590
Overall PSNR: 49.3210

[mparade]

Here is an example of PSNR testing I did recently to compare two encoders using similar input settings:U=7
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5778 1.7554
Mean Deviation: -0.2865 -0.0560 +0.1374
PSNR: 38.3900 49.3840 113.0590
Overall PSNR: 47.2469
U=6
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5727 1.8617
Mean Deviation: -0.2824 -0.0558 +0.1501
PSNR: 37.8044 49.5623 113.0590
Overall PSNR: 47.3430
U=4
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5394 1.7126
Mean Deviation: -0.2688 -0.0562 +0.1314
PSNR: 38.7745 49.9945 113.0590
Overall PSNR: 47.9252
U=2
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5183 1.6034
Mean Deviation: -0.2520 -0.0521 +0.0933
PSNR: 39.3200 50.3035 113.0590
Overall PSNR: 48.3147
U=1
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4951 1.5673
Mean Deviation: -0.2369 -0.0396 +0.0829
PSNR: 39.5391 50.6595 113.0590
Overall PSNR: 48.6991
======================================================
Ultrafast
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.6671 1.6027
Mean Deviation: -0.2224 -0.0260 +0.2059
PSNR: 39.4357 48.1060 113.0590
Overall PSNR: 46.3559
Medium
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4568 0.9306
Mean Deviation: -0.2536 -0.0321 +0.0340
PSNR: 43.1644 50.5743 113.0590
Overall PSNR: 49.0122
Slow
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4498 0.9232
Mean Deviation: -0.2435 -0.0337 +0.1690
PSNR: 43.0575 50.6748 113.0590
Overall PSNR: 49.0893
Medium 2 pass
Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.4439 0.8791
Mean Deviation: -0.2291 -0.0320 +0.0435
PSNR: 44.3512 50.7754 113.0590
Overall PSNR: 49.3210

Thank you, jdobbs, I really appreciate your help and I would like both to collect the data for statistical analysis and I am feeling ambitious, but could you please explain a bit more how to evaluate those values? There is nothing about the evaluation process in the avisynth's corefilters\compare.htm file. I have found and read an other thread on PSNR discussion from 2003 but still do not know what intervals should be considered as e.g. extremely good, very good, good, acceptable etc...Your medium and slow results seems to be extremely good. I have started testing (currently running the PSNR test of The Dark Knight Rises (236705 frames) at the CRF of 17 but I am far away from your results:

Minimum Average Maximum
Mean Absolute Deviation: 0.0000 0.5860 2.1021
Mean Deviation: -0.1443 +0.0351 +0.3783
PSNR: 38.6744 48.9061 113.0590
Overall PSNR: 48.0616

These were the results at CRF=18.

I have just made this avs file to feed into virtualdub to play (during the playback the attached compare.txt was being made).

LoadPlugin("E:\Users\X\Downloads\dgavcdec109\DGAVCDecode.dll")
clip1 = AVCSource("E:\Users\X\Desktop\Test\DARK_KNIGHT_RISES_EREDETI.dga")
clip2 = AVCSource("E:\Users\X\Desktop\Test\DARK_KNIGHT_RISES_CRF=17.dga")
Compare(clip2,clip1,"","compare.log",false)

Thank you in advance for your help!

[mparade]

No need to load DGAVCDecode.dll in that script.

Thanks, it will be deleted for my future scripts. :)

[mparade]

You need to encode that script with Huffyuv, Lagarith, UTVideo or some other lossless, intra-only codec in order to get random access (jumping around). You will probably want to Trim out one or more short segments to save time.
Trim(0, 1000) + Trim(30000, 31000)

Thank you for your help! I really appreciate it!