GMSD and SSIM Quality Metrics

[WorBry]

Originally Posted by WorBry

I see there's a GUI for rav1e....
https://github.com/moisesmcardona/rav1e_gui

I've checked out the rav1e GUI (v1.8r2) and just can't get an encoded AV1 file (webm or mkv) out of it.

Tried encoding the CrowdRun 1080/50p 'master' (lossless, intra x264) clip at the default settings - Quantizer 100, Speed 3, Quality Tuning - PSNR. It proceeded to encode (slowly) and finally declared 'Finished', but there was no output file to be found.

I'll maybe try the command-line route. Otherwise, if you want to run the tests with that particular source, I used the 2160/50p (8bit 420) y4m version of Crowd Run from:

https://media.xiph.org/video/derf/

....as the source/reference for the 2160/50p tests. For the 1080/50p tests, I converted to lossless, intra x264:

ffmpeg -i {Path}:/crowd_run_2160p50.y4m -vf scale=1920:-1 -vcodec libx264 -intra -preset slow -qp 0 {Path}:/crowd_run_1080p50_x264_lossless.mp4

Edit: Well I got the command line rav1e working (with default settings - I set --tune Psychovisual just to be sure) but I can see it will take a long time to generate a series of test encodes. If you want to run the encodes and send me the files, or run the metric tests as well and send me the results (scores & bitrates), I could combine them with my x264/x265 data if you like.

[zorr]

Originally Posted by WorBry

So it is internally converting sRGB to Linear then ?

Yes, indeed it is. It's assuming the input is sRGB and always does the conversion. No need to do retests with Butteraugli.

I also got a response from Jyrki, the author of Butteraugli:

Is this correct and a conversion to "raw intensity" is needed for the input images?
Yes.
And if so is the result of the above mentioned gamma correction that the input is in linear RGB? So for example when applied to standard definition video with matrix '601' the frames should be converted to linear RGB before applying Butteraugli? Thanks!
You need to convert into linear RGB light where RGB are sRGB. These values may be (slightly) negative for wide gamut use. The normalization value of 255 corresponds roughly to 120-200 nits.

[WorBry]

That's good to know.

[WorBry]

Originally Posted by WorBry

I've checked out the rav1e GUI (v1.8r2) and just can't get an encoded AV1 file (webm or mkv) out of it.

Tried encoding the CrowdRun 1080/50p 'master' (lossless, intra x264) clip at the default settings - Quantizer 100, Speed 3, Quality Tuning - PSNR. It proceeded to encode (slowly) and finally declared 'Finished', but there was no output file to be found.

Appears that the issue was with the final muxing (to mkv or webm) - if I uncheck Remove Temporary Files, the concatenated ivf file is there in the Temp Folder.

[ChaosKing]

I will upload some encodes soon. I made encodes from -q 100 - 200. Bitrate mode looked very ugly within the first 2 frames so I dropped it.

[WorBry]

Sounds Good. Having solved the the rav1e GUI issue (i.e. not removing the temp ivf files), I've run a couple of tests - wasn't sure if it's best to disable Low Latency, which slows things down more. I was thinking I might have to set-up for encoding on another PC so I can get other stuff done, but if you've already produced test files, all the better.

Edit: Quick test with the rav1e GUI and Crowd Run 1080/50p source - Quantizer 140, Speed 3, tune Psychovisual

'Low Latency' enabled - Bitrate 20.7 Mbps, VMAF 93.21
'Low Latency' disabled - Bitrate 15 Mbps, VMAF 88.75

So yes I think -q100 - 200 will be a good test range.

I'll run some additional x264/x265 CRF encode tests in the lower bitrate range.

[ChaosKing]

Here are my rav1e 1080/50p encodes: https://www.dropbox.com/s/vnx69xt3wb...crowd.zip?dl=1
Used rav1e.exe from here https://github.com/xiph/rav1e/releases/tag/20190219

[WorBry]

Great. I'll try and run the tests this evening.

Edit: It will take me a bit as I'll have to extend the x264/x265 series down to around CRF38 to match the low rav1e bitrates - q200 is just 2655 Kbps

[WorBry]

The results.

I ran the entire series of x264 and x265 encodes afresh with the last Zeranoe nightly build (ffmpeg-20190225-f948082-win64-static) for good measure. They were encoded with the default CRF settings. What were the rav1e encode settings, btw ?

I used VapourSynth-VMAF version r3 in model=0 (vmaf_v0.6.1.pkl) mode. 'Downsample' was applied in the muvsfunc SSIM and GMSD tests.

[ChaosKing]

I used default settings: --quantizer X --tune Psychovisual(just to be sure)
--speed <SPEED> Speed level (0 is best quality, 10 is fastest) [default: 3]

The result is more or less what I expected after I made some x265 encodes and compared it, it looked always slightly worse.
But to be more fair you should also use the -q mode in x264/5 and not crf. (I also used crf out of habit)

[WorBry]

Originally Posted by ChaosKing;1866989
But to be more fair you should also use the -q mode in x264/5 and not crf. [I

(I also used crf out of habit)[/I]

It didn't occur to me, especially as all the other tests were in CRF mode, I'll retest in -q mode when I have time.

[WorBry]

Since I'd run the tests, the ffmpeg SSIM and PSNR results:





I also ran MDSI tests on the rav1e encodes, but haven't yet on the x264/x265 series.

[WorBry]

Originally Posted by ChaosKing;1866989
But to be more fair you should also use the -q mode in x264/5 and not crf. [I

(I also used crf out of habit)[/I]

Originally Posted by WorBry

It didn't occur to me, especially as all the other tests were in CRF mode, I'll retest in -q mode when I have time.

The VMAF, muvsfunc SSIM and GMSD results updated to include x264 encoded in CQP mode (qp 28 - 46) :







Interesting that the VMAF scores for x264 in CQ mode are barely lower than CRF mode and then at around 22 Mbps CQ scores the highest of them all.

x265 in CQP mode to follow....at some point.

[WorBry]

The composite metric results, including x265 encoded in CQP mode (qp 28 - 44).

Note:

Originally Posted by WorBry

Interesting that the VMAF scores for x264 in CQ mode are barely lower than CRF mode and then at around 22 Mbps CQ scores the highest of them all.

Simple data transfer error. Corrected below.




Click on image to enlarge and on (+) cursor to enlarge further.

Clearly encoding x265 and x264 in CQP mode brings a different perspective. Now rav1e better matches CQP x265, as judged by SSIM and GMSD, and has the edge at the lowest bitrates. But VMAF still deems x265 to have higher perceptual quality over the entire bitrate range.

Knowing next to nothing about AV1, what are the prospects for CRF rate control in rav1e ?

[ChaosKing]

What enc paramteres did you use for x264/x265?
Maybe I will make also some 2pass encodes with rav1e if the bitrate mode doesn't suck like in 1pass. Just for completeness. It would be interessting to see by how much it will improve. (or not improve at all )

[WorBry]

Originally Posted by ChaosKing

What enc paramteres did you use for x264/x265?

For CRF:

ffmpeg -i  Input.mp4 -vcodec libx264 -preset slow -crf {value} -pix_fmt yuv420p -r 50/1 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 Output.mp4

ffmpeg -i Input.mp4 -vcodec libx265 -preset slow -crf {value} -pix_fmt yuv420p -r 50/1 -x265-params colorprim=1:transfer=1:colormatrix=1 Output.mp4

For CQP:

ffmpeg -i  Input.mp4 -vcodec libx264 -preset slow -qp {value} -pix_fmt yuv420p -r 50/1 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 Output.mp4

ffmpeg -i Input.mp4 -vcodec libx265 -preset slow -pix_fmt yuv420p -r 50/1 -x265-params qp={value}:colorprim=1:transfer=1:colormatrix=1 Output.mp4

In hindsight I maybe should have left at default -preset medium; I used -preset slow in all of the prior testing and didn't think to change it. I suppose I could re-test if it's felt that gave x264/x265 an unfair advantage, but can't face doing it all over just now.

I'm in the middle of some tests with visually lossless intermediate codecs. Seeing interesting results with Prores, MDSI and Butteraugli that deserve reporting when I'm done.

Originally Posted by ChaosKing

Maybe I will make also some 2pass encodes with rav1e if the bitrate mode doesn't suck like in 1pass. Just for completeness. It would be interessting to see by how much it will improve. (or not improve at all )

Sure. Bring it on

[WorBry]

Originally Posted by WorBry

I'm in the middle of some tests with visually lossless intermediate codecs. Seeing interesting results with Prores, MDSI and Butteraugli that deserve reporting when I'm done.

So I've been looking at the behaviour of these metrics when applied to discern fine differences between 'visually lossless' codecs.

For this I used a 1080/50p 10bit 422 (MagicYUV) transcode of the original Crowd Run 2160/50p SGI sequence as the test source and reference:

ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_...TER_SVTdec05_/

Encoded the source to:

- Prores HQ with VirtualDub2. The integrated (ffmpeg) encoder offers a range of 'Quality Levels' (q2-31), which appear to equate with qscale in ffmpeg CLI. I tested over q2 - 20 range.

- Cineform with VirtualDub2, using the 'native' Cineform SDK encoder. This implementation adds another quality level 'Filmscan 3' that is not normally available in other applications. I tested the full 'Filmscan 3' to 'Low' quality range.

- JPEG (RGB24, 100 - 85% compression range) using VapourSynth's very own ImageMagick Writer and the corresponding reader for re-importing the JPEG sequence:

I also encoded to Prores_ks HQ and DNxHR-HQX with ffmpeg CLI using the default settings.

Here are the muvsfunc VMAF, SSIM, GMSD MDSI and ffmpeg SSIM/PSNR results for Prores HQ and Cineform:





Nothing that remarkable about the muvsfunc VMAF, SSIM and GMSD results. For ffmpeg SSIM I also plotted the aggregate component Y, U and V scores. The Cineform 'Filmscan 3' scores were only marginally higher than 'Filmscan 2'. Clearly, the VMAF metric has no value in this context.

What is interesting in this case are the MDSI results. With Prores (and 'downsample' applied) the scores plateaued around q10 and actually decreased as the quality level was increased to q2. The MDSI scores with 'downsample' disabled also plateaued, but at a higher 'quality level' (around q4). The Cineform MDSI results do not show this behaviour.

I decided to examine this further. It made little difference if the Prores imports (and MagicYUV source) were converted to RGB48 or RGB24, with or without dithering. What did make a difference was converting the Prores and reference (MagicYUV) clips to greyscale before the RGB conversion - it dramatically relieved the apparent 'score saturation', implicating the chroma component:



The MDSI metric score is derived from pooled Gradient (Luminance) and Chromacity similarity assessments. I asked WolframRhodium if it might be possible to generate the Gradient (GS), Chromacity (CS) and pooled Gradient-Chromacity (GCS) maps and he has kindly obliged:

https://forum.doom9.org/showthread.p...62#post1867762

The generated map traces are very faint but they can be brought up quite nicely with Unsharp Mask. For the tests with downsample enabled I applied 2-passes (Radius 25, Amount 10) in Gimp.

Here are the similarity maps for ProRes at q2, q10, q16 and q20 quality levels, with 'downsample' enabled - I did also generate maps with 'downsample' disabled, but have yet to examine them.



Click image to enlarge, and then on (+) cursor to enlarge to max.

Looking at the Gradient-Chromacity Similarity (GCS) maps it's difficult to see how the final MDSI score at q2 would be less than that at q10. But the Chromacity Similarity (CS) maps definitely have an issue with the higher saturation colours on the runners shirts, right up to q2.

If anyone wants to examine them in more detail here are the matching frame grabs (VSEditor) of the imported clips (ffms2 decode) before and after conversion to RGB:



The frames in the downloaded image (right click > Save Image As) are the original 1920 x 1080 res.

And here are the similarity maps obtained with the Cineform and JPEG transcodes:





Big difference. Evidently MDSI is picking up chroma 'distortion' in the Prores encodes.

Bringing the clips and frame grabs into DaVinci Resolve I couldn't pick-up any obvious chroma shifts or saturation changes on the videoscopes, but I'll look at that in more detail with selective hue/saturation/luminance keys.

I don't think it's 'Quicktime' related issue per se - Cineform produced the same MDSI results whether encoded in AVI or MOV format. However, there are numerous reports of 'chroma shifts' associated with the ffmpeg Prores implementation.

The metric scores obtained with the ffmpeg CLI Prores encode (at default settings) put it on par with 'quality level' 12 using the VDub2 encoder.

More interesting still are the Butteraugli test results. They don't show this behaviour with the Prores encodes and converting to greyscale had much less impact on the scores:



In fact above Prores quality level 4 (the default), the greyscale scores were lower.

As mentioned above, the Butteraugli metric was tuned for detecting fine differences in JPEG images in the 90-95% compression domain. Note that the score drops quite markedly going from 90 to 85% quality.

Also interesting that ffmpeg DNxHR-HQX, whilst giving a 'bitrate-matched' MDSI score similar to that of Prores (VDub2), gives a much higher Butteraugli score, approaching that of JPEG 90%. Also the Butteraugli score for the ffmpeg Prores encode was lower than of Prores (VDub2) at an equivalent bitrate.

I've yet to scrutinize the 'heat maps' to see what more they can reveal.

I'm also wondering now if the MDSI scores seen earlier with the ffmpeg x264/x265 encodes might have been influenced in the same way as Prores.

[WorBry]

Originally Posted by WorBry

I've yet to scrutinize the 'heat maps' to see what more they can reveal.

The Butteraugli 'heat maps' and corresponding RGB24 source frames.

Click on image to enlarge, and on (+) cursor to enlarge further.

Prores HQ (VirtualDub2):



Cineform:



JPEG:



In the Prores series, there is very little difference between the 'full colour' and greyscale heat maps. Looking at the image components that Butteraugli is targeting in the highest quality encodes of each series, it appears to be mostly the blacks and whites.

Default dithering was applied in the fmtconv YUV422P10 > RGB24 conversion btw:

clip = core.ffms2.Source(source=r'{Path}:/10bit_422_Source.avi')
clip = core.fmtc.resample(clip=clip, css="444")
clip = core.fmtc.matrix(clip=clip, mat="709", col_fam=vs.RGB)
clip = core.fmtc.bitdepth(clip=clip, bits=8)

[WorBry]

Originally Posted by WorBry

Here are the MDSI similarity maps for ProRes at q2, q10, q16 and q20 quality levels, with 'downsample' enabled - I did also generate maps with 'downsample' disabled, but have yet to examine them.



Click image to enlarge, and then on (+) cursor to enlarge to max.

Looking at the Gradient-Chromacity Similarity (GCS) maps it's difficult to see how the final MDSI score at q2 would be less than that at q10. But the Chromacity Similarity (CS) maps definitely have an issue with the higher saturation colours on the runners shirts, right up to q2.

After further investigation it appears that the issue was with the (FFMS2) decoding.

The reference 10bit 422 source was in MagicYUV AVI format and the test Prores encodes were in MOV format. First tried converting the MagicYUV AVI reference clip to MOV, but it made no difference. But when I converted the Prores MOV files to (lossless) MagicYUV 10bit 422 AVI and re-ran the MDSI tests, lo and behold, it resolved the 'chroma distortion'. Just why, I don't know. But here are the revised MDSI similarity maps:



What's interesting is that when I re-ran the Butteraugli tests with the converted Prores clips it did not change the scores or 'heat maps' at all. And no change in the VMAF, SSIM and GMSD scores either. It only affected MDSI.

I'll post the revised MDSI score graphs in due course.

[WorBry]

The revised Prores metric score charts:



And the revised MDSI scores compared with those of other transcodes:



Now that looks a lot healthier.

The ffmpeg (CLI) Prores-ks-HQ and DNxHR-HQX transcodes both showed the 'chroma distortion' in the MDSI tests as well and converting the MOV files to MagicYUV 10bit 422 AVI resolved that also. So this was not specifically a Prores issue.

As mentioned above, it is interesting that the Butteraugli scores and heat maps were not changed at all by the conversion. Suggests it is unresponsive to subtle color shifts.

As this exercise has shown, MDSI is very sensitive to subtle color shifts and could prove to be a useful tool for assessing chroma sampling efficiencies.