GMSD and SSIM Quality Metrics [Archive]

zorr

13th February 2019, 21:57

While I still have the test files I'd like to see how the SSIM and GMSD metrics (in muvsfunc) compare.

The function descriptions state:

SSIM - 'The mean SSIM (MSSIM) index value of the distorted image will be stored as frame property 'PlaneSSIM' in the output clip'.

GMSD - 'The distortion degree of the distorted image will be stored as frame property 'PlaneGMSD' in the output clip. The value of GMSD reflects the range of distortion severities in an image.'

But I'm not clear how to access these results.

The easiest way probably is to use zoptilib.py which is part of the Zopti optimizer (https://forum.doom9.org/showthread.php?t=176076). But you don't need to run Zopti at all, just your script which calls zoptilib.

Here's a simple example:

from zoptilib import Zopti

# read input video
orig = core.ffms2.Source(source=r'source.avi')

# initialize output file and chosen metrics
zopti = Zopti('results.txt', metrics=['ssim', 'gmsd'])

# ... process the video ...
# alternate = some_process(orig)

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

The output file will contain frame number and the chosen metrics separated by ; and on the last line the sum of per frame values. The file will be written when all of the video frames have been processed (on the last frame).

The latest version of zoptilib is currently at here (https://pastebin.com/511BcmNp).

There's also MDSI metric but for that you need to upgrade to the latest muvsfunc version manually.

WorBry

13th February 2019, 23:57

Yes, that works. Thanks.

WorBry

17th February 2019, 03:11

I ran and added muvsfunc SSIM and GMSD to the first test series linked to above:

https://forum.doom9.org/showthread.php?p=1864770#post1864770

Crowd Run 1080/50p encoded to x264 with crf 0 to 30:

http://i.imgur.com/JhKH4Bv.png (https://imgur.com/JhKH4Bv)

http://i.imgur.com/IdYYRjL.png (https://imgur.com/IdYYRjL)

Interesting that the libvmaf, ffmpeg and muvsfunc SSIM implementations give rather different results:

http://i.imgur.com/BsGiz6q.png (https://imgur.com/BsGiz6q)

According to the documentation, the ffmpeg filter does apply the original SSIM algorithm but to to improve speed uses the standard approximation of overlapped 8x8 block sums rather than the original gaussian weights:

https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_ssim.c

As described in the original SSIM paper, one problem with the moving 8 x 8 block computation is that the resulting SSIM index map often exhibits undesirable blocking artifacts. By modifying the local statistics with gaussian weights the "quality maps exhibit a locally isotropic property" - which I take to mean it smooths out blocking artifacts in the quality map.

Page 605 - http://www.compression.ru/video/quality_measure/ssim.pdf

The muvsfunc SSIM function does apply gaussian filtering, with a default standard deviation of 1.5 as per the original recommendation:

https://github.com/WolframRhodium/muvsfunc/blob/master/muvsfunc.py

Would that explain why the muvsfunc SSIM metric gives higher scores this test series ?

Another factor might be whether preliminary downsampling is applied, as is recommended in the 'Suggested Usage":

https://ece.uwaterloo.ca/~z70wang/research/ssim/"

There's no mention of downscaling in the FFMPEG documentation. The muvsfunc SSIM filter does apply downscaling by default:

downsample: (bool) Whether to average the clips over local 2x2 window and downsample by a factor of 2 before calculation. Default is True.

And apparently VMAF "includes an empirical downsampling process, as described at the Suggested Usage" in it's elementary SSIM derivation:

https://github.com/Netflix/vmaf/issues/22

So why are the libvmaf SSIM scores even higher, if both are following the original code and applying the down-sampling process ?

The muvsfunc SSIM description does state though that it uses different size gaussian kernel to the one in the original MATLAB code.

Note that the size of gaussian kernel is different from the one in MATLAB.

Could that explain the difference?

I have to say this leaves me in a quandary about which SSIM implementation to use when comparing the inherent quality efficiency of different video formats, especially at high bitrates - for example, when comparing 'visually lossless' intermediate codecs, where the interest is not in perceptual quality under certain viewing conditions but in preservation of structural fidelity. The ffmpeg SSIM metric gives a much wider spread of values which makes it easier to judge, with some confidence, that one video is of higher quality than another based on the difference of isolated scores (the last graph presented below shows that well), but is it valid as a quotable SSIM score ?

Is there a valid case for omitting the down-sampling step in the muvsfunc SSIM metric under such conditions ?

Anyhow, I also ran muvsfunc GMSD and SSIM on the parallel x265 series:

http://i.imgur.com/lxlsHIp.png (https://imgur.com/lxlsHIp)

Including the ffmpeg SSIM results made the graph too 'busy', so here they are separately:

http://i.imgur.com/gPsF3om.png (https://imgur.com/gPsF3om)

GMSD gave consistently higher scores for x265 over the same bitrate range. The SSIM metrics did also, but by a narrower margin at the higher bitrates:

http://i.imgur.com/l8kZ7Xl.png (https://imgur.com/l8kZ7Xl)

GMSD looks like it could be very useful. I've yet to test muvsfunc SSIM and GMSD on the 2160/50p Crowd Run x264 and x265 series.

If anyone's interested here's the original GMSD paper:

https://arxiv.org/pdf/1308.3052.pdf

Edit: Came across this article that quotes from an article by RealNetworks CTO Reza Rassool:

“if a video service operator were to encode video to achieve a VMAF score of about 93 then they would be confident of optimally serving the vast majority of their audience with content that is either indistinguishable from original or with noticeable but not annoying distortion.” So a 93 VMAF score is about the same as .95 for SSIM

https://streaminglearningcenter.com/learning/mapping-ssim-vmaf-scores-subjective-ratings.html

In the above tests - at a VMAF score of 93 the corresponding muvsfunc SSIM scores were around 0.96 - 0.97 and the ffmpeg SSIM scores were around the 0.925 - 0.93. The libvmaf component SSIM scores however were way up at around 0.993, which surely suggests there's something more going

WorBry

17th February 2019, 17:40

I'd like to test muvsfunc SSIM with 'downsample=False' to see how the results compare. How can I change that setting so as to get the results through Zoptilib ?

from zoptilib import Zopti

# read input video
orig = core.ffms2.Source(source=r'source.avi')

# initialize output file and chosen metrics
zopti = Zopti('results.txt', metrics=['ssim', 'gmsd'])

# ... process the video ...
# alternate = some_process(orig)

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

ChaosKing

17th February 2019, 19:45

I made an update https://github.com/theChaosCoder/zoptilib
You can use it like this
zopti = Zopti(output_file, metrics=['ssim', 'mdsi'])
zopti.addParams('ssim', dict(downsample=False, show_map=False))
zopti.addParams('mdsi', dict(down_scale=1))

WorBry

17th February 2019, 20:02

Brilliant. Haven't got around to looking at MDSI yet. It's RGB only though, isn't it ?

ChaosKing

17th February 2019, 21:06

Yes. Also use the latest version https://raw.githubusercontent.com/WolframRhodium/muvsfunc/master/muvsfunc.py

WorBry

18th February 2019, 18:39

I'd like to test muvsfunc SSIM with 'downsample=False' to see how the results compare.

I've done that with the x264 test series:

http://i.imgur.com/EfLsZmO.png (https://imgur.com/EfLsZmO)

http://i.imgur.com/IcqcDXH.png (https://imgur.com/IcqcDXH)

Clearly removing the downsampling step has a profound effect, producing a much wider spread of scores - even more so than ffmpeg SSIM - although up at around CRF=2 (400 - 425 Mbps), the scores start to approach those obtained with downsampling applied, and lossless is still reported as such.

http://i.imgur.com/gxaKZFN.png (https://imgur.com/gxaKZFN)

As described in the 'Suggested Usage' for the original (Matlab) SSIM code, the purpose of the downsampling is to compensate for viewing the image at a typical distance from the screen:

The above (ssim_index.m) is a single scale version of the SSIM indexing measure, which is most effective if used at the appropriate scale. The precisely “right” scale depends on both the image resolution and the viewing distance and is usually difficult to be obtained. In practice, we suggest to use the following empirical formula to determine the scale for images viewed from a typical distance (say 3~5 times of the image height or width): 1) Let F = max(1, round(N/256)), where N is the number of pixels in image height (or width); 2) Average local F by F pixels and then downsample the image by a factor of F; and 3) apply the ssim_index.m program. For example, for an 512 by 512 image, F = max(1, round(512/256)) = 2, so the image should be averaged within a 2 by 2 window and downsampled by a factor of 2 before applying ssim_index.m.

http://www.cns.nyu.edu/~lcv/ssim/

In other words it is a perceptual quality modifier. Whether it's valid to remove that step when using SSIM to compare video images for structural differences that exceed visual acuity (i.e. independent of perceived quality) I'm still not sure.

WorBry

18th February 2019, 19:58

Thought it might be interesting to see how the AVISynth SSIM filter compares also. This plugin has a rather nebulous history going back to the original implementation by LeFungus in 2003:

https://forum.doom9.org/showthread.php?t=61128

His last update was version 0.24, although the results log still reports it as 0.23.

It appears the plugin then received further fixes and modifications made by others, but in the absence of associated documentation, it is not clear exactly what changes were made.

This thread attempted to make sense of it:

https://forum.doom9.org/showthread.php?p=1089303#post1089303

I decided to test both the original (as assumed) 'LeFungus' 0.24 version and the 0.25.1 version posted by Mitsubishi in that thread. They produced identical results:

http://i.imgur.com/OnifUFL.png (https://imgur.com/OnifUFL)

Wow, very different from the other SSIM implementations, with the CRF=30 x264 encode scoring way down at 33 (0.33).
Yet, according to LeFungus, it was developed from the original code.

I suspect this stems from the 'luma masking' parameter that was given as an option (Default: True) in the original (LeFungus) 0.24 plugin. In 0.25.1 that option is not accessible, as such (returns an error), but since 0.25.1 produced identical results, it's reasonable to assume that 'Luma Masking' was being applied.

Possibly it equates with the luminance normalization filtering that is applied in the original SSIM algorithm ? In the AVISynth plugin it is applied as a weighting:

This filter is designed to compute an SSIM value by two methods, the original one, and a "enhanced" one that weight these results by lumimasking........In the csv file, when lumimasking is activated, both SSIM values and its weight is written.

https://avisynth.org.ru/docs/english/externalfilters/ssim.htm

Unfortunately, I only recorded the final aggregate SSIM score reported in the log file and didn't generate the csv file that lists the individual frame scores and weightings. I'll maybe re-run some tests to see what difference the weightings made. But really, these results and the vagaries surrounding this plugin, don't exactly instill confidence in it's use.

ChaosKing

18th February 2019, 20:14

So basically every ssim implementation gave different results... which one can we trust (more)?

WorBry

18th February 2019, 20:51

Quite so ! Granted the tests were conducted with just one source clip (CrowdRun) - although a good one at that - high quality/complexity/motion/hard to compress.

The results as they stand leave me more inclined to use muvsfunc SSIM as a 'definitive' implementation of the original code. Would be nice if there were an AVISynth(+) implementation of the muvsfunc SSIM filter.

Still don't understand though why the libvaf derived SSIM figures are so much higher. Is it down to difference in Gaussian kernel size or are the reported elementary SSIM scores being further weighted by the VMAF 'model' in some way (before the final VMAF calculation, that is) ?

As for MDSI - results to follow.

zorr

18th February 2019, 21:39

I looked at sources of muvsfunc SSIM and Avisynth's SSIM (v0.25.1 by Mitsubishi).

The Avisynth version is not doing the gaussian kernel at all - it's implemented using summed area tables. That's a faster but lower quality way to calculate it, the MSU Quality measurement tool page (http://www.compression.ru/video/quality_measure/info.html#ssim) has an example of the difference.

Also muvsfunc returns SSIM calculated on one plane only, by default the luma. Avisynth SSIM has a plane argument which defaults to 0 and then it returns a weighted sum of the luma and chroma channels:
(0.8 * Y) + (0.1*(U+V))

And yes, Avisynth has the lumimask but it's disabled in the code. Muvsfunc has the variables k1 and k2, but at least they default to same values as the ones used in Avisynth version.

So there are quite a few ways to make the implementations differ, I guess there are similar small differences between the other implementations.

My opinion is that the default muvsfunc SSIM downsampling is not useful when comparing the quality of different script settings (like in the Zopti optimizer).

WorBry

18th February 2019, 22:08

Thanks for the insights. That explains a lot.

Also muvsfunc returns SSIM calculated on one plane only, by default the luma.

So presumably libvmaf is doing the same ?

Avisynth SSIM has a plane argument which defaults to 0 and then it returns a weighted sum of the luma and chroma channels:
(0.8 * Y) + (0.1*(U+V))

Is that how ffmpeg calculates an aggregate 'All' SSIM score also ? Even if it's not applying Gaussian weights, obtaining individual scores for the Luma and U, V channels can be useful in assessing whether losses are occurring in the chroma only - for examining chroma subsampling efficiencies etc.

My opinion is that the default muvsfunc SSIM downsampling is not useful when comparing the quality of different script settings (like in the Zopti optimizer).

Yes, I don't see there's anything to be gained in that context.

WorBry

18th February 2019, 23:38

Question:

For conducting these tests I've been using VirtualDub2 to run the VS scripts and generate the result files.

I'd like to change to KingChaos's Portable (Flatpack) version in future. The changelog for the next update (2019-02-xx) promises to include:

- Add VFW "install" script, so that VDub and co can read vpy files

https://forum.doom9.org/showthread.php?t=175529

Meanwhile, for running MDSI (and possibly Buttergauli) scripts, how do I vspipe the RGB24 output to ffmpeg as a null operation, purely to generate the results files ?

I suppose I could use VSEditor > Preview in place of VirtualDub2 but I can't see how to stop the playback looping when it comes to the end of the clip.

ChaosKing

18th February 2019, 23:48

I'd like to change to KingChaos's Portable (Flatpack) version in future. The changelog for the next update (2019-02-xx) promises to include:

I think I'm trapped in an alternate reality.
You can use the reg file for now https://forum.doom9.org/showthread.php?p=1864051#post1864051

I suppose I could use VSEditor > Preview in place of VirtualDub2 but I can't see how to stop the playback looping when it comes to the end of the clip.

Use the Benchmark (F7) instead.

WorBry

19th February 2019, 00:28

I think I'm trapped in an alternate reality

Really, what's the weather like there ?

Use the Benchmark (F7) instead.

That's the one. Thanks.

You can use the reg file for now https://forum.doom9.org/showthread.php?p=1864051#post1864051

Hadn't seen your other post about the reg edit. So is that basically what the 'VFW "Install" Script' will be doing ?

WorBry

19th February 2019, 03:53

I looked at sources of.....Avisynth's SSIM (v0.25.1 by Mitsubishi).

...And yes, Avisynth has the lumimask but it's disabled in the code.

That's odd - I went back to check and re-run some of the tests with v0.24 and v0.25.1. The results I posted above were definitely with Lumimask=True applied in v0.24, and v0.25.1 gave the same results. You can see both the 'original' and weighted ('enhanced') SSIM scores displayed on the output frames as the script is played through VDub2 and the per-frame scores are listed in separate columns in the generated csv file. The text file however only gives the 'global' weighted score.

The Lumimask parameter may be disabled as an option in v0.25.1, but it's definitely being applied.

That said, setting Lumimask=False in v0.24 didn't radically change the results. I only ran a couple of tests:

Lumimask=True Lumimask=False
CRF0 100 100
CRF1 98.50 98.36
CRF12 87.17 86.64
CRF30 33.80 34.09

ChaosKing

19th February 2019, 14:06

Really, what's the weather like there ?

Cloudy with a Chance of Meatballs :p

Hadn't seen your other post about the reg edit. So is that basically what the 'VFW "Install" Script' will be doing ?

Yes, it will add it to the registry with the correct path.

WorBry

19th February 2019, 17:13

As for MDSI - results to follow.

The MDSI and GMSD results for the x264 and x265 series:

http://i.imgur.com/3f9NgNc.png (https://imgur.com/3f9NgNc)

http://i.imgur.com/CI2nHXH.png (https://imgur.com/CI2nHXH)

Interesting that the MDSI scores show a fairly linear relation with bitrate plotted as base 2 log. Encoding x264 at any fractional CRF value <1 of course defaults to lossless High444Predictive. Interesting also that the difference between the (bitrate matched) x264 and x265 score plots is fairly constant down to around 24 Mbps.

Those MDSI results were with downscale applied i.e.

zopti.addParams('mdsi', dict(down_scale=2))

With downscale turned off (the default)...

zopti.addParams('mdsi', dict(down_scale=1))

....the scores are lower and lose the linear relationship at the higher bitrates.

http://i.imgur.com/cHIFRD6.png (https://imgur.com/cHIFRD6)

For those interested, here's the original paper for the MDSI (Mean Deviation Similarity Index) metric.

https://arxiv.org/pdf/1608.07433.pdf

This metric pools combined image gradient (sensitive to structural distortions) and chromacity similarity maps.

zorr

19th February 2019, 22:08

Is that how ffmpeg calculates an aggregate 'All' SSIM score also ?

I looked at the ffmpeg source. It's doing the fast version too - no gaussian kernels there. The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight. The constants 0.01 and 0.03 appear in the code and I assume that means the weights k1 and k2 are the same as in other implementations.

The Lumimask parameter may be disabled as an option in v0.25.1, but it's definitely being applied.

You're right, or should I say we're both right. :) The lumimask is disabled in the Get_SSIM_Frame() function which is the one used by Zopti (it can return the SSIM value to the calling script). But the other function which saves the file and the one you used has the lumimask functionality.

The MDSI and GMSD results for the x264 and x265 series:

Interesting results. Are the x264 and x265 results swapped in the second chart, it has x264 with better scores?

Note that if the GMSD results were calculated with the version before ChaosKing's addParams()-function it had 2x downsampling enabled for GMSD as well.

WorBry

20th February 2019, 01:12

I looked at the ffmpeg source. It's doing the fast version too - no gaussian kernels there. The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight. The constants 0.01 and 0.03 appear in the code and I assume that means the weights k1 and k2 are the same as in other implementations.

Thanks for looking at that. Shame the free version of the MSU VQM tool is limited to SD resolution and the demo of Pro version doesn't guarantee that the results will be correct; seems an odd way to promote a product - the statistical equivalent of a water mark, I guess. It would be interesting to compare the results otherwise.

A few years back someone put together a VQA tool in Java (jVQA) inspired by the MSU VQA:

https://forum.doom9.org/showthread.php?t=172876&highlight=GMSD+SSIM+Java

Looks like it didn't get beyond beta development.

Are the x264 and x265 results swapped in the second chart, it has x264 with better scores?
No, they are the right way round. It's just that the second graph plots the scores against the encode CRF and the the resulting x264 bitrates were appreciably higher (around 30-35 %) than x265 at the same CRF setting, so it biases the plots that way. I only included it really to show why the x264 MDSI score goes sharply from 9.8 (at CRF=1) to 0 (at CRF=0) and there are no in-between data points.

For comparing x264 and x265 the first graph with the scores plotted against bitrate is the one to look at.

Note that if the GMSD results were calculated with the version before ChaosKing's addParams()-function it had 2x downsampling enabled for GMSD as well.

They were. I might go back and re-run the GMSD tests with down-sampling disabled, for completeness. Just finishing off testing Butteraugli.

WorBry

20th February 2019, 18:50

And ButtUgly...I mean, Butteraugli:

http://i.imgur.com/KfySsp9.png (https://imgur.com/KfySsp9)

Somewhat at odds with the results of the SSIM, GMSD and MDSI testing. Down at the low bitrates it reports x264 and x265 to be pretty much on par. Increasing the bitrate it gives the edge to x264, but then around the 170 Mbps mark (around CRF=10 for x264 and CRF=8 for x265) it switches and starts to score x265 at higher quality:

http://i.imgur.com/IA2vXEG.png (https://imgur.com/IA2vXEG)

That said, the butteraugli developers do stress that this metric was tuned for comparing images "in the domain of barely noticeable differences" ...."We don't know how well butteraugli performs with major deformations -- we have mostly tuned it within a small range of quality, roughly corresponding to jpeg qualities 90 to 95."

https://opensource.google.com/projects/butteraugli

https://github.com/google/butteraugli

I'm not sure how that equates to x264 and x265 quality in CRF mode, but I think that "switch-over" point where x265 starts to get the edge is probably significant.

And you can see it reflected in the Butteraugli quality 'heat' maps. Really starts to warm up around the CRF 10-12 (x264) mark and by CRF 30 it is a veritable furnace.

http://i.imgur.com/rffEp4Qm.jpg (https://imgur.com/rffEp4Q)

After opening image link, click on (+) cursor to enlarge

At some point I'll pull up quality maps from other tests to see what differences each is picking up, in the high bitrate 'visually lossless' range especially.

WorBry

21st February 2019, 03:17

Note that if the GMSD results were calculated with the version before ChaosKing's addParams()-function it had 2x downsampling enabled for GMSD as well.

They were. I might go back and re-run the GMSD tests with down-sampling disabled, for completeness.

GMSD on the x264 series with and without downsampling:

http://i.imgur.com/nCkM0Ug.png (https://imgur.com/nCkM0Ug)

zorr

21st February 2019, 22:18

Looks like Butteraugli is not useful for the whole quality range just like its developer suspected.

Interesting that GMSD without downsampling has the base 2 log - linear behavior a bit like MDSI when it was downsampled.

WorBry

22nd February 2019, 17:24

Looks like Butteraugli is not useful for the whole quality range just like its developer suspected.

The results, with that one source at least, do leave me with that impression. I searched for other instances where butteraugli has been used to compare x264 and x265. The only one I could find was:

https://encode.ru/threads/2811-Psychovisual-measurements-on-modern-image-codecs

I've yet to look at the uploaded results file in detail, but the comment by the second poster, Jyrki Alakuijala (who is the butteraugli developer.), is of note:

Digging out some data from the test results to support the question:

x264 233464 bytes, butteraugli = 0.948613, ssimulacra = 0.00761919
x265 241125 bytes, butteraugli = 1.689265, ssimulacra = 0.01749584

x265 gives significantly worse results than x264. Perhaps someone who knows ffmpeg flags can help.

Also, here jpeg for reference (also digged from the result_corpus.zip). In this test x264 is a lot better than jpeg, and x265 worse?!:

libjpeg/q93/yuv444 235122, butteraugli = 1.467834, ssimulacra = 0.01439446

Interesting that GMSD without downsampling has the base 2 log - linear behavior a bit like MDSI when it was downsampled.

I have no idea why. Must surely be in the scaled vs non-scaled computations.

zorr

23rd February 2019, 01:38

Hopefully you remembered to run the Butteraugli tests in linear RGB. :)

WorBry

23rd February 2019, 01:56

WorBry

23rd February 2019, 06:32

Meanwhile, I ran muvsfunc SSIM and GMSD on the 2160/50p Crowd Run x264 and x265 series that I tested in the other thread:

https://forum.doom9.org/showthread.php?p=1865316#post1865316

http://i.imgur.com/BPrFZvQ.png (https://imgur.com/BPrFZvQ)

Note that the shape of the muvsfunc and ffmpeg SSIM curves are different to those in the 1080/50p series - the scores dip more in the mid bitrate range. I brought attention to that (with ffmpeg SSIM) in the other thread:

http://i.imgur.com/cHsJnCW.png (https://imgur.com/cHsJnCW)

https://forum.doom9.org/showpost.php?p=1865424&postcount=56

More striking though are the GMSD results. Whereas the 1080/50p series gave consistently higher GMSD scores for x265 over the entire (CRF 0-30) range, here the x265 and x264 curves converge over the lower (48 - 96 Mbps) range.

At the higher bitrates however, GMSD still clearly favours x265:

http://i.imgur.com/pJ8H1ch.png (https://imgur.com/pJ8H1ch)

It could be said that the SSIM and GMSD results are somewhat complementary in this case.

I haven't tested MDSI.

ChaosKing

23rd February 2019, 23:02

Could you also test a recent rav1e build? --tune Psychovisual is now default. VMAF and GMSD would be interesting.

WorBry

24th February 2019, 02:32

I could look at it once I'm done testing with these x264 and x265 files, but I have to say that I have zero experience encoding AV1 - so you'd need to give me a command line with the parameters for ramping over a comparable test bitrate range.

zorr

24th February 2019, 15:42

Err, no I didn't. Didn't realize I needed to. I just converted to RGB24 with Rec709 matrix:

https://github.com/fdar0536/VapourSynth-butteraugli

I see you converted to linear RGB in your Zopti studies:

https://forum.doom9.org/showthread.php?p=1865218#post1865218/

But I don't understand why one would need to in this context. Doesn't butteraugli internally convert sRGB to Linear anyway ?

It's not really well documented but I found this in the source header (https://github.com/google/butteraugli/blob/master/butteraugli/butteraugli.h):

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

This "raw intensity" I interpreted as linear RGB, I could be wrong though. But it states clearly that the caller must perform the correction.

ChaosKing

24th February 2019, 19:13

The command line looks like this vspipe.exe script.vpy - --y4m | rav1e.exe - --output out.ivf
https://github.com/xiph/rav1e/releases

The only interesting parameters are:
--quantizer <QP> Quantizer (0-255), smaller values are higher quality [default: 100]
-b, --bitrate <BITRATE> Bitrate (kbps)
-s, --speed <SPEED> Speed level (0 is best quality, 10 is fastest) [default: 3]
You can mux ivf to mkv. The ffms2 version in the portable fatpack can decode av1.

For me it would be interesting to see how the codecs perform on a lower bitrate.

WorBry

24th February 2019, 20:24

It's not really well documented but I found this in the source header (https://github.com/google/butteraugli/blob/master/butteraugli/butteraugli.h):

// Value of pixels of images rgb0 and rgb1 need to be represented as raw
// intensity. Most image formats store gamma corrected intensity in pixel
// values. This gamma correction has to be removed, by applying the following
// function:
// butteraugli_val = 255.0 * pow(png_val / 255.0, gamma);
// A typical value of gamma is 2.2. It is usually stored in the image header.
// Take care not to confuse that value with its inverse. The gamma value should
// be always greater than one.
// Butteraugli does not work as intended if the caller does not perform
// gamma correction.

This "raw intensity" I interpreted as linear RGB, I could be wrong though. But it states clearly that the caller must perform the correction.

Well, I dunno. If that is the case it should be made clear in the VapourSynth butteraugli plugin usage. Even the given example has:

import mvsfunc as mvf

clipa = core.std.Trim(src1, 0, 0)
clipa = mvf.ToRGB(clipa, depth=8)
clipb = core.std.Trim(src2, 0, 0)
clipb = mvf.ToRGB(clipb, depth=8)

diff = core.Butteraugli.butteraugli(clipa, clipb)

https://github.com/fdar0536/VapourSynth-butteraugli

I suppose I could retest but this really deserves some clarification.

WorBry

24th February 2019, 20:42

ChaosKing

24th February 2019, 22:19

Seems that Bitrate was added but not released yet https://github.com/moisesmcardona/rav1e_gui/issues/1

zorr

24th February 2019, 23:10

Well, I dunno. If that is the case it should be made clear in the VapourSynth butteraugli plugin usage.
...
I suppose I could retest but this really deserves some clarification.

I created an issue (https://github.com/google/butteraugli/issues/53) and asked about the input format. We also need to find out whether Vapoursynth-Butteraugli does it automatically.

ChaosKing

24th February 2019, 23:20

Found this https://github.com/fdar0536/VapourSynth-butteraugli/blob/master/vsbutteraugli.cpp#L98

WorBry

25th February 2019, 00:49

I haven't tested MDSI.

I have now:

http://i.imgur.com/YNOJ0i6.png (https://imgur.com/YNOJ0i6)

Compared with the 1080 50p series:

http://i.imgur.com/3f9NgNc.png (https://imgur.com/3f9NgNc)

x265 does still score marginally higher at high bitrates but we're looking at a difference of 0.1 units - or rather 0.001 in actual scores (x100).

http://i.imgur.com/inNFUxj.png (https://imgur.com/inNFUxj)

And then there's that abrupt upturn in the x264 score going from 20.5 at CRF=1 (2098 Mbps) to 0 for lossless CRF=0 (2566 Mbps). An even larger jump than seen in the 1080 50p series. Makes me wonder if there is a bitrate saturation point for this metric or if it's like ffmpeg PSNR, which has no saturation point and absolute lossless is reported as infinity (Inf).

Tried testing x264 (1080 50p) encoded at increasing 2-pass bitrates to see what more I could glean, but the maximum bitrate attainable was 615 Mbps:

http://i.imgur.com/QjFgFDM.png (https://imgur.com/QjFgFDM)

The MDSI score (with downsampling) increased just 0.02 (actual 0.0002) units going from 400 Mbps (18.30) to 615 Mbps (18.28).

WorBry

25th February 2019, 01:00

Found this https://github.com/fdar0536/VapourSynth-butteraugli/blob/master/vsbutteraugli.cpp#L98

So it is internally converting sRGB to Linear then ?

WorBry

25th February 2019, 18:21

I see there's a GUI for rav1e....
https://github.com/moisesmcardona/rav1e_gui

I've checked out the rav1e GUI (v1.8r2) and just can't get an encoded AV1 file (webm or mkv) out of it.

Tried encoding the CrowdRun 1080/50p 'master' (lossless, intra x264) clip at the default settings - Quantizer 100, Speed 3, Quality Tuning - PSNR. It proceeded to encode (slowly) and finally declared 'Finished', but there was no output file to be found.

I'll maybe try the command-line route. Otherwise, if you want to run the tests with that particular source, I used the 2160/50p (8bit 420) y4m version of Crowd Run from:

https://media.xiph.org/video/derf/

....as the source/reference for the 2160/50p tests. For the 1080/50p tests, I converted to lossless, intra x264:

ffmpeg -i {Path}:/crowd_run_2160p50.y4m -vf scale=1920:-1 -vcodec libx264 -intra -preset slow -qp 0 {Path}:/crowd_run_1080p50_x264_lossless.mp4

Edit: Well I got the command line rav1e working (with default settings - I set --tune Psychovisual just to be sure) but I can see it will take a long time to generate a series of test encodes. If you want to run the encodes and send me the files, or run the metric tests as well and send me the results (scores & bitrates), I could combine them with my x264/x265 data if you like.

zorr

25th February 2019, 21:33

So it is internally converting sRGB to Linear then ?

Yes, indeed it is. It's assuming the input is sRGB and always does the conversion. No need to do retests with Butteraugli.

I also got a response from Jyrki, the author of Butteraugli:

Is this correct and a conversion to "raw intensity" is needed for the input images?
Yes.
And if so is the result of the above mentioned gamma correction that the input is in linear RGB? So for example when applied to standard definition video with matrix '601' the frames should be converted to linear RGB before applying Butteraugli? Thanks!
You need to convert into linear RGB light where RGB are sRGB. These values may be (slightly) negative for wide gamut use. The normalization value of 255 corresponds roughly to 120-200 nits.

WorBry

25th February 2019, 22:08

That's good to know.

WorBry

26th February 2019, 18:06

I've checked out the rav1e GUI (v1.8r2) and just can't get an encoded AV1 file (webm or mkv) out of it.

Tried encoding the CrowdRun 1080/50p 'master' (lossless, intra x264) clip at the default settings - Quantizer 100, Speed 3, Quality Tuning - PSNR. It proceeded to encode (slowly) and finally declared 'Finished', but there was no output file to be found.

Appears that the issue was with the final muxing (to mkv or webm) - if I uncheck Remove Temporary Files, the concatenated ivf file is there in the Temp Folder.

ChaosKing

26th February 2019, 18:47

I will upload some encodes soon. I made encodes from -q 100 - 200. Bitrate mode looked very ugly within the first 2 frames so I dropped it.

WorBry

26th February 2019, 19:19

Sounds Good. Having solved the the rav1e GUI issue (i.e. not removing the temp ivf files), I've run a couple of tests - wasn't sure if it's best to disable Low Latency, which slows things down more. I was thinking I might have to set-up for encoding on another PC so I can get other stuff done, but if you've already produced test files, all the better.

Edit: Quick test with the rav1e GUI and Crowd Run 1080/50p source - Quantizer 140, Speed 3, tune Psychovisual

'Low Latency' enabled - Bitrate 20.7 Mbps, VMAF 93.21
'Low Latency' disabled - Bitrate 15 Mbps, VMAF 88.75

So yes I think -q100 - 200 will be a good test range.

I'll run some additional x264/x265 CRF encode tests in the lower bitrate range.

ChaosKing

26th February 2019, 20:51

Here are my rav1e 1080/50p encodes: https://www.dropbox.com/s/vnx69xt3wbzn4ot/rav1e_crowd.zip?dl=1
Used rav1e.exe from here https://github.com/xiph/rav1e/releases/tag/20190219

WorBry

26th February 2019, 23:10

Great. I'll try and run the tests this evening.

Edit: It will take me a bit as I'll have to extend the x264/x265 series down to around CRF38 to match the low rav1e bitrates - q200 is just 2655 Kbps

WorBry

27th February 2019, 19:13

The results.

I ran the entire series of x264 and x265 encodes afresh with the last Zeranoe nightly build (ffmpeg-20190225-f948082-win64-static) for good measure. They were encoded with the default CRF settings. What were the rav1e encode settings, btw ?

I used VapourSynth-VMAF version r3 in model=0 (vmaf_v0.6.1.pkl) mode. 'Downsample' was applied in the muvsfunc SSIM and GMSD tests.

http://i.imgur.com/IpT23ng.png (https://imgur.com/IpT23ng)

http://i.imgur.com/lAJH4nZ.png (https://imgur.com/lAJH4nZ)

http://i.imgur.com/QXovVYL.png (https://imgur.com/QXovVYL)

ChaosKing

27th February 2019, 19:28

I used default settings: --quantizer X --tune Psychovisual(just to be sure)
--speed <SPEED> Speed level (0 is best quality, 10 is fastest) [default: 3]

The result is more or less what I expected after I made some x265 encodes and compared it, it looked always slightly worse. :D
But to be more fair you should also use the -q mode in x264/5 and not crf. (I also used crf out of habit)

WorBry

27th February 2019, 19:58

(I also used crf out of habit)[/I]

It didn't occur to me, especially as all the other tests were in CRF mode, I'll retest in -q mode when I have time.

WorBry

27th February 2019, 20:30

Since I'd run the tests, the ffmpeg SSIM and PSNR results:

http://i.imgur.com/C5DfLvz.png (https://imgur.com/C5DfLvz)

http://i.imgur.com/LdgQvtp.png (https://imgur.com/LdgQvtp)

I also ran MDSI tests on the rav1e encodes, but haven't yet on the x264/x265 series.

WorBry

28th February 2019, 03:40

(I also used crf out of habit)[/I]

It didn't occur to me, especially as all the other tests were in CRF mode, I'll retest in -q mode when I have time.

The VMAF, muvsfunc SSIM and GMSD results updated to include x264 encoded in CQP mode (qp 28 - 46) :

http://i.imgur.com/C9muivX.png (https://imgur.com/C9muivX)

http://i.imgur.com/UxWZmcP.png (https://imgur.com/UxWZmcP)

http://i.imgur.com/hM6CvJx.png (https://imgur.com/hM6CvJx)

Interesting that the VMAF scores for x264 in CQ mode are barely lower than CRF mode and then at around 22 Mbps CQ scores the highest of them all.

x265 in CQP mode to follow....at some point.

WorBry

28th February 2019, 18:41

The composite metric results, including x265 encoded in CQP mode (qp 28 - 44).

Note:
Interesting that the VMAF scores for x264 in CQ mode are barely lower than CRF mode and then at around 22 Mbps CQ scores the highest of them all.

Simple data transfer error. Corrected below.

http://i.imgur.com/elq9QV7l.png (https://imgur.com/elq9QV7)

Click on image to enlarge and on (+) cursor to enlarge further.

Clearly encoding x265 and x264 in CQP mode brings a different perspective. Now rav1e better matches CQP x265, as judged by SSIM and GMSD, and has the edge at the lowest bitrates. But VMAF still deems x265 to have higher perceptual quality over the entire bitrate range.

Knowing next to nothing about AV1, what are the prospects for CRF rate control in rav1e ?

ChaosKing

2nd March 2019, 02:38

What enc paramteres did you use for x264/x265?
Maybe I will make also some 2pass encodes with rav1e if the bitrate mode doesn't suck like in 1pass. Just for completeness. It would be interessting to see by how much it will improve. (or not improve at all :D)

WorBry

2nd March 2019, 03:28

What enc paramteres did you use for x264/x265?

For CRF:

ffmpeg -i Input.mp4 -vcodec libx264 -preset slow -crf {value} -pix_fmt yuv420p -r 50/1 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 Output.mp4

ffmpeg -i Input.mp4 -vcodec libx265 -preset slow -crf {value} -pix_fmt yuv420p -r 50/1 -x265-params colorprim=1:transfer=1:colormatrix=1 Output.mp4

For CQP:

ffmpeg -i Input.mp4 -vcodec libx264 -preset slow -qp {value} -pix_fmt yuv420p -r 50/1 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 Output.mp4

ffmpeg -i Input.mp4 -vcodec libx265 -preset slow -pix_fmt yuv420p -r 50/1 -x265-params qp={value}:colorprim=1:transfer=1:colormatrix=1 Output.mp4

In hindsight I maybe should have left at default -preset medium; I used -preset slow in all of the prior testing and didn't think to change it. I suppose I could re-test if it's felt that gave x264/x265 an unfair advantage, but can't face doing it all over just now.

I'm in the middle of some tests with visually lossless intermediate codecs. Seeing interesting results with Prores, MDSI and Butteraugli that deserve reporting when I'm done.

Maybe I will make also some 2pass encodes with rav1e if the bitrate mode doesn't suck like in 1pass. Just for completeness. It would be interessting to see by how much it will improve. (or not improve at all :D)

Sure. Bring it on ;)

WorBry

8th March 2019, 22:04

I'm in the middle of some tests with visually lossless intermediate codecs. Seeing interesting results with Prores, MDSI and Butteraugli that deserve reporting when I'm done.

So I've been looking at the behaviour of these metrics when applied to discern fine differences between 'visually lossless' codecs.

For this I used a 1080/50p 10bit 422 (MagicYUV) transcode of the original Crowd Run 2160/50p SGI sequence as the test source and reference:

ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat/2160p50_CgrLevels_Master_SVTdec05_/1_CrowdRun_2160p50_CgrLevels_MASTER_SVTdec05_/

Encoded the source to:

- Prores HQ with VirtualDub2. The integrated (ffmpeg) encoder offers a range of 'Quality Levels' (q2-31), which appear to equate with qscale in ffmpeg CLI. I tested over q2 - 20 range.

- Cineform with VirtualDub2, using the 'native' Cineform SDK encoder. This implementation adds another quality level 'Filmscan 3' that is not normally available in other applications. I tested the full 'Filmscan 3' to 'Low' quality range.

- JPEG (RGB24, 100 - 85% compression range) using VapourSynth's very own ImageMagick Writer and the corresponding reader for re-importing the JPEG sequence:

I also encoded to Prores_ks HQ and DNxHR-HQX with ffmpeg CLI using the default settings.

Here are the muvsfunc VMAF, SSIM, GMSD MDSI and ffmpeg SSIM/PSNR results for Prores HQ and Cineform:

http://i.imgur.com/EywOupol.png (https://imgur.com/EywOupo)

http://i.imgur.com/ICbJOIMl.png (https://imgur.com/ICbJOIM)

Nothing that remarkable about the muvsfunc VMAF, SSIM and GMSD results. For ffmpeg SSIM I also plotted the aggregate component Y, U and V scores. The Cineform 'Filmscan 3' scores were only marginally higher than 'Filmscan 2'. Clearly, the VMAF metric has no value in this context.

What is interesting in this case are the MDSI results. With Prores (and 'downsample' applied) the scores plateaued around q10 and actually decreased as the quality level was increased to q2. The MDSI scores with 'downsample' disabled also plateaued, but at a higher 'quality level' (around q4). The Cineform MDSI results do not show this behaviour.

I decided to examine this further. It made little difference if the Prores imports (and MagicYUV source) were converted to RGB48 or RGB24, with or without dithering. What did make a difference was converting the Prores and reference (MagicYUV) clips to greyscale before the RGB conversion - it dramatically relieved the apparent 'score saturation', implicating the chroma component:

http://i.imgur.com/d6GBbKdl.png (https://imgur.com/d6GBbKd)

The MDSI metric score is derived from pooled Gradient (Luminance) and Chromacity similarity assessments. I asked WolframRhodium if it might be possible to generate the Gradient (GS), Chromacity (CS) and pooled Gradient-Chromacity (GCS) maps and he has kindly obliged:

https://forum.doom9.org/showthread.php?p=1867762#post1867762

The generated map traces are very faint but they can be brought up quite nicely with Unsharp Mask. For the tests with downsample enabled I applied 2-passes (Radius 25, Amount 10) in Gimp.

Here are the similarity maps for ProRes at q2, q10, q16 and q20 quality levels, with 'downsample' enabled - I did also generate maps with 'downsample' disabled, but have yet to examine them.

http://i.imgur.com/NtIYYbLm.png (https://imgur.com/NtIYYbL)

Click image to enlarge, and then on (+) cursor to enlarge to max.

Looking at the Gradient-Chromacity Similarity (GCS) maps it's difficult to see how the final MDSI score at q2 would be less than that at q10. But the Chromacity Similarity (CS) maps definitely have an issue with the higher saturation colours on the runners shirts, right up to q2.

If anyone wants to examine them in more detail here are the matching frame grabs (VSEditor) of the imported clips (ffms2 decode) before and after conversion to RGB:

http://i.imgur.com/8OC28Tsm.jpg (https://imgur.com/8OC28Ts)

The frames in the downloaded image (right click > Save Image As) are the original 1920 x 1080 res.

And here are the similarity maps obtained with the Cineform and JPEG transcodes:

http://i.imgur.com/NNFPVlFm.png (https://imgur.com/NNFPVlF)

http://i.imgur.com/pUdCInWm.png (https://imgur.com/pUdCInW)

Big difference. Evidently MDSI is picking up chroma 'distortion' in the Prores encodes.

Bringing the clips and frame grabs into DaVinci Resolve I couldn't pick-up any obvious chroma shifts or saturation changes on the videoscopes, but I'll look at that in more detail with selective hue/saturation/luminance keys.

I don't think it's 'Quicktime' related issue per se - Cineform produced the same MDSI results whether encoded in AVI or MOV format. However, there are numerous reports of 'chroma shifts' associated with the ffmpeg Prores implementation.

The metric scores obtained with the ffmpeg CLI Prores encode (at default settings) put it on par with 'quality level' 12 using the VDub2 encoder.

More interesting still are the Butteraugli test results. They don't show this behaviour with the Prores encodes and converting to greyscale had much less impact on the scores:

http://i.imgur.com/Mk9ghqCl.png (https://imgur.com/Mk9ghqC)

In fact above Prores quality level 4 (the default), the greyscale scores were lower.

As mentioned above, the Butteraugli metric was tuned for detecting fine differences in JPEG images in the 90-95% compression domain. Note that the score drops quite markedly going from 90 to 85% quality.

Also interesting that ffmpeg DNxHR-HQX, whilst giving a 'bitrate-matched' MDSI score similar to that of Prores (VDub2), gives a much higher Butteraugli score, approaching that of JPEG 90%. Also the Butteraugli score for the ffmpeg Prores encode was lower than of Prores (VDub2) at an equivalent bitrate.

I've yet to scrutinize the 'heat maps' to see what more they can reveal.

I'm also wondering now if the MDSI scores seen earlier with the ffmpeg x264/x265 encodes might have been influenced in the same way as Prores.

WorBry

9th March 2019, 06:01

I've yet to scrutinize the 'heat maps' to see what more they can reveal.

The Butteraugli 'heat maps' and corresponding RGB24 source frames.

Click on image to enlarge, and on (+) cursor to enlarge further.

Prores HQ (VirtualDub2):

http://i.imgur.com/jvzL8C6m.jpg (https://imgur.com/jvzL8C6)

Cineform:

http://i.imgur.com/auU6yQAm.jpg (https://imgur.com/auU6yQA)

JPEG:

http://i.imgur.com/GtTYHdjm.jpg (https://imgur.com/GtTYHdj)

In the Prores series, there is very little difference between the 'full colour' and greyscale heat maps. Looking at the image components that Butteraugli is targeting in the highest quality encodes of each series, it appears to be mostly the blacks and whites.

Default dithering was applied in the fmtconv YUV422P10 > RGB24 conversion btw:

clip = core.ffms2.Source(source=r'{Path}:/10bit_422_Source.avi')
clip = core.fmtc.resample(clip=clip, css="444")
clip = core.fmtc.matrix(clip=clip, mat="709", col_fam=vs.RGB)
clip = core.fmtc.bitdepth(clip=clip, bits=8)

WorBry

9th March 2019, 19:48

Here are the MDSI similarity maps for ProRes at q2, q10, q16 and q20 quality levels, with 'downsample' enabled - I did also generate maps with 'downsample' disabled, but have yet to examine them.

http://i.imgur.com/NtIYYbLm.png (https://imgur.com/NtIYYbL)

Click image to enlarge, and then on (+) cursor to enlarge to max.

Looking at the Gradient-Chromacity Similarity (GCS) maps it's difficult to see how the final MDSI score at q2 would be less than that at q10. But the Chromacity Similarity (CS) maps definitely have an issue with the higher saturation colours on the runners shirts, right up to q2.

After further investigation it appears that the issue was with the (FFMS2) decoding.

The reference 10bit 422 source was in MagicYUV AVI format and the test Prores encodes were in MOV format. First tried converting the MagicYUV AVI reference clip to MOV, but it made no difference. But when I converted the Prores MOV files to (lossless) MagicYUV 10bit 422 AVI and re-ran the MDSI tests, lo and behold, it resolved the 'chroma distortion'. Just why, I don't know. But here are the revised MDSI similarity maps:

http://i.imgur.com/YynjVDbm.png (https://imgur.com/YynjVDb)

What's interesting is that when I re-ran the Butteraugli tests with the converted Prores clips it did not change the scores or 'heat maps' at all. And no change in the VMAF, SSIM and GMSD scores either. It only affected MDSI.

I'll post the revised MDSI score graphs in due course.

WorBry

10th March 2019, 06:24

The revised Prores metric score charts:

http://i.imgur.com/WZ1HkDLm.png (https://imgur.com/WZ1HkDL)

And the revised MDSI scores compared with those of other transcodes:

http://i.imgur.com/xgaEanem.png (https://imgur.com/xgaEane)

Now that looks a lot healthier.

The ffmpeg (CLI) Prores-ks-HQ and DNxHR-HQX transcodes both showed the 'chroma distortion' in the MDSI tests as well and converting the MOV files to MagicYUV 10bit 422 AVI resolved that also. So this was not specifically a Prores issue.

As mentioned above, it is interesting that the Butteraugli scores and heat maps were not changed at all by the conversion. Suggests it is unresponsive to subtle color shifts.

As this exercise has shown, MDSI is very sensitive to subtle color shifts and could prove to be a useful tool for assessing chroma sampling efficiencies.

Boulder

10th March 2019, 14:17

Have you compared MDSI to GMSD? Based on my Bicubic resize parameter tests with Zopti, MDSI seems to favour a sharper image than GMSD. I don't know if there are other differences.

WorBry

10th March 2019, 17:17

Well this is how GMSD and MDSI compare for this particular source:

http://i.imgur.com/r5bI4h1m.png (https://imgur.com/r5bI4h1)

That said - whilst Crowd Run serves as a good test reference for it's hard-to-compress complex/colorful content, I wouldn't say it's an especially sharp image, at least by contemporary 4K standards; there's a fair bit of motion and pan blur going on there and it was downscaled (Spline36) to 1080p for these tests also.

MDSI seems to favour a sharper image than GMSD

Might well explain why the MDSI scores plots don't taper down to approach 0 as they do with GMSD.

I'll look and see what other HQ sharp sources I can test.

Boulder

10th March 2019, 17:56

WorBry

10th March 2019, 19:11

OK thanks. I'll have a look at it.

zorr

10th March 2019, 22:54

After further investigation it appears that the issue was with the (FFMS2) decoding.

Great detective work WorBy. MDSI looks pretty good in my tests too because it's more "picky" than the other metrics. Even if that's not the most important feature it can be an advantage.

WorBry

10th March 2019, 23:25

I spent a good while staring at images, videoscopes and selective H,S,V keys in DaVinci Resolve and could not see any chroma shifts in the Prores clips - came to the conclusion it must be occurring on VapourSynth import.

MDSI looks pretty good in my tests too because it's more "picky" than the other metrics.

I need to do more tests to reach a firm conclusion about MDSI. GMSD is really growing on me - as you've observed in your Zopti studies it does appear to be slightly more focused than SSIM (muvsfunc). But it's good to have a sensitive metric like MDSI that takes into account luminance (gradient) and chromacity also. Pity it doesn't provide separate luminance and chromacity measures. Still, a lot of can be gleaned from the similarity maps, as in this case. Running a parallel test in greyscale also.

I'm looking at Boulder's clip just now.

Incidentally, what would be good in-line option for 'normalizing' the very faint/nigh-on-invisible MDSI similarity maps in VapourSynth? Two Unsharp Mask passes in Gimp does it nicely, but it's a bit tedious copying frame grabs, especially when there's little visibly to guide the frame selection.

Boulder

11th March 2019, 04:55

Incidentally, what would be good in-line option for 'normalizing' the very faint/nigh-on-invisible MDSI similarity maps in VapourSynth? Two Unsharp Mask passes in Gimp does it nicely, but it's a bit tedious copying frame grabs, especially when there's little visibly to guide the frame selection.

Maybe something like this would do?

result = core.std.Levels(result, min_in=127, max_in=132, min_out=0, max_out=255, planes=0)

For other than 8-bit clips, the values must be bigger. For 16-bit I use result = core.std.Levels(result, min_in=125*256, max_in=132*256, min_out=0, max_out=255*256, planes=0).

WorBry

12th March 2019, 04:58

Here's one from Black Sails season 1, which is a very high quality 1080p release. The texture of characters skins is very detailed-looking and sharp. A lot of the content in the series has shaky camera movement but this scene is rather still compared to others.

https://drive.google.com/open?id=1mZz-woQvyl949qCnZIXqKQE3igWDeRHg

So I took a 12.5 sec section (frames 563 - 864) from the clip that spanned the head shots of the three actors, and converted to lossless MagicYUV YV12.

Since it's 8-bit 420 thought I may as well do a x264 vs x265 comparison but limited to the 'high-end" CF range (x264 CRF 1-5 and x265 CRF 0 -3) and Intra-frame only - I anticipated taking frame shots.

To avoid any decode issues, I converted the encodes to MagicYUV YV12 for the GMSD and MDSI tests. Ran the tests with and without downsample. Also ran a parallel series (with down-sampling) with the source and test clips converted to greyscale, which I did in AVISynth.

Here are the results:

http://i.imgur.com/PmeSy7Pm.png (https://imgur.com/PmeSy7P)

Both GMSD and MDSI consistently found differences between x264 and x265. The pattern is actually quite similar to that seen earlier with Crowd Run at the high bitrates:

http://i.imgur.com/QjFgFDMm.png (https://imgur.com/QjFgFDM)

Running the tests with greyscale clips improved the MDSI scores marginally. As noted in the Crowd Run 10bit 422 test series, converting to greyscale had no effect on the GMSD scores, as you might expect.

So what differences are GMSD and MDSI picking up between x264 and x265 ? Thought I'd see what the 'quality maps' could reveal. Fortuitously, x264 CRF=1 and x265 CRF=0 were both 161 Mbps. So I picked a frame (#778, I think, in the original mkv clip) that showed a visible map trace with both GMSD and MDSI (no downsample) in the x264 clip, generated the quality maps and amplified them with a two-pass Unsharp Mask in Gimp.

Here are the maps from the tests with downsample applied:

http://i.imgur.com/auyrVRsm.png (https://imgur.com/auyrVRs)

Very faint aren't they? I was tempted to run a third Unsharp pass but, for consistency, left it there.

And with no downsample:

http://i.imgur.com/QAA09zqm.png (https://imgur.com/QAA09zq)

And the matching frames (VSEditor YUV420P8 output) from the source and test clips:

Source:
http://i.imgur.com/R0D4zvdm.png (https://imgur.com/R0D4zvd)

x264:
http://i.imgur.com/8geESDam.png (https://imgur.com/8geESDa)

x265:
http://i.imgur.com/TygrruUm.png (https://imgur.com/TygrruU)

Clearly (in the 'no downsample' series at least), both the GMSD quality map and MDSI Gradient Similarity (GS) map are picking up distortions in the x264 clip that represent more than fine detail. The GMSD map for the x265 clip is markedly less dense. The MDSI GS map for x265 shows more of a selective improvement - see how traces representing fine detail on the actors forehead, around the eyes and moustache are greatly diminished - the striped shirt sleeve pattern also. In the pooled Gradient-Chromacity (GCS) map that gets offset by the faint Chromacity (CS) map, but there also you can see an improvement in the x265 clip trace.

Anyhow, there it is. I was thinking about applying some controlled distortions to the source clip to see how GMSD and MDSI respond but I'm not sure I'll have time.

Boulder

12th March 2019, 05:16

Quite interesting that the maps show such an amount of difference at that bitrate. I think that there you can see the fundamental difference between x264 and x265, the first one has blocking/is more focused on enhancing the edges and higher frequencies by creating "fake detail" at default settings while the latter one likes to blur more.

WorBry

12th March 2019, 05:55

The maps also support Zorr's perception that MDSI tends to be more 'picky':

MDSI looks pretty good in my tests too because it's more "picky" than the other metrics.

BTW, I tried:

result = core.std.Levels(result, min_in=127, max_in=132, min_out=0, max_out=255, planes=0)

But inserted after....

result = core.std.SetFrameProp(result, prop='_Matrix', delete=True)

...to generate the MDSI maps, it outputs a black frame.

WorBry

13th March 2019, 16:02

I think that there you can see the fundamental difference between x264 and x265, the first one has blocking/is more focused on enhancing the edges and higher frequencies by creating "fake detail" at default settings while the latter one likes to blur more.

I think you're right. Here are crops from another frame.

http://i.imgur.com/NR57nspm.png (https://imgur.com/NR57nsp)

Click image to enlarge and (+) cursor to enlarge further

The x265 image definitely has more blur than x264 (notably on the skin textures), which MDSI deems more acceptable.

ifb

13th March 2019, 22:24

How about a quick XAVC and/or AVC Intra test? It always seemed to me that prores_aw/ks was blurry compared to DNxHD, but DNxHD would get blocky pretty easily. I'm curious how the AVC variants do since they forbid deblocking.

WorBry

13th March 2019, 23:39

I don't have means for encoding bone fide XAVC or AVC-Intra, if that's what you mean ?

ifb

14th March 2019, 02:07

I don't have means for encoding bone fide XAVC or AVC-Intra, if that's what you mean ?
Something like:
x264 foo -o xavc.h264 --output-csp i422 --output-depth 10 --avcintra-class 100 --avcintra-flavor sony --level 4.2 --sar 1:1

The "panasonic" flavor will generate AVC-Intra. There's also Class 200 (doubles the bitrate).

WorBry

14th March 2019, 04:28

ifb

14th March 2019, 15:19

I'm not that familiar with x264.exe command line encoding - only ever used ffmpeg for encoding pseudo AVC-Intra 100 and XAVC-I Class 480 in the past, and that was not using this -avcintra-flavor option.

Transcoding the Crowd Run 1080/50p 10bit 422 master with your command as is, the resulting bitrate is 227 Mbps - shouldn't it be 100 Mbps for avcintra-class 100?

Edit: Ah, that's because its 50p ?

http://www.xavc-info.org/xavc/share/data/XAVC_Profiles_and_OperatingPoints_120_Amd1.pdf
The names break down for 50p/60p rates. Bitrate switches to 2x the class name. Same for UHD where Class 300/480 is 600/960 Mbps, but there's no support in x264 for that right now anyway.

For ffmpeg, you can pass custom params to libx264:
-c:v libx264 -x264-params avcintra-class=100:avcintra-flavor=sony

With avfs, you can set enable_v210 = True and encode with Adobe Media Encoder if you have that available. It was terrible at XAVC the last time I tried it. YMMV.

WorBry

14th March 2019, 21:56

OK, so I tested CrowdRun 1080/50p 10bit 422 encoded to AVC-Intra 100 (227 Mbps) and AVC-Intra 200 (436 Mbps), both 'Sony flavour'.

Here are the SSIM, GMSD, and MDSI scores (with and without downsample) compared with those of the other formats:

http://i.imgur.com/VCITk0sm.png (https://imgur.com/VCITk0s)

I was quite surprised.

Didn't test Butteraugli. I don't think it provides more useful information, in this context at least.

Interesting that ffmpeg DNxHR-HQX gives higher scores then Prores HQ when no downsampling is applied.

ifb

15th March 2019, 03:44

I'm not too surprised. AVC is more advanced than the other intra codecs you're testing, even if they limit the codec a bit (no deblocking, fixed slice sizes, custom CQM, no CABAC). I wasn't sure how it would turn out given the two bitrates you can choose from.

It'd be interesting to see an XF-HEVC comparison too.

WorBry

15th March 2019, 04:31

It'd be interesting to see an XF-HEVC comparison too.

Can ffmpeg even ingest Canon XF-HEVC (10bit 422) camera footage yet ?

ifb

16th March 2019, 02:45

Can ffmpeg even ingest Canon XF-HEVC (10bit 422) camera footage yet ?
Dunno. I have an XF705 sample but haven't looked at it yet.

I did see a (flawed) comparison between the in-camera HEVC on the Fuji X-T3 and an external recorder (ProRes). I thought HEVC won easily.

I'm not sure that intra coding in HEVC is that much better than AVC? Of course it might be a moot point if XF-HEVC was crippled for the sake of reducing complexity/power consumption. I haven't seen any docs on the format, so I just don't know.

Iron_Mike

16th March 2019, 03:59

from zoptilib import Zopti

# read input video
orig = core.ffms2.Source(source=r'source.avi')

# initialize output file and chosen metrics
zopti = Zopti('results.txt', metrics=['ssim', 'gmsd'])

# ... process the video ...
# alternate = some_process(orig)

# measure similarity of original and alternate videos, save results to output file
zopti.run(orig, alternate)

quick question:

trying to run some GMSD, SSIM and MDSI metrics via Zopti - same as outlined on page 1 in the thread...

when I specify all three metrics as a list to Zopti, it creates a log file w/ the three results... when I then run the same distorted clip again via Zopti but only use ONE of the metrics (e.g.) GMSD, I get a log file with only one metric but the result does not match any of the results in the log file that contains all three metrics...

Example - all on the same distorted clip

GMSD, SSIM, MDSI
stop 123.25129597713656 228.20727309992276 230.492983448053

GMSD
stop 83.77192685888292

SSIM
stop 431.9516453872494

What am I missing ?

I'm running Zopti via a .vpy file executed by vspipe...

Thanks.

WorBry

16th March 2019, 06:40

Would you mind posting your script(s) for the 3-in-1 and individual metric tests ?

poisondeathray

16th March 2019, 06:46

Works for me . Same values in the same colorspace

MDSI requires RGB , so I converted to RGB for the 3 way test . Same values for the individual runs in RGB

But I got the same values within YUV for 2 way test GMSD ,SSIM, as single runs (within YUV, different values than converted RGB)

What source filter were you using ? When something is off like that, usually it's a frame mismatch

Iron_Mike

16th March 2019, 08:32

Would you mind posting your script(s) for the 3-in-1 and individual metric tests ?

I'm using same file you've used for some of your tests: Crowdrun 1080p50 420, 500 frames (crowd_run_1080p50.y4m)

The encoded file was created via ffmpeg x265 CRF 28 preset slow

vid_ref = core.ffms2.Source(source=ref_fp)
vid_enc = core.ffms2.Source(source=enc_fp)
log_fp = r'd:\zopti.log'

metrics = ('gmsd', 'ssim')
matrix = None

zopti = Zopti(log_fp, metrics=metrics, matrix=matrix)
zopti.addParams('ssim', dict(downsample=False, show_map=False))

zopti.run(vid_ref, vid_enc)

Works for me . Same values in the same colorspace

MDSI requires RGB , so I converted to RGB for the 3 way test . Same values for the individual runs in RGB

But I got the same values within YUV for 2 way test GMSD ,SSIM, as single runs (within YUV, different values than converted RGB)

What source filter were you using ? When something is off like that, usually it's a frame mismatch

thank for the hint that MDSI gets converted to RGB... that changes all values for GMSD and SSIM (if requested in the same run), which makes this absolutely pointless...

ONLY MDSI should be done in RGB, everything else in YUV, so that results stay consistent...

but even with that, the results using the Zopti class are as unstable as I've seen...

look at this... consecutive execution of the exact same script, just the requested metrics have been changed in each run - only using GMSD and SSIM... this is odd to say the least.... :confused:

GMSD #1
stop 82.89543157743563

SSIM #1
stop 436.64183973524297

GMSD #2
stop 95.29160100865921

GMSD #3
stop 82.89543157743563

GMSD #4
stop 82.89543157743563

GMSD #5
stop 82.89543157743563

GMSD #6
stop 82.89543157743563

GMSD, SSIM - #1
stop 122.23059696996059 236.93252855842488

GMSD, SSIM - #2
stop 121.9632105098167 237.4638095431857

SSIM #2
stop 436.641839735243

GMSD, SSIM - #3
stop 82.89543157743563 436.641839735243

GMSD, SSIM - #4
stop 122.14837149649146 236.40679167570883

GMSD, SSIM - #5
stop 123.8819626481414 228.86426005799092

GMSD, SSIM - #6
stop 120.18331285114424 246.86257738654993

ChaosKing

16th March 2019, 10:00

Iron_Mike

16th March 2019, 12:46

Check your video file with https://github.com/theChaosCoder/vapoursynth-portable-FATPACK/blob/master/VapourSynth64Portable/VapourSynth64/seek-test.py
python.exe seek-test.py video.mp4 0 100
0 100 = start_frame end_frame

tested the encoded/distorted clip, result

Press 1 for FFMS2000
2 for L-SMASH-Works
3 for D2V Source
4 for AVISource
5 for FFMS2000(seekmode=0) [slow but more safe]
Number: 5
ffms2seek0
Clip has 500 frames.
Hashing: 80%
Clip hashed.

Test complete. No seeking issues found :D

ChaosKing

16th March 2019, 12:56

Have you also tested with number 1 (without seekmode=0)?

WorBry

16th March 2019, 16:32

I'm using same file you've used for some of your tests: Crowdrun 1080p50 420, 500 frames (crowd_run_1080p50.y4m)

The encoded file was created via ffmpeg x265 CRF 28 preset slow

Actually I converted the crowd_run_2160p50.y4m file to lossless 1080/50p x264 CRF0 Intra mp4 to serve as the 'Master' source and reference for the 1080/50p tests.

For the 2160/50p tests I used the original crowd_run_2160p50.y4m file as the 'Master'.

Re-testing the files I have archived, I get the following raw scores for 1080/50p x265 CRF28 (default settings, except -preset slow):

SSIM:GMSD (together): 478.4192336998458;43.506163861971956

SSIM(alone) : 478.4192336998458
GMSD (alone): 43.506163861971956

Exact same scores whether tested together or separately.

I'm using ChaosKing's Portable Fatpack VapourSynth and running the scripts with VSEditor > (F7) Benchmark. Now using Wolfberry's FFMS2 build:

https://forum.doom9.org/showthread.php?t=176198

poisondeathray

16th March 2019, 16:56

thank for the hint that MDSI gets converted to RGB... that changes all values for GMSD and SSIM (if requested in the same run), which makes this absolutely pointless...

ONLY MDSI should be done in RGB, everything else in YUV, so that results stay consistent...

The point is you should get the same consistent results in RGB (the multi run should be same as individual runs in RGB). If you didn't - it would add support that there was something wrong .

You did not clarify how you converted to RGB or any of the other procedures, so maybe there were other errors in your procedure?

So I even added the 2way YUV runs, which were the same within YUV as the individual YUV runs - and it looks like you're still getting different inconsistent results

I did multiple 5 runs of the same script, everything is consistent here

One thing that is odd, is sometimes the frame number will out of numerical order in the log e.g it might go 45,44,46 . I assume it has to do with the threading. But the actual values are the same.

Another difference is I did not prevent SSIM downsample (I went by the 1st script you quoted) , but that shouldn't affect GMSD , and I used vsedit to run the benchmark like WorBry . But I cannot see how using vspipe would alter the results .

Try another ffms build.

If seeking is accurate, maybe something else is up with your machine. Maybe start looking at memory integrity tests, hardware checks, temps, overheating

WorBry

16th March 2019, 19:28

Another difference is I did not prevent SSIM downsample (I went by the 1st script you quoted) , but that shouldn't affect GMSD

Disabling downsample changes the GMSD scores profoundly - examples:

https://forum.doom9.org/showthread.php?p=1866314#post1866314

...and

https://forum.doom9.org/showthread.php?p=1868858#post1868858

These are the raw SSIM and GMSD scores I get testing the x265 CRF28 encode with Downsample=False:

SSIM:GMSD (together): 446.2983305603785; 75.80431917408734

SSIM (alone): 446.2983305603785
GMSD (alone): 75.80431917408734

poisondeathray

16th March 2019, 21:32

Disabling downsample changes the GMSD scores profoundly - examples:

SSIM:GMSD (together): 446.2983305603785; 75.80431917408734

SSIM (alone): 446.2983305603785
GMSD (alone): 75.80431917408734

He only specified addParams for SSIM . Are you saying that changes GMSD too ? Don't you have to addParams('msdn' etc...) explicitly too ?

You you don't expect downsampling to change the consistency of the results .

You would expect any operation resize, or filter, or blur etc.. to affect the actual values.

But you do not expect it to change the values between multi testing at once vs. single testing; unless that filter uses some random property (e.g. a noise generator with random seed)

ie. When you run it you should get the same values when the settings are the same. You expect repeatable, consistent results within the same testing parameters. It shouldn't give you different values based on the time of the day or the phase of the moon

Iron_Mike

16th March 2019, 22:03

Actually I converted the crowd_run_2160p50.y4m file to lossless 1080/50p x264 CRF0 Intra mp4 to serve as the 'Master' source and reference for the 1080/50p tests.

For the 2160/50p tests I used the original crowd_run_2160p50.y4m file as the 'Master'.

ah, thank you for clarifying that. I was looking at some scores I got for the 1080p50 src I used and they did not perfectly match yours...

okay, I'm gonna convert the 2160p src to see if I can match your results throughout the thread - as a little (although not perfect) unit test. :D

Could you state the command how you converted to "1080/50p x264 CRF0 Intra mp4" ? (just to be sure I match your result)

I'm using ChaosKing's Portable Fatpack VapourSynth and running the scripts with VSEditor > (F7) Benchmark. Now using Wolfberry's FFMS2 build:

https://forum.doom9.org/showthread.php?t=176198

I'm using CK's Fatpack as well.

Btw, when I ran extensive tests before (using CK's Fatpack and VSPipe) on the VS VMAF implementation (vs. ffmpeg VMAF), there were never any inconsistencies...

Iron_Mike

16th March 2019, 22:14

Have you also tested with number 1 (without seekmode=0)?

just did that, and yes it found many errors... seeking seems to be 3 frames off

Press 1 for FFMS2000
2 for L-SMASH-Works
3 for D2V Source
4 for AVISource
5 for FFMS2000(seekmode=0) [slow but more safe]
Number: 1
ffms2
Clip has 500 frames.
Hashing: 80%
Clip hashed.

Requested frame 478, got frame 481.
Previous requests: 478
Requested frame 358, got frame 361.
Previous requests: 478 358
Requested frame 284, got frame 287.
Previous requests: 478 358 83 284
Requested frame 274, got frame 277.
Previous requests: 478 358 83 284 274
Requested frame 242, got frame 245.
Previous requests: 478 358 83 284 274 242
Requested frame 214, got frame 217.
Previous requests: 478 358 83 284 274 242 98 214
Requested frame 195, got frame 198.
Previous requests: 478 358 83 284 274 242 98 214 30 195
Requested frame 200, got frame 203.
Previous requests: 358 83 284 274 242 98 214 30 195 11 200
Requested frame 381, got frame 384.
Previous requests: 83 284 274 242 98 214 30 195 11 200 381
Requested frame 184, got frame 187.
Previous requests: 284 274 242 98 214 30 195 11 200 381 184
Requested frame 405, got frame 408.
Previous requests: 98 214 30 195 11 200 381 184 5 55 405
Requested frame 436, got frame 439.
Previous requests: 30 195 11 200 381 184 5 55 405 33 436
Requested frame 275, got frame 278.
Previous requests: 195 11 200 381 184 5 55 405 33 436 275
Requested frame 425, got frame 428.
Previous requests: 200 381 184 5 55 405 33 436 275 47 425
Requested frame 247, got frame 250.
Previous requests: 381 184 5 55 405 33 436 275 47 425 247
Requested frame 395, got frame 398.
Previous requests: 184 5 55 405 33 436 275 47 425 247 395
Requested frame 346, got frame 349.
Previous requests: 5 55 405 33 436 275 47 425 247 395 346
Requested frame 427, got frame 430.
Previous requests: 55 405 33 436 275 47 425 247 395 346 427
Requested frame 323, got frame 326.
Previous requests: 436 275 47 425 247 395 346 427 3 499 323
Requested frame 160, got frame 163.
Previous requests: 275 47 425 247 395 346 427 3 499 323 160
Requested frame 111, got frame 114.
Previous requests: 47 425 247 395 346 427 3 499 323 160 111
Requested frame 145, got frame 148.
Previous requests: 395 346 427 3 499 323 160 111 76 52 145
Requested frame 316, got frame 319.
Previous requests: 346 427 3 499 323 160 111 76 52 145 316
Requested frame 158, got frame 161.
Previous requests: 427 3 499 323 160 111 76 52 145 316 158
Requested frame 122, got frame 125.
Previous requests: 3 499 323 160 111 76 52 145 316 158 122
Requested frame 230, got frame 233.
Previous requests: 499 323 160 111 76 52 145 316 158 122 230
Requested frame 251, got frame 254.
Previous requests: 160 111 76 52 145 316 158 122 230 94 251
Requested frame 253, got frame 256.
Previous requests: 111 76 52 145 316 158 122 230 94 251 253
Requested frame 166, got frame 169.
Previous requests: 76 52 145 316 158 122 230 94 251 253 166

... (many more)

Test complete. Seeking issues found :-(

so why has this encoded file seeking issues ?

or is it my ffms2 version ? (it's the one from your Fatpack)

WorBry

16th March 2019, 22:19

He only specified addParams for SSIM . Are you saying that changes GMSD too ? Don't you have to addParams('msdn' etc...) explicitly too ?

Sorry, misunderstanding on my part.

When you run it you should get the same values when the settings are the same. You expect repeatable, consistent results within the same testing parameters. It shouldn't give you different values based on the time of the day or the phase of the moon

Absolutely.

ChaosKing

16th March 2019, 22:20

I added the lastest build from the ffms2 thread. You could try your old ffms2 version. Maybe something broke. L-SMASH is usually a bit more safe and is frame accurate for many more containers like m2ts or even vob.

p.s. that is why it is always good to test for seeking issues first. I have a h264 mkv but every ffms2 version has seeking issues with that file for some reason. lsmash no problemo.

WorBry

16th March 2019, 22:27

Could you state the command how you converted to "1080/50p x264 CRF0 Intra mp4" ? (just to be sure I match your result)

Probably better that I run the tests again with crowd_run_1080p50.y4m as source and post my results for x265 CRF28, rather than introducing another variable.

poisondeathray

16th March 2019, 22:51

I have a h264 mkv but every ffms2 version has seeking issues with that file for some reason. lsmash no problemo.

Is that with seekmode=0, threads=1 too ?

Did you examine that mkv to see what characteristics cause the problem ?

Can you share the file ? It would be a good debugging test clip

zorr

16th March 2019, 22:54

Iron_Mike

16th March 2019, 23:01

Now using Wolfberry's FFMS2 build:

https://forum.doom9.org/showthread.php?t=176198

I just exchanged the FFMS2 plug in the Fatpack w/ Wolfberry's latest version...

same result run CK's seek test script (on the same encoded video file), it is always 3 frames off...

I also ran the #1 FFMS2 seek test on the original download of "crowd_run_1080p50.y4m" - zero issues found.

Alright, so I assume my encodes are bad...

did multiple re-encodes using Wolfberry's build or Zeranoes build... all have frame seeking issues, always 3 frames off...

here's the ffmpeg command:

<ffmpeg path> -y -thread_queue_size 8092 -i <path/to/crowd_run_1080p50.y4m> -c:v libx265 -preset slow -crf 24 -color_range tv -color_primaries bt709 -color_trc bt709 -colorspace bt709
-pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:rc-lookahead=100" -r 50 <path/to/output.mp4>

I also tested w/o specifying colorspace and keyframes... same result, 3 frames off

any pointers what I'm doing wrong in my encoding settings ?

ifb

16th March 2019, 23:03

Can ffmpeg even ingest Canon XF-HEVC (10bit 422) camera footage yet ?Yes. It was added (http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=f95aee2b72535e14b7463750fd7afb6d1cdbe4d4) to the MXF demuxer 8 days ago.

Samples (https://www.dropbox.com/sh/uam3s1bvralba07/AADfc7RvwmhEA-rLJ8pDjz8la?dl=0)

ChaosKing

16th March 2019, 23:16

Is that with seekmode=0, threads=1 too ?

Did you examine that mkv to see what characteristics cause the problem ?

Can you share the file ? It would be a good debugging test clip

Not yet. I will see if I can make a cut or maybe if remuxxing the file solves the issue :D

poisondeathray

16th March 2019, 23:20

I did some tests too. With SSIM and GMSD the first two runs gave identical results, but the third one gave a different one. The difference is not big - starts with the 6th decimal - but it should not happen.

stop 49.08846907182173 3.484012159548981 3762.3334999999825
stop 49.088468378240414 3.4840229354201178 3428.1400000000417

There's no RGB conversion involved here so it has to be something else.

Looking at the individual frame results:

0; 0.9794317688604798; 0.07659219540647606; 0.0;
1; 0.9793688841540404; 0.07778895381762808; 0.0;
2; 0.9801722825175584; 0.07465046759649577; 0.0;
3; 0.9805196896947995; 0.07238442343729419; 0.0;

0; 0.9794317688604798; 0.0765921875457907; 0.0; <--- GMSD different from 8th digit
1; 0.9793688841540404; 0.07778895381762808; 0.0;
2; 0.9801722825175584; 0.07465046759649577; 0.0;
3; 0.9805189190488873; 0.07238807894499658; 0.0; <--- SSIM and GMSD different from 6th digit

Looks like there are differences on some frames only. Does not look like a seeking issue either, the errors are much smaller than what would happen if the frame was different.

I remember seeing this kind of issue when I first tested SSIM metric with Vapoursynth but then the error disappeared and I forgot about it.

I triple checked this on 2 different sources, and I cannot reproduce

The only "weird" thing is sometimes the log file frame number is misordered , but values match every run when you check in a spread sheet

same result run CK's seek test script (on the same encoded video file), it is always 3 frames off...

I also ran the #1 FFMS2 seek test on the original download of "crowd_run_1080p50.y4m" - zero issues found.

Alright, so I assume my encodes are bad...

The Y4M is I-frame only , so you should not get any seek issues

I doubt your encodes are "bad" . More likely that ffms2 build is "bad"

Iron_Mike

16th March 2019, 23:27

I triple checked this on 2 different sources, and I cannot reproduce

The only "weird" thing is sometimes the log file frame number is misordered , but values match every run when you check in a spread sheet

you need to run many tests... as shown in one of my earlier posts, sometimes it outputs the exact same result 4-5 times in a row and then it's off...

The Y4M is I-frame only , so you should not get any seek issues

I doubt your encodes are "bad" . More likely that ffms2 build is "bad"

hah, that would explain it ! :D

@WorBry:

(1) what OS are you using to do your tests ?
(2) can you run the seek-test.py script from CK on one of your crowdrun encodes and see if it reports seeking errors ?

poisondeathray

16th March 2019, 23:31

you need to run many tests... as shown in one of my earlier posts, sometimes it outputs the exact same result 4-5 times in a row and then it's off...

Well I assumed 5 times was enough...I'll run 5 more for a total of 10 and report back if there are differences

But I don't have seek errors either...

WorBry

16th March 2019, 23:35

Probably better that I run the tests again with crowd_run_1080p50.y4m as source and post my results for x265 CRF28, rather than introducing another variable.

So, I encoded crowd_run_1080p50.y4m to x265 CRF28 mp4:

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx265 -preset slow -crf 28 -pix_fmt yuv420p -r 50/1 -x265-params colorprim=1:transfer=1:colormatrix=1 {Path):/crowd_run_1080p50_x265_CRF28.mp4

Used Zeranoe nightly build - ffmpeg-20190312-d227ed5-win64-static.

Ran the SSIM and GMSD tests as before:

Downsample=True:

SSIM;GMSD (together): 475.68958616657005; 45.96858528255874
SSIM (alone): 475.68958616657005
GMSD (alone): 45.96858528255874

Downsample=False:

SSIM;GMSD (together): 436.07619188850276; 83.3646771904963
SSIM (alone): 436.07619188850276
GMSD (alone): 83.3646771904963

Edit: Observed no change in any of the scores over ten consecutive runs.

ChaosKing

16th March 2019, 23:40

OK, so with the "old" ffms2 from here https://github.com/FFMS/ffms2/releases my h264 file has no seeking issues. So I guess the newer ffmpeg version or something else is causing this problem. But other h264 files are ok.
5 sec test file link (https://www.dropbox.com/s/tdaaygh1wqfgtax/ffms2_seeking_issue.mkv?dl=1)

Maybe it would be a good idea to gather some files and make a script to test new releases against them.

poisondeathray

16th March 2019, 23:45

10 runs all exactly the same, not even the 6th or 8th decimal place variation zorr is reporting

I used 2 different test clips, a 300 frame 1920x1080, 1000 frame clip 1920x1080 (not crowdrun) . Both x265 encoded with default settings

@Iron_Mike - What ffms2 version are you using that has seek issues ?

Can you post a "problem" file that exhibits seek issues ?

Thanks CK for the h264 "seek" problem testclip . But to be clear, Iron_Mike is using h265

Iron_Mike

17th March 2019, 00:38

@Iron_Mike - What ffms2 version are you using that has seek issues ?

can VS output the version number of a plugin ? if so, what is the command to get the version number of a specific VS plugin ?

OK, so with the "old" ffms2 from here https://github.com/FFMS/ffms2/releases my h264 file has no seeking issues. So I guess the newer ffmpeg version or something else is causing this problem. But other h264 files are ok.

Thank you for that, this FFMS2 v2.23.1 does not have any seek issues. Ran your seek test script in mode #1 and zero issues !

So, I encoded crowd_run_1080p50.y4m to x265 CRF28 mp4:

Used Zeranoe nightly build - ffmpeg-20190312-d227ed5-win64-static.

Ran the SSIM and GMSD tests as before:

Downsample=True:

SSIM;GMSD (together): 475.68958616657005; 45.96858528255874
SSIM (alone): 475.68958616657005
GMSD (alone): 45.96858528255874

Downsample=False:

SSIM;GMSD (together): 436.07619188850276; 83.3646771904963
SSIM (alone): 436.07619188850276
GMSD (alone): 83.3646771904963

Edit: Observed no change in any of the scores over ten consecutive runs.

used same Zeranoe build (20190312-d227ed5) and encoded with same settings, then tried to run the GMSD/SSIM tests...

with the old FFMS2 version (that does not have seeking issues), vspipe crashes now and vseditor crashes as well when executing the .vpy script...

how are you reading the encoded files, are you using ffms2 as well ? if so, the ffms2 Wolfberry build you mentioned earlier does have seeking issues...

so which source filter to use to read in the files ?

ifb

17th March 2019, 00:50

I'm not sure that intra coding in HEVC is that much better than AVC? Of course it might be a moot point if XF-HEVC was crippled for the sake of reducing complexity/power consumption.
To sorta answer my own question, HEVC intra is supposed to need between 17.3% (https://www.researchgate.net/profile/Detlev_Marpe/publication/241631765_Performance_analysis_of_HEVC-based_intra_coding_for_still_image_compression/links/58de431daca27206a8a4e8b9/Performance-analysis-of-HEVC-based-intra-coding-for-still-image-compression.pdf) and 22.3% (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.352.3008&rep=rep1&type=pdf)/22.4% (https://www.researchgate.net/publication/327987700_Intra_Coding_Performance_Comparison_of_HEVC_H264AVC_Motion-JPEG2000_and_JPEGXR_Encoders/download) less bitrate than AVC intra for equivalent quality (BD-BR). I haven't found any numbers for AV1 that aren't paywalled yet.

WorBry

17th March 2019, 01:42

how are you reading the encoded files, are you using ffms2 as well ? if so, the ffms2 Wolfberry build you mentioned earlier does have seeking issues...

Yes, I'm using Wolfberry's ffms2 build.

I'm using ChaosKing's Portable Fatpack VapourSynth and running the scripts with VSEditor > (F7) Benchmark. Now using Wolfberry's FFMS2 build..

Iron_Mike

17th March 2019, 02:09

Yes, I'm using Wolfberry's ffms2 build.

I've used the one from your link, and it also had seeking issues...

can you test one of your encodes w/ CK's seek test script to see if they're also 3 frames off ?

ChaosKing

17th March 2019, 02:17

And have you tried lsmash, that should be ok.

WorBry

17th March 2019, 02:22

can you test one of your encodes w/ CK's seek test script to see if they're also 3 frames off ?

Yes, I've just done that with the Crowd Run x265 CRF28.mp4 encode and it (#5 - seekmode=0) reports 'No seeking issues found'. I'll test #1 also.

Edit: With #1, it's reporting the 3 frames off thing.

Iron_Mike

17th March 2019, 02:44

Yes, I've just done that with the Crowd Run x265 CRF28.mp4 encode and it (#5 - seekmode=0) reports 'No seeking issues found'. I'll test #1 also.

Edit: With #1, it's reporting the 3 frames off thing.

alright, so that matches my results over here.

so, does that mean we should not use this ffms2 version to read source files ?

how many of your tests in this thread were done w/ this ffms2 version ? I believe you initially stated you were reading files via vdub ?

is there a VS command to print the version of a specific plug ?

WorBry

17th March 2019, 02:49

Iron_Mike

17th March 2019, 03:53

And have you tried lsmash, that should be ok.

just did that, it reports "[hevc @ 0000009525d3eb20] missing picture in access unit"

but the seek test was successful (no errors).

WorBry

17th March 2019, 04:41

I only started using Wolberry's ffms2 build today, for these tests. All previous tests were with the ffms2 build that came with the VapourSynth Fatpack Portable package, including the time that I was running the metric tests with the installed version of VapourSynth and VirtualDub2.

I've re-tested the CrowdRun x265 CRF28.mp4 encode from..

https://forum.doom9.org/showthread.php?p=1869094#post1869094

....with the original ffms2 version that came with VapourSynth Fatpack Portable and L-Smash Source (LWLibavSource) and both produce exactly the same SSIM and GMSD scores as Wolfberry's ffms2 did:

Downsample=True:

SSIM;GMSD (together): 475.68958616657005; 45.96858528255874
SSIM (alone): 475.68958616657005
GMSD (alone): 45.96858528255874

Downsample=False:

SSIM;GMSD (together): 436.07619188850276; 83.3646771904963
SSIM (alone): 436.07619188850276
GMSD (alone): 83.3646771904963

And the seek-test script reports 'No seeking issues found' for option # 2 'L-Smash-Works'.

WorBry

17th March 2019, 05:44

Can ffmpeg even ingest Canon XF-HEVC (10bit 422) camera footage yet ?

Yes. It was added (http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=f95aee2b72535e14b7463750fd7afb6d1cdbe4d4) to the MXF demuxer 8 days ago.

Samples (https://www.dropbox.com/sh/uam3s1bvralba07/AADfc7RvwmhEA-rLJ8pDjz8la?dl=0)

Nice.

Iron_Mike

17th March 2019, 06:09

Alright, I made some progress in figuring out what causes the issue w/ these fluctuations in GMSD/SSIM results.

it appears that when using the keyframes option (specifying a GOP size), e.g. -x265-params "keyint=100:min-keyint=100:rc-lookahead=100", it produces a file that some VS plug/module has an issue with, and therefore produce these inconsistent GSMD/SSIM metrics...

As a test to match WorBrys results, I made two encodes - in both cases I used the same source (crowdrun 1080p50), the same ffmpeg Zeranoe build, and the same ffmpeg settings that WorBry used.

But in one encode I added the keyframes/GOPsize, which is case (A).

# (A) here are the GSMD/SSIM (Downsample=False) results of video encoded w/ keyframes being set - 4 consecutive metrics calculation using the exact same vpy script

stop 122.04975401899155 236.93252855842488
stop 119.92695987646071 248.23415308822817
stop 82.89543157743563 436.64183973524297
stop 124.03537883254465 227.42573587299864

--> results in random metrics score

# (B) here are the GSMD/SSIM (Downsample=False) results of video encoded w/o setting keyframes - 10 consecutive metrics calculation using the exact same vpy script

stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.0761547550152
stop 83.36467719049627 436.07615475501524
stop 83.3646771904963 436.0761547550152

--> while the encode w/o keyframes (B) produces pretty much the same metrics repeatedly, two things to note:

(1) 5 out of 10 tests have more precision - more decimal points
(2) the SSIM only matches the first few digits of WorBrys results...

Downsample=False:

SSIM;GMSD (together): 436.07619188850276; 83.3646771904963
SSIM (alone): 436.07619188850276
GMSD (alone): 83.3646771904963 [/CODE]

Edit: Observed no change in any of the scores over ten consecutive runs.

why do these plugs not consistently produce the same result ?

Iron_Mike

17th March 2019, 06:28

I've re-tested the CrowdRun x265 CRF28.mp4 encode from..

https://forum.doom9.org/showthread.php?p=1869094#post1869094

....with the original ffms2 version that came with VapourSynth Fatpack Portable and L-Smash Source (LWLibavSource) and both produce exactly the same SSIM and GMSD scores as Wolfberry's ffms2 did:

And the seek-test script reports 'No seeking issues found' for option # 2 'L-Smash-Works'.

option #2 always resulted in success, #1 is the one that reports seek errors...

question is if the seek-test script is not correct, or if all plugs u mentioned above all are 3 frames off... then they would all produce the same results, which are ultimately incorrect

WorBry

17th March 2019, 07:06

option #2 always resulted in success, #1 is the one that reports seek errors...

question is if the seek-test script is not correct, or if all plugs u mentioned above all are 3 frames off... then they would all produce the same results, which are ultimately incorrect

On the contrary, it shows that regardless of what seek errors the seek-test script is reporting for ffms2, the metric scores come out correct - i.e. they are exactly the same as obtained with LWLibavSource.

Why do you say 'or if all plugs u mentioned above all are 3 frames off' when the seek-test reports no issues with LWLibavSource ?

Also what is the conclusion when (ffms2) option #5 (seekmode=0) reports 'No seeking issues found' but (ffms2) option #1 reports seek errors - in this case the "3 frames off" thing ?

WorBry

17th March 2019, 07:15

Encoded a file w/ the same settings and same ffmpeg Zeranoe build as WorBry did using the 1080p50 crowdrun as ref file. Made two encodes, one with and one without keyframes/GOPsize specified.

When I encode w/ the keyframes option (specifying a GOP size) -x265-params "keyint=100:min-keyint=100:rc-lookahead=100".....

(2) the SSIM only matches the first few digits of WorBrys results...

Were they the exact same settings that I used:

So, I encoded crowd_run_1080p50.y4m to x265 CRF28 mp4:

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx265 -preset slow -crf 28 -pix_fmt yuv420p -r 50/1 -x265-params colorprim=1:transfer=1:colormatrix=1 {Path):/crowd_run_1080p50_x265_CRF28.mp4

...or did you add "-x265-params rc-lookahead=100" ?

Iron_Mike

17th March 2019, 09:12

On the contrary, it shows that regardless of what seek errors the seek-test script is reporting for ffms2, the metric scores come out correct - i.e. they are exactly the same as obtained with LWLibavSource.

Why do you say 'or if all plugs u mentioned above all are 3 frames off' when the seek-test reports no issues with LWLibavSource ?

must have missed the connection of the LWLibavSource, I don't know what that is... is that the lib for LSmash-Works ? (I'm not familiar w/ many of these plugs)

Alright, so, if the seek script reports no error for using LSmash, but does report errors via ffms2, yet the GMSD/SSIM scores are the exact same when the vid files are read via ffms2 or Lsmash, then one would assume that the seek test script has either flawed logic or at least the way it uses ffms2 in the #1 test is different from the way it is used when the vid files are read in your vpy script...

What is the command syntax for reading a vid file via Lsmash/LWLibavSource ?

Also what is the conclusion when (ffms2) option #5 (seekmode=0) reports 'No seeking issues found' but (ffms2) option #1 reports seek errors - in this case the "3 frames off" thing ?

+1 on that question. Since #5 is labeled as "slow, but more safe" I assumed that this is the better verification but then CK said to test via #1 as well...

Were they the exact same settings that I used:

...or did you add "-x265-params rc-lookahead=100" ?

...
edit: I edited my original post for clarity, I hope it makes more sense now ;-) https://forum.doom9.org/showthread.php?p=1869119#post1869119
...

one encode was exactly your settings - as you can see the GMSD matches but the SSIM only matches on the first 5 decimal points... (that again is odd, considering we used same source, same ffmpeg, same settings)

then another encode w/ your settings but in addition setting the keyframes/GOPsize... the resulting file always produces different GMSD/SSIM results, meaning whatever plugs/modules/classes are utilized to obtain these metrics, at least one of them has a problem w/ that file (or the keyframes in that file)... I just have zero idea why it produces always different results - maybe the frame seek returns a random frame, hence the random metric score...

btw, I ran a verification test on that encode to check if all keyframes are in the correct position (every 100 frames, hence every 2 secs) - and they are... I doin't think it's a bad encode, it's just triggers bad logic in at least one of the involved plugs/modules/classes...

ChaosKing

17th March 2019, 11:30

+1 on that question. Since #5 is labeled as "slow, but more safe" I assumed that this is the better verification but then CK said to test via #1 as well...

seekmode is a ffms2 parameter, see here https://github.com/FFMS/ffms2/blob/master/doc/ffms2-avisynth.md#int-seekmode--1

Iron_Mike

17th March 2019, 13:17

seekmode is a ffms2 parameter, see here https://github.com/FFMS/ffms2/blob/master/doc/ffms2-avisynth.md#int-seekmode--1

ah, thanks for that.

so I just re-ran the GMSD/SSIM tests but opened both files w/ ffms and seekmode=0, and the metrics score is the same.

so I guess seekmode is not used when these tests are run (since the seek test script indicated that seek mode is 3 frames off, the scores would be different)...

ChaosKing

17th March 2019, 13:48

Can you upload a small cut of your video file that you're testing?

WorBry

17th March 2019, 15:21

must have missed the connection of the LWLibavSource.

I've re-tested the CrowdRun x265 CRF28.mp4 encode from..

https://forum.doom9.org/showthread.php?p=1869094#post1869094

....with the original ffms2 version that came with VapourSynth Fatpack Portable and L-Smash Source (LWLibavSource) and both produce exactly the same SSIM and GMSD scores as Wolfberry's ffms2 did

I don't know what that is... is that the lib for LSmash-Works ? (I'm not familiar w/ many of these plugs)

You said you tested it, but got an error.

And have you tried lsmash, that should be ok.

just did that, it reports "[hevc @ 0000009525d3eb20] missing picture in access unit"

but the seek test was successful (no errors).

What is the command syntax for reading a vid file via Lsmash/LWLibavSource ?

clip = core.lsmas.LWLibavSource(source=r'{Path}:/video')

edit: I edited my original post for clarity, I hope it makes more sense now ;-) https://forum.doom9.org/showthread.php?p=1869119#post1869119

I don't know why you get inconsistent results on consecutive metric testing. I don't.

Please, no more requests for me to test this and that.

Over to you ChaosKing.

Can you upload a small cut of your video file that you're testing?

Edit: One last test.

I ran the SSIM:GMSD script with the 1080/50p x265 CRF28.mp4 encode imported with ffms2 and LWLibavSource - both plugins being the versions that came with VapourSynth Fatpack Portable.

I used the updated Zoptilib.py that provides the per frame metric scores:

https://forum.doom9.org/showthread.php?p=1868821#post1868821

In both tests, opened up the Previews in VSEditor and 'sent' to Frame #50. Here are the frame grabs:

http://i.imgur.com/xHJRr0Zm.jpg (https://imgur.com/xHJRr0Z)

Click on image to enlarge and (+) cursor to enlarge further.

Both tests exported exact same frame and per-frame SSIM/GMSD scores. Same thing stepping through the frames.

I also examined the x265 CRF26 encode in Elecard StreamEye and TMPGEnc SmartRenderer5 and they display the same frame for frame #50 i.e. frame #50 in the display sequence, but #52 in the stream sequence - it's a B frame.

Iron_Mike

17th March 2019, 23:52

You said you tested it, but got an error.
yes, using the option in the seek-test script, as CK suggested that. I couldn't run a metric test b/c I didn't have the command syntax.

clip = core.lsmas.LWLibavSource(source=r'{Path}:/video')

thank you for that.

Please, no more requests for me to test this and that.

thanks for running a couple of tests and confirming your settings.

I don't know why you get inconsistent results on consecutive metric testing. I don't.

as outlined in my other post, it's FFMS2 w/ seekmode=1 (default) that has a problem with keyframes - which will provide random, inconsistent GMSD/SSIM results.

This can be a problem for peeps running GMSD/SSIM tests on files they did not encode themselves and may not be aware of the issues w/ seekmode=1 (default).

So I ran a few more tests to confirm this.

Using ffmpeg Zeranoe 20190312-d227ed5 (https://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20190312-d227ed5-win64-static.zip), CK's Fatpack (https://forum.doom9.org/showthread.php?t=175529), and FFMS2 from Wolfberry (https://drive.google.com/open?id=1UKBIocylymTJY_Q5-OjnDTIDzGfdyvG0) and the crowd_run_1080p50.y4m (https://media.xiph.org/video/derf/y4m/crowd_run_1080p50.y4m) 1080p50 source (500 frames / 10 secs), I created three encodes.

(1) encode w/o keyint, min-keyint, rc-lookahead parameters:

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

GMSD/SSIM results - 5 consecutive metrics tests each - source filter varied

Lsmash:
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.36467719049627 436.0761547550152
stop 83.3646771904963 436.07615475501524

FFMS2 (Wolfberry) - seekmode=0:
stop 83.3646771904963 436.0761547550152
stop 83.36467719049627 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 83.36467719049627 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152

--> as you can see, w/o keyframes specified in the encode, seekmode causes no problem.

(2) encode w/ keyint, min-keyint, rc-lookahead:

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Lsmash:
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.64183973524297

FFMS2 (Wolfberry) - seekmode=0:
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243
stop 82.89543157743563 436.641839735243

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 123.93944837285996 227.9372376995322
stop 123.74699115224307 228.86426005799092
stop 82.89543157743563 436.641839735243
stop 121.66415998563879 239.07092253508384
stop 122.1422683865401 236.40679167570886

--> keyframes were specified in the encode, seekmode=1 (default) provides inconsistent metrics

And as you can see ONE of the 5 metrics (w/ seekmode=1) matches the consistent metrics from Lsmash, so this can be very misleading if you only run one test and it happens to match...

(3) encode w/ keyint, min-keyint but w/o rc-lookahead:

--> wanted to see if it is specifically the rc-lookahead setting that causes the issues w/ seekmode=1

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Lsmash:
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165

FFMS2 (Wolfberry) - seekmode=0:
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841787 435.99382191599165

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 108.17858520287257 307.76386120454794
stop 124.03602892099701 228.48869961962274
stop 83.19534706841786 435.99382191599165
stop 122.13314288222833 236.93921207380862
stop 123.97733586255691 229.28604315863697

--> same issues as (2)

Notes:

(a) you need to use seekmode=0 if using FFMS2

(b) the code does provide sometimes higher precision, or very minor variations, see above decimal points highlighted in red

(c) although the same source and only the keyframes parameter as a difference in the three encodes, the consistent metrics via Lsmash or FFMS2 w/ seekmode=0 in all three tests provide different GMSD/SSIM results

83.3646771904963 vs. 82.89543157743563 vs. 83.19534706841786 # GMSD
436.07615475501524 vs. 436.641839735243 vs. 435.99382191599165 # SSIM

although these are minor variations considering one would divide the total metric by num frames, it is a another factor that affects the score, and it shouldn't IMO

ChaosKing

18th March 2019, 01:15

I recreated your tests and came to the same conclusion.

I checked it also like this in excel:

frame gmsd ssim run2 frame run1 - run2
255 0.1665046186678081 0.8689233699845679 255 0.1665046186678081 0.8689233699845679 0 0
256 0.30940736771850696 0.8740746768904321 256 0.16450349612365942 0.8740746768904321 0,144903872 0
257 0.3089268132506728 0.8666172357253087 257 0.16778360296358616 0.8666172357253087 0,14114321 0
...
264 0.2645300712830965 0.8644133391203703 264 0.16818244844402383 0.8644133391203703 0,096347623 0
265 0.2663959966212291 0.3675497926311728 265 0.1650834566291641 0.8708016251929013 0,10131254 -0,503251833

I noticed what the first 255 frames are ok. This is also why the seek test said no seeking issues found, bcs it tests (see bat file) only the first 100 frames. Setting it to 500 showed that ffms2 has also seeking issues with the encoded x265 mp4 file.

So basically only DGDecodeNV was 100% correct, but it cuts off the last 2 frames for some reason, so I had to test with 498 frames. (maybe ffmpeg produces a "broken" stream?)

p.s. I only tested the with latest ffms2 version.
pp.s. I would be good if zopti would sort by frame before saving the log file :D
pppppp.s
stop 83.36467719049627 436.0761547550152
This indicates that it could be a decoding issue. Like 1 pixel off or something. This could be a case for inspector butteraugli.

Iron_Mike

18th March 2019, 01:49

This is also why the seek test said no seeking issues found, bcs it tests (see bat file) only the first 100 frames. Setting it to 500 showed that ffms2 has also seeking issues with the encoded x265 mp4 file.

Not sure I understand you correctly, but using Wolfberry's FFMS2 w/ this command

py seek-test.py "/path/to/crowd_run_1080p50_encode.mp4" 0 499

option #1 - seekmode=1: showed seeking issues, always 3 frames off
option #5 - seekmode=0: showed no seeking issues

pp.s. I would be good if zopti would sort by frame before saving the log file :D

+1

even better: include a header line that states the Zopti and muvsfunc version that were used in the test and specifically state which metrics are listed in the file (GMSD|SSIM|MDSI|etc) and the order in which they are listed...

otherwise nobody knows what is what unless you wrote down the metrics collection passed to the Zopti class... ;-)

zorr

18th March 2019, 01:52

This indicates that it could be a decoding issue. Like 1 pixel off or something. This could be a case for inspector butteraugli.

A small difference like that can happen when summing floating point numbers in a different order. Most likely the frame order is a bit different sometimes. That could also be solved by sorting by frame numbers before summing them up.

zorr

18th March 2019, 01:59

even better: include a header line that states the Zopti and muvsfunc version that were used in the test and specifically state which metrics are listed in the file (GMSD|SSIM|MDSI|etc) and the order in which they are listed...

otherwise nobody knows what is what unless you wrote down the metrics collection passed to the Zopti class... ;-)

Zoptilib was written for the Zopti optimizer and the log file was not meant for human consumption. :) Zopti reads the used metrics from the source so it knows. But I can see that Zoptilib can be useful without Zopti and the header would make it better. I just need to update Zopti so that it doesn't choke on the header.

Iron_Mike

18th March 2019, 02:19

Iron_Mike

18th March 2019, 02:42

Zoptilib was written for the Zopti optimizer and the log file was not meant for human consumption. :) Zopti reads the used metrics from the source so it knows. But I can see that Zoptilib can be useful without Zopti and the header would make it better. I just need to update Zopti so that it doesn't choke on the header.

btw, if it is possible for you to identify whether the metrics were calculated via YUV or RGB - since once MDSI is in the metrics list, everything goes RGB and the values change - that would also be an important parameter to state in the header...

WorBry

18th March 2019, 06:02

Using ffmpeg Zeranoe 20190312-d227ed5 (https://ffmpeg.zeranoe.com/builds/win64/static/ffmpeg-20190312-d227ed5-win64-static.zip), CK's Fatpack (https://forum.doom9.org/showthread.php?t=175529), and FFMS2 from Wolfberry (https://drive.google.com/open?id=1UKBIocylymTJY_Q5-OjnDTIDzGfdyvG0) and the crowd_run_1080p50.y4m (https://media.xiph.org/video/derf/y4m/crowd_run_1080p50.y4m) 1080p50 source (500 frames / 10 secs), I created three encodes.

(1) encode w/o keyint, min-keyint, rc-lookahead parameters:

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

GMSD/SSIM results - 5 consecutive metrics tests each - source filter varied

Lsmash:
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.36467719049627 436.0761547550152
stop 83.3646771904963 436.07615475501524

FFMS2 (Wolfberry) - seekmode=0:
stop 83.3646771904963 436.0761547550152
stop 83.36467719049627 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.07615475501524

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 83.36467719049627 436.07615475501524
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152
stop 83.3646771904963 436.07615475501524
stop 83.3646771904963 436.0761547550152

Well, i can only reiterate that in my tests I do not get these inconsistencies.

Same source, same x265 CRF28.mp4 command line (i.e. mine). Five consecutive runs of SSIM:GMSD (default, with downsample):

LWLibavSource:

Downsample=True (Default)

stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874

Downsample=False

stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963

FFMS2 (Wolfberry) -default

Downsample=True (Default)

stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874
stop 475.68958616657005 45.96858528255874

Downsample=False

stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963
stop 436.07619188850276 83.3646771904963

I assume you tested with Downsample=False. Interesting that your SSIM scores 436.0761547550152(4 ) are also marginally lower than mine 436.07619188850276, whereas the GMSD scores are the same, allowing for those inconsistencies in the last digit.

I should mention also that I ran the series' twice - first, retaining the ffindex and lwi files from the first test for the subsequent tests and then creating a fresh index file for each successive test (deleting the last one, of course). There was absolutely no difference in the results. What did you do in your tests - use the same index files or generate a fresh one for each test ?

And as a double check, I've just now imported the score data from the log files (I retained), into Excel (well, LibreOffice Calc) and, like ChaosKing, run difference calcs on the per frame scores - within each series and between the ffms2 and LWLibavSource series - and I find zero differences.

I didn't run any metric tests with ffms2 seekmode=0, but why would I ? Nor have I run any tests with x265 encoded with custom key frame and rc-lookahead settings, because I have no interest in doing so.

I would be good if zopti would sort by frame before saving the log file :D

The frames in my SSIM:GMSD test log files are all in sequence.

ChaosKing

18th March 2019, 10:17

But have you also used the fmpeg Zeranoe 20190312-d227ed5 build?

EDIT
re-muxxing the mp4 to mkv fixes everything.

The frames in my SSIM:GMSD test log files are all in sequence.
I have a Ryzen 1700 with 16 (8 real) cores. As far as I remember vsedit makes a get_frameAsync(i) call. So a unordered list can be expected.

Iron_Mike

18th March 2019, 11:32

Well, i can only reiterate that in my tests I do not get these inconsistencies.

well, did you actually read my post... ? ;-)

it is the x265 KEYFRAMES parameter that creates an encode that will throw off seekmode... all explained in detail above... simply use the provided ffmpeg commands in my post to re-create the problem

Iron_Mike

18th March 2019, 11:35

ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

https://i.imgur.com/d6WMXYh.png

ChaosKing

18th March 2019, 13:39

https://github.com/theChaosCoder/zoptilib
Zoptilib now writes an ordered log file.

But sadly it still shows small differences at the end sometimes. I will look for a more precise sum() function.

stop 83.3646771904963 stop 83.36467719049628

WorBry

18th March 2019, 13:55

well, did you actually read my post... ? ;-)

it is the x265 KEYFRAMES parameter that creates an encode that will throw off seekmode... all explained in detail above... simply use the provided ffmpeg commands in my post to re-create the problem

Yes, I read your post in great detail. And did you read mine ? I only referred to your first set of data with the x265 CRF26 created with the command line I gave you. "Simply use the provided ffmpeg commands in my post to re-create the problem" indeed - you've got a nerve. Yes I did, and I cannot replicate your inconsistent results - I get the exact same scores every time and a marginally higher SSIM score to you.

Read:
Well, I can only reiterate that in my tests I do not get these inconsistencies.

Interesting that your SSIM scores 436.0761547550152(4 ) are also marginally lower than mine 436.07619188850276, whereas the GMSD scores are the same, allowing for those inconsistencies in the last digit.

Yet you seem intent on asserting that your experience is somehow the norm i.e. inconsistencies are to be expected - when there is quite clearly something wrong.

And you didn't answer my question:

I should mention also that I ran the series' twice - first, retaining the ffindex and lwi files from the first test for the subsequent tests and then creating a fresh index file for each successive test (deleting the last one, of course). There was absolutely no difference in the results. What did you do in your tests - use the same index files or generate a fresh one for each test ?

Do you get the same dodgy results retaining the ffindex and lwi index files from the first test in a series vs creating a fresh index file for each ?

WorBry

18th March 2019, 14:14

But have you also used the fmpeg Zeranoe 20190312-d227ed5 build?

It's the other way round. He used Zeranoe 20190312-d227ed5 to replicate my test procedure:

https://forum.doom9.org/showthread.php?p=1869094#post1869094

As a test to match WorBrys results, I made two encodes - in both cases I used the same source (crowdrun 1080p50), the same ffmpeg Zeranoe build, and the same ffmpeg settings that WorBry used.

I get consistent results on serial testing. He get's these dodgy inconsistent results. How so ?

WorBry

18th March 2019, 16:05

...it is the x265 KEYFRAMES parameter that creates an encode that will throw off seekmode...

...Nor have I run any tests with x265 encoded with custom key frame and rc-lookahead settings, because I have no interest in doing so

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

And these are the results I get on serial testing with SSIM:GMSD (Downsample=False):

LWLibavSource:

stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786

ffms2- Wolfberry (default, i.e. seekmode=1)

stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786

No inconsistencies at all. And the same results regardless of whether the ffindex and lwi files from the first test in each series were retained or a fresh index file was generated for each test.

Compare that to your results.

(3) encode w/ keyint, min-keyint but w/o rc-lookahead:

--> wanted to see if it is specifically the rc-lookahead setting that causes the issues w/ seekmode=1

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Lsmash:
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165

FFMS2 (Wolfberry) - seekmode=0:
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841786 435.99382191599165
stop 83.19534706841787 435.99382191599165

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 108.17858520287257 307.76386120454794
stop 124.03602892099701 228.48869961962274
stop 83.19534706841786 435.99382191599165
stop 122.13314288222833 236.93921207380862
stop 123.97733586255691 229.28604315863697

Also, as before, I extracted the log file data into LibreOffice Calc to run difference checks on the per-frame scores and I could find zero differences at all between the serial tests and between the LWLibavSource and ffms2 test series. And the frames were listed in correct sequence (0-499) in the log files.

Go figure.

poisondeathray

18th March 2019, 16:48

What hardware were you guys running on ?

It might partially have to do with threads and cores ; The more threads, the more requests, the higher chance of frame mismatches and seek errors if you're not using a robust seek method or indexing like dgsource or seekmode=0, threads=1

CK's seektest is like wild random seeks to simulate very bad case; but usually "normal" encoding scenarios , "normal scripts", even "normal" metric testing occurs in linear order .

But small inconsistencies, like 7th decimal place etc... suggest something else wrong. Unless you have duplicate frames in the test sequence (and the mild variation is just lossy encoding differences of duplicate frames), there is something else going on. I would start to look at other things, hardware issues, run memory diagnostics

WorBry

18th March 2019, 17:26

What hardware were you guys running on ?

Actually, I've been running these tests on an older PC (6-core AMD FX-6300 Vishera 3500 Mhz, NVidia GeForce GT730, Win10 x64) that I keep for grunt encoding/processing and linux stuff (dual boot). A bit sluggish with 4K , but it gets there. For the metric tests I've let it run on 6 threads.

But small inconsistencies, like 7th decimal place etc... suggest something else wrong. Unless you have duplicate frames in the test sequence (and the mild variation is just lossy encoding differences of duplicate frames), there is something else going on. I would start to look at other things, hardware issues, run memory diagnostics

There are no duplicate frames in the 10 sec Crowd Run sequence.

poisondeathray

18th March 2019, 17:36

Note the files you guys encoded are going to be slightly different too. Even with the same commandline . Because of encoding threads. Unless you guys have the same core and thread count on your hardware, or you explictly set threads

ChaosKing

18th March 2019, 20:02

I added Decimal() for testing purposes https://github.com/theChaosCoder/zoptilib
The metric values are now much longer, sum should always be the same between runs. Please test.

Iron_Mike

18th March 2019, 21:05

Yes, I read your post in great detail. And did you read mine ? I only referred to your first set of data with the x265 CRF26 created with the command line I gave you. "Simply use the provided ffmpeg commands in my post to re-create the problem" indeed - you've got a nerve. Yes I did, and I cannot replicate your inconsistent results - I get the exact same scores every time and a marginally higher SSIM score to you.

well, if you read the post then you may wanna allude to the point that the keyframes encode setting does not trip up seekmode on your system, b/c clearly as I outlined, the keyframes setting is the only differentiating factor in the encodes I made.

if you followed exactly the commands I posted and used the exact same sw packages I posted and linked to, then it may be a system difference, as poisondeathray pointed to. I asked you pages ago, what OS you're using...

But, point remains:
I can consistently repeat the problem w/ seekmode=1 (and the keyframes encode).
CK, who repeated the procedure from my post, ended up experiencing the exact same issues on his system.

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

btw, this is not a ffmpeg specific version problem, I tested w/ three different ffmpeg versions..

Iron_Mike

18th March 2019, 21:11

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script

did you use the download of crowd_run_1080p50.y4m or did you use your own encode from the 2160p version as ref file ?

Iron_Mike

18th March 2019, 21:13

Do you get the same dodgy results retaining the ffindex and lwi index files from the first test in a series vs creating a fresh index file for each ?

tried both ways, did not make a difference w/ seekmode=1 on the keyframes encoded file...

WorBry

18th March 2019, 21:44

if you followed exactly the commands I posted and used the exact same sw packages I posted and linked ..

Good grief. You mean the sw packages I pointed you to in the first place.

I asked you pages ago, what OS you're using...

Actually, I've been running these tests on an older PC (6-core AMD FX-6300 Vishera 3500 Mhz, NVidia GeForce GT730, Win10 x64) that I keep for grunt encoding/processing and linux stuff (dual boot).

And what OS and hardware are you running the tests on ?

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

Not denying there is an issue, but I'm not seeing it.

WorBry

18th March 2019, 21:46

did you use the download of crowd_run_1080p50.y4m or did you use your own encode from the 2160p version as ref file ?

The crowd_run_1080p50.y4m file of course :rolleyes:

zorr

18th March 2019, 21:51

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 123.93944837285996 227.9372376995322
stop 123.74699115224307 228.86426005799092
stop 82.89543157743563 436.641839735243
stop 121.66415998563879 239.07092253508384
stop 122.1422683865401 236.40679167570886

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

Iron_Mike

18th March 2019, 21:56

Not denying there is an issue, but I'm not seeing it.

another pointer is that, on my end, when running CK's test script option #1 and option #5 showed both success if the file was NOT encoded w/ keyframes setting... if it was encoded w/ keyframes, then option #1 (seekmode=0) showed seek errors, always 3 frames off...

meaning: seek errors on my system are repeatable (if triggered), and are both confirmed by CK's test script and the inconsistent GMSD/SSIM results.

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

Iron_Mike

18th March 2019, 22:00

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

does that require the latest Zoptilib ?

where do I pass in the showstats parameter ? when I create an instance of the class, e.g.

zopti = Zopti(log_fp, metrics=metrics_list, showstats=True)

is that correct ?

ChaosKing

18th March 2019, 22:47

@Iron_Mike, yes it's correct.

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

showstats doesn't affect the score since the text overlay is done after the ssim gsm filters. Plus I just tested it again, just to be sure :D

alt_clip = muv.SSIM(alt_clip, clip, **filter_args).text.FrameProps()
Frameprops are showing me this:
https://i.imgur.com/TsrzNz7.png

The values are much shorter. That means Decimal() is also totally unnecessary!? And we need to round even more?
Or is FrameProps() rounding them?

WorBry

18th March 2019, 23:07

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

Iron_Mike

18th March 2019, 23:09

WorBry

18th March 2019, 23:22

Through Zoptilib

WorBry

18th March 2019, 23:48

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

And these are the results I get on serial testing with SSIM:GMSD (Downsample=False):

LWLibavSource:

stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786

ffms2- Wolfberry (default, i.e. seekmode=1)
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786

I added Decimal() for testing purposes https://github.com/theChaosCoder/zoptilib
The metric values are now much longer, sum should always be the same between runs. Please test.

Retested with Wolfberry's ffms2 (default, seekmode=1) and zoptilib with the Decimal update:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

Consistent results on consecutive testing (still), but slightly different scores to previous results when rounded up to 14 decimal places.....not that it really matters.

ChaosKing

18th March 2019, 23:49

Iron_Mike

19th March 2019, 02:24

alright, more tests, results are the same (for seekmode=1 on keyframes encoded files) - updated to Zoptilib 1.0.8

this is the .vpy script

import vapoursynth as vs
from zoptilib_108 import Zopti

core = vs.get_core()

# set vars
ref_fp = r'/path/to/crowd_run_1080p50.y4m'
enc_fp = r'/path/to/encode_with_keyframes.mp4'
log_fp = r'/path/to/zopti.log'

metrics = ('gmsd', 'ssim')

# load files using default seekmode=1
vid_ref = core.ffms2.Source(source=ref_fp)
vid_enc = core.ffms2.Source(source=enc_fp)

# initialize output file and chosen metrics
zopti = Zopti(log_fp, metrics=metrics)

# run
zopti.run(vid_ref, vid_enc)

--- VSPipe ---

stop 123.8387661168988139459568034 228.0649240752797068665369550
stop 122.1054561320879287833474084 236.3004045199170526647680646
stop 123.9634227532677362149104996 227.1636231975790896908673488
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 120.4371235439986860815686501 244.7757634970582563960217470

--- VSedit ---

I did not put the script in a loop, but: open script > F7 > press START (every time for a new run)

stop 120.4699959471850678194293496 245.2682722981770835624093506
stop 122.9025251845727162713917127 231.9192097981770835091186457
stop 93.01091081028496621030399182 389.5661511983989193086408413
stop 91.15626372903303242800632232 399.5109242681809407216064747
stop 85.74332191455660112655579041 428.0741357421874995203836523

> deleted the ffindex files, which did not get re-created, 12 more runs of the same script

stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 101.1081413762829624147787655 346.7421019904996144500586296
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 100.0318691978474301140789038 351.6773840784143521642768346
stop 93.44757297579696733724752276 387.2203145194347990720729052
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 87.01267141164586690904592101 421.4470892107928237657610272

--> as u can see sometimes it gets 3 in a row right...

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?[/B].

file # 1

0; 0.163505585855354740321132567260065115988254547119140625; 0.89805531442901231908848558305180631577968597412109375;
1; 0.1769067779219471170382149693978135474026203155517578125; 0.85627302758487655598429455494624562561511993408203125;
2; 0.1792310442991052743533231250694370828568935394287109375; 0.8481661723572531119685891098924912512302398681640625;
3; 0.1784714411393954203166600791519158519804477691650390625; 0.84848427854938268666273870621807873249053955078125;
4; 0.1779983423827615485190989375041681341826915740966796875; 0.85007233796296299832562226583831943571567535400390625;
5; 0.170543962458093500966782585237524472177028656005859375; 0.87440025800540122080661831205361522734165191650390625;
6; 0.1773295845776405865734659528243355453014373779296875; 0.85208224826388889727724063050118274986743927001953125;
7; 0.179107019975634884634274612835724838078022003173828125; 0.8447753906249999555910790149937383830547332763671875;
8; 0.17649390964808808757169344971771351993083953857421875; 0.85264449508101847197139022682677023112773895263671875;
9; 0.17707103545260505228498004726134240627288818359375; 0.85212275752314814436516599016613326966762542724609375;
10; 0.1701688806650836494060996528787654824554920196533203125; 0.87342773437499998667732370449812151491641998291015625;
...
490; 0.274298550587420308932706802806933410465717315673828125; 0.28572524836033952322367213128018192946910858154296875;
491; 0.27483107733594713817382171328063122928142547607421875; 0.27932755835262346177927383905625902116298675537109375;
492; 0.27492352759235305637020019275951199233531951904296875; 0.2829961178626543283343153234454803168773651123046875;
493; 0.273530875811458684676580332961748354136943817138671875; 0.300018687307098741445798850691062398254871368408203125;
494; 0.273138560134608965324076734759728424251079559326171875; 0.29670530719521603923993779972079209983348846435546875;
495; 0.2725773231787707029383227563812397420406341552734375; 0.313249662422839525444118180530495010316371917724609375;
496; 0.273262551506504303144851064644171856343746185302734375; 0.301779996141975315193661799639812670648097991943359375;
497; 0.2638890744928075182684779065311886370182037353515625; 0.39238395785108026725396257461397908627986907958984375;
498; 0.245544133215554405413882932407432235777378082275390625; 0.5239230685763889283634853200055658817291259765625;
499; 0.177711940866405015970741487763007171452045440673828125; 0.84061951437114201812761393739492632448673248291015625;
stop 93.44757297579696733724752276 387.2203145194347990720729052

file # 2

0; 0.163505585855354740321132567260065115988254547119140625; 0.89805531442901231908848558305180631577968597412109375;
1; 0.1769067779219471170382149693978135474026203155517578125; 0.85627302758487655598429455494624562561511993408203125;
2; 0.1792310442991052743533231250694370828568935394287109375; 0.8481661723572531119685891098924912512302398681640625;
3; 0.1784714411393954203166600791519158519804477691650390625; 0.84848427854938268666273870621807873249053955078125;
4; 0.1779983423827615485190989375041681341826915740966796875; 0.85007233796296299832562226583831943571567535400390625;
5; 0.170543962458093500966782585237524472177028656005859375; 0.87440025800540122080661831205361522734165191650390625;
6; 0.1773295845776405865734659528243355453014373779296875; 0.85208224826388889727724063050118274986743927001953125;
7; 0.179107019975634884634274612835724838078022003173828125; 0.8447753906249999555910790149937383830547332763671875;
8; 0.17649390964808808757169344971771351993083953857421875; 0.85264449508101847197139022682677023112773895263671875;
9; 0.17707103545260505228498004726134240627288818359375; 0.85212275752314814436516599016613326966762542724609375;
10; 0.1701688806650836494060996528787654824554920196533203125; 0.87342773437499998667732370449812151491641998291015625;
...
490; 0.164946739428928845772048816797905601561069488525390625; 0.871590350115740708503153655328787863254547119140625;
491; 0.173819541766866370036126454579061828553676605224609375; 0.849342146508487605416348742437548935413360595703125;
492; 0.1763885578142398158352222026223898865282535552978515625; 0.84145887586805556690450202950160019099712371826171875;
493; 0.1748831330143497397866525489007472060620784759521484375; 0.84578281732253091096396246939548291265964508056640625;
494; 0.1741334644004921827775689280315418727695941925048828125; 0.846259825906635843040248801116831600666046142578125;
495; 0.1677916978418648452642258916966966353356838226318359375; 0.86350929542824073070761414783191867172718048095703125;
496; 0.173813412948995438256361012463457882404327392578125; 0.84659740306712960578039428582997061312198638916015625;
497; 0.1727639159829111592525663354535936377942562103271484375; 0.84863715277777773682288398049422539770603179931640625;
498; 0.17653796953829969940130695249536074697971343994140625; 0.8404566936728394832556432447745464742183685302734375;
499; 0.177711940866405015970741487763007171452045440673828125; 0.84061951437114201812761393739492632448673248291015625;
stop 85.74332191455660112655579041 428.0741357421874995203836523

had to trim the files as the post ended up having too many chars, but thee two example files are attached (see below)...

WorBry

19th March 2019, 15:15

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

Maybe you have a consistent inconsistency :D
I mean your log is always in perfect order... I tested vspipe with 3 frames, and even then sometimes the output is 0 2 1 (before the update)

Wasn't sure if you were joking there, but I've examined this further and I have an explanation.

When the seek-test script is run on the Crowd Run 1080p50 x265 CRF28.mp4 encode (0-499 frames) it reports errors (3 frames off) with ffms2 mode #1 (seekmode=1) but no errors with ffms2 mode# 5 (seekmode=0) or L-Smash Works.

The source/reference crowd_run_1080p50.y4m gives no errors with any of these modes, as you would expect, being raw, intra-frame video.

Logically then you would expect to see frame mismatches between the x265 encode and the reference clip when imported with ffms2 in default seekmode=1.

So I ran a script to compare the output frames.

import vapoursynth as vs
core = vs.get_core()
Ref = core.ffms2.Source(source=r'X:/crowd_run_1080p50.y4m')
Ref = core.std.Crop(Ref, left=0,top=0,right=960,bottom=0)
Test = core.ffms2.Source(source=r'X:/crowd_run_1080p50_x265_CRF28_KeyInt100.mp4')
Test = core.std.Crop(Test, left=0,top=0,right=960,bottom=0)
Stack = core.std.StackHorizontal([Ref, Test])
Stack.set_output()

Note I used the x265 encode with the custom key interval (100) here.

Opened the script in VSEditor > Preview and made random frame selections with the mouse cursor (left click) on the timeline and also using the frame selection box (to the left of the timeline). Up to frame #99 there were no frame mismatches, but from #100 there were indeed frame offsets.

However, when I reset to frame #0, closed and re-opened Preview, played the video (from frame #0) and paused at any point, the frames always matched.

With ffms2 set to seekmode=0, there were no frame mismatches on random frame seeking or playback, and likewise with LWLibavSource as the import filter.

So it is only random seeking with ffms2 (default, seekmode=1), which is what the seek-test script does, that gives these frame offsets, not sequential output from frame #0, as is the case when the metric test scripts are run with Benchmark to generate the score log files.

This perfectly explains why the seek-test script gave errors with the encoded x265 file and ffms2 mode #1 (seekmode=1), yet the results of the SSIM:GMSD metric tests with ffms2 (default, seekmode=1) were in perfect agreement (aggregate and per frame scores) with those produced with LWLibavSource - in my tests at least.

Clearly, if there are issues with the indexed frame ordering then it's going to complicate matters further.

WorBry

19th March 2019, 17:03

As for Iron_Mike's case - purely a stab in the dark, but maybe try using LibavSMASHSource, which doesn't create an index file. It won't open the reference crowd_run_1080p50.y4m as such, but try it for importing the x265 test clip:

clip = core.lsmas.LibavSMASHSource(source=r'{Path}:/video')

And use LWLibavSource for the reference crowd_run_1080p50.y4m clip.

In my tests at least, using LibavSMASHSource for the test clip produces exactly the same metric scores as LWLibavSource. But YMMV.

Otherwise, try converting the crowd_run_1080p50.y4m clip to lossless x264 Intra, use that as the source for your x265 encode, and run the metric tests with LibavSMASHSource as the import filter for both files.

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx264 -preset slow -crf 0 -intra -r 50/1 -pix_fmt yuv420p -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/crowd_run_1080p50_x264_lossless_intra.mp4

Incidentally, if you run the seek-test script in ffms2 mode #1 on the crowd_run_1080p50.y4m clip does it report any seeking errors? It shouldn't, being all intra, but if it does, it suggests there's something else going on.

ChaosKing

19th March 2019, 17:09

crowd_run_1080p50.y4m is fine with ffms2
ffms2 has only problems with the x265 mp4 encode. Remux it to mkv and all is fine.
lsmash is also fine.

p.s. you should also run more than 100 frames with seek-test with our x265 mp4 file, just to be sure. (edit the bat file or use cmd)

WorBry

19th March 2019, 17:36

Remux it to mkv and all is fine.

Actually, I still get seek errors (the 3 frames off) in the seek-test (ffms2 seekmode=1, all 500 frames) with an mkv remux of the x265.mp4 file. And likewise re-encoding crowd_run_2160p50.y4m to x265.mkv.

ChaosKing

19th March 2019, 18:11

hmm strange. I used mkvtoolnix v32 to remux the mp4. Just tested the mkv again -> no seeking issues.

WorBry

19th March 2019, 21:07

I tried several re-muxing methods - ffmpeg, mkvtoolnix, (TMPGEnc) SmartRenderer 5 - and the mkv files all give seek errors with seek-test and ffms2 (searchmode=1).

ChaosKing

19th March 2019, 21:31

Iron_Mike

19th March 2019, 22:02

I'm out of ideas then. The only factor seems to be the cpu then. AMD FX-6300 vs Ryzen 1700

I used this ffms2 version https://forum.doom9.org/showthread.php?t=176198. And as far as I can see you too...

that's the version I used as well in all my tests.

seekmode=0 and Lsmash give consistent results.

seekmode=1 on a file that was NOT encoded w/ keyframes setting also works just fine.

so, as I stated before:

it's the keyframes intval in the encoded file that throws off ffms2, which evidently starts at frame 100, which is where the first keyframe is (if you used my keyframe setting: keyint=100:min-keyint=100:rc-lookahead=100)

man, it would be desperately needed to be able to easily output the versions of the plugs used via VS.. there may be a difference in sw packages, which would explain why WorBry gets different results

being able to create the exact same dev environments for all peeps involved is key, and with a plethora of packages and setup/config options here, it'll be hard to figure this one out...

Iron_Mike

19th March 2019, 22:14

poisondeathray

19th March 2019, 22:44

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)

You guys should upload your files, because --frame-threads will be different on a 6 core , vs. 4C/8T , vs.8C/16T . That's why Worbry's values are consistently slightly higher. By default he will only have 2 Frame threads from 6 cores; 8T will have 3, 16 will have 5 . The files are slightly different. Does that affect what you are seeing in terms of seeking here ( besides the actual values)

The big errors are clearly frame mismatches. But the 10th ,12 etc.. tiny decimal place errors are something else Iron_Mike report that with seekmode=0 without keyint too - so that's another separate issue that shouldn't happen and is still worrisome

ChaosKing

19th March 2019, 23:15

btw, I wanna add since I saw CK mention this a few times:

my Zopti logs - whether there are seeking issues or not via ffms - are always in correct frame order (at least at quick glance) - see the two attached example log files above in the other post (https://forum.doom9.org/showpost.php?p=1869294&postcount=161)...

this machine here (notebook): Win 8.1 x64, Intel i7-4810, 24GB RAM

Because the log is now always written in correct order. https://github.com/theChaosCoder/zoptilib/commit/d99ff1e72d913a89dc99ee05c45b60f39e2148fa

I get the VS notice "script ran out of memory, consider increasing cache size"... but not sure if this can be a factor since the notice always comes, even when there are no seeking errors
This is normal. VS uses a percentage of the total allowed memory. But I don't know what the default % is. You can set it higher with http://www.vapoursynth.com/doc/pythonreference.html?highlight=core#Core.max_cache_size

Iron_Mike

19th March 2019, 23:36

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)

if the encode was done via a specified key intval setting, then I verify afterwards via ffprobe that all IDR keyframes are in the correct position - and they are on my encodes, so @ frame 100 is a real IDR keyframe (unless these are labeled as IDR keyframes but are not ?)

The big errors are clearly frame mismatches. But the 10th ,12 etc.. tiny decimal place errors are something else Iron_Mike report that with seekmode=0 without keyint too - so that's another separate issue that shouldn't happen and is still worrisome

just ran another 10x tests w/ seekmode=0 on a keyframe encoded file via Zopti 1.0.8 - GMSD/SSIM

stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523

edit: and another 10x tests w/ seekmode=0 on a non-keyframe encoded file (different encoding settings than file above, results will differ) via Zopti 1.0.8 - GMSD/SSIM

stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585

WorBry

20th March 2019, 01:12

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)

Elecard StreamEye identifies frame #100 (display sequence, #97 stream sequence) as a 'Key Frame', which according to their definition is an IDR frame:

"...key frames (also referred to as IDR frames; all the other frames positioned in between IDR frames cannot make reference or be dependent to/on the frames outside of this interval)."

http://i.imgur.com/szAS3Rsm.png (https://imgur.com/szAS3Rs)

As for the x265 CRF28.mp4 file encoded at default settings, the first key frame appears at frame #250 in the display sequence, as expected.

poisondeathray

20th March 2019, 01:25

WorBry

20th March 2019, 01:57

You guys should upload your files

No problem:

crowd_run_1080p50_y4m_x265_CRF28 (https://drive.google.com/file/d/1H4cpcE2Q70z-g0SN6FMK88iYDVK4k0Fl/view?usp=sharing)

Zipped folder contains two x265 (CRF28) files, encoded without and with custom key interval (100), as per:

(1) encode w/o keyint, min-keyint, rc-lookahead parameters:

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

(3) encode w/ keyint, min-keyint but w/o rc-lookahead:

--> wanted to see if it is specifically the rc-lookahead setting that causes the issues w/ seekmode=1

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Iron_Mike

20th March 2019, 02:06

What version of streameye ? Note that earlier versions misidentified HEVC IDR frames . It might be corrected in newer versions, not sure.

FFmpeg/FFProbe do not distinguish between them either - they mark them all as "key"

A freebie that correctly identifies them is hevcesbrowser
https://github.com/virinext/hevcesbrowser
https://www.codeproject.com/Tips/896030/The-Structure-of-HEVC-Video

You can check by encoding with --no-open-gop and you will see it's working . (Or not working in the case of FFprobe) .

x265 almost always places "i" frames by default . Very few frames will be IDR unless it's a scenechange or completely different content

Note there are 2 types of IDR frames in HEVC . The true strict type is an IDR_N_LP or "no leading picture"

Does this affect seeking ? Possibly. Open gop's can be problematic in some situation

x265 encoded with keyframes setting (open gop)

ffprobe positions of all found IDR keyframes: 0, 100, 200, 300, 400 (as expected)
HEVC browser GUI, I Slices found: 5 (1x IDR slice, 4x I slice)

x265 encoded with keyframes setting (using param: no-open-gop)

<ffmpeg> -y -i "/path/tocrowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:open-gop=0:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_key_frames_closed_gop.mp4"

ffprobe positions of all found IDR keyframes: 0, 100, 200, 300, 400 (as expected)
HEVC browser GUI, I Slices found: 5 (5x IDR slice)

x265 encoded without keyframes setting

ffprobe positions of all found IDR keyframes: 0, 250 (as expected)
HEVC browser GUI, I Slices found: 2 (1x IDR slice, 1x I slice)

re HEVC browser: I assume that an "IDR Slice" is a real IDR keyframe and every other "I slice" is as mere I frame...

WorBry

20th March 2019, 02:16

What version of streameye ? Note that earlier versions misidentified HEVC IDR frames . It might be corrected in newer versions, not sure.

StreamEye 4 (demo). Actually, their definition of key frames is given in the context of the leading key frame in a sequence, which is an IDR frame, so it's a bit confusing whether it applies to I-frames designated as 'key frames' downstream. The full quote:

".. Decoding of the whole video sequence, including scenarios at positioning, starts with key frames (also referred to as IDR frames; all the other frames positioned in between IDR frames cannot make reference or be dependent to/on the frames outside of this interval). For instance, video startsplaying only after receiving a key frame at network broadcasting."

A freebie that correctly identifies them is hevcesbrowser
https://github.com/virinext/hevcesbrowser
https://www.codeproject.com/Tips/896030/The-Structure-of-HEVC-Video

Thanks, I'll have a look at that.

poisondeathray

20th March 2019, 02:24

You have to use ES (elementary stream) with hevcesbrowser

Iron_Mike

20th March 2019, 02:27

and here are 3 encodes from me: https://drive.google.com/file/d/1-Jl8iVYc-PI6PLwob2vwbu4dY-h7Isun/view?usp=sharing

Iron_Mike

20th March 2019, 02:29

You have to use ES (elementary stream) with hevcesbrowser

I only tested the GUI, how is this done ?

or are you using the console ?

edit: ah. gotta convert... more PAIN ;-)

poisondeathray

20th March 2019, 02:30

ES (elementary stream) - just demux it from the container . Open the "naked" :) hevc stream with the GUI

e.g

ffmpeg -i input.mp4 -c:v copy output.hevc

poisondeathray

20th March 2019, 02:43

@Iron_Mike - your "closed" gop mp4 isn't "closed" for some reason . The command wasn't passed. You can verify with mediainfo (view=>text)

Iron_Mike

20th March 2019, 02:45

ES (elementary stream) - just demux it from the container . Open the "naked" :) hevc stream with the GUI

e.g

ffmpeg -i input.mp4 -c:v copy output.hevc

ok, edited my original post, see here (https://forum.doom9.org/showpost.php?p=1869391&postcount=177)

num keyframes match what ff probe reports...

how can I determine the frame number of the HEVC reported I slices ?

Iron_Mike

20th March 2019, 03:00

and... 10x GMSD/SSIM metrics test on x265 closed-gop encode - Zopti 1.0.8 w/ seekmode=1:

stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194457 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309

since these are all fine, seekmode=1 trips up on x265 open-gop encodes with custom keyframes interval...

WorBry

20th March 2019, 03:04

and here are 3 encodes from me: https://drive.google.com/file/d/11IzqeEUl91Bw4XxlN6xIwQcsd6XzgG4Z/view?usp=sharing

I assume the script for the crowdrun_x265_keyframes-100_closed_gop.mp4 encode was:

x265 encoded with keyframes setting (using param: no-open-gop)

<ffmpeg> -y -i "/path/tocrowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:no-open-gop:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_key_frames_closed_gop.mp4"

But what about the other two - crowdrun_x265_keyframes-100_open_gop.mp4 and crowdrun_x265_keyframes-none.mp4 ?

Were they also encoded with -preset fast and rc-lookahead=100 ? Please post the scripts.

Iron_Mike

20th March 2019, 03:07

@Iron_Mike - your "closed" gop mp4 isn't "closed" for some reason . The command wasn't passed. You can verify with mediainfo (view=>text)

hah... all for nothing ;-)

re-doing the tests...

Iron_Mike

20th March 2019, 03:34

alright...

> updated correct HEVC browser results, see here (https://forum.doom9.org/showpost.php?p=1869391&postcount=177)

> uploaded a new .zip w/ the three encodes, see here (https://forum.doom9.org/showpost.php?p=1869394&postcount=180)

> updated GMSD/SSIM results for closed-gop encode, see here (https://forum.doom9.org/showpost.php?p=1869401&postcount=185)

seems like @poisondeathray has been right all along that the default open-gop flag on x265 could possibly trip up seekmode=1 when custom keyframe intervals are set... interesting that the default keyframe intval does NOT trip up seekmode=1 on x265...

I assume the script for the crowdrun_x265_keyframes-100_closed_gop.mp4 encode was:
But what about the other two - crowdrun_x265_keyframes-100_open_gop.mp4 and crowdrun_x265_keyframes-none.mp4 ?

Were they also encoded with -preset fast and rc-lookahead=100 ? Please post the scripts.

(1) keyframes none

<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-r 50/1 "/path/to/encode_keyframes_none.mp4"

(2) custom keyframes intval 100 (2 secs) - default open-gop

<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_keyframes_open_gop.mp4"

(3) custom keyframes intval 100 (2 secs) - open-gop=0 (closed gop)

<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:open-gop=0:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_keyframes_closed_gop.mp4"

WorBry

20th March 2019, 03:51

OK, thanks.

Here are the SSIM:GMSD (Downsample=False) results I get with your files using ffms2 (Wolfberry) in default seekmode=1. I created fresh ffindex files for each run.

crowdrun_x265_keyframes-none.mp4

stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421

crowdrun_x265_keyframes-100_open_gop.mp4

428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041

crowdrun_x265_keyframes-100_closed_gop.mp4

425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660

Edit: Oh, I see you uploaded a new set of encodes while I was testing. Ah well.
Edit: But judging from the file sizes (in bytes) it looks like it's just the crowdrun_x265_keyframes-100_closed_gop.mp4 that's different - so I'll only test that one.
Edit:
crowdrun_x265_keyframes-100_closed_gop.mp4 Updated Version

stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456

Do I really need to test with LWLibavSource.

Iron_Mike

20th March 2019, 04:02

Edit: But judging from the file sizes (in bytes) it looks like it's just the crowdrun_x265_keyframes-100_closed_gop.mp4 that's different - so I'll only test that one.

only the closed gop file has changed, the other two are the same.

poisondeathray

20th March 2019, 04:11

Also, I would expect WorBry's frame-threads to be 2 for a 6C/6T . But mediainfo says 6 . Were those encoded on a different computer ?

WorBry

20th March 2019, 04:18

only the closed gop file has changed, the other two are the same.

Results added to my last post.

WorBry

20th March 2019, 04:33

Also, I would expect WorBry's frame-threads to be 2 for a 6C/6T . But mediainfo says 6 . Were those encoded on a different computer ?

No, same PC. But looking back at the scripts I did actually specify -threads 6. So to be clear:

crowd_run_1080p50_y4m_x265_CRF28.mp4:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -threads 6 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -threads 6 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

Iron_Mike

20th March 2019, 04:43

Results added to my last post.

can you post your .vpy script used in these tests ?

poisondeathray

20th March 2019, 04:43

Iron_Mike

20th March 2019, 04:58

Did you upload that one ?
The custom keyframe version with open gop is introducing sublayer leading pictures . That might have to do with it, causing ffms2 /ffmpeg/ffprobe to mix things up

yes, there were 3 encoded files in the .zip (I've updated the post with a new zip a bit ago, replacing the closed_gop encode, since it was faulty, as you discovered)

(1) crowdrun_x265_keyframes-100_closed_gop.mp4 - custom keyframes intval of 100 (2 secs @ 50fps), with open-gop=0

(2) crowdrun_x265_keyframes-100_open_gop.mp4 - custom keyframes intval of 100 (2 secs @ 50fps), with default open-gop=1

(3) crowdrun_x265_keyframes-none.mp4 - no custom keyframes set, with default open-gop=1, which will result in 2 keyframes

file (3), although open-gop, does NOT trip up ffms2 w/ seekmode=1

edit:

another thing I just saw via HEVC browser...

re (1): the encode places all 5x IDR keyframes @ 100 frames interval (as specified)...

re (2) the encode places the 4x simple I frames (following the only real IDR keyframe) in incorrect position, they are a few frames off

re (3) the encode places the 1x simple I frame (following the only real IDR keyframe) in correct position

so, the ffmpeg encode of (2) is bad...

so how come this does not happen to WorBry using the same ffmpeg version and the same command ?

poisondeathray

20th March 2019, 05:00

(3) crowdrun_x265_keyframes-none.mp4 - no custom keyframes set, with default open-gop=1, which will result in 2 keyframes

file (3), although open-gop, does NOT trip up ffms2 w/ seekmode=1

Ok, sorry I misinterpreted that ; I thought "none" meant "infinite" . basically 1 keyframe for any stream. But this is just "default" setting

poisondeathray

20th March 2019, 05:04

What about threading on the vpy side ? How is that handled ? I know you can set core = vs.get_core(threads = something) , but what is the "auto" value ? And does this affect script execution or frame requests in that script?

WorBry

20th March 2019, 05:14

@WorBry - for x265 it's --frame-threads . I guess ffmpeg -threads get passed as libx265 frame threads. A value of 6 would correspond to a 32 core machine . You can see on the chart
https://x265.readthedocs.io/en/latest/threading.html

It was really oversight on my part :o I have my commands on a text file - copied another script for the file paths and neglected to delete the -threads 6.

I have crowdrun y4m on a HDD somewhere. I'll test it too for a 3rd set of data later probably tomorrow

Otherwise, it's here:

https://media.xiph.org/video/derf/

WorBry

20th March 2019, 05:27

can you post your .vpy script used in these tests ?

Well it's nothing special:

import vapoursynth as vs
import muvsfunc as muv
from zoptilib import Zopti
core = vs.get_core()
orig = core.ffms2.Source(source=r'{Path}:/crowd_run_1080p50.y4m')
zopti = Zopti('{Path}:/SSIM_GMSD_NDS_crowdrun_x265_keyframes-100_closed_gop_v2_#5.txt', metrics=['ssim', 'gmsd'])
zopti.addParams('ssim', dict(downsample=False))
zopti.addParams('gmsd', dict(downsample=False))
alternate = core.ffms2.Source(source=r'{Path}:/crowdrun_x265_keyframes-100_closed_gop_v2.mp4')
zopti.run(orig, alternate)

Iron_Mike

20th March 2019, 09:25

another thing I just saw via HEVC browser...

re (1): the encode places all 5x IDR keyframes @ 100 frames interval (as specified)...

re (2) the encode places the 4x simple I frames (following the only real IDR keyframe) in incorrect position, they are a few frames off

re (3) the encode places the 1x simple I frame (following the only real IDR keyframe) in correct position

HEVC browser stats of the 3 files I uploaded - note: frame count starts at 1, not at 0 in the below stats

re (1) - x265 w/ closed-gop and custom keyframes intval 100 - IDR keyframes at positions: 1, 101, 201, 301, 401 | simple (non-IDR) I frames: none

re (2) - x265 w/ open-gop and custom keyframes intval 100 - IDR keyframes at positions: 1 | simple (non-IDR) I frames at positions: 97, 197, 297, 397

re (3) - x265 w/ open-gop and no custom keyframes intval - IDR keyframes at positions: 1 | simple (non-IDR) I frames at positions: 247

not sure how ffms2 works, but if it reads file (2), then sees the specified GOP size with keyintval of 100 (in the encoding settings), and then does not find the IDR frames in the correct positions, or matter of fact does not find any IDR frames at all (b/c ffmpeg w/ open-gop only places simple I frames), then that may be the reason why it acts up...

ChaosKing

20th March 2019, 09:50

There are new ffms2 / lsmash builds up. But seems to behave the same.
And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z
(This is the second file what has no problems with the old version)

I tested also a build by HolyWu (ffmpeg 4.1.1) https://forum.doom9.org/showthread.php?p=1866411#post1866411
-> Seeking issues

sneaker_ger

20th March 2019, 10:29

not sure how ffms2 works, but if it reads file (2), then sees the specified GOP size with keyintval of 100 (in the encoding settings), and then does not find the IDR frames in the correct positions
It doesn't work like that. No decoder cares about keyint in custom/optional "encoding settings" SEI that x264/x265 write.

poisondeathray

20th March 2019, 15:27

It's ok display order wise.

The offset position and encoding order does not necessarily reflect the display order of the frames. That utility does not display the GOP number , display frame number, encoded frame number , but hey it's free

There is another analyzer, GitlHEVCAnalyzer on sourceforge but it only supports 8bit HEVC . But you guys are using 8bit here, but it will display a graph like elecard with "I" slices in red. But you can see they are placed correctly. Another method, with avisynth ffms2 is to use FFInfo() and it will display the pic type. But again, it does not distinguish between IDR types , or non IDR "i". Just simple I,P,B slices . Or commercial analyzers ($) .

ChaosKing

20th March 2019, 16:11

It's ok display order wise.

with avisynth ffms2 is to use FFInfo() and it will display the pic type. But again, it does not distinguish between IDR types , or non IDR "i". Just simple I,P,B slices . Or commercial analyzers ($) .

VS version: core.ffms2.Source(source=r"D:\Download\crowd_run_1080p50.y4m").text.FrameProps(props=['_PictType'])

or just text.FrameProps() to show all frame props

WorBry

20th March 2019, 16:14

@WorBry - for x265 it's --frame-threads . I guess ffmpeg -threads get passed as libx265 frame threads. A value of 6 would correspond to a 32 core machine . You can see on the chart
https://x265.readthedocs.io/en/latest/threading.html

The frame thread effect is smaller than with x264, but it's the reason why your values were slightly higher, because it should have been 2 with a 6C/6T machine. But if you specified -threads 6 , and that got passed to libx265 as frame-threads, then I would have expected the scores to be lower...

It was really oversight on my part :o I have my commands on a text file - copied another script for the file paths and neglected to delete the -threads 6.

So, I've re-encoded the x265 files omitting the "-threads 6". MediaInfo reports 'frame-threads=2'.

These are the SSIM:GMSD (Downsample=False) obtained with Wolfberry's ffms2 (default seekmode=1)

crowd_run_1080p50_y4m_x265_CRF28_v2.mp4:

stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100_v2.mp4:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

Compared with the x265 files that were encoded with '-threads 6' (MediaInfo reports 'frame-threads=6')

crowd_run_1080p50_y4m_x265_CRF28.mp4:

stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

So it actually made no difference to the results.

In fact the two crowd_run_1080p50_y4m_x265_CRF28.mp4 files are byte identical and the crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4 files differ by only 5 bytes.

Still, here are the re-encoded files (omitting -threads 6):

crowd_run_1080p50_y4m_x265_CRF28_v2 (https://drive.google.com/file/d/1j0A4ImFL2c1xV9a1RX-KdOB-8pS6KlT8/view?usp=sharing)

poisondeathray

20th March 2019, 16:54

So it actually made no difference to the results.

Maybe not enough vertical motion to exhibit the frame threads effect in this shot. No idea why your values where consistently slightly higher than Iron_Mikes' (frame threads would have been a nice way to explain it) . Maybe some more inconsistencies between runs or systems :(

But beyond that , look at the bitrate / filesize between your uploads and his. Way different. Some settings are actually different too looking at mediainfo . psy-rdoq is 0 for his 1 for yours for the keyint 100 encode, rd4 vs 2 etc... . Need to compare apples to apples . Or at least look at the same physical file

WorBry

20th March 2019, 17:30

Iron_Mike

20th March 2019, 23:21

He modified the 'original' test encode settings for those uploads - changed -preset slow to fast and added -rc-lookahead=100:

https://forum.doom9.org/showthread.php?p=1869402#post1869402

Why, I don't know.

So he should upload x265 files encoded with the 'original' scripts for valid comparison:

https://forum.doom9.org/showthread.php?p=1869390#post1869390

you misread my post (the last post you linked to):

rc-lookahead was ALWAYS a parameter I included if I specified custom keyframes... but I did ONE TEST to see if specifically the rc-lookahead parameter was the problem for ffms2, hence I removed that param in that ONE encode...

which is stated in my original post (https://forum.doom9.org/showpost.php?p=1869188&postcount=128), and as you can see in point (2) in that post - which you removed from the quote - I have used the rc-lookahead param...

All files uploaded by me were done w/ the rc-lookahead param (if custom keyframes were specified), only reason I had to re-upload once is b/c I used incorrect syntax for the no-open-gop flag on one of the three files...

so, if a custom keyframes intval was specifed these three params were added to the x265-params: keyint=100:min-keyint=100:rc-lookahead=100

WorBry

21st March 2019, 00:41

WorBry

21st March 2019, 01:24

And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z
(This is the second file what has no problems with the old version)

Trouble is, VS crashes when I try to test it.

Think I'll switch to LWLibavSource for future metric testing.

poisondeathray

21st March 2019, 01:30

@Poisondeathray, I'm interested to know how your results with the files I uploaded compare anyway.

Not repeatable results here , unless seekmode=0 . I only tested your keyint 100, 6 frame thread version . Larger errors on some runs, clearly frame mismatches

Not able to replicate the 12th or whatever tiny decimal error with seekmode=0 over 10 runs . 100% consistent

Tried 2 computers, but both were 4C/8T Intels . I also have a really old AMD (with XP!) lying around but I didn't test on that

EDIT #1: also on the other file, with default settings (no custom keyint) , it was inconsistent too. Iron_Mike reported "big" inconsistent results on some runs only when using custom keyint . I'm seeing inconsistent everywhere with ffms2 default seekmode on these.

EDIT #2: remuxing to mkv with mkvmerge made everything consistent for the keyint 100 file (10 runs matched) with default seekmode, as CK also reported . This makes me think it has something to do with ffms2 reading the timebase and timecodes in the container , or container mux, or interpretation issue

ffmpeg -i input.ext

mkv
Stream #0:0: Video: hevc (Main), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR
16:9], 50 fps, 50 tbr, 1k tbn, 50 tbc (default)

mp4
Stream #0:0(und): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt709,
progressive), 1920x1080 [SAR 1:1 DAR 16:9], 15411 kb/s, 50 fps, 50 tbr, 12800 t
bn, 50 tbc (default)

ffmpeg "thinks" the tbn is 12800 for the mp4 . tbn is the container time base

Iron_Mike

21st March 2019, 02:01

No I didn't misread your post. You changed the preset to fast also. First you are following my encode script, as a common ground - now you're not. Have you even taken the trouble to test the files I uploaded and post the results, as I did yours ? Really can't be bothered with this.

@Poisondeathray, I'm interested to know how your results with the files I uploaded compare anyway.

Cheers.
You did misread as your own post clearly proves, b/c I state in detail in my post that I left out rc-lookadhead in one encode...... !?

The preset does NOT matter in any of this - no idea why you say any of that.

I also don't care if you test ffms2 or not. I posted and proved that there are issues w/ seekmode=1, other people confirmed.

Don't care whether you personally test this or not.

The only reason why I even bothered to write here is b/c you spend some time doing some tests and if you ever are subject to this behavior - or other inconsistent behavior we've not yet discovered - then your posted results here would be pointless.

Hence, it's always good if another source confirms general trends of test results of these metrics... which I did.

Iron_Mike

21st March 2019, 02:23

Not repeatable results here , unless seekmode=0 . I only tested your keyint 100, 6 frame thread version . Larger errors on some runs, clearly frame mismatches

Not able to replicate the 12th or whatever tiny decimal error with seekmode=0 over 10 runs . 100% consistent

Tried 2 computers, but both were 4C/8T Intels . I also have a really old AMD (with XP!) lying around but I didn't test on that

EDIT #1: also on the other file, with default settings (no custom keyint) , it was inconsistent too. Iron_Mike reported "big" inconsistent results on some runs only when using custom keyint . I'm seeing inconsistent everywhere with ffms2 default seekmode on these.

tested WorBry's files...

WorBry - crowd_run_1080p50_y4m_x265_CRF28.mp4 - seekmode=1 - GMSD/SSIM

stop 83.36467719049633404160282170 436.0761547550154326735594167
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166

WorBry - crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4 - seekmode=1 - GMSD/SSIM

stop 124.0039187143357256870590535 227.9924869038146211797091216
stop 83.19534706841788751496835637 435.9938219159915119282899813
stop 120.1705765900797210909978223 248.1439760410638495358170989
stop 122.2412107559194712758099402 236.4289261580102229909350346
stop 122.2412107559194712758099402 236.4289261580102229909350346

matches my previously reported behavior...

Iron_Mike

21st March 2019, 02:26

And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z

As I posted before, this old ffms2 version crashes for me w/ your Fatpack... I simply copied the x64 plug into the Fatpack's VS plugins folder... anything else I need to do ?

WorBry

21st March 2019, 03:10

The only reason why I even bothered to write here is b/c you spend some time doing some tests and if you ever are subject to this behavior - or other inconsistent behavior we've not yet discovered - then your posted results here would be pointless.

Which clearly is not the case - unlike your comparison of medium vs fast presets which, if the attachment ever gets approved, must be viewed as suspect in the light of your unresolved issues. How can you post the results of study knowing that your test system is unreliable? Very unscientific ?

ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

https://forum.doom9.org/attachment.php?attachmentid=16770&stc=1&d=1552905262

OK, are we done ?

WorBry

21st March 2019, 03:18

Iron_Mike

21st March 2019, 03:23

Which clearly is not the case - unlike your comparison of medium vs fast presets which, if the attachment ever gets approved, must be viewed as suspect in the light of your unresolved issues. How can you post the results of study knowing that your test system is unreliable? Very unscientific ?

b/c contrary to you - who has not tested his system and only learned from me that there are potential problems - I've only run all metrics w/ seekmode=0 and confirmed each metric multiple times (consistent, repeatable) before posting them...

yes, you are done, Brian. go take a nap. :D

WorBry

21st March 2019, 03:38

b/c contrary to you - who has not tested his system and only learned from me that there are potential problems ..

Ridiculous statement.

..go take a nap. :D

Very juvenile, but I had that sense already.

Iron_Mike

21st March 2019, 03:47

So ffms2 is unreliable on some machines. What about LWLibavSource ?

well... that took a while. :sly:

as far as I can see, the ONLY one that it has been fully "reliable" w/ seekmode=1 w/ custom keyframes is you...

I posted it, CK confirmed and poisondeathray confirmed...

As I stated pages ago, it would not surprise me if it turns out to be a package difference, and by that I mean a specific VS plugin difference, but w/o a direct, convenient method to list version/release data of all VS plugins involved in the VS chain, it'll be hard to compare...

easiest way to confirm it is to create a minimal VS venv - only what's needed for this ffms2 read test - zip it, upload it and then everybody test in that venv on their machines...

for VS, I created an isolated venv, but I used CK's Fatpack as a base, and there's too many other things included that are not needed for this minimal test...

good news is that seekmode=0 is rock solid on my end (and not slower), and Lsmash as well, so it's not a show stopper...

poisondeathray

21st March 2019, 04:05

good news is that seekmode=0 is rock solid on my end (and not slower), and Lsmash as well, so it's not a show stopper...

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.php?p=1869188#post1869188

And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ? So some weirdness and inconsistencies between setups too

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

WorBry

21st March 2019, 04:15

well... that took a while. :sly:

You just don't stop do you.

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

Not denying there is an issue, but I'm not seeing it.

WorBry

21st March 2019, 04:23

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.php?p=1869188#post1869188

Nor I.

LSmash versions can share these little bugs too...

What about LWLibavSource ?

Did you test LWLibavSource also ?

poisondeathray

21st March 2019, 04:35

Did you test LWLibavSource also ?

Not on this series

And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test

Iron_Mike

21st March 2019, 04:36

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

yeah, in some cases it provided more precision, in a perfectly stable, repeatable system that should not happen

And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ?

I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?

poisondeathray

21st March 2019, 04:40

I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?

I used mkvmerge (mkvtoolnix is the GUI)

CK reported this finding here
https://forum.doom9.org/showthread.php?p=1869333#post1869333

Sorry I got mixed up; it was WorBry that still got seek errors after remuxing to MKV with several methods

Iron_Mike

21st March 2019, 04:51

@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?

Wolfberry

21st March 2019, 04:54

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

Read: it may have nothing to do with the encoder or the decoder, but about the ability of FFMS2 to use the FFmpeg API to parse the bitstream in the correct way, and certain encoder/decoder combinations expose errant behavior in the parser. This would explain why ffplay (or mpv) works fine on the same sample and the same build of libdav1d that FFMS2 uses, but FFMS2 exhibits problems.

I think that the reason behind the different kinds of quirks and bugs is the different FFmpeg version that FFMS2 / L-SMASH is built with (like 3.x or 4.x)

A newer FFmpeg version can fix/add support for some formats but can also introduce new bugs in some other formats in FFMS2 / L-SMASH due to their ability to use the FFmpeg API to parse the bitstream in the correct way.

WorBry

21st March 2019, 04:55

And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test

https://forum.doom9.org/showthread.php?p=1869332#post1869332

WorBry

21st March 2019, 04:57

it was WorBry that still got seek errors after remuxing to MKV with several methods

With the seek-test script yes, but not in the metric tests.

poisondeathray

21st March 2019, 05:21

With the seek-test script yes, but not in the metric tests.
But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other . And I'm getting a 3rd different :(

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV

Another question is why does this vpy script request frames out of order, instead of linearly? You can use the same decoder, the same ffms2 source filter, and use -f vapoursynth in Wolfberry's ffmpeg and the ssim / psnr there and it's consistent 10/10 times because frames are requested linearly in order . I realize from WorBry's earlier discussion the ssim calcs are slightly different; but ideally this should be a linear operation.

In terms of difficulty the synthetic seek test is like a torture, worse case sceneario .

WorBry

21st March 2019, 06:38

With the seek-test script yes, but not in the metric tests.

But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other .

Right, and this is why I looked at the alignment of the reference and test file output frames when applying random frame searches vs playback from frame #0 in VSEditor:

https://forum.doom9.org/showthread.php?p=1869323#post1869323

So it is only random seeking with ffms2 (default, seekmode=1), which is what the seek-test script does, that gives these frame offsets, not sequential output from frame #0, as is the case when the metric test scripts are run with Benchmark to generate the score log files.

This perfectly explains why the seek-test script gave errors with the encoded x265 file and ffms2 mode #1 (seekmode=1), yet the results of the SSIM:GMSD metric tests with ffms2 (default, seekmode=1) were in perfect agreement (aggregate and per frame scores) with those produced with LWLibavSource - in my tests at least.

And I'm getting a 3rd different :(

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV

The difference between me and thee being that I also get consistent metrics with mp4 and ffms2 in default seekmode=1

Iron_Mike

21st March 2019, 06:42

But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

correct, the seek script predicted things correctly for me.

alright, copied video stream of the mp4 encode w/ custom keyframes and open-gop into mkv container via ffmpeg

<ffmpeg> -i encode_with_keyfranmes_open_gop.mp4 -c copy encode_with_keyfranmes_open_gop.mkv

(lmk if that is not the correct way to create the mkv file)

> 1st metrics test run: MDSI - mkv - seekmode=1

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 202.9107845033357604980217559

> 2nd metrics test run: MDSI - mkv - seekmode=1 (same script)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 3rd metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 4th metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 194.8927784337109077039684736
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 5th metrics test run: GMSD/SSIM - mkv - seekmode=1

stop 105.8009960111949401551090232 321.3940602997202934110188006
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 114.6468917102554510301093416 275.2515876977237659073427521
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524

> seek-test.py from CK, test #1 on mkv: seek errors, 3 frames off

> the corresponding .mp4 file (encode w/ custom keyframes and open-gop) - seekmode=1

> MDSI:

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> GMSD/SSIM:

stop 123.9634227532677362149104996 227.1636231975790896908673488
stop 122.1886438717397119491092642 235.7664674057195217937721310
stop 123.9120154727625848778416187 228.0649240752797068665369550
stop 123.6926367223807698425552817 228.4161432110821759733365609
stop 123.7779981257600182276146230 228.0649240752797068665369550

--> so why is the MDSI metric much more stable (although not perfect) w/ seekmode=1 ?

ChaosKing

21st March 2019, 09:37

As I posted before, this old ffms2 version crashes for me w/ your Fatpack... I simply copied the x64 plug into the Fatpack's VS plugins folder... anything else I need to do ?
Nope, all you need is to replace the "old" ffms2.dll. You probably missing a vs runtime.
You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case :D But I think msvcrt.dll should fix it.

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

How about a test folder with some video files and a folder with different ffms2/lsmash dlls + a test script wich just tests all files with all dlls via seek-test? Like an automated mass seek-test if you will. Easy to make... the results could be written into a txt or csv file.

@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?

Sure why not. Will upload it a bit later.

Another question is why does this vpy script request frames out of order, instead of linearly?
Because the requests are made in parallel wich makes your script run faster.

Wolfberry

21st March 2019, 10:57

You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case :D But I think msvcrt.dll should fix it.

Well, I don't think downloading arbitrary runtime dlls is a good way to solve runtime dependency problems.

If you are lazy you can just install an AIO pack (https://repacks.net/viewtopic.php?t=247) as mentioned in this thread (http://forum.doom9.org/showthread.php?t=176026).

ChaosKing

21st March 2019, 11:55

Correct. It's just to quickly confirm that this dll is the culprit. I checked with ldd and dependency walker and msvcrt stood out. Or maybe it's ADVAPI32.dll but I think its a system dll.

EDIT
ok lol, the "old" ffms2 crashes with the y4m file!!! I only tested it via seek-test and the mp4 file in vsedit.

ldd ffms2.dll <--- old ffms2.dll
ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
apphelp.dll => /cygdrive/c/WINDOWS/SYSTEM32/apphelp.dll (0x7ffd2c780000)
ADVAPI32.dll => /cygdrive/c/WINDOWS/System32/ADVAPI32.dll (0x7ffd2fa60000)
msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)

$ ldd ffms2_.dll <-- (dll by Wolfberry)
ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
bcrypt.dll => /cygdrive/c/WINDOWS/System32/bcrypt.dll (0x7ffd2f3e0000)
msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)

WorBry

21st March 2019, 19:20

With the seek-test script yes, but not in the metric tests.

Just to mention that the seek-test script (Wolfberry ffms2 seekmode=1) reports no issues with x264 mp4/mkv

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx264 -preset slow -crf 28 -pix_fmt yuv420p -r 50/1 -x264-params colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/crowd_run_1080p50_y4m_x264_CRF28.mp4

The rav1e.mkv encodes of Crowd Run that ChaosKing uploaded earlier...

https://forum.doom9.org/showthread.php?p=1866894#post1866894

....do give errors (1 frame off) with the seek-test (ffms2 seekmode=1) but the SSIM:GMSD results are consistent on consecutive runs and the log files give the frames in correct order.

Iron_Mike

22nd March 2019, 07:40

ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

https://forum.doom9.org/attachment.php?attachmentid=16775&stc=1&d=1553236484

more stats - x265 - preset: medium vs. fast

Butteraugli / MDSI per CRF

https://i.imgur.com/6QVs6Ru.png

VMAF / SSIM / GMSD per bitrate (Mbps)

https://i.imgur.com/PM5RWBo.png

Butteraugli / MDSI per bitrate (Mbps)

https://i.imgur.com/EzIjj93.png

WorBry

22nd March 2019, 19:38

Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?

VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.

But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.

I ran self tests in all of the studies I carried out earlier, as well as duplicate tests on representative encodes (typically at the highest and lowest quality settings), and encode duplicates, (except for the rav1e series, which I didn't encode) to check for reproducibility before proceeding to test the entire series. And found no inconsistencies.

Iron_Mike

22nd March 2019, 22:32

Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?
But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.

so run a control test on a x265 encode that has custom keyframes w/ open-gop ? seekmode=0 or seekmode=1 ? or what are we looking for here ?

VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.

the "first frame" thing is only specific to the crowdrun file, we discussed this a while ago. When I ran extensive VMAF control tests on other files than crowdrun, the score was never 99.999 and it wasn't just the first frame score that threw it off...

as you know, NF does state that control test score of 98.x are to be expected...

WorBry

22nd March 2019, 23:44

so run a control test on a x265 encode that has custom keyframes w/ open-gop ? seekmode=0 or seekmode=1 ? or what are we looking for here ?

If that's how you encoded your x265 files, yes, and using whatever seekmode you used in your metric tests. You don't need to test the entire series.

You should also run a 'self-test' on the source/reference clip though - I assume it was crowd_run_1080p50.y4m.

the "first frame" thing is only specific to the crowdrun file, we discussed this a while ago. When I ran extensive VMAF control tests on other files than crowdrun, the score was never 99.999 and it wasn't just the first frame score that threw it off...

as you know, NF does state that control test score of 98.x are to be expected...

All I was saying is that VMAF is an exception.

Iron_Mike

23rd March 2019, 00:25

If that's how you encoded your x265 files, yes, and using whatever seekmode you used in your metric tests. You don't need to test the entire series.

You should also run a 'self-test' on the source/reference clip though - I assume it was crowd_run_1080p50.y4m.

control - crowd_run_1080p50.y4m - seekmode=0 - GMSD/SSIM

stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500

control - (one example file) x265 CRF 28, preset medium, open-gop, no custom keyframes - seekmode=0 - GMSD/SSIM

--> this is how I encoded my series of encodes for the preset medium|fast comparison

stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500

WorBry

23rd March 2019, 00:54

That's good then.

ChaosKing

23rd March 2019, 11:05

Ok here's my idea for testing for frame seeking issues. VSNTSF - VapourSynths Next Top Source Filter

Download https://www.dropbox.com/s/illkfly7wq72ay7/VapourSynth64PortableTEST.rar?dl=1

Put your video files in test_chamber\video_test_files
ffms2 & lsmas dlls in test_chamber\ffms2 & test_chamber\lsmas

Edit the bat file if needed
set start=0
set end=300
set tests=ffms2 lsmas lsmasav ffms2seek0

Result is a csv file (open in excel or similar app)
https://i.imgur.com/YP6vAxa.png

Testing only the first 10 frames!
https://i.imgur.com/0Sn66gc.png

EDIT:
- Added LibavSMASHSource() which doesn't create an index, but works only with mp4? (From Avisynth wiki: " Recommended for MP4, MOV, ISO Base Media and its derived file formats.")

set tests=ffms2 lsmas lsmasav ffms2seek0
means:
- ffms2.Source()
- lsmas.LWLibavSource()
- lsmas.LibavSMASHSource()
- ffms2.Source(seekmode=0)

WorBry

24th March 2019, 05:48

Works as expected. Results consistent with seek-test script. Nice that that can just drop test files in the 'test_chamber', and of course having the results compiled in a csv file. Very useful.

"VSNTSF" is surely a provisional title ? ;)

Iron_Mike

24th March 2019, 06:22

@Ck: good stuff

here are my results with the 3 files I shared as an upload

tests I ran (default): ffms2 lsmas lsmasav ffms2seek0

https://i.imgur.com/swtYdP5.png

ChaosKing

24th March 2019, 09:35

Attachments approval takes forever. Better upload it somewhere else, like imgur.com :)

"VSNTSF" is surely a provisional title ? ;)
Maybe "Torture File Chamber"? :devil:

Frame Seeker?

Iron_Mike

24th March 2019, 10:01

Attachments approval takes forever

was this for me ? Are you saying you cannot see the image ?

ChaosKing

24th March 2019, 10:12

Yes
https://i.imgur.com/gArMvcU.png

Iron_Mike

24th March 2019, 10:16

Yes
https://i.imgur.com/gArMvcU.png

wow. thanks for that side note... 2 weeks too late, I may add CK but better late than never... ;-)))

alright, just posted the csv snapshot on Imgur, u should see it now...

will post the other test result images now there as well...

funny bit is that when I post and/or preview the post it shows me all attachments and linked to images, INSIDE the post... had no idea other members could not see it... :sly:

Iron_Mike

24th March 2019, 10:23

alright, updated my other two posts that show the metrics test results for x265 preset medium vs. fast

https://forum.doom9.org/showpost.php?p=1869215&postcount=138
https://forum.doom9.org/showpost.php?p=1869644&postcount=238

ChaosKing

24th March 2019, 11:44

Nice results.
But lsmash 2017 has AV1 support? That can't be correct. Could you check manually?

ChaosKing

24th March 2019, 12:21

Replace this file in VapourSynth64: https://www.dropbox.com/s/ivtbc83g3yyibj0/seek-test.zip?dl=1
I only checked for vs.Error.

Edit: argh I should have tested it more, still doesn't work.
Edit2: It's working now. lsmash doesn't throw an exception with invalid files...

Wolfberry

24th March 2019, 12:45

New builds for testing (http://forum.doom9.org/showthread.php?t=176198)

ChaosKing

24th March 2019, 13:12

Thx for the new dlls
Update https://www.dropbox.com/s/illkfly7wq72ay7/VapourSynth64PortableTEST.rar?dl=1
- Fixed spaces in filenames are now allowed
- Added Wolfberrys dlls

Wolfberry

24th March 2019, 13:54

The new ffms2 with dav1d is bascially the same as the one labeled 2.31.0.0, the difference is 1 commit in dav1d and openssl.

You can keep only one for testing and results should be the same.

L-SMASH with libvpx has no seeking issues for the vp9 files I tested.

WorBry

24th March 2019, 16:27

Attachments approval takes forever. Better upload it somewhere else, like imgur.com :)

Maybe "Torture File Chamber"? :devil:

Frame Seeker?

'You've Been Framed' ? :p

poisondeathray

24th March 2019, 16:40

Effect of muxer on seeking ?

When you guys a chance , can you test this crowd run encode ? It's encoded with the modded yuuki build (default settings, with --keyint 100 --min-keyint 100 ) . It is compiled with lsmash support , so can export MP4 directly instead of ES

https://www.mediafire.com/file/mwij210cqk4sybn/yuuki.mp4/file
https://down.7086.in/x265-Yuuki-Asuna/

1) using default seekmode, seek-test ok here, and metric results consistent using the --min-keyint 100 --keyint 100 settings. Verified that open GOP's present as well, same as other x265 CLI and ffmpeg libx265 builds

2) But I cannot mux a "normal" x265 ES with independent lsmash muxer and have it work (I tried differen x265 binaries) . You can't even output an ES from the same yuuki mod and mux with the lsmash muxer separately and have it working for seek-test and consistent metrics with default seekmode . I tried several versions of lsmash muxer. Something to do with muxing as it encodes, or some muxer setting (I played with some settings)

I tried other muxers, mp4box (different settings in the full help) , other x265 builds (in case ffmpeg muxer, which has been known to have problems in the past, was causing issues; or if ffmpeg libx265 was causing issues)

3) Newer MP4Box / GPAC muxers will change the brand, because it auto detects open GOP's . And the "codec id" and compatible brands for the container will be "iso4 (iso4/iso6)" instead of "mp42 (mp42/mp41/isom)" . But it does not affect the seek test results

Stream uses forward prediction - stream CTS offset: 2 frames
OpenGOP detected - adjusting file brand

4) I tried looking at the box dumps both with different mp4box and lsmash muxes, but I cannot figure out what causes the difference for seeking. There are notable timescale differences between muxers for the movie timescale vs. track timescale. But even changing those settings to match the "seeking" yuuki file, it does not work for default seekmode with the other encodes

ChaosKing

24th March 2019, 20:19

'You've Been Framed' ? :p
- Frame & Seek
- Frame Me If You Can

Edit
The new lsmash is better:
https://i.imgur.com/PL8dYXo.png

WorBry

26th March 2019, 17:13

more stats - x265 - preset: medium vs. fast

Butteraugli / MDSI per CRF

https://i.imgur.com/6QVs6Ru.png

VMAF / SSIM / GMSD per bitrate (Mbps)

https://i.imgur.com/PM5RWBo.png

Butteraugli / MDSI per bitrate (Mbps)

https://i.imgur.com/EzIjj93.png

So, little, if any difference in the bitrate-matched metric scores, for this particular source at least.

Couple of points to note:

I see the muvsfunc SSIM and GMSD tests were run with no downsampling. VMAF does apply downsampling in deriving the component SSIM and MS-SSIM scores. That said in my initial tests with Crowd Run the VMAF derived SSIM scores were still higher than muvsfunc SSIM with downsampling applied - I'm still not sure why.

The MDSI tests on the other hand were run with downsampling applied (down_scale=2). Any particular reason for that ?

For graphic presentation of the results obtained over a wide bitrate range I found it more informative to plot the bitrate (x axis) as log base 2 - gives better separation of the scores at the lower bitrates. I think it's helpful to include the actual data points also.

Just something to bear in mind if you are going to be posting more of these tests.

@Wolfberry. Any chance you could add an implementation of the MS-SSIM metric to muvsfunc ?

https://ece.uwaterloo.ca/~z70wang/publications/msssim.pdf

ChaosKing

26th March 2019, 18:35

The vmaf plugin can also calc "psnr, ssim and ms_ssim" https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF#usage

WorBry

26th March 2019, 19:14

I know, but for some reason VMAF derives markedly higher aggregate (elementary) SSIM scores than muvsfunc SSIM (downsample=True).

I speculated earlier in the thread whether that might stem from differences in the size of the gauss kernel ?

https://forum.doom9.org/showthread.php?p=1865872#post1865872

Or is VMAF adding some other weighting in the SSIM and MS-SSIM calculations ?

Given these unknowns, I just think it would be useful to have a muvsfunc version of MS-SSIM that is implemented in like manner to (SS-) SSIM.

Iron_Mike

27th March 2019, 04:23

The MDSI tests on the other hand were run with downsampling applied (Downsample=2). Any particular reason for that ?

I thought Downsample=2 means Downsample=False... the MDSI implementation is more than confusing, should be 0/1 for False/True...

can somebody clarify ?

WorBry

27th March 2019, 05:21

I did clarify. Look at the MDSI function in the muvsfunc script and the Zoptilib default for MDSI.

.... should be 0/1 for False/True...

No it shouldn't. With the MDSI metric, the 'down-scale' parameter specifies the factor of downsampling (downscaling). Quote:

down_scale: (int) Factor of downsampling before quality assessment.
Default is 1.

So down_scale=1 (default) applies no downsampling and down-scale=2 downsamples by a factor of two.

You only need to run a test to see that down_scale=1 gives a lower score (i.e. higher figure) than down_scale=2.

WorBry

27th March 2019, 05:34

Thank-you HolyWu. That would explain it.

Iron_Mike

29th March 2019, 04:21

alright, here are some more tests I ran on these IQ metrics...

this is from a 16-bit float EXR source (15 secs, 360 frames), made nine (9) x265 encodes, all CRF 10, in these formats (using Wolfberry ffmpeg build): yuv420p, yuv422p, yuv444p, yuv420p10le, yuv422p10le, yuv444p10le, yuv420p12le, yuv422p12le, yuv444p12le

A few interesting results, but maybe some folks here can chime in on why things are the way they are...

Pixel Format vs. Bitrate

https://i.imgur.com/g6pObHL.png

--> why does 422 variants result in highest bit rates across 8/10/12 bit ? and yuv420p has higher bit rate than yuv444p12le ?

Libvmaf via FFMPEG (converts to yuv444p10le internally)

https://i.imgur.com/lK1bCMg.png

Libvmaf via VS (requires conversion to yuv444p10, if higher format)

https://i.imgur.com/o5b4UNC.png

--> while the general VMAF score trend/indication is the same in ffmpeg/VS, the SSIM/MS-SSIM results differ... VS ones are flat across all formats, ffmpeg 8-bit has higher score than 10/12-bit... (?)

GMSD / SSIM (muvsfunc)

https://i.imgur.com/XvTbjEQ.png

--> flat across 8-bit, and then finds no difference across the 10/12 bit formats...

MDSI / Butteraugli (requires 8-bit files)

https://i.imgur.com/DZMJFUe.png

--> Butteraugli does not seem to find a difference in any of the encodes... MDSI, from 10-bit on no more differences...

WorBry

29th March 2019, 17:14

Given the source....

this is from a 16-bit float EXR source

...the VS metric results don't surprise me.

Libvmaf via FFMPEG (converts to yuv444p10le internally)

https://i.imgur.com/lK1bCMg.png

Libvmaf via VS (requires conversion to yuv444p10, if higher format)

https://i.imgur.com/o5b4UNC.png

--> while the general VMAF score trend/indication is the same in ffmpeg/VS, the SSIM/MS-SSIM results differ... VS ones are flat across all formats, ffmpeg 8-bit has higher score than 10/12-bit... (?)

Looks like there is a slight increase in the VS libvmaf SSIM/MS-SSIM scores going from 8 to 10-bit - you'd likely see that more clearly with the SSIM/MS-SSIM scores plotted on a secondary Y axis that expands out the 99.75 - 100 range.

The drop in FFMPEG libvmaf SSIM/MS-SSIM scores is interesting though. Perhaps HolyWu has a perspective on that ?

Libvmaf via FFMPEG (converts to yuv444p10le internally)....

Libvmaf via VS (requires conversion to yuv444p10, if higher format)

I thought libvmaf internally converts to 8bit for all of component (elementary) metrics which are applied to the luma plane only. Correct or no?

Iron_Mike

29th March 2019, 22:31

I thought libvmaf internally converts to 8bit for all of component (elementary) metrics which are applied to the luma plane only. Correct or no?

not sure.

ffmpeg prints yuv444p10 as the format when calculating via libvmaf - u can feed it anything and it will convert internally.

VS VMAF allows input of up to yuv444p10 (u need to convert to that if source is higher), not sure what it does internally then.

I did ran one test in which I converted everything to 8-bit before feeding it to VS VMAF and the SSIM and MS-SSIM lines on the plot were absolutely straight across the 8/10/12 bit formats.

WorBry

29th March 2019, 22:38

I did ran one test in which I converted everything to 8-bit before feeding it to VS VMAF and the SSIM and MS-SSIM lines on the plot were absolutely straight across the 8/10/12 bit formats.

Presumably in that case you converted the EXR source/reference to 8bit also ?

WorBry

29th March 2019, 22:52

I thought libvmaf internally converts to 8bit for all of component (elementary) metrics, which are applied to the luma plane only. Correct or no?

But not in the ffmpeg libvmaf implementation, according to HolyWu here:

...because FFmpeg filter doesn't normalize 10 bit to 8 bit like what Netflix does for calculation, hence the inconsistency.

https://forum.doom9.org/showthread.php?p=1868436#post1868436

...same thread :

do you have a link to where they state that they downsample the master to 8bit ?

Netflix doesn't explicitly mention that in the documentation. It's simply done this way in their source code.

https://forum.doom9.org/showthread.php?p=1868966#post1868966

Iron_Mike

29th March 2019, 23:08

Presumably in that case you converted the EXR source/reference to 8bit also ?

yes, in that one test that I mentioned (which is not on the pics posted) everything higher than yuv444p was converted to yuv444p (including the 16-bit EXR reference) before feeding it to VS VMAF - if the encode that is evaluated was lower format than yuv444p (e.g. yuv420p, yuv422p), it was converted to yuv444p before feeding it to VS VMAF - as VMAF requires matching pixel formats

--> result was a flat line for libvmaf SSIM / MS-SSIM

the results on the posted pic followed the same workflow. but foimat fed to VS VMAF was yuv444p10 (highest supported)

--> result on the posted pic is almost the same, tiny bit lower score in 8bit formats.

So, any idea why the bit rate for 422 formats is the highest across 8/10/12 bit ? and why do 12-bit formats generally have lower bit rates than 8/10 bit ?

it also can bee seen that from 10-bit on, these metrics don't pick up differences any longer... And Butteraugli sees no difference at all...

Iron_Mike

29th March 2019, 23:14

But not in the ffmpeg libvmaf implementation, according to HolyWu here:

https://forum.doom9.org/showthread.php?p=1868436#post1868436

...same thread :

https://forum.doom9.org/showthread.php?p=1868966#post1868966

right, but that does not explain why FFMPEG libvmaf SSIM/MS-SSIM for 8bit has higher scores than for 10/12 bit

WorBry

29th March 2019, 23:53

That's what perplexes me also.

Perhaps it would be better to address this point in the VMAF thread, as this is really a continuation of your posts there.

I'm not set-up for ffmpeg libvmaf and so can't verify myself. But just to be clear, when you say....

ffmpeg prints yuv444p10 as the format when calculating via libvmaf - u can feed it anything and it will convert internally.

....does that include the ffmpeg vmaf tests with the 8-bit x265 formats also (i.e. they are up-sampled to yuv444p10 for the vmaf calcs) or does that only apply to the 10/12 bit formats ?

So, any idea why the bit rate for 422 formats is the highest across 8/10/12 bit ? and why do 12-bit formats generally have lower bit rates than 8/10 bit ?

Can't give a precise answer off the top of my head. 422 has more chroma information than 420 and so needs more bits, but why 444 has the lowest bitrate (across 8/10/12 bit depth) at the same CRF level, I'm not sure. Perhaps others could weigh in on that.

Iron_Mike

30th March 2019, 02:56

That's what perplexes me also.

Perhaps it would be better to address this point in the VMAF thread, as this is really a continuation of your posts there.

posted here (https://forum.doom9.org/showpost.php?p=1870364&postcount=96)

I'm not set-up for ffmpeg libvmaf and so can't verify myself. But just to be clear, when you say....

....does that include the ffmpeg vmaf tests with the 8-bit x265 formats also (i.e. they are up-sampled to yuv444p10 for the vmaf calcs) or does that only apply to the 10/12 bit formats ?

(maybe I'm misinterpreting the output) it seems ffmpeg internally evaluates in the same format as the distorted input file, see here...

ref RGB48le, dist yuv420p
https://i.imgur.com/wyNdEy6.png

ref RGB48le, dist yuv420p10le
https://i.imgur.com/Ak3Arqj.png

ref RGB48le, dist yuv444p12le - as you can see it down-converts to yuv444p10 as that is the max supported format
https://i.imgur.com/GB7bY82.png

as stated before, the problem w/ this approach is (IF the ref and dist formats differ), that you lose precision converting the ref down to whatever the format of the dist is, distorting results... it would be a better approach to up-convert the dist to the format of the ref, as you should never ever change the ref, b/c that compromises the whole comparison...

in case of crowdrun 1080p that was never an issue as the ref was already bare bottom yuv420p...

WorBry

30th March 2019, 03:19

Well, I don't know. Probably best to see what HolyWu has to say about it.

poisondeathray

30th March 2019, 03:26

as stated before, the problem w/ this approach is (IF the ref and dist formats differ), that you lose precision converting the ref down to whatever the format of the dist is, distorting results... it would be a better approach to up-convert the dist to the format of the ref, as you should never ever change the ref, b/c that compromises the whole comparison...

in case of crowdrun 1080p that was never an issue as the ref was already bare bottom yuv420p...

Another problem is vmaf does not support 16bit RGB , and ffmpeg does not support EXR float (only 16bit int) .

You are going to have compromises if you want to use vmaf eitherway - you have to change the ref, no way around it

You can make a request to the vmaf developers to extend support for other pixel types

Iron_Mike

30th March 2019, 03:38

Another problem is vmaf does not support 16bit RGB , and ffmpeg does not support EXR float (only 16bit int) .

You are going to have compromises if you want to use vmaf eitherway - you have to change the ref, no way around it

yeah, unfortunately.

I convert 16bit float RGB to 32-bit YUV 444 (to keep precision in VS for some plugins that only support 16bit int) - and that conversion is a small compromise at best IMO compared to converting 16-bit float RGB to yuv420p to then be used as a ref in VMAF comparison...

these metrics are just indications anyways, but the approach of distorting the ref that much is flawed...

You can make a request to the vmaf developers to extend support for other pixel types

the problem is they converting the ref down to the dist, instead of converting the dist up to the ref - independent of supported formats. if your ref is yuv444p and dist is yuv420p (both supported), it makes no sense to down-convert the ref to yuv420p to then compare...

poisondeathray

30th March 2019, 03:48

WorBry

30th March 2019, 03:50

Bear in mind also that the muvsfunc MDSI and Butteraugli tests require converting both the reference and test clip to RGB, and RGB24 specifically in the case of Butteraugli.

poisondeathray

30th March 2019, 04:06

I thought libvmaf internally converts to 8bit for all of component (elementary) metrics which are applied to the luma plane only. Correct or no?

HolyWu in the vmaf thread said the Netflix implementation converts to 8bit in their source code

And this suggests chroma isn't looked at

Currently, VMAF does not use chroma features, and does not fully express the perceptual advantage of HDR / WCG videos.

https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12

For the other Iron_Mike tests, they were Y only , right ? For example, not SSIM-U , or SSIM-V . Because you would expect 444 to score better than 420 in terms of color in general . Or what was the point of 444,422,420 testing ?

I don't think the netflix implementation can , but the muvsfunc can specify the plane processed

WorBry

30th March 2019, 04:37

I thought libvmaf internally converts to 8bit for all of component (elementary) metrics which are applied to the luma plane only. Correct or no?

HolyWu in the vmaf thread said the Netflix implementation converts to 8bit in their source code

Yes, I made reference to that subsequently:

https://forum.doom9.org/showthread.php?p=1870341#post1870341

But he also said that the 'ffmpeg filter' (which in context I assume referred to the ffmpeg libvmaf filter) doesn't normalize 10bit to 8bit for the calculations.

And this suggests chroma isn't looked at

https://medium.com/netflix-techblog/vmaf-the-journey-continues-44b51ee9ed12

Was sure I'd seen that stated somewhere.

For the other Iron_Mike tests, they were Y only , right ? For example, not SSIM-U , or SSIM-V . Because you would expect 444 to score better than 420 in terms of color in general . Or what was the point of 444,422,420 testing ?

He did test MDSI and Butteraugli also.

Seems ironic that Butteraugli requires RGB24 yet appears to be quite unresponsive to subtle chroma/chromacity distortions - as seen in the Crowd Run metric tests where I was comparing different 10bit 422 'intermediate formats':

https://forum.doom9.org/showthread.php?p=1868159#post1868159

..and the 3 posts that followed.

I get the impression that Butteraugli is primarily tuned for JPEG compression artifacts. Personally, I don't think it brings any 'added value' in this context, so I've stopped testing it. MDSI on the other hand is very sensitive to chroma distortions.

Iron_Mike

30th March 2019, 05:30

I don't like it either.

But vmaf is for a specific scenario, viewing distance etc... Streaming web video delivery typically won't be 12bit or 16bit ...

And typically displays won't be 10bit either. Sure, they are becoming more common, but >99% of displays will still be 8bit or lower. vmaf is all about the perception of the video from the end user point of view

display tech does not matter (and will change anyways), it's about the approach in general. as I wrote before: when you deliver to NF, Amazon, Hulu, Vimeo or Youtube - you want to make sure you deliver the highest quality file that they allow you to deliver (some limit you in upload size, NF doesn't). if you deliver in 8-bit then you must have shot on a camera from the 90's, or a consumer camera - which is already amateur, meaning NOT a professional production. But even then, your grade will be at least viewed on 10-bit equipment, if it's not then you have no clue what you're doing. Bare bones minimum export out of the grading app is 10-bit, standard is 16-bit float master export b/c the grade and VFX etc are at 32/64 bit internally...

All of these streaming services then take your ref master (that you deliver to them), and make X encodes from it for different resolutions, bit rates etc.... THAT is the process. so, ultimately for metrics you want to compare to the ref master, which is almost never in a professional production 8-bit - but more importantly the bit-depth and the chroma subsampling should not matter in the comparison - u just up-convert the dist to the ref, it won't gain or lose precision.

so with a proper approach, u can compare RGB48 to yuv420p, or yuv422p10 to yuv444p10 - it don't matter.

I understand some of these metrics are just starting out, but the fact that VMAF distorts the ref to the dist format in order to compare is very, very flawed...

As suggested earlier, you can control the conversions and algorithms , instead of letting "auto" conversions do it for you . For example if 10bit444 was the max supported pixel type, you could have controlled the RGB48le conversion to 10bit444 , and the upconversion of yuv420p to 10bit444 in that 1st test . It would make me feel better that it's not as a low downconversion for the source.

I did do that - I mentioned that. Since the ref master was RGB48le, which needed to be converted to yuv444p10le (max supported format by VMAF), every encode was conformed to yuv444p10le as well.

The problem is if VS VMAF internally then converts everything to 8-bit anyways (and the way I interpreted it, Holy Wu confirmed that), then the question is shouldn't I just bring everything down to yuv444p... ? (which is why I made that one test mentioned on the last page, but then the results were completely useless)

there are two problems with this approach of 8-bit in VS VMAF (if res src is higher than 8-bit):

(1) if I have to convert RGB48le to yuv444p in order to run a metric test, the scores will be very close together for a variety of formats which makes all of this meaningless... look at the Buttetraugli results, it requires 8-bit, which I did provide by converting everything down to it... look at the results... there is ZERO difference in the score between yuv420p and yuv444p12le... absolutely idiotic... I can see differences by easily eye (as you can imagine, 12bit 444 vs. 8bit 420)

same for VS VMAF SSIM/MS-SSIM... the more you compromise the ref master (by converting), the more you contaminate the test

(2) it is just very confusing for VS VMAF to state "max supported format is yuv444p10" if you ultimately internally then convert anyways to 8-bit... then just state the exact format your using internally, so that users can control the conversion to it

WorBry

30th March 2019, 05:50

I see HolyWu responded to your query in the VMAF thread:

https://forum.doom9.org/showthread.php?p=1870370#post1870370

I have to confess, I'm now thoroughly confused re: the behaviors of VS VMAF vs ffmpeg VMAF, and so will stick with the former.

poisondeathray

30th March 2019, 06:29

display tech does not matter (and will change anyways), it's about the approach in general. as I wrote before: when you deliver to NF, Amazon, Hulu, Vimeo or Youtube - you want to make sure you deliver the highest quality file that they allow you to deliver (some limit you in upload size, NF doesn't). if you deliver in 8-bit then you must have shot on a camera from the 90's, or a consumer camera - which is already amateur, meaning NOT a professional production. But even then, your grade will be at least viewed on 10-bit equipment, if it's not then you have no clue what you're doing. Bare bones minimum export out of the grading app is 10-bit, standard is 16-bit float master export b/c the grade and VFX etc are at 32/64 bit internally...

Yes, I agree in general. But you don't need a metric to tell you what is the "highest" quality for the submission according to the guidelines (at least I hope you don't...) .

But you were referring to VMAF - You're talking about a specific implementation of a specific metric that deals with a specific viewing conditions on current displays, not ones 20 years from now. That's what VMAF is about now . Maybe VMAF in 2039 will be different. It's not about acquisition formats or any post production or any of that. They didn't train their models to account for whether or not it was a full 32bpc float pipeline. Irrelevant to their models and testing and viewing conditions. Remember, VMAF wasn't trained on master formats, so it's not valid for what your master submission format anyways. Maybe they can expand it one day, add more models and extensions .

All of these streaming services then take your ref master (that you deliver to them), and make X encodes from it for different resolutions, bit rates etc.... THAT is the process. so, ultimately for metrics you want to compare to the ref master, which is almost never in a professional production 8-bit - but more importantly the bit-depth and the chroma subsampling should not matter in the comparison - u just up-convert the dist to the ref, it won't gain or lose precision.

so with a proper approach, u can compare RGB48 to yuv420p, or yuv422p10 to yuv444p10 - it don't matter.

Yes, the proper approach is to convert to the uniform format, as stated way back. So you can precisely control all the variables about how the conversions are done , what algorithms are used. Many of the earlier netflix ffmpeg issues in the github tracker were because of using the wrong resizing flags.

Ideally, if your "master" was 16bit float, then you'd use a metric that could measure 16bit float, and you upconvert your other distribution versions to measure at the reference format. But we don't really live in an ideal world. I don't know of any that can properly measure float formats

I understand some of these metrics are just starting out, but the fact that VMAF distorts the ref to the dist format in order to compare is very, very flawed...

Yes . I think that's this newer patch for ffmpeg2vmaf's "bandaid" solution, because you needed a common format . The older implementation would throw an error. You had to do the conversion

The problem is if VS VMAF internally then converts everything to 8-bit anyways (and the way I interpreted it, Holy Wu confirmed that), then the question is shouldn't I just bring everything down to yuv444p... ? (which is why I made that one test mentioned on the last page, but then the results were completely useless)

But apparently normalized values are stored in FP...
https://forum.doom9.org/showthread.php?p=1870370#post1870370

vmafossexec (the CLI of libvmaf) also does this normalization for 10-bit input. The normalized values are stored in floating-point, hence you needn't worry about precision lost.

same for VS VMAF SSIM/MS-SSIM... the more you compromise the ref master (by converting), the more you contaminate the test

I agree ; ideally the reference pixel format should be what you're measuring against.

(2) it is just very confusing for VS VMAF to state "max supported format is yuv444p10" if you ultimately internally then convert anyways to 8-bit... then just state the exact format your using internally, so that users can control the conversion to it

Still not clear on that because of the FP comments

I did some earlier tests and there were differences between a yuv420p10 and it's yuv420p8 derivative . But now that I think about it, it might have been the ffmpeg vmaf implementation which does not downconvert to 8bit . I'll have to check later

Iron_Mike

30th March 2019, 08:22

Yes, I agree in general. But you don't need a metric to tell you what is the "highest" quality for the submission according to the guidelines (at least I hope you don't...) .

highest quality will be lossless 16-bit float EXR, but you won't provide that to these folks... because you would provide terabytes of data...

so, you want a master codec that brings down file size as much as possible and still looks visually lossless... along the lines of WorBry's recent codec test w/ Prores etc...

then, sometimes they restrict you in upload size etc etc, hence the need to use a codec/bit depth/chroma combo that performs as good as possible given the allowed bit rate...

and then - and I can only stress this - they will decode your delivery and re-encode... funny things can happen there...

in the CRF tests w/ crowdrun we saw that CRF 16 (or better) was the point that got you there... but that was 8-bit 420 on all stages...

in my last tests, one would assume from 10 bit on there is no difference (as the curves flat out), but I think this is only b/c most of these metrics require down-conversion to 8-bit...

which brings me to another point, maybe I could improve the way I down-convert:

RGB48le conversion to YUV 32-bit 444
vid_ref = core.fmtc.matrix(clip=vid_ref, mat=matrix, col_fam=vs.YUV, bits=32)

Conversion to 8-bit
vid_ref = core.fmtc.bitdepth(clip=vid_ref, bits=8)

Matching another video format
vid_enc = core.resize.Bicubic(vid_enc, format=vid_ref.format.id, matrix_s=matrix)

anything wrong with that ?

WorBry

30th March 2019, 16:48

No, libvmaf doesn't convert the format. It's FFmpeg automatically doing the format conversion when the format of the two input clips differs, before sending frames into libvmaf. Because FFmpeg puts the distorted clip as the first argument and the reference clip as the second argument, hence FFmpeg converts the format of the second clip to that of the first clip when they differ.

Same as the ffmpeg SSIM and PSNR filters then.

poisondeathray

30th March 2019, 18:10

so, you want a master codec that brings down file size as much as possible and still looks visually lossless... along the lines of WorBry's recent codec test w/ Prores etc...

Applying a test designed for lower delivery bitrates to qualify a high bitrate "master" doesn't seem right either.

Remember , the current VMAF was trained from CRF 22 to CRF 28 (they don't say whether that was x264 or x265, 8bit or 10bit) . But either way, this is certainly not "master" quality.

(Also , beware the quantizer scaling is different between x264, x265, 8bit and 10bit . The numbers don't mean the same thing)

in the CRF tests w/ crowdrun we saw that CRF 16 (or better) was the point that got you there... but that was 8-bit 420 on all stages...

in my last tests, one would assume from 10 bit on there is no difference (as the curves flat out), but I think this is only b/c most of these metrics require down-conversion to 8-bit...

But clearly the "results" are problematic . You can "see" the difference, say, at CRF 10 than CRF 16 . You can see more details, fewer artifacts. Or at least you should be able to if you are in that 4% . Production/Post-production people should be able to. You're not "Joe Public" . That early plateau indicates VMAF is not suitable for assessing "visually lossless" scenarios

I think it' s more that the test is invalid for that bitrate range ~ CRF10 .

Iron_Mike

30th March 2019, 22:12

But clearly the "results" are problematic . You can "see" the difference, say, at CRF 10 than CRF 16 . You can see more details, fewer artifacts. Or at least you should be able to if you are in that 4% . Production/Post-production people should be able to. You're not "Joe Public" . That early plateau indicates VMAF is not suitable for assessing "visually lossless" scenarios

I think it' s more that the test is invalid for that bitrate range ~ CRF10 .

yeah, I agree. hence u need to run various of these metrics and then look at trends/indication.

Also , beware the quantizer scaling is different between x264, x265, 8bit and 10bit . The numbers don't mean the same thing

can u eloborate on this ?

poisondeathray

30th March 2019, 22:51

can u eloborate on this ?

crf "x" is really a rate control method, but it can be used (often abused) as a rough estimate of "quality"

But the value "x" doesn't mean the same thing in 8 bit, 10bit , x264 or x265 . eg. You can't say CRF 16 with x264 8bit produces roughly the same thing as CRF 16 x265 8bit . It's really just an arbitrary number

A hint is the default value is 23 for x264 8bit but 28 for x265 8bit . The scale is 0-51 in 8bit, 0-63 in 10bit x264 10bit .
"0" is lossless for x264 (both 8,10bit) , but a qp of "4" is lossless for x265

In the past they've rescaled the crf calculations several times too. There were (-) crf values allowed at one point for higher bitdepths .

Iron_Mike

30th March 2019, 23:42

crf "x" is really a rate control method, but it can be used (often abused) as a rough estimate of "quality"

But the value "x" doesn't mean the same thing in 8 bit, 10bit , x264 or x265 . eg. You can't say CRF 16 with x264 8bit produces roughly the same thing as CRF 16 x265 8bit . It's really just an arbitrary number

A hint is the default value is 23 for x264 8bit but 28 for x265 8bit . The scale is 0-51 in 8bit, 0-63 in 10bit x264 10bit .
"0" is lossless for x264 (both 8,10bit) , but a qp of "4" is lossless for x265

In the past they've rescaled the crf calculations several times too. There were (-) crf values allowed at one point for higher bitdepths .

makes sense. so what's ur take on (re my last pics uploaded here) that the bit rates for 12-bit were generally the lowest, and that 422 flavor in all bit-depths had the highest bit rate ?

the higher pixel formats did outscore 8-bit (in the metrics) but the bit rates were lower... I expected bit rates trends to be roughly the same for a given chroma flavor 420|422|444...

poisondeathray

31st March 2019, 00:03

so what's ur take on (re my last pics uploaded here) that the bit rates for 12-bit were generally the lowest, and that 422 flavor in all bit-depths had the highest bit rate ?

I don't know .

Interesting trend , but not sure it means anything . You have to be careful how you interpret anything done with CRF . As soon as you change 1 thing, all bets are off

the higher bit rates did outscore 8-bit (in the metrics) but the bit rates were lower... I expected bit rates trends to be roughly the same for a given chroma flavor 420|422|444...

Yes, I expect higher bitrates ,in general, to outscore lower bitrates

But I don't have expectation of bitrate at a given CRF using different settings (different pixel format is a different setting) . In the end, CRF is just a rate control method.

Iron_Mike

31st March 2019, 04:55

Yes, I expect higher bitrates ,in general, to outscore lower bitrates

But I don't have expectation of bitrate at a given CRF using different settings (different pixel format is a different setting) . In the end, CRF is just a rate control method.

sorry, typo on my end:

I meant to say the higher pixel formats (which ended up having lower bit rates than 8-bit) still outscored 8-bit in these metrics...

I don't care about the CRF setting at all, but the bit rate does indicate how much data is in the stream (and all encodes are using the same format here: x265), and lower bit rates did outscore higher bit rates...

so that is interesting.

and then the 422 trend - independent of bit-depth - having the highest bit rate...

maybe this has to do w/ how ffmpeg specifically encodes files...

poisondeathray

31st March 2019, 06:09

I meant to say the higher pixel formats (which ended up having lower bit rates than 8-bit) still outscored 8-bit in these metrics...

I don't care about the CRF setting at all, but the bit rate does indicate how much data is in the stream (and all encodes are using the same format here: x265), and lower bit rates did outscore higher bit rates...

so that is interesting.

In general, you'd expect 10bit encodes to score higher, against 10bit (or even scaled down against 8bit) in general for typical or high bitrate ranges. The compression efficiency and precision of 10bit vs 8bit is higher . This is very well established , many tests , some academic papers. It's the SSIM/MS-SSIM libvmaf dip that seems out of order .

At very low bitrate ranges, however, it reverses. 10bit will cost more, produce lower quality. It's worse with x265 than x264. (or another way to put it is x264 benefits more from 10bit vs. 8bit advantage)

12bit is less researched . And experience /usage is very low. I think the additional gains are low.

and then the 422 trend - independent of bit-depth - having the highest bit rate...

maybe this has to do w/ how ffmpeg specifically encodes files...

Not sure, it's only one test series, one source .

WorBry

31st March 2019, 16:10

and then the 422 trend - independent of bit-depth - having the highest bit rate...

maybe this has to do w/ how ffmpeg specifically encodes files...

Crowd Run shows the same pattern. Here the 1080/50p 10-bit 422 version I used in the ProRes tests converted to x265 420,422 and 444 (8 and 10bit) at CRF=10.

Bitrates:

8bit 4:2:0 175 Mbps
8bit 4:2:2 214 Mbps
8bit 4:4:4 176 Mbps

10bit 4:2:0 174 Mbps
10bit 4:2:2 211 Mbps
10bit 4:4:4 175 Mbps

Haven't run any metrics on them. Been looking at other stuff.

It's the SSIM/MS-SSIM libvmaf dip that seems out of order

It does seem odd, especially when the aggregate VMAF scores show the opposite trend. Evidently there is sufficient bias in the other component metrics to swing it the other way.

That said, if the metric results had been presented with the scores matched for bits i.e. as a ratio of score/bitrate or score/(bits per pixel), which is probably more valid, we'd be seeing a different pattern with respect to 420 vs 422 vs 444.....judging from your data, the bit-matched metric scores for 422 would be lower than 420 and 444 at each bit depth.

BTW - you didn't state the resolution of your EXR source, and also the specific VMAF 'model' used in your tests - in turn dictated by whether you ran in CI=False or True mode. Was it the same in the VS and ffmpeg VMAF tests ?

WorBry

31st March 2019, 21:19

Been looking at other stuff.

Meanwhile, back at the ranch, I've been continuing where I left off:

...whilst Crowd Run serves as a good test reference for it's hard-to-compress complex/colorful content, I wouldn't say it's an especially sharp image, at least by contemporary 4K standards; there's a fair bit of motion and pan blur going on there and it was downscaled (Spline36) to 1080p for these tests also.

......I'll look and see what other HQ sharp sources I can test.

So here I used some UHD (2160/29.970p) footage shot in XAVC-S (8bit 420) format on a Sony AX100.

I used TMPGenc SmartRenderer 5 to sample 4 scenes each of 4 GOP length. The cuts were made on key frames so there was no re-encoding. The resulting test clip was 8.5 sec duration (256 frames).

I encoded the clip to x264 and x265 but limited to CRF 1-5 range - I was more interested in seeing what fine differences the metrics could pick up in the near-lossless domain than quality efficiency at lower bitrates, and especially in light of the results obtained with Boulder's Black Sails clip, where x265 had the edge:

https://forum.doom9.org/showthread.php?p=1868499#post1868499

In this instance however I didn't encode 'All Intra':

ffmpeg -i {Path}:/AX100.mp4 -vcodec libx264 -preset slow -crf {Value} -pix_fmt yuv420p -r 30000/1001 -x264-params colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/AX100_x264_CRFx.mp4

ffmpeg -i {Path}:/AX100.mp4 -vcodec libx265 -preset slow -crf {Value} -pix_fmt yuv420p -r 30000/1001 -x265-params colorprim=1:transfer=1:colormatrix=1 {Path}:/AX100_x265_CRFx.mp4

Here are the muvsfunc SSIM, GMSD and MDSI test results, with and without downsampling. I used LWLibavSouce for import.

http://i.imgur.com/TntEcu3l.png (https://imgur.com/TntEcu3)

Now isn't that interesting. In this case, x264 gives higher (bitrate-matched) scores than x265 for all three metrics. So I turned to the quality maps to see what more could be gleaned. For this I selected frames from the four scenes that I knew to be I frames i.e. the leading I frame at each scene change. The bitrates of the x264 CRF1 and x265 CRF2 encodes were very close (457.4 and 458 Mbps respectively), so I think it's valid to compare the equivalent frames from each.

Here are the quality maps obtained with no downsample applied. As before, I amplified the map traces with two Unsharp Mask passes in Gimp. Note that the original 2160p maps and source frames were downsized to 1080p for these composites and there's a fair bit of aliasing going on in the downsized source frames. So I've included grabs of the original 2160p frames also for reference and close scrutiny.

As usual, click on the image to enlarge, on the (+) cursor to enlarge further and right click 'Save Image As' to download at original resolution.

Frame #0:

http://i.imgur.com/g9mutYKl.jpg (https://imgur.com/g9mutYK)

http://i.imgur.com/eDNcD0El.jpg (https://imgur.com/eDNcD0E)

Frame #65:

http://i.imgur.com/2k252Qzl.jpg (https://imgur.com/2k252Qz)

http://i.imgur.com/6Dx6xZRl.jpg (https://imgur.com/6Dx6xZR)

Frame #129 (It was a different scene)

http://i.imgur.com/aur4XYel.jpg (https://imgur.com/aur4XYe)

http://i.imgur.com/5D6N4iGl.jpg (https://imgur.com/5D6N4iG)

Frame #193

http://i.imgur.com/laKV2yTl.jpg (https://imgur.com/laKV2yT)

http://i.imgur.com/DK8IisSl.jpg (https://imgur.com/DK8IisS)

Feel free to scrutinize and interpret.

In the Black Sails clip tests the higher metric scores with x265 were put down to:

Quite interesting that the maps show such an amount of difference at that bitrate. I think that there you can see the fundamental difference between x264 and x265, the first one has blocking/is more focused on enhancing the edges and higher frequencies by creating "fake detail" at default settings while the latter one likes to blur more.

....and:

I think you're right. Here are crops from another frame.

http://i.imgur.com/NR57nspm.png (https://imgur.com/NR57nsp)

Click image to enlarge and (+) cursor to enlarge further

The x265 image definitely has more blur than x264 (notably on the skin textures), which MDSI deems more acceptable.

With this source however that appears to work against x265, at least in terms of the metric scores. Which looks better from psychovisual perspective of course is another matter that might be open to debate.

Iron_Mike

3rd April 2019, 04:24

BTW - you didn't state the resolution of your EXR source, and also the specific VMAF 'model' used in your tests - in turn dictated by whether you ran in CI=False or True mode. Was it the same in the VS and ffmpeg VMAF tests ?

source/ref resolution and encoding resolution was 1920x1080p.

Model was 0.6.1 as stated on the pics - I used the exact same model in both VS|ffmpeg tests, meaning: the exact same .pkl file.

Iron_Mike

3rd April 2019, 04:30

For the other Iron_Mike tests, they were Y only , right ? For example, not SSIM-U , or SSIM-V . Because you would expect 444 to score better than 420 in terms of color in general . Or what was the point of 444,422,420 testing ?

I don't think the netflix implementation can , but the muvsfunc can specify the plane processed

Could you elaborate on this ?

which SSIM implementation (libvmaf SSIM | libvmaf MS-SSIM | SSIM muvsfunc) processes Y plane only ?

poisondeathray

3rd April 2019, 05:03

I encoded the clip to x264 and x265 but limited to CRF 1-5 range - I was more interested in seeing what fine differences the metrics could pick up in the near-lossless domain than quality efficiency at lower bitrates, and especially in light of the results obtained with Boulder's Black Sails clip, where x265 had the edge:

Interesting CRF 1-5 results.

For lossless encoding, x264 compresses better than x265 by few % ; at least for 8bit 4:2:0 . I've compared them head to head on about 30 sources over the last few years (different types of content), and 100% of the time x264 compresses better .

For moderate to high bitrates , in scenarios where you have adequate bandwidth to attempt to preserve details, I would disable SAO for x265 . It tends to be a detail killer, acting like a smoothing filter. I always do this for my own usage.

Could you elaborate on this ?

which SSIM implementation (libvmaf SSIM | libvmaf MS-SSIM | SSIM muvsfunc) processes Y plane only ?

for muvsfunc, you specify the plane= . You might be able to do all 3 with plane=[0,1,2]

Not sure about netflix libvmaf SSIM, but the earlier Netflix comments say color isn't looked at for VMAF. And you would expect the log should show U, V (or Cb,Cr) channel info - but it doesn't

ffmpeg ssim log will print out SSIM-U, SSIM-V, SSIM-Y , so you know all 3 channels are measured

Then, there are different weighting formulas for Y,U,V . ie. What is the "correct" way to weight the channels ? Simple mean? "Y" is much more important to human vision . You can optimize for some metrics by tweaking chroma offsets for encoders (stack the "Y" by underallocating, U,V) . Since you know Neflix VMAF doesn't look at color, you could allocate more bitrate into Y for example, and score higher

Also I did 8bit/10bit 4:2:0,4:2:2,4:4:4 test on another source at CRF21 and the 4:2:2 was significantly larger , so that's another data point . Still not sure if it means anything, but still interesting

Iron_Mike

3rd April 2019, 06:25

for muvsfunc, you specify the plane= . You might be able to do all 3 with plane=[0,1,2]

Zoptilib and muvsfunc do not unfortunately, u have to do all three planes individually

Also I did 8bit/10bit 4:2:0,4:2:2,4:4:4 test on another source at CRF21 and the 4:2:2 was significantly larger , so that's another data point . Still not sure if it means anything, but still interesting

as of right now, I would assume this has to do with the way the encoder app is handling things...

WorBry

3rd April 2019, 06:33

For moderate to high bitrates , in scenarios where you have adequate bandwidth to attempt to preserve details, I would disable SAO for x265 . It tends to be a detail killer, acting like a smoothing filter. I always do this for my own usage.

Thanks, I'll look at that.

for muvsfunc, you specify the plane= . You might be able to do all 3 with plane=[0,1,2]

Not sure. The function definition only gives:

plane: (int) Specify which plane to be processed. Default is None.

And looking at the function code, I can't see provision for other plane options. I got the impression that it's hard coded for luma only. Same goes for GMSD.

Not sure about netflix libvmaf SSIM, but the earlier Netflix comments say color isn't looked at for VMAF. And you would expect the log should show U, V (or Cb,Cr) channel info - but it doesn't

Simplest way to test is to convert the source and test clips to greyscale and see if it changes the scores. I did that in the Crowd Run 10bit 422 tests with ProRes and Cineform and the muvsfunc SSIM and GMSD scores were not affected at all. Didn't test VMAF on the greyscale clips in that instance, but I'm pretty sure all the component metrics are luma plane only.

ffmpeg ssim log will print out SSIM-U, SSIM-V, SSIM-Y , so you know all 3 channels are measured

Right. Zorr checked out the code for ffmpeg SSIM (AVISynth SSIM also) earlier in the thread. In essence, it's akin to the 'fast' SSIM metric in the MSU Quality Tool:

https://forum.doom9.org/showthread.php?p=1866030#post1866030

https://forum.doom9.org/showthread.php?p=1866162#post1866162

Couple of examples where I plotted out the Y,U,V channel scores.

http://i.imgur.com/ICbJOIMm.png (https://imgur.com/ICbJOIM)

http://i.imgur.com/WZ1HkDLm.png (https://imgur.com/WZ1HkDL)

WorBry

3rd April 2019, 06:36

Zoptilib and muvsfunc do not unfortunately, u have to do all three planes individually

How have you tested that ???

WorBry

3rd April 2019, 06:46

source/ref resolution and encoding resolution was 1920x1080p.

Model was 0.6.1 as stated on the pics - I used the exact same model in both VS|ffmpeg tests, meaning: the exact same .pkl file.

OK thanks.

Iron_Mike

3rd April 2019, 09:14

And looking at the function code, I can't see provision for other plane options. I got the impression that it's hard coded for luma only. Same goes for GMSD.

yes, it does default to plane=0 (Y), if the plane param is None, meaning: not passed in

How have you tested that ???

looked at the code in muvsfunc, the plane param needs to be of type int...

and u can also simply test this by running these params when passing config to Zopti:

'ssim': {'downsample': False, 'show_map': False, 'plane': (0, 1, 2)}

it will throw error from muvsfunc that plane param needs to be int... setting plane=2 (for example) will measure the 'v' plane...

'ssim': {'downsample': False, 'show_map': False, 'plane': 2}

so currently, u need to run all three planes individually...

WorBry

3rd April 2019, 15:03

Oh I see now. It uses the GetPlane (ShufflePlanes based) helper function from mvsfunc to extract and convert the selected plane to greyscale (Y Luma) for processing.

I'd be more interested in deriving an aggregate value for the UV chroma planes than all three planes.

Actually, in the MDSI function there is provision to adjust the weight given to the pooled gradient and chromacity similarity maps:

alpha: (float, 0~1) Weight used to merge gradient similarity (GS) map and chromaticity similarity (CS) map.

Default is 0.6.

I'll look at that also.

WorBry

3rd April 2019, 16:03

I'm pretty sure all the component metrics are luma plane only.

https://forum.doom9.org/showthread.php?p=1870779#post1870779

....Note that libvmaf only uses luma plane for calculating scores.

WorBry

4th April 2019, 02:12

Oh I see now. It uses the GetPlane helper function from mvsfunc to extract and convert the chosen plane to greyscale (Y Luma) for processing.

Tested it out using the x264 CRF1 transcode of the Sony 2160/30p XAVC clip. Appears to work as expected. Plane=1 and Plane=2 produce different SSIM and GMSD scores to default plane=0 (None). Also, when the reference and test clips are converted to greyscale, the SSIM and GMSD scores are absolute (1 and 0) with Plane=1 and Plane=2, as you would expect.

In the MDSI function there is provision to adjust the weight given to the pooled gradient and chromacity similarity maps:

alpha: (float, 0~1) Weight used to merge gradient similarity (GS) map and chromaticity similarity (CS) map.

Default is 0.6.

I'll look at that also.

I was rather hoping that setting alpha=0 would exclude the gradient (structural/luminance) similarity component and setting alpha=1 would exclude the chromacity similarity component, but that does not appear to be the case. Based on the outcomes observed with greyscale clips, it does shift the bias to some degree, but it's not absolute.

WorBry

4th April 2019, 16:34

Incidentally, came across this study:

https://www.intechopen.com/books/proceedings-of-the-3rd-czech-china-scientific-conference-2017/influence-of-chroma-subsampling-on-objective-video-quality-assessment-for-high-resolutions

They used the MSU Quality Measurement Tool for the SSIM and PSNR metrics.

WorBry

4th April 2019, 18:48

Oh I see now. It uses the GetPlane (ShufflePlanes based) helper function from mvsfunc to extract and convert the selected plane to greyscale (Y Luma) for processing.

Tested it out using the x264 CRF1 transcode of the Sony 2160/30p XAVC clip. Appears to work as expected. Plane=1 and Plane=2 produce different SSIM and GMSD scores to default plane=0 (None). Also, when the reference and test clips are converted to greyscale, the SSIM and GMSD scores are absolute (1 and 0) with Plane=1 and Plane=2, as you would expect.

I also checked whether setting the selected plane internally gives the same score as converting the plane to Gray8 externally with ShufflePlanes, e.g.

clip = core.std.ShufflePlanes(clips=clip, planes=1, colorfamily=vs.GRAY)

And it does - the scores are identical. For the x264 CRF1 clip (256 frames):

SSIM (no downsample):

Plane=0; 255.9883826437114205942791044
Plane=1; 255.9511353443287036135700421
Plane=2; 255.9553550588348764804891293

GMSD (no downsample):

Plane=0 1.342334158110490287003219167
Plane=1 2.221467708712367798208099638
Plane=2 2.180245724244588914348796231

However, it needs to be borne in mind the luma-converted U (Plane 1) and V (Plane 2) planes are being processed at the chroma resolution. In this case, the 4:2:0 chroma > luma was processed at 1920 x 1080 resolution. For 4:2:2, it would be 1920 x2160 and for 4:4:4, 3840 x 2160.

So expect that chroma resolution of the reference and test clip will affect the scores.

In the FFMPEG SSIM metric:

The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight.

https://forum.doom9.org/showthread.php?p=1866162#post1866162

Iron_Mike

4th April 2019, 23:41

However, it needs to be borne in mind the luma-converted U (Plane 1) and V (Plane 2) planes are being processed at the chroma resolution. In this case, the 4:2:0 chroma > luma was processed at 1920 x 1080 resolution. For 4:2:2, it would be 1920 x2160 and for 4:4:4, 3840 x 2160.

is the original src clip UHD format or why do u end up @ 3840x2160 in 444 for the chroma planes ?

WorBry

4th April 2019, 23:56

is the original src clip UHD format or why do u end up @ 3840x2160 in 444 for the chroma planes ?

Yes, the original source clip and x264 transcode are UHD resolution.

Tested it out using the x264 CRF1 transcode of the Sony 2160/30p XAVC clip.

WorBry

7th April 2019, 16:03

Quick question - how do you convert a full color RGB or YUV clip (whether 4:4:4, 4:2:2 or 420) to greyscale RGB in VapourSynth ?
Couldn't figure it out when I ran those parallel greyscale MDSI tests in the CrowdRun > Prores series and ended up using the 'Greyscale' filter in AVISynth+ and exporting the output to MagicYUV.

Must surely be possible with fmtconv/shuffleplanes, but I just can't figure it.

ChaosKing

7th April 2019, 16:44

Maybe like this core.std.ShufflePlanes(clip, planes=[0,0,0], colorfamily=vs.RGB)

EDIT
import mvsfunc as mvf
clip = mvf.GrayScale(clip)

WorBry

7th April 2019, 16:50

Thank-you. :)

Edit:
Actually, core.std.ShufflePlanes(clip, planes=[0,0,0], colorfamily=vs.RGB) works fine, but mvf.GrayScale throws the error - Error on frame 0 request: Resize error 1026: RGB color family cannot have YUV matrix coefficients - when trying to convert a full color YUV source to greyscale RGB with:

clip = core.fmtc.resample (clip, css="444")
clip = core.fmtc.matrix (clip, mat="709", col_fam=vs.RGB)
clip = core.fmtc.bitdepth (clip, bits=8)
clip = mvf.GrayScale(clip)

or

clip = mvf.ToRGB(clip, depth=8)
clip = mvf.GrayScale(clip)

WorBry

8th April 2019, 03:19

For moderate to high bitrates , in scenarios where you have adequate bandwidth to attempt to preserve details, I would disable SAO for x265 . It tends to be a detail killer, acting like a smoothing filter. I always do this for my own usage.

Thanks, I'll look at that.

I ran a parallel series of x265 encodes (CRF 1 -5 ) of the Sony 2160/30p XAVC clip, with SAO disabled:

ffmpeg -i {Path}:/AX100.mp4 -vcodec libx265 -preset slow -crf {Value} -pix_fmt yuv420p -r 30000/1001 -x265-params no-sao=1:colorprim=1:transfer=1:colormatrix=1 {Path}:/AX100_x265_No_SAO_CRFx.mp4

It had negligible effect on the encode bitrates and SSIM and GMSD scores in this case:

http://i.imgur.com/JGqJcsxm.png (https://imgur.com/JGqJcsx)

WorBry

15th April 2019, 18:56

I also checked whether setting the selected plane internally gives the same score as converting the plane to Gray8 externally with ShufflePlanes, e.g.

clip = core.std.ShufflePlanes(clips=clip, planes=1, colorfamily=vs.GRAY)

And it does - the scores are identical. For the x264 CRF1 clip (256 frames):

SSIM (no downsample):

Plane=0; 255.9883826437114205942791044
Plane=1; 255.9511353443287036135700421
Plane=2; 255.9553550588348764804891293

GMSD (no downsample):

Plane=0 1.342334158110490287003219167
Plane=1 2.221467708712367798208099638
Plane=2 2.180245724244588914348796231

However, it needs to be borne in mind the luma-converted U (Plane 1) and V (Plane 2) planes are being processed at the chroma resolution. In this case, the 4:2:0 chroma > luma was processed at 1920 x 1080 resolution. For 4:2:2, it would be 1920 x2160 and for 4:4:4, 3840 x 2160.

So expect that chroma resolution of the reference and test clip will affect the scores.

In the FFMPEG SSIM metric:

https://forum.doom9.org/showthread.php?p=1866162#post1866162

In light of the discussion that ensued in the Zopti thread...

https://forum.doom9.org/showthread.php?p=1870880#post1870880...

...I thought it would be interesting to see how the raw (muvsfunc) SSIM scores for the chroma planes compare with those generated by ffmpeg SSIM.

For that I turned to the Crowd Run 1080 50p series of x264 (CRF 0 -30) encodes that I retained from the earlier metric tests; I had already collated the ffmpeg SSIM Y, U and V results.

I also wanted look at the outcomes when the 8bit 4:2:0 chroma (of both the test and reference clips) is up-sampled to (YUV) 444 before testing, and whether converting the original chroma planes to Gray8 and then up-scaling to 1920 x 1080 produces similar results, which in theory it should.

I used Resize.Bicubic for both the chroma up-sampling and Gray8 up-scaling. Downsample=False was applied in the chroma plane tests. Here are the results:

http://i.imgur.com/SZ5NSCTm.png (https://imgur.com/SZ5NSCT)

The 'raw' muvsfunc SSIM scores obtained for the Y, U and V planes (top right chart) show a similar pattern to those produced by ffmpeg SSIM, but they are lower and propotionately more so in the lower bitrate range.

Looking at the results for the individual planes.

In the Y plane results I also included the scores obtained previously with 'Down-sample=True' i.e. the default settings. As seen in all of the test series, the initial 2x2 down-sampling always produces higher scores, and in this case higher than ffmpeg SSIM - I don't think the ffmpeg SSIM metric applies any internal down-sampling.

As for the U and V planes. First thing to note is that the upsampled (444) and upscaled (Gray8) scores are indeed very close. Secondly, the 'upsampled/upscaled' scores are higher than the 'raw' scores and also the ffmpeg SSIM scores. Which leaves me wondering....

I looked at the ffmpeg source. It's doing the fast version too - no gaussian kernels there. The total SSIM takes into account all the planes but the weighting is different, each plane is scaled by the resolution is has. So for example with YUV420 the color planes have 4 times smaller weight. .

Does that mean that the reported (aggregate) scores for the individual Y, U and V planes are also weighted (i.e. scaled by the resolution each has) or is that weighting only being applied in calculating a total ('All') score ? Or is it simply that the muvsfunc SSIM is more accurate than the 'fast' ffmpeg implementation ?

Thinking about Poisondeathray's comments in the Zopti thread:

The idea of "weights" for a combined Y/U/V aggregate metric score, should reflect that the Y plane should recieve a proportionally higher weighting due to human perceptions - I think everyone will agree on the general idea, but might disagree on the actual formula for the weighting

All I'm saying is the relative weighting shouldn't change because of subsampling . Your perception of the proportion of black/white vs. color importance doesn't suddenly change if you watch a 4:4:4 video or a 4:2:0 video.

That lower quality of 4:2:0 should already be reflected in the lower U, V scores for 4:2:0 . That up/down converison is the "penalty" already incurred . The "different treatment" of chroma is exactly what you're measuring in the first place when you measure U-SSIM, V-SSIM, or U-PSNR, V-PSNR or whatever metric

Yes you can have other weights, other categories, combine /mix/match in any way you want , analyze it in whatever way you want, call it whatever you want ; but either way you're measuring Y,U,V separately - so you should have the "raw" scores

I'm inclined to agree that, for the purpose of objective metric analysis at least, you want to have the raw scores for the individual Y, U and V planes. Personally, I don't see that much value in composite 'Total' score, however derived. If there are differences, you want to know if they occurring in the Luma (structural distortions) or the Chroma. A weighted 'Total' score might also obscure subtle differences that are occurring in the chroma. You can see that in the above ffmpeg SSIM results, at the lower bitrates especially; the weighted 'All' score curve shows a greater bias to the 'Y' plane curve, as you would expect.

That said, when it comes to 'perceptual quality' is there maybe a case for 'normalizing' the chroma to 444 ? After all, that is what we are looking at when viewing a video on a display. Is it more meaningful to 'normalize' the chroma before the metric is applied than to apply some compensatory weighting to the raw measures derived from the native chroma ? I'm not necessarily saying that's how it should be done - just some food for thought.

I also ran simultaneous GMSD tests but have yet to collate the data.