VMAF - Video Multi-Method Assessment Fusion [Archive]

HolyWu

19th January 2019, 15:42

Can you add an "average score" for each metric at the end of the log?

It's already there, albeit not at the end. For example a xml log:

<?xml version="1.0" encoding="ISO-8859-1"?>
<VMAF version="1.3.7">
<params subsample="1" scaledHeight="1080" scaledWidth="1920" model=""/>
<fyi execFps="2.3972" aggregateMS_SSIM="0.997629" aggregateSSIM="0.998861" aggregatePSNR="48.5527" aggregateVMAF="98.452" numOfFrames="6"/>
<frames>
<frame vmaf="95.1147" vif_scale3="0.999067" vif_scale2="0.997998" vif_scale1="0.986572" vif_scale0="0.711273" ssim="0.998833" psnr="48.1752" ms_ssim="0.997502" motion2="0" adm2="0.990135" frameNum="0"/>
<frame vmaf="98.0908" vif_scale3="0.999122" vif_scale2="0.998097" vif_scale1="0.987361" vif_scale0="0.725121" ssim="0.998876" psnr="48.5654" ms_ssim="0.997635" motion2="2.08641" adm2="0.991341" frameNum="1"/>
<frame vmaf="98.6133" vif_scale3="0.999151" vif_scale2="0.998111" vif_scale1="0.987686" vif_scale0="0.737886" ssim="0.998892" psnr="48.9502" ms_ssim="0.997769" motion2="2.4176" adm2="0.991782" frameNum="2"/>
<frame vmaf="99.0617" vif_scale3="0.999089" vif_scale2="0.998038" vif_scale1="0.98654" vif_scale0="0.71651" ssim="0.99883" psnr="48.3169" ms_ssim="0.997532" motion2="2.96657" adm2="0.990675" frameNum="3"/>
<frame vmaf="100" vif_scale3="0.999108" vif_scale2="0.998094" vif_scale1="0.987838" vif_scale0="0.738668" ssim="0.998885" psnr="48.9485" ms_ssim="0.997766" motion2="3.60763" adm2="0.991847" frameNum="4"/>
<frame vmaf="100" vif_scale3="0.999115" vif_scale2="0.99804" vif_scale1="0.986751" vif_scale0="0.720485" ssim="0.99885" psnr="48.371" ms_ssim="0.997572" motion2="4.2859" adm2="0.991102" frameNum="5"/>
</frames>
</VMAF>

lansing

19th January 2019, 16:04

Oh, I was using the json format and it doesn't have it

lansing

20th January 2019, 05:50

So I finally have time to play with this plugin, it's very good tool to help find optimal encode settings for my contents.

My test clip is a 1000 frames of 1440x1080 anime with no grain and very little motion. Here are the comparison between x264 and x265 I made with vmaf :

Aggregate VMAF Percentage
source 98.8374

x264 300 kb/s medium animation 79.9185
x265 300 kb/s medium 85.6454
x265 300 kb/s slower 86.9443

x264 2500 kb/s medium animation 95.1616
x265 2500 kb/s medium 94.9675

x264 CRF 25 medium animation 91.3914
x265 CRF 25 medium 91.1363

x264 CRF 18 medium animation 96.1513
x265 CRF 21 medium 93.6972

x264 CRF 18 medium animation 96.1513
x264 CRF 18 slower animation 96.3279

According to the FAQ, comparing the source clip to itself won't gives 100% score, so 98.8% here is the highest quality for this clip.

For encoding in low bitrate, x265 clearly wins. But for high/transparent encoding, it seems that x264 is still better if encode in same bitrate or same CRF. I don't have good 4k contents to see how that goes.

I have heard people said that x264 crf 18 is equivalent to x265 crf 21 so I did a comparison on those too, and the score shows that there is a difference.

On the last comparison I did it between two different presets. So with high bitrate encoding for my content, using slower preset is just a waste of time with insignificant amount of gains. It only make sense to use it on low bitrate encodes.

Boulder

20th January 2019, 21:23

gonca

20th January 2019, 22:48

Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself.

I think it is the screen, otherwise the --phone-model switch makes no sense

Boulder

21st January 2019, 04:46

I think it is the screen, otherwise the --phone-model switch makes no sense

Yes, looking at the models page suggests that.. Looking at the linked slideshow there, it also seems that for proper analysis, you have to upscale to 4K.

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

HolyWu

21st January 2019, 06:45

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

Can't reproduce. Provide the exact script you used.

Boulder

21st January 2019, 17:21

Here's the one I used:
import vapoursynth as vs

core = vs.get_core()

orig = core.dgdecodenv.DGSource(r'O:\Testclips\test2.dgi', fulldepth=True)
orig = core.f3kdb.Deband(orig, preset="medium", output_depth=10)

clp = core.ffms2.Source(source=r'c:\x265\aq\aqmode3.hevc')
orig = core.resize.Bicubic(orig, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)
clp = core.resize.Bicubic(clp, width=3840, height=2160, filter_param_a=0, filter_param_b=0.5)

result = core.vmaf.VMAF(orig, clp, model=1, log_path="c:\x265\aq\clp1.log", log_fmt=0, pool=1, ci=True)

result.set_output()
I also noticed that there's no logfile written if I run that one through "vspipe script.vpy ."

Boulder

21st January 2019, 21:11

The process also hangs without returning to the command prompt. The values are output but I need to use CTRL+C to get the prompt back.

C:\>vspipe c:\x265\aq\compare.vpy .
Start calculating VMAF score...
Exec FPS: 0.501642
VMAF score (harmonic_mean) = 98.925030

The Exec FPS must be a wrong value, the clips are 1385 frames long and it only took a couple of seconds to produce that output. If I ran with the -p option, I got those out of memory errors quite soon.

lansing

21st January 2019, 21:34

C:\>vspipe c:\x265\aq\compare.vpy .
Start calculating VMAF score...
Exec FPS: 0.501642
VMAF score (harmonic_mean) = 98.925030

The Exec FPS must be a wrong value, the clips are 1385 frames long and it only took a couple of seconds to produce that output.

Have you try running it with vs editor's benchmark? My 1000 frames 1080p anime took about a minute and 20 seconds, yours should be doubling or tripling my time comparing 4k clips.

Boulder

21st January 2019, 21:40

Just tried - it crashed by vanishing without a warning almost right after it started. I raised the cache size to 2048 MB and it hung at frame 30 which it probably what happens with vspipe as well.

lansing

21st January 2019, 23:03

Just tried - it crashed by vanishing without a warning almost right after it started. I raised the cache size to 2048 MB and it hung at frame 30 which it probably what happens with vspipe as well.

I tried your script, mine stop moving about frame 14 too. I try lowering the resize resolution to 3000x2000, it was able to run with RAM usage jumping between 3.8G to 4.8G. So it should be an insufficient memory problem?

ChaosKing

22nd January 2019, 02:23

No problems here. Max ram usage was 10gb.
Maybe it is also a path problem? I used log_path=r"D:\test\clp1.log"
The x265 folder exists on c? Bcs if not vapoursynth can't create one without admin rights.

gonca

22nd January 2019, 02:52

Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself.

Could someone expand on this.
Is it the device or video?
Does the video have to be resized?

HolyWu

22nd January 2019, 03:09

I also can't reproduce the issues you two encountered with a 2000 frames 1080p upscaled to 4K, no matter using vspipe or vsedit. And not prefixing the string containing a single backslash with r is always a bad idea.

HolyWu

22nd January 2019, 03:44

Could someone expand on this.
Is it the device or video?
Does the video have to be resized?

It's the viewing device. See https://github.com/Netflix/vmaf/blob/master/FAQ.md#q-when-computing-vmaf-on-low-resolution-videos-480-height-for-example-why-the-scores-look-so-high-even-when-there-are-visible-artifacts.

lansing

22nd January 2019, 03:59

Boulder

22nd January 2019, 04:50

It's the viewing device. See https://github.com/Netflix/vmaf/blob/master/FAQ.md#q-when-computing-vmaf-on-low-resolution-videos-480-height-for-example-why-the-scores-look-so-high-even-when-there-are-visible-artifacts.

And this seems to confirm that upscaling to that resolution is the way to go, although getting a VMAF score from an originally 1080p source on a 4K display is not optimal.

If, say, for a distorted video of 480 resolution, we still want to predict its quality viewing from 3 times the height (not 6.75), how can this be achieved?

If the 480 distorted video comes with a source (reference) video of 1080 resolution, then the right way to do it is to upsample the 480 video to 1080, and calculate the VMAF at 1080, together with its 1080 source.

If the 480 distorted video has only a 480 reference, then you can still upsample both distorted/reference to 1080, and calculate VMAF. A caveat is, since the VMAF model was not trained with upsampled references,
the prediction would not be as accurate as 1).

Boulder

22nd January 2019, 04:56

I did another test, my PC has 16G RAM and I start the benchmark with all my apps closed dropping total ram usage to 3.5G. And now the benchmark ran without chocking with RAM usage for the process peaks at 6G rarely.

And then I did another benchmark with starting ram usage at 8.4G, this time the process stopped after the first few frames with its ram usage start from 200MB -> 3GB -> 500MB.

So this seems like a not-enough-memory issue, but with 16G, I should still have 1G excessive memory to run the second benchmark.

When I first tested things, I didn't have any other big processes running and I had no cache size restrictions in the script. The memory usage jumped to around 8GB and still got the out of memory error. Task Manager showed that the total memory usage was around 65-70% when it happened, so there was still more memory available but maybe Vapoursynth restricted it.

It looks like I cannot run an encode simultaneously because it seems end abruptly quite often when the other VS process grabs hold of most of the memory on the computer. I'm getting 8GB more this week which is good for testing this issue.

gonca

22nd January 2019, 23:38

Doing a test run
4K display, 4K sample
Using VDub2 64 bit for playback of script
So far it is slow and CPU is at near 100%
RAM for VDub2 is +/- 12 GB, but overall I see spikes to near 16GB overall, VDub2 has spiked to about 14GB on some occasions.
When this run is over I will try VSEditor

gonca

23rd January 2019, 00:17

VDub2 seems to run fine
Tried VSEditor and got the out of memory error early
Added one line and it seems to go through fine
Numbers (RAM) are about the same as VDub2

Original script

import vapoursynth as vs
core = vs.get_core()
core.std.LoadPlugin("C:/Program Files (Portable)/dgdecnv/x64 Binaries/DGDecodeNV.dll")
core.std.LoadPlugin(r'C:\Users\LUIS\Desktop\VMAF-r2\plugins64\VMAF.dll')
orig = core.dgdecodenv.DGSource(r'I:\original.dgi', fieldop=0, fulldepth=True)
orig = core.resize.Point(orig, format=vs.YUV420P10)
dist = core.dgdecodenv.DGSource(r'I:\distorted.dgi', fieldop=0, fulldepth=True)
dist = core.resize.Point(dist, format=vs.YUV420P10)
result = core.vmaf.VMAF(orig, dist, model=1, log_path=r'I:\log.log', log_fmt=1, pool=1, ci=True)
result.set_output ()

Modified script

import vapoursynth as vs
core = vs.get_core()
core.max_cache_size = 16384
core.std.LoadPlugin("C:/Program Files (Portable)/dgdecnv/x64 Binaries/DGDecodeNV.dll")
core.std.LoadPlugin(r'C:\Users\LUIS\Desktop\VMAF-r2\plugins64\VMAF.dll')
orig = core.dgdecodenv.DGSource(r'I:\original.dgi', fieldop=0, fulldepth=True)
orig = core.resize.Point(orig, format=vs.YUV420P10)
dist = core.dgdecodenv.DGSource(r'I:\distorted.dgi', fieldop=0, fulldepth=True)
dist = core.resize.Point(dist, format=vs.YUV420P10)
result = core.vmaf.VMAF(orig, dist, model=1, log_path=r'I:\log.log', log_fmt=1, pool=1, ci=True)
result.set_output ()

Seems minimum amount of RAM for 4K is 16GB + all background processes

lansing

25th January 2019, 04:53

HolyWu

25th January 2019, 06:47

I'm wondering, right now with my machine I'm doing 12 fps on 1440x1080 clips with 3G ram usage. If I'm to upgrade to a faster machine that do 3 times this speed, will my ram usage also go up by 3 times?

I guess the ram usage should only be relevant to the video resolution, bit depth, and number of threads.

lansing

25th January 2019, 21:37

I did another test on this, I disabled the core in task manager and ran benchmark on 1 to 8 threads, speed rises as more threads were used, but the ram usage are the same on the all of them.

A side note, setting the thread number in the script doesn't work
core = vs.get_core(threads=1)

HolyWu

26th January 2019, 03:08

A side note, setting the thread number in the script doesn't work
core = vs.get_core(threads=1)

Ah, libvmaf uses pthreads for its internal multithreading, and I didn't set it according to core's number of threads but leave it at its default which uses all threads available. I'll fix it at the next release.

HolyWu

1st February 2019, 06:01

Update r3.

Update libvmaf to v1.3.13, which includes performance improvement.
Change default pool to 1.
Set libvmaf's threads according to core's number of worker threads.

lansing

1st February 2019, 19:28

I couldn't update it through vsrepo?

WorBry

1st February 2019, 22:25

I've been testing the Vapour Synth VMAF (r3) plugin with a high quality 1080/50p source (CrowdRun, lossless x264 8bit 420 Intra) encoded to x264 over a range of CRF values.

When testing the source file against self (as a control), which should be lossless, I was surprised to find that the VMAF score is not 100.

http://i.imgur.com/DldW0wj.png (https://imgur.com/DldW0wj)

Is this normal ?

Script:

import vapoursynth as vs
core = vs.get_core()
clip = core.ffms2.Source(source=r'X:/CrowdRun_x264_lossless.mp4')
result = core.vmaf.VMAF(clip, clip, ssim=True, ms_ssim=True, psnr=True, model=0, log_path=r'X:/VMAF_r3.log' )
result.set_output()

Also the log reports VMAF version="1.3.11", not 1.3.13

Update r3.

[LIST]
Update libvmaf to v1.3.13, which includes performance improvement.

ChaosKing

1st February 2019, 22:29

https://github.com/Netflix/vmaf/blob/master/FAQ.md#q-when-i-compare-a-video-with-itself-as-reference-i-expect-to-get-a-perfect-score-of-vmaf-100-but-what-i-see-is-a-score-like-987-is-there-a-bug

A: VMAF does not guarantee that you get a perfect score in this case, but you should get a score close enough. Similar things would happen to other machine learning-based predictors (another example is VQM-VFD).

WorBry

1st February 2019, 22:32

Thanks.

WorBry

2nd February 2019, 20:16

When testing the source file against self (as a control), which should be lossless, I was surprised to find that the VMAF score is not 100.

http://i.imgur.com/DldW0wj.png (https://imgur.com/DldW0wj)

Is this normal ?

https://github.com/Netflix/vmaf/blob/master/FAQ.md#q-when-i-compare-a-video-with-itself-as-reference-i-expect-to-get-a-perfect-score-of-vmaf-100-but-what-i-see-is-a-score-like-987-is-there-a-bug

Actually, looking at the per-frame scores in that same log, it is just the VMAF score for the first frame that skews the aggregate result, and it looks like it's the motion2 metric (which measures temporal difference) score of 0 that is responsible for that. All of the remaining 499 frames have a VMAF score of 100.

http://i.imgur.com/H17lsxVh.png (https://imgur.com/H17lsxV)

To enlarge open image and click (+) cursor.

Perhaps there should be an option to exclude the first frame from the aggregate scores?

WorBry

5th February 2019, 05:46

I've been testing the Vapour Synth VMAF (r3) plugin with a high quality 1080/50p source (CrowdRun, lossless x264 8bit 420 Intra) encoded to x264 over a range of CRF values.

Interesting results....having not tested VMAF before.

Here I encoded the CrowdRun 1080/50p 'master' to x264 over CRF 0 - 30. This was using the default vmaf_v0.6.1.pkl model (i.e. Predict Quality on a 1080p HDTV screen at distance 3x the screen height). The VMAF, SSIM and MS-SSIM scores are the aggregate values. The 'classic' SSIM tests were run on Zeranoe ffmpeg win64-static nightly build (20190131).

http://i.imgur.com/SumN4ba.png (https://imgur.com/SumN4ba)

http://i.imgur.com/z3s3fgG.png (https://imgur.com/z3s3fgG)

Big difference in the libvmaf SSIM and ffmpeg SSIM scores. Apparently, the vmaf SSIM implementation "includes an empirical downsampling process, as described at the Suggested Usage section of https://ece.uwaterloo.ca/~z70wang/research/ssim/", whereas the FFMPEG implementation does not have this step:

https://github.com/Netflix/vmaf/issues/22

As for the VMAF metric itself; well, I can appreciate it's value in context of 'perceptual quality'. In this example it effectively declares the x264 transcodes to be visually lossless from CRF 0 to around CRF 16, whereas the ffmpeg-SSIM scores show a progressive decline over the entire CRF/bitrate range.

And here I ran a parallel series encoded to x265 for comparison.

http://i.imgur.com/LfAjzft.png (https://imgur.com/LfAjzft)

Clearly VMAF judges x265 to have significantly higher perceptual quality than x264 at the lower bitrate range and more so than revealed by SSIM.

That said, I think 'classic' (ffmpeg) SSIM is still a useful tool for analyzing fine differences at the pixel peeping level and beyond visual acuity, and (by virtue of the differential Y, U and V scores) for determining whether the luma and/or chroma are affected.

I did record the libvmaf and ffmpeg PSNR scores also, but they are not as interesting.

@HolyWu, btw, thanks for the plugin.

Boulder

5th February 2019, 07:25

Has anyone else noticed that the VMAF scores in some cases tend to be "too perfect" to measure?

https://forum.doom9.org/showthread.php?p=1864721#post1864721

lansing

5th February 2019, 23:09

Clearly VMAF judges x265 to have significantly higher perceptual quality than x264 at the lower bitrate range and more so than revealed by SSIM.

Good comparison to show that x265 really has no advantage over x264 on 1080p materials if we're going for transparent encoding.

Now we'll just have to wait for people with high end computer to do the 4K comparison.

WorBry

6th February 2019, 00:41

I started off testing at original (Crowd Run) 2160/50p resolution but could see I would be in for a long haul ;)

ChaosKing

6th February 2019, 13:58

I couldn't update it through vsrepo?

Updates via vsrepo will never be available immediately. In addition, the new version was not recognized by the update script so it had to be done by hand. After what Myrsloik need to upload the new compiled repo file to his site. There are many steps as you can see.

But it's available now :)

WorBry

7th February 2019, 07:52

Actually, looking at the per-frame scores in that same log, it is just the VMAF score for the first frame that skews the aggregate result, and it looks like it's the motion2 metric (which measures temporal difference) score of 0 that is responsible for that. All of the remaining 499 frames have a VMAF score of 100.

http://i.imgur.com/H17lsxVh.png (https://imgur.com/H17lsxV)

To enlarge open image and click (+) cursor.

Perhaps there should be an option to exclude the first frame from the aggregate scores?

There again, that's not always the case. Here, the original 2160/50p Crowd Run (8bit 420, y4m) reference clip encoded to x264 CRF=0 (i.e. lossless with switch to qp 0 and High 444 Predictive profile), and the clips compared with VMAF v3 in Model=1 mode:

<VMAF version="1.3.11">
<params model="" scaledWidth="3840" scaledHeight="2160" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="500" aggregateVMAF="100" aggregatePSNR="60" aggregateSSIM="1" aggregateMS_SSIM="1"......
....
<frame frameNum="0" adm2="1" motion2="0" ms_ssim="1" psnr="60" ssim="1" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999999" vif_scale3="0.999998" vmaf="100" />
<frame frameNum="1" adm2="1" motion2="8.42311" ms_ssim="1" psnr="60" ssim="1" vif_scale0="1" vif_scale1="0.999999" vif_scale2="0.999999" vif_scale3="0.999998" vmaf="100" />

So I guess you have to let it do it's thing and take the scores as they come.

WorBry

11th February 2019, 05:20

Good comparison to show that x265 really has no advantage over x264 on 1080p materials if we're going for transparent encoding.

Now we'll just have to wait for people with high end computer to do the 4K comparison.

Done. So I ran a parallel test series using the original 2160/50p Crowd Run clip (8bit 420 y4m) as source and reference for the metric tests. For VMAF tests I used the VapourSynth (v3) plugin with the vmaf_4k_v0.6.1.pk model (model=1) which aims to "predict the subjective quality of video displayed on a 4KTV and viewed from the distance of 1.5 times the height of the display device (1.5H)"

The x264 results:

http://i.imgur.com/ajUApds.png (https://imgur.com/ajUApds)

http://i.imgur.com/QLT0D6W.png (https://imgur.com/QLT0D6W)

Interesting that the shape of ffmpeg-SSIM vs bitrate plot is quite different to that in the 1080/50p series and the differential between the VMAF and ffmpeg-SSIM scores is larger. The x265 encodes show the same behaviour:

http://i.imgur.com/On2NNyf.png (https://imgur.com/On2NNyf)

Again the VMAF scores deem that x265 has higher perceptual quality over the lower bitrate range.

As to whether there is an advantage over x264 for 'transparent' encoding; well, I looked more closely at what point at which the VMAF plots hit the maximum score of 100.

http://i.imgur.com/qHeRVnr.png (https://imgur.com/qHeRVnr)

For x264 it was at CRF=8 (1296 Mbps) and for x265 at CRF=10 (976 Mbps). So on that basis it could be concluded that x265 is significantly more efficient. That said, if you look at the per-frame VMAF scores, it is clear that the first frame skews the outcome somewhat.

Taking the x264 series first; going from CRF=9 to 16, all frames bar the first frame in each test scored VMAF=100. And in the x265 series also, going from CRF 11 to 17 only the first frame scored less than VMAF=100:

http://i.imgur.com/sVGXdAC.png (https://imgur.com/sVGXdAC)

So, if the aggregate VMAF scores are calculated with the first frame excluded (simple average across the remaining 499 frames), CRF=16 (392 Mbps) becomes the point at which VMAF=100 is reached in the x264 series, and CRF=17 (306 Mbps) in the x265 series:

http://i.imgur.com/s19XMYe.png (https://imgur.com/s19XMYe)

Makes quite a difference. x265 still has the edge on bit savings, but not by as much. I don't have time to calculate 'adjusted' aggregate VMAF scores for the other (lower bitrate) CRF data points. In the 1080/50p series an aggregate VMAF=100 score was never attained for precisely the same reason - the VMAF score of the first frame skewed the aggregate score.

Here are the ffmpeg-SSIM scores obtained in the 2160/50p series at these 'significant' CRF points though:

http://i.imgur.com/0vb8SAu.png (https://imgur.com/0vb8SAu)

Edit:

Interesting that the shape of ffmpeg-SSIM vs bitrate plot is quite different to that in the 1080/50p series and the differential between the VMAF and ffmpeg-SSIM scores is larger...

I'll maybe see how the 2160/50p and 1080/50p series compare when plotted against bits/pixel.

lansing

11th February 2019, 08:00

x265 still has the edge on bit savings, but not by as much.

I would say there's no advantage at all. The difference is like 0.03 between the two scores at crf 17. I thought it would be like a 5 or 6 point difference...

VS_Fan

11th February 2019, 17:49

So, if the aggregate VMAF scores are calculated with the first frame excluded (simple average across the remaining 499 frames), CRF=16 (392 Mbps) becomes the point at which VMAF=100 is reached in the x264 series, and CRF=17 (306 Mbps) in the x265 series ... x265 still has the edge on bit savings, but not by as much.Considering your criteria to obtain a 'transparent' encode, with x265 you are reducing the required bitrate from 392 (with x264) to 306 Mbps; you get 28% savings in storage space and/or streaming bandwidth, that's a significant amount!

ChaosKing

11th February 2019, 18:04

You save 22% not 28% which still is very good.

WorBry

11th February 2019, 18:25

For better (more valid ?) interpretation of these fine differences at high bitrates I'm thinking now it might have been prudent to run these tests with the 'Confidence Interval' (ci) option.

That said, I see that the log generates ci95_high, ci95_low and stddev values for the individual frames but does not derive aggregate values as I was expecting - according to the VDK documentation, the command line tool reports aggregate values:

https://github.com/Netflix/vmaf/blob/master/resource/doc/conf_interval.md

So is this a limitation of the libvmaf implementation ?

WorBry

12th February 2019, 07:17

I'll maybe see how the 2160/50p and 1080/50p series compare when plotted against bits/pixel.

The VMAF results:

http://i.imgur.com/RBzTFoB.png (https://imgur.com/RBzTFoB)

Perhaps not surprising, given that the vmaf_4k_v0.6.1 model predicts the subjective quality of video displayed on a 4KTV and viewed from the distance of 1.5 times the height of the screen whereas the vmaf_v0.6.1 model predicts the subjective quality of video displayed displayed a 1080p HDTV screen at distance 3 times the screen height.

In the 1080/50p series an aggregate VMAF=100 score was never attained for precisely the same reason - the VMAF score of the first frame skewed the aggregate score.

http://i.imgur.com/syJWJ0V.png (https://imgur.com/syJWJ0V)

As seen there, the maximum VMAF score achieved in the 1080 50p series was 99.947 with the lossless (crf0) x264 encode,

What intrigues me more are the FFMPEG-SSIM results:

http://i.imgur.com/cHsJnCW.png (https://imgur.com/cHsJnCW)

It's reasonable to assume that down-scaling of the original 2160/50p Crowd Run clip for the 1080/50p tests incurred some loss of fidelity in the 1080/50p source (and reference) clip, making it more 'compressible'. But why is the differential between the bit-matched 1080p and 2160p SSIM scores so much larger at 32-64 bits/pixel than it is down at around 6-8 bits/pixel ?

HolyWu

12th February 2019, 08:26

So is this a limitation of the libvmaf implementation ?

Yes. I just send a pull request to improve that.

WorBry

12th February 2019, 17:47

Great. Thanks.

https://github.com/Netflix/vmaf/pull/304

WorBry

12th February 2019, 23:42

I did record the libvmaf and ffmpeg PSNR scores also, but they are not as interesting.

Actually they are quite interesting.

PSNR scores obtained with the VapourSynth VMAF filter:

http://i.imgur.com/sSFW1rQ.png (https://imgur.com/sSFW1rQ)

60 is the maximum score, achieved only with the lossless x264 CRF=0 encodes.

And the equivalent ffmpeg PSNR results:

http://i.imgur.com/iEq4pkP.png (https://imgur.com/iEq4pkP)

I excluded the CRF=0 encode results because ffmpeg-PSNR reports lossless as Infinity (Inf).

The libvmaf PSNR scores are in general a little lower than the ffmpeg PSNR scores but show the same overall pattern. The 1080/50p series encodes gave higher bit-matched SSIM scores than the 2160/50p series at the higher bit/pixel range but at the lower end (< 24bits/pixel), that is reversed. Interesting also that the libvmaf PSNR metric gives wider separation of the x264 and x265 scores in the higher bit/pixel domain.

Iron_Mike

1st March 2019, 05:51

Update r2.

Scale 10-bit pixel values to 8-bit range for correct score calculation.
Use stricter linear frame request since VMAF score will change if frame order is different.
Report aggregate PSNR, SSIM, and MS-SSIM scores in addition to VMAF score.

could you elaborate on the first two points of this update ?

does that mean we have to scale both inputs to 8bit before calculating VMAF, or does VMAF do it automatically ?

I'm comparing a 16bit exr sequence to itself (as control) and I get a 98.2 VMAF score... I know that the FAQ states that this is normal, just making sure I do this correctly... most of the examples here use 8bit source/ref footage...

When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...

Also, how do I use a "stricter linear frame request" ?

Thanks.

WorBry

1st March 2019, 15:19

could you elaborate on the first two points of this update ?

does that mean we have to scale both inputs to 8bit before calculating VMAF, or does VMAF do it automatically ?

.....Also, how do I use a "stricter linear frame request" ?

These are internal improvements that were made in update r2 - you don't need to do anything.

When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...

Bear in mind though that all of those tests were done on a single source. Could be any number of factors weighing in there. What scores to you get if you get if you test the ref and x265 clips against themselves ?

poisondeathray

1st March 2019, 16:43

When I compare a 12bit 444 (yuv444p12le) CRF 10 x265 encode to the 16 bit exr ref footage I get a 96.4 VMAF score... little bit low considering the tests that WorBry has done...

I'm not sure how valid that test would be. EXR is usually sRGB linear , and 16bit half float .

For any metric you usually need a common ground to compare. This means same pixel format (same colorspace, same bit depth, same chroma subsampling) . Otherwise you introduce other variables that are not controlled for. e.g. if one run uses one algorithm to scale (e.g. bicubic vs. bilinear, vs...) , or another dithers down using one algorithm, but another does not... or if you convert to RGB using different matrix, etc... there are many factors that invalidate your testing

WorBry

1st March 2019, 22:12

Not to mention the potential for frame shifts/misalignment when using different decoders for the reference and test clips, although the filter will report an error if the number of frames is different.

Also needs to be appreciated that the VMAF models are 'trained' for predicting perceptual quality at streaming bitrates primarily. Came across this quote from Netflix:

"VMAF has been trained using encodes spanning from CRF 22 @ 1080 (highest quality) to CRF 28 @ 240 (lowest quality). The former is mapped to score 100 and the latter is mapped to score 20. Anything in between is mapped in the middle (for example, SD encode at 480 is typically mapped to 40 ~ 70)."

https://streaminglearningcenter.com/codecs/finding-the-just-noticeable-difference-with-netflix-vmaf.html

I would assume a similar focus was applied in training the 4K model.

So at CRF 10 you are well into uncharted territory.

Personally, I'd be more inclined to look at other metrics available for VapourSynth that are (maybe) better attuned for VQA in the visually lossless domain - GMSD, MDSI and yes, SSIM....Butteraugli, possibly. My own journey of discovery in that vein continues:

https://forum.doom9.org/showthread.php?t=176101

Iron_Mike

1st March 2019, 22:42

Bear in mind though that all of those tests were done on a single source. Could be any number of factors weighing in there. What scores to you get if you get if you test the ref and x265 clips against themselves ?

I ran all test w/ the ffmpeg libvmaf filter, but I assume since it uses the exact same VMAF models the result would be the same...

16bit EXR ref clip is 400 frames - control test to itself via 0.6.1 results in 98.2

x265 12bit 444 encode (from EXR), control tested to itself via 0.6.1 results in 98.08

x265 12bit 444 encode (from EXR) tested against it's source (16 bit EXR) via 0.6.1 results in 96.4

interesting that the clip you tested had 399 out of 400 frames a perfect 100 in the control test...

Iron_Mike

1st March 2019, 22:51

I'm not sure how valid that test would be. EXR is usually sRGB linear , and 16bit half float .

For any metric you usually need a common ground to compare. This means same pixel format (same colorspace, same bit depth, same chroma subsampling) . Otherwise you introduce other variables that are not controlled for. e.g. if one run uses one algorithm to scale (e.g. bicubic vs. bilinear, vs...) , or another dithers down using one algorithm, but another does not... or if you convert to RGB using different matrix, etc... there are many factors that invalidate your testing

EXR contains whatever you put into it (has nothing to do with sRGB) - there are no assumptions here: this is the master of the movie in HD in lossless 16 bit EXR, I took 400 frames as a test sequence

for streaming the movie was encoded via ffmpeg and x265 from EXR (rgb48le) to x265 12bit 444 (yuv444p12le) - final result looks very good, we're using VMAF to compare various encodes (presets/CRF/etc) against each other... exact same as NF does it w/ VMAF... the master one delivers to NF is obviosly also not 8bit 420, they encode from that (high quality) master for NF streaming...

so the "common ground" you state is the same movie in the same resolution, which is the only thing that NF states in their VMAF instructions...

the whole reason for comparison is different output bit depth w/ different output chroma subsampling, on top of different encoding settings, so I do not understand your point...

Iron_Mike

1st March 2019, 22:59

Not to mention the potential for frame shifts/misalignment when using different decoders for the reference and test clips, although the filter will report an error if the number of frames is different.

not sure what "different encoders" here relates to ?

ffmpeg reads an EXR frame and then reads a frame from the mp4 encode (that was done from the EXR via ffmpeg) and then passes the decoded frames to VMAF for comparison...

is there a setting to avoid frame shifts/misalignment ?

So at CRF 10 you are well into uncharted territory.

you yourself validated the test clip up to CRF 0 and results on the charts make sense... your encodes went up to VMAF 100... not sure I understand your concern for CRF 10 ?

Thanks

WorBry

1st March 2019, 23:09

..interesting that the clip you tested had 399 out of 400 frames a perfect 100 in the control test...

For that particular source, yes...actually it was 499 out of 500 frames that scored 100 - it was just that first frame with the motion2 score of 0 that skewed the aggregate score. But I've yet to test other sources.

Go through the list of per-frame VMAF scores from your 'self' tests and you'll be able to identify which frames are skewing the aggregate score.

poisondeathray

1st March 2019, 23:29

for streaming the movie was encoded via ffmpeg and x265 from EXR (rgb48le) to x265 12bit 444 (yuv444p12le) - final result looks very good, we're using VMAF to compare various encodes (presets/CRF/etc) against each other... exact same as NF does it w/ VMAF... the master one delivers to NF is obviosly also not 8bit 420, they encode from that (high quality) master for NF streaming...

so the "common ground" you state is the same movie in the same resolution, which is the only thing that NF states in their VMAF instructions...

the whole reason for comparison is different output bit depth w/ different output chroma subsampling, on top of different encoding settings, so I do not understand your point...

You are posting in the vapoursynth vmaf thread. Only certain pixel formats are supported.

https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF/

Clips to calculate VMAF score. Only YUV420P8, YUV422P8, YUV444P8, YUV420P10, YUV422P10, and YUV444P10 are supported.

Those are the common pixel formats supported by vmaf.VMAF .

My point is strive to be more scientific. To eliminate all those confounding variables in a controlled environment. How you perform the various conversions will affect the results that are calculated.

But now it's clear you're using ffmpeg vmaf. Did you look at the ffmpeg log to see what other conversions were occurring ? There might be other stuff going on behind your back

ffmpeg reads an EXR frame and then reads a frame from the mp4 encode (that was done from the EXR via ffmpeg) and then passes the decoded frames to VMAF for comparison...

is there a setting to avoid frame shifts/misalignment ?

Sometimes ffmpeg can "mix" up frames, less often with I-frame formats. But if your x265 encode used long GOP, there is a higher chance of a mixup than if it used I-frames only. EXR sequence will be I-frame only

For the vapoursynth , the source filter can be indexed, and is more robust method for frame accuracy. For ffmpeg you can reset the PTS which might help

you yourself validated the test clip up to CRF 0 and results on the charts make sense... your encodes went up to VMAF 100... not sure I understand your concern for CRF 10 ?

Look at the results WorBy has been posting . They all plateau off below crf 18 or so. crf16 has the "same quality" as crf 10 or crf 1 if you blindly believe VMAF. ie. Everything looks "the same" to VMAF at higher bitrate ranges. ie. It's not a useful metric for distinguishing between higher quality streams - only for streaming lowish bitrate delivery ranges

Iron_Mike

1st March 2019, 23:30

Go through the list of per-frame VMAF scores from your 'self' tests and you'll be able to identify which frames are skewing the aggregate score.

I did that, only a few frames out of the 400 are VMAF 100 - all others have scores in the 97-99 range (in the EXR control test), hence my question...

This all seems normal considering that NF themselves state that in their FAQs but when I saw you get a perfect 100 in 499/500 frames I thought maybe they've updated their model to make control tests perform close to 100...

WorBry

1st March 2019, 23:40

not sure what "different encoders" here relates to ?

I said 'decoders'.

Iron_Mike

1st March 2019, 23:42

Those are the common pixel formats supported by vmaf.VMAF.

is this for the VS flavor or for VMAF in general... where did you get this list ?

But now it's clear you're using ffmpeg vmaf. Did you look at the ffmpeg log to see what other conversions were occurring ? There might be other stuff going on behind your back

since I wasn't sure whether the bitrate was an issue (for the VMAF calculation), I converted the main and ref clips to yuv444p (8bit) before passing them into ffmpeg libvmaf (by specifying the -pix_fmt)... ffmpeg VMAF will tell the format it uses to compare in the console output, for my main/ref clips (16bit/12bit) it defaults to yuv444p10le, but once u pass clips in as 8bit it uses that format...

the VMAF score whether using original bit depth, 12 bit, 10 bit or 8 bit for the main/ref clips was always ~ 96.x (real test, not control)

Sometimes ffmpeg can "mix" up frames, less often with I-frame formats. But if your x265 encode used long GOP, there is a higher chance of a mixup than if it used I-frames only. EXR sequence will be I-frame only

GOP size on the encode is a fixed 48 frames, fps is 24

Look at the results WorBy has been posting . They all plateau off below crf 18 or so. crf16 has the same quality as crf 10 or crf 1 if you blindly believe VMAF. ie. Everything looks "the same" to VMAF at higher bitrate ranges. ie. It's not a useful metric for distinguishing higher quality - only for streaming lowish bitrate delivery ranges

well, or in other words:
those results could easily be interpreted that from a certain CRF on, the encode is perceptually identical, which is the whole point of VMAF...

their samples are based on humans reporting perceived quality differences...

ChaosKing

1st March 2019, 23:48

is this for the VS flavor or for VMAF in general... where did you get this list ?

Read the Readme: https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF/#usage

poisondeathray

1st March 2019, 23:50

is this for the VS flavor or for VMAF in general... where did you get this list ?

vapoursynth vmaf,

https://github.com/HomeOfVapourSynthEvolution/VapourSynth-VMAF/

since I wasn't sure whether the bitrate was an issue (for the VMAF calculation), I converted the main and ref clips to yuv444p (8bit) before passing them into ffmpeg libvmaf (by specifying the -pix_fmt)... ffmpeg VMAF will tell the format it uses to compare in the console output, for my main/ref clips (16bit/12bit) it defaults to yuv444p10le, but once u pass clips in as 8bit it uses that format...

the VMAF score whether using original bit depth, 12 bit, 10 or or 8bit for the main/ref clips was always ~ 96.x (real test, not control)

VMAF is probably less picky about those sorts of things, but it makes a significant difference on other metrics. There are a bunch of uncontrolled variables and operations there can cause wildly different results with other metrics - how it's scaled, dithering algo, etc...

well, or in other words:
those results could easily be interpreted that from a certain CRF on, the encode is perceptually identical, which is the whole point of VMAF...

their samples are based on humans reporting perceived quality differences...

Yes , that's a good way of phrasing it

I personally haven't used VMAF enough to be comfortable with it yet

I personally don't find that particularly useful. I guess it might be good enough for "joe public" , they might not be able to tell the difference. But you can bet people that deal frequently with encoding, codecs, compression ; ie. people that post here - they can tell the difference between say, a crf 10 vs. crf 18 encode.

Maybe a conspiracy theory, but it's almost like a Netflix scheme trying to justify their low delivery bitrate practices :devil:

Iron_Mike

2nd March 2019, 01:18

I personally don't find that particularly useful. I guess it might be good enough for "joe public" , they might not be able to tell the difference. But you can bet people that deal frequently with encoding, codecs, compression ; ie. people that post here - they can tell the difference between say, a crf 10 vs. crf 18 encode.

well "joe public" ultimately watches the content... I can tell the diff between CRF 10 and CRF 18 (frame per frame pixel peeping), but I also evaluate content on fully calibrated screens...

problem is w/ "scientific metrics" is that they often not relate a lot to the HVS (Human Vision System), which is the only thing that matters when humans watch the streamed content...

VMAF attempts to address that with their sample data... question always are if enough people were sampled, what kind of people (gender/age/race/ethicity - diff between European and Asian samples etc) and the sample procedure was done as best as possible...

Maybe a conspiracy theory, but it's almost like a Netflix scheme trying to justify their low delivery bitrate practices :devil:

hah ! probably the reason to start the project.. :cool:

poisondeathray

2nd March 2019, 02:11

well "joe public" ultimately watches the content... I can tell the diff between CRF 10 and CRF 18 (frame per frame pixel peeping), but I also evaluate content on fully calibrated screens...

It's an assumption . Audience might videophiles, or doom9ers, or you might be doing these tests for you

problem is w/ "scientific metrics" is that they often not relate a lot to the HVS (Human Vision System), which is the only thing that matters when humans watch the streamed content...

Yes, pros/cons to every measure , but there are other HVS modelled metrics.

VMAF attempts to address that with their sample data... question always are if enough people were sampled, what kind of people (gender/age/race/ethicity - diff between European and Asian samples etc) and the sample procedure was done as best as possible...

It's just that the RD curve characteristics limit VMAF's usefulness in some situations , since it's trained on higher CRF ranges.

So another way to phrase it - is the data set is not valid at higher bitrates. You cannot apply VMAF at higher bitrates because it was trained at CRF 22-28

WorBry

2nd March 2019, 02:18

VMAF attempts to address that with their sample data... question always are if enough people were sampled, what kind of people (gender/age/race/ethicity - diff between European and Asian samples etc) and the sample procedure was done as best as possible...

And what biases were introduced into the model by the choice of video codecs used in the subjective DMOS testing. Hmmm :cool:

https://forum.doom9.org/showthread.php?p=1867137#post1867137

Makes me wonder.

https://www.reddit.com/r/netflix/comments/9r75hm/netflix_starting_to_use_hevc_codec_for_hd_titles/

HolyWu

2nd March 2019, 04:18

Update r4.

Update libvmaf to v1.3.14, which reports aggregate CI scores and fix empty model name in log.

WorBry

2nd March 2019, 05:55

Cool. Should be interesting to see what statistical significance VMAF gives to those superfine score differences seen at the very high bitrates that I brought attention to earlier, which now, in light of the present discussion, I wish I hadn't ;)

https://forum.doom9.org/showthread.php?p=1865424#post1865424

Seeing that comment from Netflix changed my perspective somewhat:

"VMAF has been trained using encodes spanning from CRF 22 @ 1080 (highest quality) to CRF 28 @ 240 (lowest quality). The former is mapped to score 100 and the latter is mapped to score 20. Anything in between is mapped in the middle (for example, SD encode at 480 is typically mapped to 40 ~ 70)."

WorBry

4th March 2019, 07:08

Cool. Should be interesting to see what statistical significance VMAF gives to those superfine score differences seen at the very high bitrates that I brought attention to earlier....

I've tested the v4 update with the Crowd Run 1080/50p x264 (CRF 0 - 30) encodes I retained from the first tests with v3:

https://forum.doom9.org/showthread.php?p=1864770#post1864770

Here are the VMAF results, together with the aggregate 95% confidence interval (CI95_Low and CI95-High) scores i.e. the aggregate derived from the individual frame confidence intervals. I didn't generate the component SSIM, MS-SSIM and PSNR scores.

http://i.imgur.com/V2VKdry.png (https://imgur.com/V2VKdry)

http://i.imgur.com/MrYWJ8Q.png (https://imgur.com/MrYWJ8Q)

First thing to note is that the VMAF v4 scores are lower than the scores I obtained previously (with the exact same x264 encodes) with v3. The same default pool=1 (harmonic mean) setting was applied in both cases, so I can only assume this reflects changes in the VMAF model itself.

And homed in on the higher bitrate range.

http://i.imgur.com/SIR1U9O.png (https://imgur.com/SIR1U9O)

As noted in the v3 test series, the VMAF score for the lossless x264 CRF=0 encode (99.9954) didn't quite reach 100, and for the same reason - the component motion2=0 score for the first frame skewed an otherwise perfect 100 score for the other 499 frames.

I've yet to test the parallel x265 series with VMAF v4 but looking at the aggregate CI scores obtained with the x264 files I think I can confidently say that what minor differences were seen at the high bitrates in the first test series are not statistically significant. Seems odd though that the CI95_Low intervals for the CRF 22 - 30 encodes are actually smaller than those of CRF 20 despite being beyond the scope of the trained vmaf_v0.6.1.pkl model. Would have thought they would be larger. I suppose it depends on the content and quality of the source/reference video also.

Jamaika

4th March 2019, 07:48

I did my VMAF test on BPG files.
Vmaf is already embedded in the SVT encoder, not as a json file tester.
Pictures for I frames are better because they have a larger size by the same QP values for different encoders. And so much on the topic photos .
Ma should add codec X265 with VMAF metric .
http://forum.doom9.org/showthread.php?p=1867419#post1867419

WorBry

4th March 2019, 21:51

First thing to note is that the VMAF v4 scores are lower than the scores I obtained previously (with the exact same x264 encodes) with v3. The same default pool=1 (harmonic mean) setting was applied in both cases, so I can only assume this reflects changes in the VMAF model itself.

I see what the issue is now. When I ran the first series of tests with v3 I left it set for CI=False because it did generate the aggregate CI scores (only the per-frame CI scores). Now that aggregate CI scores are available in v4 I set CI=True which switches from 'vmaf_v0.6.1.pkl' model to 'vmaf_b_v0.6.3.pkl'. Testing the x264 series again with v4 and CI=False, the aggregate VMAF scores are exactly the same as those obtained previously with v3.

So actually the issue is that CI=True (model vmaf_b_v0.6.3.pkl) is giving lower aggregate VMAF scores than CI=False (model vmaf_v0.6.1.pkl) ! Why is that ? Surely they should be giving the same VMAF score ?

Edit: There was no mix-up of the 'model' folders , btw - when I updated to v4 I replaced the VMAF.dll and 'model folder' that came with it.

The above graphs amended accordingly:

http://i.imgur.com/qC0jFq2.png (https://imgur.com/qC0jFq2)

http://i.imgur.com/acrt2MK.png (https://imgur.com/acrt2MK)

http://i.imgur.com/tTaUoXu.png (https://imgur.com/tTaUoXu)

HolyWu

5th March 2019, 03:21

So actually the issue is that CI=True (model vmaf_b_v0.6.3.pkl) is giving lower aggregate VMAF scores than CI=False (model vmaf_v0.6.1.pkl) ! Why is that ? Surely they should be giving the same VMAF score ?

I don't remember seeing any official document mentioning that the VMAF score between non-CI model and CI model should be the same :confused:

WorBry

5th March 2019, 05:40

I guess the VMAF Confidence Interval doc does explain why there are differences:

https://github.com/Netflix/vmaf/blob/master/resource/doc/conf_interval.md

Note that the CI=False VMAF scores are within or at the limits of the CI95_High interval.

Edit: btw, testing the parallel series of x265 encodes with v4 CI=True gives exactly the same pattern.

HolyWu

6th March 2019, 02:52

I guess the VMAF Confidence Interval doc does explain why there are differences:

You are right. See #316 (https://github.com/Netflix/vmaf/issues/316).

WorBry

6th March 2019, 03:34

Thanks for raising the issue. I have a better understanding of what's going on now.:)

Iron_Mike

11th March 2019, 10:59

ran some control tests (same file validated to itself) via ffmpeg libvmaf and VS libvmaf to test whether both implementations return the same data.

since libvmaf only supports up to yuv444p10le, higher quality formats need to be down-converted - ffmpeg does that automatically.

4 sources used for the control tests: RGB48, yuv444p12le, yuv444p10le, yuv444p

VMAF SDK 1.3.14 - Model 0.6.1 - pool: mean

EXR RGB48 VMAF Note
ffmpeg 98.2549 converts internally to yuv444p10le
VS (1) 97.747 converted to yuv444p10le via FMTC
VS (2) 97.7475 converted to yuv444p10le via resize.bicubic

MP4 yuv444p12le VMAF Note
ffmpeg 98.1216 converts internally to yuv444p10le
VS (1) 97.7106 converted to yuv444p10le via FMTC
VS (2) 97.7101 converted to yuv444p10le via resize.bicubic

MP4 yuv444p10le VMAF Note
ffmpeg 98.1044 no conversion needed
VS 97.7144 no conversion needed

MP4 yuv444p VMAF Note
ffmpeg 97.7363 no conversion needed
VS 97.7363 no conversion needed

While it can be expected that the 16bit and 12bit sources will not return the same VMAF scores (ffmpeg internal down-conversion may not match the chosen VS conversion method), it is interesting to see that only w/ the 8-bit src the VMAF scores match.

The VMAF scores of the 10-bit src (although no conversion being done) still differ.

what is the reason for that ?

HolyWu

11th March 2019, 16:32

The VMAF scores of the 10-bit src (although no conversion being done) still differ.

what is the reason for that ?

Because FFmpeg filter doesn't normalize 10 bit to 8 bit like what Netflix does for calculation, hence the inconsistency.

Iron_Mike

11th March 2019, 22:14

Because FFmpeg filter doesn't normalize 10 bit to 8 bit like what Netflix does for calculation, hence the inconsistency.

which is the better approach

if a 10bit ref/src is provided. up-converting an encoded/distorted clip to 10bit does not lose precision, but down-converting a 10bit ref src to 8bit to then compare to the inferior 8-bit encode loses precision/accuracy...

NF does up-res a lower res encoded clip before comparing to the higher res ref/src (same logic), so this seems odd...

do you have a link to where they state that they downsample the master to 8bit ?

Thanks.

WorBry

13th March 2019, 04:48

Finally got around to re-testing the Crowd Run 2160/50p x264 series that I kept from earlier tests with v3:

https://forum.doom9.org/showthread.php?p=1865316#post1865316

So this was testing with VapourSynth VMAF v4 in Model=1 mode which uses vmaf_4k_v0.6.1 by default (CI=False) and vmaf_4k_rb_v0.6.2 when set to CI=True.

Now in this case CI=False and CI=True produced the exact same aggregate VMAF scores, which came as a surprise:

http://i.imgur.com/f2Py225l.png (https://imgur.com/f2Py225)

Now how is that ? The Confidence Interval doc doesn't mention 4K models specifically but I would assume the 'rb' in 'vmaf_4k_rb_v0.6.2' means 'residue bootstrapping', in which case why is residue bootstrapping used to derive CI scores for 4K video, whereas the CI model for HD/SD (vmaf_b_v0.6.3) uses plain bootstrapping ? All rather confusing.

HolyWu

16th March 2019, 04:12

do you have a link to where they state that they downsample the master to 8bit ?

Netflix doesn't explicitly mention that in the documentation. It's simply done this way in their source code.

Now in this case CI=False and CI=True produced the exact same aggregate VMAF scores, which came as a surprise:

I think the non-bootstrapping 4K model should have been named v0.6.2 rather than v0.6.1, as the model was released after VMAF algorithm v0.6.2. And the VMAF score won't be different between residue bootstrapping and plain bootstrapping. Only the CI-related scores will be affected.

WorBry

16th March 2019, 05:02

... And the VMAF score won't be different between residue bootstrapping and plain bootstrapping. Only the CI-related scores will be affected.

OK, but still - why in the 4K (2160/50p) tests does CI=True (vmaf_4k_rb_v0.6.2) give exactly the same aggregate VMAF scores as CI=False (vmaf_4k_v0.6.1), when in the 1080/50p tests CI=True (vmaf_b_v0.6.3.pkl) gave consistently lower aggregate VMAF scores than CI=False (vmaf_v0.6.1.pkl) ?

HolyWu

16th March 2019, 05:22

OK, but still - why in the 4K (2160/50p) tests does CI=True (vmaf_4k_rb_v0.6.2) give exactly the same aggregate VMAF scores as CI=False (vmaf_4k_v0.6.1), when in the 1080/50p tests CI=True (vmaf_b_v0.6.3.pkl) gave consistently lower aggregate VMAF scores than CI=False (vmaf_v0.6.1.pkl) ?

If vmaf_4k_v0.6.1 is actually trained with the same underlying environment as vmaf_4k_rb_v0.6.2, they are expected to have the same VMAF scores then. vmaf_v0.6.1.pkl was trained with different underlying environment compared to vmaf_b_v0.6.3.pkl, hence they don't give the same VMAF scores.

WorBry

16th March 2019, 06:03

So why don't they update the 'classic' non-bootstrapping HD/SD model, trained in the same environment as vmaf_b_v0.6.3, so that CI=False and CI=True produce the same aggregate VMAF scores as well? Surely it's important to have consistent outcomes ?

HolyWu

16th March 2019, 06:11

WorBry

16th March 2019, 06:38

Fair enough ;)

Iron_Mike

30th March 2019, 02:35

already posted this in another thread (https://forum.doom9.org/showpost.php?p=1870245&postcount=269), but thought I'd post VMAF results here as well

from a 16-bit EXR source (15 secs, 360 frames), made nine (9) x265 encodes, all CRF 10, in these formats (using Wolfberry ffmpeg build): yuv420p, yuv422p, yuv444p, yuv420p10le, yuv422p10le, yuv444p10le, yuv420p12le, yuv422p12le, yuv444p12le

VS VMAF results (sources were down-converted to yuv444p10, if higher, since that is the highest input format supported)

https://i.imgur.com/o5b4UNC.png

FFMPEG VMAF results (internally converts to yuv444p10, if higher source)

https://i.imgur.com/lK1bCMg.png

as you can see VMAF score indication is the same in both, but the SSIM/MS-SSIM differ... now besides that FFMPEG has that odd dip (scoring 8-bit higher than 10/12-bit), the VS VMAF results are almost flat...

so does VS VMAF internally convert everything to 8-bit (although it supports up to yuv444p10 input format) ?

poisondeathray

30th March 2019, 03:29

so does VS VMAF internally convert everything to 8-bit (although it supports up to yuv444p10 input format) ?

That's what HolyWu said, above - as per Netflix's source code

Netflix doesn't explicitly mention that in the documentation. It's simply done this way in their source code.

And a difference is that ffmpeg's vmaf implementation does not

HolyWu

30th March 2019, 03:33

as you can see VMAF score indication is the same in both, but the SSIM/MS-SSIM differ... now besides that FFMPEG has that odd dip (scoring 8-bit higher than 10/12-bit), the VS VMAF results are almost flat...

so does VS VMAF internally convert everything to 8-bit (although it supports up to yuv444p10 input format) ?

Yes. vmafossexec (the CLI of libvmaf) also does this normalization for 10-bit input. The normalized values are stored in floating-point, hence you needn't worry about precision lost. If you enable PSNR calculation in both VS libvmaf and FFmpeg libvmaf as well, you'll probably see bigger difference.

Iron_Mike

30th March 2019, 07:49

@HolyWu:

since everything gets down-converted to 8-bit internally, why are you guys not making 8-bit input mandatory in VS VMAF ?

I mean the 10-bit input support is pointless, and VS VMAF already requires same format for ref/dist, so the user is already required to convert in most cases before calling VS VMAF...

and btw, I mentioned this in the other thread:

when I use yuv444p (8-bit) as input format (coming from RGB48le) in VS VMAF compared to using yuv444p10 (10-bit), the range of VMAF/SSIM/MS-SSIM values is compressed (closer together)... since everything gets converted to 8-bit internally anyways, the range of values should pretty much be the same... unless the result of the filters I use to down-convert to 8-bit is substantially different to what VMAF uses internally... (I use fmtc or vs.resize)

alongside the other VMAF results, this is the result if I use 8-bit input w/ VS VMAF:

VS VMAF results (sources were down-converted to yuv444p)
https://i.imgur.com/KlXZtPd.png

HolyWu

3rd April 2019, 15:39

Update r5.

Accept clips of any planar format with integer sample type of 8-16 bit depth, except RGB. Note that libvmaf only uses luma plane for calculating scores.
Remove parameter psnr.

Iron_Mike

3rd April 2019, 23:11

Update r5.

Accept clips of any planar format with integer sample type of 8-16 bit depth, except RGB. Note that libvmaf only uses luma plane for calculating scores.
Remove parameter psnr.

is this an update of your VS implementation of libvmaf, or an official update ?

so YUV 8-16 bit input supported, but internally everything gets converted to 8-bit before measuring score, correct ?

Update r5.

Note that libvmaf only uses luma plane for calculating scores.

is there a way to specify to score any of the color planes ?

Thanks.

ChaosKing

4th April 2019, 00:13

Iron_Mike

4th April 2019, 01:54

Just extract your plane and pass it as a GRAY clip to vmaf http://www.vapoursynth.com/doc/functions/shuffleplanes.html

c1 = core.std.ShufflePlanes(clip, planes=1, colorfamily=vs.GRAY)

@CK: thank you, as always. I'll try that.

Edit: this does work with r5.

Iron_Mike

4th April 2019, 02:42

@HolyWu:

so the .dll size changed from 16.3MB (r4) to 2.2MB (r5)...

considering that the changelog states input format validation and removal of psnr input param as the only changes, that must have either been the most code written ever for these two simple tasks or u must have hidden a few 8-bit video games as easter eggs in r4 that none of us have discovered... ;-)

that is quite a reduction in filesize...

HolyWu

4th April 2019, 03:25

is this an update of your VS implementation of libvmaf, or an official update ?

Neither. Nothing has to be changed in libvmaf. The format restriction is simply for the constants needed for PSNR calculation (https://github.com/Netflix/vmaf/blob/8f41503605e6e55b34945e49cba864697e1ca683/feature/src/psnr_tools.c#L26). Hence I disabled it as only 8 bit and 10 bit are defined.

so YUV 8-16 bit input supported, but internally everything gets converted to 8-bit before measuring score, correct ?

Yes. But they are not clamped to 0-255 range. For example the peak of 10 bit is 1023/4.=255.75, 12 bit is 4095/16.=255.9375, and so on.

considering that the changelog states input format validation and removal of psnr input param as the only changes, that must have either been the most code written ever for these two simple tasks or u must have hidden a few 8-bit video games as easter eggs in r4 that none of us have discovered... ;-)

Unfortunately both guesses are wrong! It's just because I changed to another compiler.

Iron_Mike

4th April 2019, 03:36

Yes. But they are not clamped to 0-255 range. For example the peak of 10 bit is 1023/4.=255.75, 12 bit is 4095/16.=255.9375, and so on.

well, it should not be 1023/4... since we're zero index based it's 1024 (total values in 10-bit, 1023+1)/4 and/or 4096 (total values in 12-bit, 4095+1)/16 which both result in 256, which is in 8-bit the value of 255 (256-1)

or what am I missing in your example ?

Iron_Mike

4th April 2019, 03:40

Neither. Nothing has to be changed in libvmaf. The format restriction is simply for the constants needed for PSNR calculation (https://github.com/Netflix/vmaf/blob/8f41503605e6e55b34945e49cba864697e1ca683/feature/src/psnr_tools.c#L26). Hence I disabled it as only 8 bit and 10 bit are defined.

yeah, the var u define for *peak needs to accommodate the zero index... peak in 8-bit is always 255, meaning 256 total values starting at 0

u need to adjust +/- 1 when u go from bit values 0-255/0-1023/0-4095 to number of distinct values per bit-depth 256 (2^8)/1024 (2^10)/4096(2^12)

HolyWu

4th April 2019, 04:44

yeah, the var u define for *peak needs to accommodate the zero index... peak in 8-bit is always 255, meaning 256 total values starting at 0

I didn't define it. It's in libvmaf. So you are implying that Netflix does it wrong. :devil:

u need to adjust +/- 1 when u go from bit values 0-255/0-1023/0-4095 to number of distinct values per bit-depth 256 (2^8)/1024 (2^10)/4096(2^12)

Actually I don't totally agree with you. How come the value needs to be +1 to make the range 0-256? The MSU page (http://www.compression.ru/video/quality_measure/info.html#ssim) also describes how the peak value is defined in the same way.

Maximum value of a color component is calculated in the same way as for PSNR:
videoMax = 255 for 8 bit color components
videoMax = 255 + 3/4 for 10 bit color components
videoMax = 255 + 63/64 for 14 bit color components
videoMax = 255 + 255/256 for 16 bit color components

Iron_Mike

4th April 2019, 05:12

I didn't define it. It's in libvmaf. So you are implying that Netflix does it wrong. :devil:

sorry, I thought it was a link to your VS VMAF github... ;-)

Actually I don't totally agree with you. How come the value needs to be +1 to make the range 0-256? The MSU page (http://www.compression.ru/video/quality_measure/info.html#ssim) also describes how the peak value is defined in the same way.

there is no range 0-256 in common bit depths.

8-bit is 0-255, with a total of 256 distinct values: 2^8 = 256

the bit of code that you linked to, converts incorrectly. there's various ways to do this, but here's the most simple way, very easy to follow:

the code tries to convert from 10-bit or 12-bit to 8-bit, at least that how it appears.

approach #1:

(1) take any 10/12 bit value, e.g 912 in 10-bit
(2) normalize the value: 912+1/10^2 = 0.8916015625 --> meaning 89.16% stimuli in that channel/plane
(3) convert to 8-bit: (2^8-1)*0.8916015625 = 227.3583984375, rounded to int would be 227

run the same example with 0 or 1023/4095 (which would be the extreme case scenarios) and u see it works just fine...

and here's an excerpt from our Russian buddies MSU page:

In MSU VQMT there are four PSNR implementations. "PSNR" and "APSNR" use the correct way of PSNR calculation and take maximum possible absolute value of color difference as MaxErr. However, this way of calculation gives an unpleasant effect after color depth conversion. If color depth is simply increased from 8 to 16 bits, the "PSNR" and "APSNR" will change, because MaxErr should change according to maximum possible absolute value of color difference (255 for 8 bit components and 255 + 255/256 for 16 bit components). Thus, "PSNR (256)" and "APSNR (256)" are implemented. They would not change because they use upper boundary of color difference as MaxErr. The upper boundary is 256. This approach is less correct, but it is used often because it is fast. Here are the rules of MaxErr definition:

"PSNR" and "APSNR" – MaxErr varies on color components bits usage:
255 for 8 bit components
255 + 3/4 for 10 bit components
255 + 63/64 for 14 bit components
255 + 255/256 for 16 bit components
100 for L component of LUV color space

HolyWu

4th April 2019, 07:04

Firstly, that doesn't apply in the metric calculation. We are not really doing format conversion like YUV420P16 to YUV420P8. The purpose of the normalization is not to map every pixel value into exact 0-255 range. As I said before, the normalized value is stored in floating-point and not integer so there is no rounding issue.

Secondly, taking your example, 227 in 8 bit converts to 10 bit with zero LSBs is 908. (908+1)/(2^10)*255=226.3623. How come 227 converts to 10 bit then converts back to 8 bit becomes 226.

Lastly, as you already seen on the MSU page, that approach is less correct. So why bother?

Iron_Mike

4th April 2019, 08:54

Secondly, taking your example, 227 in 8 bit converts to 10 bit with zero LSBs is 908. (908+1)/(2^10)*255=226.3623. How come 227 converts to 10 bit then converts back to 8 bit becomes 226.

because u're taking the rounded value of 227, which obviously already lost precision... use the full float value and it will convert to 10-bit fine... there are apps - like many color apps - that only take integer R|G|B values, hence u then have to round

and once u round and u convert back/forth obviously the color drifts...

Lastly, as you already seen on the MSU page, that approach is less correct. So why bother?

I just mentioned it b/c I just quickly glanced over ur link and thought it was your VS VMAF code that u use to downconvert all input to 8-bit in VS VMAF before passing it into libvmaf...

Iron_Mike

5th April 2019, 01:53

HolyWu

5th April 2019, 03:40

just ran a test on r5 using a 16-bit source and compared VMAF score to r4 (where I had to scale the clips to 10-bit) and results are very, very close... small differences in 3rd or 4th decimal place...

No surprise to me at all. Bear in mind that VMAF is a perceptual quality assessment and takes the viewing distance(3H for 1080p and 1.5H for 4K) into account. The most observable difference by human eyes between 8 bit and 10 bit in video encoding is banding. But do human eyes be able to observe significant difference between 10 bit and 16 bit from the mentioned viewing distance?

Iron_Mike

5th April 2019, 05:24

No surprise to me at all. Bear in mind that VMAF is a perceptual quality assessment and takes the viewing distance(3H for 1080p and 1.5H for 4K) into account. The most observable difference by human eyes between 8 bit and 10 bit in video encoding is banding. But do human eyes be able to observe significant difference between 10 bit and 16 bit from the mentioned viewing distance?

yeah, I would say 12-bit at most is upper limit... I've compared same image on a 12-bit post prod panel to the standard 10-bit panels, tough to see any differences, and if if so, then it could also be b/c of the cal or the display tech

WorBry

7th April 2019, 23:55

is there a way to specify to score any of the color planes ?

Just extract your plane and pass it as a GRAY clip to vmaf http://www.vapoursynth.com/doc/functions/shuffleplanes.html

c1 = core.std.ShufflePlanes(clip, planes=1, colorfamily=vs.GRAY)

Incidentally, I see same question was asked on the Netflix vmaf github:

https://github.com/Netflix/vmaf/issues/100

That was 18 months ago - I wonder when they will get around to 'extending to chroma' and how they will go about it?

poisondeathray

11th October 2019, 03:42

Are the results supposed to be identical to vmafossexec.exe ?

I'm getting some discrepancies, but the vmafossexec I'm using is based on 1.3.15, vapoursynth-vmaf 1.3.14 . Not sure if this accounts for the differences

I understand ffmpeg libvmaf is going to be a bit different from the discussion above, but's a 3rd different

HolyWu

11th October 2019, 04:10

Are the results supposed to be identical to vmafossexec.exe ?

I'm getting some discrepancies, but the vmafossexec I'm using is based on 1.3.15, vapoursynth-vmaf 1.3.14 . Not sure if this accounts for the differences

Did you use the same pooling method? VS-VMAF defaults to harmonic mean while vmafossexec defaults to arithmetic mean.

poisondeathray

11th October 2019, 04:19

Did you use the same pooling method? VS-VMAF defaults to harmonic mean while vmafossexec defaults to arithmetic mean.

No . Nice catch . I'll verify if both AM's match and post back if they don't.

Is there a switch for vmafossexec to use harmonic mean? There doesn't seem to be in the help

The blog says "our empirical results suggest that simple arithmetic mean (AM) is the best way of averaging, in that it yields highest correlation with subjective scores."

But then I saw some issues on the issue tracker that seem to suggest HM is better in some ways.

Any comments or user experiences on HM vs AM ?

HM seems to catch cases were traditional SSIM/PSNR fail better than AM ; it seems to match visually better to my eyes. It's actually the reason I posted now, I'm seeing a flip flop in expected results with vmafossexec AM . I've been using the vapoursynth VMAF for the last few months with HM and it seems to correlate pretty well

HolyWu

11th October 2019, 04:38

Is there a switch for vmafossexec to use harmonic mean? There doesn't seem to be in the help

Yes, --pool harmonic_mean. The developer seemed to forget to add this switch in the help.

poisondeathray

11th October 2019, 04:39

Awesome thanks

poisondeathray

11th October 2019, 05:23

Using raw YUV files produces matching results (vsrawsource in vapoursynth-vmaf vs. vmafossexec)

But using lsmash does not on the compressed files. So now I'm thinking seek issues

EDIT: it turns out it was just a stupid error - wrong path/file !!!

But nice to know that vmafossexec and vapoursynth-vmaf match

I noticed vmafossexec seems to process significantly faster, even when comparing raw yuv compared to to vsrawsource/vmaf . What is the reason for the difference?

Thanks again for the help and plugin

Rumbah

11th October 2019, 12:04

Thanks for noticing differences between raw and compressed sources (before your edit ;) ).

I used ffmpeg for vmaf and never checked and it seems it has this problem, too.
I get very low vmaf scores for a compressed input and looking at the logs it seems the b frames are handled wrong resulting in scores for them below 10.

If i use raw sources everything is fine and I get high scores.

poisondeathray

11th October 2019, 15:17

Thanks for noticing differences between raw and compressed sources (before your edit ;) ).

Mine was a bonehead mistake; what's your excuse :)

I would double check everything especially source filter

The LSmash version I used passed seek-test.py for the videos, so I rechecked everything and the most simple explanation was the answer

I used ffmpeg for vmaf and never checked and it seems it has this problem, too.
I get very low vmaf scores for a compressed input and looking at the logs it seems the b frames are handled wrong resulting in scores for them below 10.

If i use raw sources everything is fine and I get high scores.

FFMpeg vmaf is buggy for this -

1) for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

2) There is a 8/10 bitdepth issue on the tracker. It's easy too reproduce (esp. the 2nd post) . Vapoursynth-vpy seems ok with this
https://github.com/Netflix/vmaf/issues/211

And even then it does not match for frame accurate, 8bit to 8bit case maybe because of the reasons earlier in the thread

My thinking is vmafossexec is the "official" one, and if vapoursynth-vmaf matches it, that's 2 against 1 anyways

HolyWu

12th October 2019, 12:44

Update r6:

Update to libvmaf 1.3.15.
Add csv log format.
Minor optimization.

I noticed vmafossexec seems to process significantly faster, even when comparing raw yuv compared to to vsrawsource/vmaf . What is the reason for the difference?

How significant is it? Any numbers?

My wild guess is that it's because vs-vmaf has to use mutex lock due to the way libvmaf receives frames and that probably incur additional overheads.

richardpl

12th October 2019, 20:13

FFMpeg vmaf is buggy for this -

1) for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

2) There is a 8/10 bitdepth issue on the tracker. It's easy too reproduce (esp. the 2nd post) . Vapoursynth-vpy seems ok with this
https://github.com/Netflix/vmaf/issues/211

And even then it does not match for frame accurate, 8bit to 8bit case maybe because of the reasons earlier in the thread

My thinking is vmafossexec is the "official" one, and if vapoursynth-vmaf matches it, that's 2 against 1 anyways

What FUD you share with us? Please STOP!

poisondeathray

12th October 2019, 20:16

How significant is it? Any numbers?

My wild guess is that it's because vs-vmaf has to use mutex lock due to the way libvmaf receives frames and that probably incur additional overheads.

Thanks for the update

vmafossexec is about 3x faster. ~ 15-16fps vs. ~5fps . Setup was the same with the raw yuv files on different HDD's . Similar results with r6 for speed

poisondeathray

12th October 2019, 20:47

What FUD you share with us? Please STOP!

Which part do you believe is FUD ? Can you expand on that a bit ? Maybe it's just user error

Here are the results of an 8bit raw YUV test with Harmonic Mean (raw YUV , so in theory no potential seek issues. 8bit to 8bit, so no 8/10 bit FFmpeg libvmaf issue)

vmafossexec 95.3836
vapoursynth-vmaf 95.3836
ffmpeg libvmaf 95.38745502791545

The 8bit/10bit FFMpeg libvmaf issue on the tracker is easily reproducible, and you can try different dithering algorithms, comparing at 10 vs. 8, different combinations

HolyWu

13th October 2019, 03:39

vmafossexec is about 3x faster. ~ 15-16fps vs. ~5fps . Setup was the same with the raw yuv files on different HDD's . Similar results with r6 for speed

I can't replicate that huge difference. Where did you download vmafossexec? What's the resolution and length of your test video?

poisondeathray

13th October 2019, 03:47

I can't replicate that huge difference. Where did you download vmafossexec? What's the resolution and length of your test video?

From AppVeyor . I think this is the same one 2 days ago
https://ci.appveyor.com/project/li-zhi/vmaf/builds/28022740/artifacts

1920x1080 , 8bit 4:2:0, 2400 frames

(And FFMpeg libvmaf was about the same as vpy-vmaf in terms of speed for the same files)

HolyWu

13th October 2019, 05:40

From AppVeyor . I think this is the same one 2 days ago
https://ci.appveyor.com/project/li-zhi/vmaf/builds/28022740/artifacts

1920x1080 , 8bit 4:2:0, 2400 frames

Still doesn't make much difference.

start /b /high /wait vmafossexec yuv420p 1920 1080 E:\ref.yuv E:\dis.yuv model/vmaf_v0.6.1.pkl --log E:\vmaf1.xml --pool harmonic_mean

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1662" timeTaken="51.6417" />

ref = core.raws.Source(r'E:\ref.yuv', 1920, 1080)
dis = core.raws.Source(r'E:\dis.yuv', 1920, 1080)
clip = core.vmaf.VMAF(ref, dis, log_path=r'E:\vmaf2.xml', pool=1)

start /b /high /wait vspipe -p test.vpy .

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1263" timeTaken="51.8453" />

poisondeathray

13th October 2019, 05:48

I/O bottleneck ? Are your YUV files on separate physical drives ? or SSD ? I notice it was slower on the same drive . Otherwise I can't explain it. I'll check again but other physical files were about the same 3x speed too . Unless my vapoursynth setup and ffmpeg libvmaf are both slower for some reason ?

My results are the same, so it's unlikely to be some sort of other error like wrong file

poisondeathray

13th October 2019, 06:03

It looks like both your files are in E:\

I have "ref" on an SSD and "dis" on a HDD

HolyWu

13th October 2019, 07:10

Tested again with two files on separate SSD drives. It's marginally faster but still not a huge difference.

vmafossexec

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.3853" timeTaken="50.5524" />

vs-vmaf

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1431" timeTaken="51.7592" />

poisondeathray

13th October 2019, 07:19

Thank you for investigating it, I cannot explain why I observe those results . I'll look into it again in the morning and report back if I find anything

poisondeathray

13th October 2019, 18:57

The difference was other metrics in the calculation , eg. ssim , ms-ssim also being calculated with vmaf . Same speed when everything is equivalent

Another bonehead mistake. Sorry for wasting your time :(

richardpl

26th October 2019, 12:07

I can not reproduce finding you have. Perhaps you use libvmaf filter in invalid ways? Also non-8bit bug in libvmaf filter have been already fixed.

poisondeathray

26th October 2019, 18:29

I can not reproduce finding you have. Perhaps you use libvmaf filter in invalid ways? Also non-8bit bug in libvmaf filter have been already fixed.

Great news

I repeated and FFmpeg libvmaf results match for the 8bit case

(1) is still a potential issue -

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times

-This one is easier to demonstrate with PSNR, because it can measure "perfect", while VMAF cannot.
-An unsuspecting user, that just follows the instructions and examples can get the wrong results

MasterNobody

27th October 2019, 22:39

(1) is still a potential issue -

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times
Have you tried to use "-vsync 0" option? With it ffmpeg should output all frames without any duplicated and dropped frames.

poisondeathray

28th October 2019, 01:28

Have you tried to use "-vsync 0" option? With it ffmpeg should output all frames without any duplicated and dropped frames.

Yes;

In some cases forcing -r with that might work (cases where it doesn't work - timestamp jitter) . In other cases it does not. Sometimes you have to reset PTS along with it . Sometimes remuxing helps (same container) - the timebase differences and jitter timestamps between MP4 and MKV container is a common problem that causes wrong values in the metrics

Or just use avisynth or vapoursynth with a frame accurate source filter; that 100% works if you use assumefps and the frames are aligned. That resets all the timestamps and jitter, both ref and distorted will be equal

richardpl

28th October 2019, 09:19

Again FUD spreading from you! You also never provided commands you use and input files.

poisondeathray

28th October 2019, 16:04

Again FUD spreading from you! You also never provided commands you use and input files.

Again, which part do you believe is FUD ? This is a known issue with FFmpeg.

(1) is still a potential issue -

for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times

Here are 2 examples. I chose PSNR, because it's easy to demonstrate the issue (should get inf). In both cases "0" is the reference (but it shouldn't matter, since they are "lossless" equivalents)

If you were to follow the docs/ instructions, without making adjustments - you will get the wrong values
https://ffmpeg.org/ffmpeg-filters.html#psnr

*In both cases (and other cases, where there are timestamp issues, container timebase differences, "off" timecodes etc...) , if you just use avisynth or vapoursynth, it works . If the source filter you are using reads "off" frame rate (e.g. 23.976 vs. 24000/1001 exactly, or similar, you need to use AssumeFPS to make them equivalent if using FFmpeg; as long as the frames are aligned with a frame accurate source filter, it will work and give correct values)

https://www.mediafire.com/file/rd2zrbs6xq18b8f/ffmpeg_psnr_example1.7z/file
https://www.mediafire.com/file/c9iebz72cwxi97r/ffmpeg_psnr_example2.7z/file

example 1)
In the first case both are "23.976" by mediainfo (or 23.98 by ffmpeg) but there are tbn container differences . This is known as MP4 and MKV express timestamps differently

Also notice the non zero start time in "0.mkv" and duration difference because of it

ffmpeg -i 0.mkv
Duration: 00:00:05.01, start: 0.084000, bitrate: 38321 kb/s
Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080
[SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

ffmpeg -i 1.mp4
Duration: 00:00:04.92, start: 0.000000, bitrate: 74719 kb/s
Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), y
uv420p, 1920x1080 [SAR 1:1 DAR 16:9], 74723 kb/s, 23.98 fps, 23.98 tbr, 24k tbn,
47.95 tbc (default)

If you were to follow the docs, you will get wrong values

ffmpeg -i 1.mp4 -vcodec rawvideo -vf "movie=0.mkv,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=stats1.log" [out]" -f rawvideo -y /NUL

If you were to follow the docs that specific different containers, you will still get wrong values

ffmpeg -i 1.mp4 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

-i avs , or -i vpy works, or forcing the framerate -r 24000/1001 before each -i works in this particular case (but not necessarily others)

ffmpeg -r 24000/1001 -i 1.mp4 -r 24000/1001 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

example 2)
ffmpeg -i 0.mkv
Duration: 00:00:10.26, start: 0.000000, bitrate: 18583 kb/s
Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1
:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

ffmpeg -i 1.mp4
Duration: 00:00:10.26, start: 0.000000, bitrate: 120532 kb/s
Stream #0:0(eng): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), y
uv420p, 1920x1080 [SAR 1:1 DAR 16:9], 120539 kb/s, 23.98 fps, 23.98 tbr, 24k tbn
, 47.95 tbc (default)

Same issue with container tbn difference, but start time is zero in this case

If you were to follow the docs, you will get wrong values

ffmpeg -i 1.mp4 -vcodec rawvideo -vf "movie=0.mkv,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=stats2.log" [out]" -f rawvideo -y /NUL

If you were to follow the docs that specific different containers, you will still get wrong values

ffmpeg -i 1.mp4 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

If you were to follow the docs that specific different containers, and use the -r force framerate adjustment that worked in example #1 above, you will still get wrong values

ffmpeg -r 24000/1001 -i 1.mp4 -r 24000/1001 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

-i avs, or -vpy works

richardpl

28th October 2019, 17:36

poisondeathray

28th October 2019, 17:48

I downloaded both your uploaded files.
First file contains two video files that have same decoded video frames hashes. And for this files example in documentation about using psnr with different containers works just fine (I dunno what was your point in trying command that works only with same containers and with nice timestamps). Perhaps you used wrong command or faulty build?

I used this command: ffmpeg -i 0.mkv -i 1.mp4 -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr=shortest=1" -f null -

Extra shortest is for cases when user supplied videos of different duration.

Second file contain two video files with different video frames hashes. One file have duplicated first two frames with normal timestamps. No point in using psnr on them. If you have same results with vpy or avs then there is something seriously broken there with them :)

Shortest works for the 1st one, as does forcing -r .

The 2nd one has open GOP's with leading b's . Different decoders can handle open GOP's differently. But the same decoder should handle it consistently.

In fact, FFmpeg was used to create 1.mp4 from 0.mkv. And the decoded elementary streams are identical

ffmpeg -i 0.mkv -c:v rawvideo 0.yuv

ffmpeg -i 1.mp4 -c:v rawvideo 1.yuv

ffmpeg -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 1.yuv -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 0.yuv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

Or, if you use avs or vpy input, works too

This is why you need to make adjustments for some types of files. The docs don't cover these cases or others like them; if a user just followed the instructions, they'd get the wrong values

richardpl

28th October 2019, 19:11

No, 2nd one is completely invalid, and they are not identical.

poisondeathray

28th October 2019, 19:20

No, 2nd one is completely invalid, and they are not identical.

Explain.

How is that possible, when 1.mp4 was created from 0.mkv

ffmpeg -i 0.mkv -c:v libx264 -qp 0 1.mp4

Also, rawvideo says they are identical

ffmpeg -i 0.mkv -c:v rawvideo 0.yuv

ffmpeg -i 1.mp4 -c:v rawvideo 1.yuv

ffmpeg -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 1.yuv -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 0.yuv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

richardpl

29th October 2019, 12:54

They are not identical, and you know it as you created such files.

There is visible gap in pts of first and second frame of 0.mkv. Making it not CFR but VFR. rawvideo will just make CFR from VFR.

poisondeathray

29th October 2019, 14:38

They are not identical, and you know it as you created such files.

There is visible gap in pts of first and second frame of 0.mkv. Making it not CFR but VFR. rawvideo will just make CFR from VFR.

They are identical.

It's a lossless encode -qp 0 , encoded by FFmpeg. The commandlines are posted. Do you see conceptually the problem ? Someone uses FFmpeg to encode a "lossless" video in the same colorspace/sampling/bit depth - they expect it to be "lossless", they expect PSNR to say "lossless"

It's not an encoding issue, since the YUV streams match, encoded frames are identical.

It should have been CFR, because it was cut from a BD. BD's are CFR only.

This has open GOP with 2 leading b's. Decoders are supposed to either duplicate the leading b's or drop them (risk of desync). Eitherway , it should be consistently handled

If you go back, that PTS example for different container in the documentation is new - It wasn't there even 2 months ago if I look at my local documentation for a FFmpeg build. That's great, the docs need to be improved with more examples

That's why I said -

(1) is still a potential issue -

for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times

If you think that's "FUD", we wouldn't have these types of issues . Everything would just "work" without having to do these workarounds. The warning is so people don't get "burned"

Note that 1st example was a lossless encode from "0.mkv" too. The lengths are actually the same in terms of framecount. The -shortest or -r workaround are not mentioned in the docs. People expect things to just work; a casual user would have been "burned" there too

This illustrates some of the issues with how FFmpeg handles this calculation. There should be a way to reset all the PTS, you can improve it farther.

richardpl

29th October 2019, 18:20

It's not an encoding issue, since the YUV streams match, encoded frames are identical.

It should have been CFR, because it was cut from a BD. BD's are CFR only.

This has open GOP with 2 leading b's. Decoders are supposed to either duplicate the leading b's or drop them (risk of desync). Eitherway , it should be consistently handled

You are terribly mistaken. Closely inspect timestamp and you will see that 2 frames are completely missing from 0.mkv
There is gap created by ffmpeg decoder.
It works with vpy, avs just by pure luck because they duplicate right frames in those gaps. fps filter duplicates currently always first frame encountered so far across pts gap, which does not mach with frame hashes in 1.mp4

See for yourself with -vf showinfo,trim=end_frame=5 filter on both files.
The -r option is hacky beacuse it tries to round to nearest frame but is slightly off for some strange reason.

poisondeathray

29th October 2019, 18:26

You are terribly mistaken. Closely inspect timestamp and you will see that 2 frames are completely missing from 0.mkv
There is gap created by ffmpeg decoder.
It works with vpy, avs just by pure luck because they duplicate right frames in those gaps. fps filter duplicates currently always first frame encountered so far across pts gap, which does not mach with frame hashes in 1.mp4

See for yourself with -vf showinfo,trim=end_frame=5 filter on both files.
The -r option is hacky beacuse it tries to round to nearest frame but is slightly off for some strange reason.

Those 2 are leading b's frames from open GOP. The ffmpeg AVC libavcodec decoder places 2 duplicate frames there as "placeholder frames" to keep sync (had it been a real example with audio). This occurs with DVD / mpeg2 as well . Same handling with leading b's open GOP

(Other decoders might handle it differently; the point is it should be consistent with the same decoder; if it drops, then drop in all cases, not just some of the time or depending on the phase of the moon. Most AVC decoders will keep them as duplicates)

But that's why the encode has them as well, and the decoded YUV frames are identical, the 1st 2 are duplicates from the open gop . Same framecount. Frames are the same . Input = output. ( The encode was made from decoding first AVC stream with open GOP; and the decode of the lossless encode is also decoding AVC but they are not longer leading b's, they've been decoded and reencoded - both are consistent and have those 2 "placeholder" frames)

The problem is the FFmpeg timestamp handling . At the very least , you have to admit it's inconsistent behaviour here. On one hand FFmpeg places those b-frames. Physically decodes and encodes therm. On the other hand, when doing the calculations it drops them

Just take a step back and look at the big picture for a second: Someone supposedly encodes a lossless video with ffmpeg and it's "not lossless" according to FFmpeg PSNR . Or -
FFmpeg VMAF is not going to match vmafossexec or vapoursynth-vmaf in cases like this - unless you decode to raw YUV first for use in FFMpeg .

Shevach

16th December 2019, 11:23

Yes, looking at the models page suggests that.. Looking at the linked slideshow there, it also seems that for proper analysis, you have to upscale to 4K.

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

Dear experts

i am faced with same problem while running libvmaf by means of ffmpeg:

.\ffmpeg.exe -hide_banner -r 60 -s 1920x1080 -i test60fps.yuv -r 60 -s 1920x1080 -i ref.yuv -vframes 2372 -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl:log_path=test30Mbps60fps.xml" -report -f null -

...
frame= 81 fps= 19 q=-0.0 size=N/A time=00:00:03.24 bitrate=N/A speed=0.761x
frame= 89 fps= 19 q=-0.0 size=N/A time=00:00:03.56 bitrate=N/A speed=0.748x
frame= 98 fps= 19 q=-0.0 size=N/A time=00:00:03.92 bitrate=N/A speed=0.74x
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.

If i downscale raw video resolution to 720x480 the ffmpeg succeeds to compute vmaf scores for all frames.

Task Manager does not reveal any problem (the memory usage does not exceed 5%).

How can i resolve this issue?