VMAF - Video Multi-Method Assessment Fusion

[Iron_Mike]

Update r5.


Accept clips of any planar format with integer sample type of 8-16 bit depth, except RGB. Note that libvmaf only uses luma plane for calculating scores.
Remove parameter psnr.


is this an update of your VS implementation of libvmaf, or an official update ?

so YUV 8-16 bit input supported, but internally everything gets converted to 8-bit before measuring score, correct ?

Update r5.

Note that libvmaf only uses luma plane for calculating scores.


is there a way to specify to score any of the color planes ?


Thanks.

[ChaosKing]

is there a way to specify to score any of the color planes ?


Just extract your plane and pass it as a GRAY clip to vmaf http://www.vapoursynth.com/doc/functions/shuffleplanes.html

c1 = core.std.ShufflePlanes(clip, planes=1, colorfamily=vs.GRAY)

[Iron_Mike]

Just extract your plane and pass it as a GRAY clip to vmaf http://www.vapoursynth.com/doc/functions/shuffleplanes.html

c1 = core.std.ShufflePlanes(clip, planes=1, colorfamily=vs.GRAY)

@CK: thank you, as always. I'll try that.

Edit: this does work with r5.

[Iron_Mike]

@HolyWu:

so the .dll size changed from 16.3MB (r4) to 2.2MB (r5)...

considering that the changelog states input format validation and removal of psnr input param as the only changes, that must have either been the most code written ever for these two simple tasks or u must have hidden a few 8-bit video games as easter eggs in r4 that none of us have discovered... ;-)

that is quite a reduction in filesize...

[HolyWu]

is this an update of your VS implementation of libvmaf, or an official update ?

Neither. Nothing has to be changed in libvmaf. The format restriction is simply for the constants needed for PSNR calculation (https://github.com/Netflix/vmaf/blob/8f41503605e6e55b34945e49cba864697e1ca683/feature/src/psnr_tools.c#L26). Hence I disabled it as only 8 bit and 10 bit are defined.



so YUV 8-16 bit input supported, but internally everything gets converted to 8-bit before measuring score, correct ?

Yes. But they are not clamped to 0-255 range. For example the peak of 10 bit is 1023/4.=255.75, 12 bit is 4095/16.=255.9375, and so on.



considering that the changelog states input format validation and removal of psnr input param as the only changes, that must have either been the most code written ever for these two simple tasks or u must have hidden a few 8-bit video games as easter eggs in r4 that none of us have discovered... ;-)

Unfortunately both guesses are wrong! It's just because I changed to another compiler.

[Iron_Mike]

Yes. But they are not clamped to 0-255 range. For example the peak of 10 bit is 1023/4.=255.75, 12 bit is 4095/16.=255.9375, and so on.


well, it should not be 1023/4... since we're zero index based it's 1024 (total values in 10-bit, 1023+1)/4 and/or 4096 (total values in 12-bit, 4095+1)/16 which both result in 256, which is in 8-bit the value of 255 (256-1)

or what am I missing in your example ?

[Iron_Mike]

Neither. Nothing has to be changed in libvmaf. The format restriction is simply for the constants needed for PSNR calculation (https://github.com/Netflix/vmaf/blob/8f41503605e6e55b34945e49cba864697e1ca683/feature/src/psnr_tools.c#L26). Hence I disabled it as only 8 bit and 10 bit are defined.


yeah, the var u define for *peak needs to accommodate the zero index... peak in 8-bit is always 255, meaning 256 total values starting at 0

u need to adjust +/- 1 when u go from bit values 0-255/0-1023/0-4095 to number of distinct values per bit-depth 256 (2^8)/1024 (2^10)/4096(2^12)

[HolyWu]

yeah, the var u define for *peak needs to accommodate the zero index... peak in 8-bit is always 255, meaning 256 total values starting at 0

I didn't define it. It's in libvmaf. So you are implying that Netflix does it wrong. :devil:



u need to adjust +/- 1 when u go from bit values 0-255/0-1023/0-4095 to number of distinct values per bit-depth 256 (2^8)/1024 (2^10)/4096(2^12)

Actually I don't totally agree with you. How come the value needs to be +1 to make the range 0-256? The MSU page (http://www.compression.ru/video/quality_measure/info.html#ssim) also describes how the peak value is defined in the same way.


Maximum value of a color component is calculated in the same way as for PSNR:
videoMax = 255 for 8 bit color components
videoMax = 255 + 3/4 for 10 bit color components
videoMax = 255 + 63/64 for 14 bit color components
videoMax = 255 + 255/256 for 16 bit color components

[Iron_Mike]

I didn't define it. It's in libvmaf. So you are implying that Netflix does it wrong. :devil:

sorry, I thought it was a link to your VS VMAF github... ;-)




Actually I don't totally agree with you. How come the value needs to be +1 to make the range 0-256? The MSU page (http://www.compression.ru/video/quality_measure/info.html#ssim) also describes how the peak value is defined in the same way.

there is no range 0-256 in common bit depths.

8-bit is 0-255, with a total of 256 distinct values: 2^8 = 256

the bit of code that you linked to, converts incorrectly. there's various ways to do this, but here's the most simple way, very easy to follow:

the code tries to convert from 10-bit or 12-bit to 8-bit, at least that how it appears.

approach #1:

(1) take any 10/12 bit value, e.g 912 in 10-bit
(2) normalize the value: 912+1/10^2 = 0.8916015625 --> meaning 89.16% stimuli in that channel/plane
(3) convert to 8-bit: (2^8-1)*0.8916015625 = 227.3583984375, rounded to int would be 227

run the same example with 0 or 1023/4095 (which would be the extreme case scenarios) and u see it works just fine...



and here's an excerpt from our Russian buddies MSU page:

In MSU VQMT there are four PSNR implementations. "PSNR" and "APSNR" use the correct way of PSNR calculation and take maximum possible absolute value of color difference as MaxErr. However, this way of calculation gives an unpleasant effect after color depth conversion. If color depth is simply increased from 8 to 16 bits, the "PSNR" and "APSNR" will change, because MaxErr should change according to maximum possible absolute value of color difference (255 for 8 bit components and 255 + 255/256 for 16 bit components). Thus, "PSNR (256)" and "APSNR (256)" are implemented. They would not change because they use upper boundary of color difference as MaxErr. The upper boundary is 256. This approach is less correct, but it is used often because it is fast. Here are the rules of MaxErr definition:

"PSNR" and "APSNR" – MaxErr varies on color components bits usage:
255 for 8 bit components
255 + 3/4 for 10 bit components
255 + 63/64 for 14 bit components
255 + 255/256 for 16 bit components
100 for L component of LUV color space

[HolyWu]

Firstly, that doesn't apply in the metric calculation. We are not really doing format conversion like YUV420P16 to YUV420P8. The purpose of the normalization is not to map every pixel value into exact 0-255 range. As I said before, the normalized value is stored in floating-point and not integer so there is no rounding issue.

Secondly, taking your example, 227 in 8 bit converts to 10 bit with zero LSBs is 908. (908+1)/(2^10)*255=226.3623. How come 227 converts to 10 bit then converts back to 8 bit becomes 226.

Lastly, as you already seen on the MSU page, that approach is less correct. So why bother?

[Iron_Mike]

Secondly, taking your example, 227 in 8 bit converts to 10 bit with zero LSBs is 908. (908+1)/(2^10)*255=226.3623. How come 227 converts to 10 bit then converts back to 8 bit becomes 226.

because u're taking the rounded value of 227, which obviously already lost precision... use the full float value and it will convert to 10-bit fine... there are apps - like many color apps - that only take integer R|G|B values, hence u then have to round

and once u round and u convert back/forth obviously the color drifts...


Lastly, as you already seen on the MSU page, that approach is less correct. So why bother?

I just mentioned it b/c I just quickly glanced over ur link and thought it was your VS VMAF code that u use to downconvert all input to 8-bit in VS VMAF before passing it into libvmaf...

[Iron_Mike]

just ran a test on r5 using a 16-bit source and compared VMAF score to r4 (where I had to scale the clips to 10-bit) and results are very, very close... small differences in 3rd or 4th decimal place...

[HolyWu]

just ran a test on r5 using a 16-bit source and compared VMAF score to r4 (where I had to scale the clips to 10-bit) and results are very, very close... small differences in 3rd or 4th decimal place...

No surprise to me at all. Bear in mind that VMAF is a perceptual quality assessment and takes the viewing distance(3H for 1080p and 1.5H for 4K) into account. The most observable difference by human eyes between 8 bit and 10 bit in video encoding is banding. But do human eyes be able to observe significant difference between 10 bit and 16 bit from the mentioned viewing distance?

[Iron_Mike]

No surprise to me at all. Bear in mind that VMAF is a perceptual quality assessment and takes the viewing distance(3H for 1080p and 1.5H for 4K) into account. The most observable difference by human eyes between 8 bit and 10 bit in video encoding is banding. But do human eyes be able to observe significant difference between 10 bit and 16 bit from the mentioned viewing distance?

yeah, I would say 12-bit at most is upper limit... I've compared same image on a 12-bit post prod panel to the standard 10-bit panels, tough to see any differences, and if if so, then it could also be b/c of the cal or the display tech

[WorBry]

is there a way to specify to score any of the color planes ?


Just extract your plane and pass it as a GRAY clip to vmaf http://www.vapoursynth.com/doc/functions/shuffleplanes.html

c1 = core.std.ShufflePlanes(clip, planes=1, colorfamily=vs.GRAY)

Incidentally, I see same question was asked on the Netflix vmaf github:

https://github.com/Netflix/vmaf/issues/100

That was 18 months ago - I wonder when they will get around to 'extending to chroma' and how they will go about it?

[poisondeathray]

Are the results supposed to be identical to vmafossexec.exe ?

I'm getting some discrepancies, but the vmafossexec I'm using is based on 1.3.15, vapoursynth-vmaf 1.3.14 . Not sure if this accounts for the differences

I understand ffmpeg libvmaf is going to be a bit different from the discussion above, but's a 3rd different

[HolyWu]

Are the results supposed to be identical to vmafossexec.exe ?

I'm getting some discrepancies, but the vmafossexec I'm using is based on 1.3.15, vapoursynth-vmaf 1.3.14 . Not sure if this accounts for the differences

Did you use the same pooling method? VS-VMAF defaults to harmonic mean while vmafossexec defaults to arithmetic mean.

[poisondeathray]

Did you use the same pooling method? VS-VMAF defaults to harmonic mean while vmafossexec defaults to arithmetic mean.

No . Nice catch . I'll verify if both AM's match and post back if they don't.

Is there a switch for vmafossexec to use harmonic mean? There doesn't seem to be in the help

The blog says "our empirical results suggest that simple arithmetic mean (AM) is the best way of averaging, in that it yields highest correlation with subjective scores."

But then I saw some issues on the issue tracker that seem to suggest HM is better in some ways.


Any comments or user experiences on HM vs AM ?

HM seems to catch cases were traditional SSIM/PSNR fail better than AM ; it seems to match visually better to my eyes. It's actually the reason I posted now, I'm seeing a flip flop in expected results with vmafossexec AM . I've been using the vapoursynth VMAF for the last few months with HM and it seems to correlate pretty well

[HolyWu]

Is there a switch for vmafossexec to use harmonic mean? There doesn't seem to be in the help

Yes, --pool harmonic_mean. The developer seemed to forget to add this switch in the help.

[poisondeathray]

Awesome thanks

[poisondeathray]

Using raw YUV files produces matching results (vsrawsource in vapoursynth-vmaf vs. vmafossexec)

But using lsmash does not on the compressed files. So now I'm thinking seek issues


EDIT: it turns out it was just a stupid error - wrong path/file !!!

But nice to know that vmafossexec and vapoursynth-vmaf match


I noticed vmafossexec seems to process significantly faster, even when comparing raw yuv compared to to vsrawsource/vmaf . What is the reason for the difference?

Thanks again for the help and plugin

[Rumbah]

Thanks for noticing differences between raw and compressed sources (before your edit ;) ).

I used ffmpeg for vmaf and never checked and it seems it has this problem, too.
I get very low vmaf scores for a compressed input and looking at the logs it seems the b frames are handled wrong resulting in scores for them below 10.

If i use raw sources everything is fine and I get high scores.

[poisondeathray]

Thanks for noticing differences between raw and compressed sources (before your edit ;) ).


Mine was a bonehead mistake; what's your excuse :)

I would double check everything especially source filter

The LSmash version I used passed seek-test.py for the videos, so I rechecked everything and the most simple explanation was the answer


I used ffmpeg for vmaf and never checked and it seems it has this problem, too.
I get very low vmaf scores for a compressed input and looking at the logs it seems the b frames are handled wrong resulting in scores for them below 10.

If i use raw sources everything is fine and I get high scores.


FFMpeg vmaf is buggy for this -

1) for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

2) There is a 8/10 bitdepth issue on the tracker. It's easy too reproduce (esp. the 2nd post) . Vapoursynth-vpy seems ok with this
https://github.com/Netflix/vmaf/issues/211


And even then it does not match for frame accurate, 8bit to 8bit case maybe because of the reasons earlier in the thread

My thinking is vmafossexec is the "official" one, and if vapoursynth-vmaf matches it, that's 2 against 1 anyways

[HolyWu]

Update r6:

Update to libvmaf 1.3.15.
Add csv log format.
Minor optimization.



I noticed vmafossexec seems to process significantly faster, even when comparing raw yuv compared to to vsrawsource/vmaf . What is the reason for the difference?

How significant is it? Any numbers?

My wild guess is that it's because vs-vmaf has to use mutex lock due to the way libvmaf receives frames and that probably incur additional overheads.

[richardpl]

FFMpeg vmaf is buggy for this -

1) for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

2) There is a 8/10 bitdepth issue on the tracker. It's easy too reproduce (esp. the 2nd post) . Vapoursynth-vpy seems ok with this
https://github.com/Netflix/vmaf/issues/211


And even then it does not match for frame accurate, 8bit to 8bit case maybe because of the reasons earlier in the thread

My thinking is vmafossexec is the "official" one, and if vapoursynth-vmaf matches it, that's 2 against 1 anyways

What FUD you share with us? Please STOP!

[poisondeathray]

How significant is it? Any numbers?

My wild guess is that it's because vs-vmaf has to use mutex lock due to the way libvmaf receives frames and that probably incur additional overheads.

Thanks for the update

vmafossexec is about 3x faster. ~ 15-16fps vs. ~5fps . Setup was the same with the raw yuv files on different HDD's . Similar results with r6 for speed

[poisondeathray]

What FUD you share with us? Please STOP!



Which part do you believe is FUD ? Can you expand on that a bit ? Maybe it's just user error

Here are the results of an 8bit raw YUV test with Harmonic Mean (raw YUV , so in theory no potential seek issues. 8bit to 8bit, so no 8/10 bit FFmpeg libvmaf issue)

vmafossexec 95.3836
vapoursynth-vmaf 95.3836
ffmpeg libvmaf 95.38745502791545

The 8bit/10bit FFMpeg libvmaf issue on the tracker is easily reproducible, and you can try different dithering algorithms, comparing at 10 vs. 8, different combinations

[HolyWu]

vmafossexec is about 3x faster. ~ 15-16fps vs. ~5fps . Setup was the same with the raw yuv files on different HDD's . Similar results with r6 for speed

I can't replicate that huge difference. Where did you download vmafossexec? What's the resolution and length of your test video?

[poisondeathray]

I can't replicate that huge difference. Where did you download vmafossexec? What's the resolution and length of your test video?

From AppVeyor . I think this is the same one 2 days ago
https://ci.appveyor.com/project/li-zhi/vmaf/builds/28022740/artifacts

1920x1080 , 8bit 4:2:0, 2400 frames

(And FFMpeg libvmaf was about the same as vpy-vmaf in terms of speed for the same files)

[HolyWu]

From AppVeyor . I think this is the same one 2 days ago
https://ci.appveyor.com/project/li-zhi/vmaf/builds/28022740/artifacts

1920x1080 , 8bit 4:2:0, 2400 frames

Still doesn't make much difference.


start /b /high /wait vmafossexec yuv420p 1920 1080 E:\ref.yuv E:\dis.yuv model/vmaf_v0.6.1.pkl --log E:\vmaf1.xml --pool harmonic_mean

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1662" timeTaken="51.6417" />



ref = core.raws.Source(r'E:\ref.yuv', 1920, 1080)
dis = core.raws.Source(r'E:\dis.yuv', 1920, 1080)
clip = core.vmaf.VMAF(ref, dis, log_path=r'E:\vmaf2.xml', pool=1)

start /b /high /wait vspipe -p test.vpy .

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1263" timeTaken="51.8453" />

[poisondeathray]

I/O bottleneck ? Are your YUV files on separate physical drives ? or SSD ? I notice it was slower on the same drive . Otherwise I can't explain it. I'll check again but other physical files were about the same 3x speed too . Unless my vapoursynth setup and ffmpeg libvmaf are both slower for some reason ?

My results are the same, so it's unlikely to be some sort of other error like wrong file

[poisondeathray]

It looks like both your files are in E:\

I have "ref" on an SSD and "dis" on a HDD

[HolyWu]

Tested again with two files on separate SSD drives. It's marginally faster but still not a huge difference.

vmafossexec

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.3853" timeTaken="50.5524" />


vs-vmaf

<params model="vmaf_v0.6.1.pkl" scaledWidth="1920" scaledHeight="1080" subsample="1" num_bootstrap_models="0" bootstrap_model_list_str="" />
<fyi numOfFrames="525" aggregateVMAF="93.5213" poolMethod="harmonic_mean" execFps="10.1431" timeTaken="51.7592" />

[poisondeathray]

Thank you for investigating it, I cannot explain why I observe those results . I'll look into it again in the morning and report back if I find anything

[poisondeathray]

The difference was other metrics in the calculation , eg. ssim , ms-ssim also being calculated with vmaf . Same speed when everything is equivalent

Another bonehead mistake. Sorry for wasting your time :(

[richardpl]

I can not reproduce finding you have. Perhaps you use libvmaf filter in invalid ways? Also non-8bit bug in libvmaf filter have been already fixed.

[poisondeathray]

I can not reproduce finding you have. Perhaps you use libvmaf filter in invalid ways? Also non-8bit bug in libvmaf filter have been already fixed.

Great news

I repeated and FFmpeg libvmaf results match for the 8bit case



(1) is still a potential issue -

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times

-This one is easier to demonstrate with PSNR, because it can measure "perfect", while VMAF cannot.
-An unsuspecting user, that just follows the instructions and examples can get the wrong results

[MasterNobody]

(1) is still a potential issue -

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times
Have you tried to use "-vsync 0" option? With it ffmpeg should output all frames without any duplicated and dropped frames.

[poisondeathray]

Have you tried to use "-vsync 0" option? With it ffmpeg should output all frames without any duplicated and dropped frames.

Yes;

In some cases forcing -r with that might work (cases where it doesn't work - timestamp jitter) . In other cases it does not. Sometimes you have to reset PTS along with it . Sometimes remuxing helps (same container) - the timebase differences and jitter timestamps between MP4 and MKV container is a common problem that causes wrong values in the metrics

Or just use avisynth or vapoursynth with a frame accurate source filter; that 100% works if you use assumefps and the frames are aligned. That resets all the timestamps and jitter, both ref and distorted will be equal

[richardpl]

Again FUD spreading from you! You also never provided commands you use and input files.

[poisondeathray]

Again FUD spreading from you! You also never provided commands you use and input files.


Again, which part do you believe is FUD ? This is a known issue with FFmpeg.


(1) is still a potential issue -

for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times





Here are 2 examples. I chose PSNR, because it's easy to demonstrate the issue (should get inf). In both cases "0" is the reference (but it shouldn't matter, since they are "lossless" equivalents)

If you were to follow the docs/ instructions, without making adjustments - you will get the wrong values
https://ffmpeg.org/ffmpeg-filters.html#psnr

*In both cases (and other cases, where there are timestamp issues, container timebase differences, "off" timecodes etc...) , if you just use avisynth or vapoursynth, it works . If the source filter you are using reads "off" frame rate (e.g. 23.976 vs. 24000/1001 exactly, or similar, you need to use AssumeFPS to make them equivalent if using FFmpeg; as long as the frames are aligned with a frame accurate source filter, it will work and give correct values)

https://www.mediafire.com/file/rd2zrbs6xq18b8f/ffmpeg_psnr_example1.7z/file
https://www.mediafire.com/file/c9iebz72cwxi97r/ffmpeg_psnr_example2.7z/file

example 1)
In the first case both are "23.976" by mediainfo (or 23.98 by ffmpeg) but there are tbn container differences . This is known as MP4 and MKV express timestamps differently

Also notice the non zero start time in "0.mkv" and duration difference because of it

ffmpeg -i 0.mkv
Duration: 00:00:05.01, start: 0.084000, bitrate: 38321 kb/s
Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080
[SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

ffmpeg -i 1.mp4
Duration: 00:00:04.92, start: 0.000000, bitrate: 74719 kb/s
Stream #0:0(und): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), y
uv420p, 1920x1080 [SAR 1:1 DAR 16:9], 74723 kb/s, 23.98 fps, 23.98 tbr, 24k tbn,
47.95 tbc (default)


If you were to follow the docs, you will get wrong values


ffmpeg -i 1.mp4 -vcodec rawvideo -vf "movie=0.mkv,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=stats1.log" [out]" -f rawvideo -y /NUL


If you were to follow the docs that specific different containers, you will still get wrong values

ffmpeg -i 1.mp4 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -


-i avs , or -i vpy works, or forcing the framerate -r 24000/1001 before each -i works in this particular case (but not necessarily others)


ffmpeg -r 24000/1001 -i 1.mp4 -r 24000/1001 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -



example 2)
ffmpeg -i 0.mkv
Duration: 00:00:10.26, start: 0.000000, bitrate: 18583 kb/s
Stream #0:0(eng): Video: h264 (High), yuv420p(progressive), 1920x1080 [SAR 1
:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)

ffmpeg -i 1.mp4
Duration: 00:00:10.26, start: 0.000000, bitrate: 120532 kb/s
Stream #0:0(eng): Video: h264 (High 4:4:4 Predictive) (avc1 / 0x31637661), y
uv420p, 1920x1080 [SAR 1:1 DAR 16:9], 120539 kb/s, 23.98 fps, 23.98 tbr, 24k tbn
, 47.95 tbc (default)

Same issue with container tbn difference, but start time is zero in this case

If you were to follow the docs, you will get wrong values

ffmpeg -i 1.mp4 -vcodec rawvideo -vf "movie=0.mkv,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=stats2.log" [out]" -f rawvideo -y /NUL


If you were to follow the docs that specific different containers, you will still get wrong values

ffmpeg -i 1.mp4 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -


If you were to follow the docs that specific different containers, and use the -r force framerate adjustment that worked in example #1 above, you will still get wrong values

ffmpeg -r 24000/1001 -i 1.mp4 -r 24000/1001 -i 0.mkv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -


-i avs, or -vpy works

[richardpl]

I downloaded both your uploaded files.
First file contains two video files that have same decoded video frames hashes. And for this files example in documentation about using psnr with different containers works just fine (I dunno what was your point in trying command that works only with same containers and with nice timestamps). Perhaps you used wrong command or faulty build?

I used this command: ffmpeg -i 0.mkv -i 1.mp4 -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr=shortest=1" -f null -

Extra shortest is for cases when user supplied videos of different duration.

Second file contain two video files with different video frames hashes. One file have duplicated first two frames with normal timestamps. No point in using psnr on them. If you have same results with vpy or avs then there is something seriously broken there with them :)

[poisondeathray]

I downloaded both your uploaded files.
First file contains two video files that have same decoded video frames hashes. And for this files example in documentation about using psnr with different containers works just fine (I dunno what was your point in trying command that works only with same containers and with nice timestamps). Perhaps you used wrong command or faulty build?

I used this command: ffmpeg -i 0.mkv -i 1.mp4 -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr=shortest=1" -f null -

Extra shortest is for cases when user supplied videos of different duration.

Second file contain two video files with different video frames hashes. One file have duplicated first two frames with normal timestamps. No point in using psnr on them. If you have same results with vpy or avs then there is something seriously broken there with them :)


Shortest works for the 1st one, as does forcing -r .





The 2nd one has open GOP's with leading b's . Different decoders can handle open GOP's differently. But the same decoder should handle it consistently.

In fact, FFmpeg was used to create 1.mp4 from 0.mkv. And the decoded elementary streams are identical


ffmpeg -i 0.mkv -c:v rawvideo 0.yuv

ffmpeg -i 1.mp4 -c:v rawvideo 1.yuv

ffmpeg -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 1.yuv -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 0.yuv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -


Or, if you use avs or vpy input, works too


This is why you need to make adjustments for some types of files. The docs don't cover these cases or others like them; if a user just followed the instructions, they'd get the wrong values

[richardpl]

No, 2nd one is completely invalid, and they are not identical.

[poisondeathray]

No, 2nd one is completely invalid, and they are not identical.

Explain.

How is that possible, when 1.mp4 was created from 0.mkv

ffmpeg -i 0.mkv -c:v libx264 -qp 0 1.mp4


Also, rawvideo says they are identical


ffmpeg -i 0.mkv -c:v rawvideo 0.yuv

ffmpeg -i 1.mp4 -c:v rawvideo 1.yuv

ffmpeg -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 1.yuv -pix_fmt yuv420p -s 1920x1080 -r 24000/1001 -i 0.yuv -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

[richardpl]

They are not identical, and you know it as you created such files.

There is visible gap in pts of first and second frame of 0.mkv. Making it not CFR but VFR. rawvideo will just make CFR from VFR.

[poisondeathray]

They are not identical, and you know it as you created such files.

There is visible gap in pts of first and second frame of 0.mkv. Making it not CFR but VFR. rawvideo will just make CFR from VFR.


They are identical.

It's a lossless encode -qp 0 , encoded by FFmpeg. The commandlines are posted. Do you see conceptually the problem ? Someone uses FFmpeg to encode a "lossless" video in the same colorspace/sampling/bit depth - they expect it to be "lossless", they expect PSNR to say "lossless"

It's not an encoding issue, since the YUV streams match, encoded frames are identical.


It should have been CFR, because it was cut from a BD. BD's are CFR only.

This has open GOP with 2 leading b's. Decoders are supposed to either duplicate the leading b's or drop them (risk of desync). Eitherway , it should be consistently handled



If you go back, that PTS example for different container in the documentation is new - It wasn't there even 2 months ago if I look at my local documentation for a FFmpeg build. That's great, the docs need to be improved with more examples


That's why I said -

(1) is still a potential issue -

for some types of input formats (seek issues with some open gop files, and those with "off" timecodes, you might need to reset PTS) - you get mismatched frames. Or use an indexed frame accurate input like avisynth or vapoursynth into ffmpeg

That you need to make adjustments when there is jitter in timestamps, or slightly different container timebases, or slightly "off" timestamps such as non zero start times


If you think that's "FUD", we wouldn't have these types of issues . Everything would just "work" without having to do these workarounds. The warning is so people don't get "burned"

Note that 1st example was a lossless encode from "0.mkv" too. The lengths are actually the same in terms of framecount. The -shortest or -r workaround are not mentioned in the docs. People expect things to just work; a casual user would have been "burned" there too

This illustrates some of the issues with how FFmpeg handles this calculation. There should be a way to reset all the PTS, you can improve it farther.

[richardpl]

It's not an encoding issue, since the YUV streams match, encoded frames are identical.


It should have been CFR, because it was cut from a BD. BD's are CFR only.

This has open GOP with 2 leading b's. Decoders are supposed to either duplicate the leading b's or drop them (risk of desync). Eitherway , it should be consistently handled


You are terribly mistaken. Closely inspect timestamp and you will see that 2 frames are completely missing from 0.mkv
There is gap created by ffmpeg decoder.
It works with vpy, avs just by pure luck because they duplicate right frames in those gaps. fps filter duplicates currently always first frame encountered so far across pts gap, which does not mach with frame hashes in 1.mp4

See for yourself with -vf showinfo,trim=end_frame=5 filter on both files.
The -r option is hacky beacuse it tries to round to nearest frame but is slightly off for some strange reason.

[poisondeathray]

You are terribly mistaken. Closely inspect timestamp and you will see that 2 frames are completely missing from 0.mkv
There is gap created by ffmpeg decoder.
It works with vpy, avs just by pure luck because they duplicate right frames in those gaps. fps filter duplicates currently always first frame encountered so far across pts gap, which does not mach with frame hashes in 1.mp4

See for yourself with -vf showinfo,trim=end_frame=5 filter on both files.
The -r option is hacky beacuse it tries to round to nearest frame but is slightly off for some strange reason.


Those 2 are leading b's frames from open GOP. The ffmpeg AVC libavcodec decoder places 2 duplicate frames there as "placeholder frames" to keep sync (had it been a real example with audio). This occurs with DVD / mpeg2 as well . Same handling with leading b's open GOP

(Other decoders might handle it differently; the point is it should be consistent with the same decoder; if it drops, then drop in all cases, not just some of the time or depending on the phase of the moon. Most AVC decoders will keep them as duplicates)

But that's why the encode has them as well, and the decoded YUV frames are identical, the 1st 2 are duplicates from the open gop . Same framecount. Frames are the same . Input = output. ( The encode was made from decoding first AVC stream with open GOP; and the decode of the lossless encode is also decoding AVC but they are not longer leading b's, they've been decoded and reencoded - both are consistent and have those 2 "placeholder" frames)

The problem is the FFmpeg timestamp handling . At the very least , you have to admit it's inconsistent behaviour here. On one hand FFmpeg places those b-frames. Physically decodes and encodes therm. On the other hand, when doing the calculations it drops them

Just take a step back and look at the big picture for a second: Someone supposedly encodes a lossless video with ffmpeg and it's "not lossless" according to FFmpeg PSNR . Or -
FFmpeg VMAF is not going to match vmafossexec or vapoursynth-vmaf in cases like this - unless you decode to raw YUV first for use in FFMpeg .

[Shevach]

Yes, looking at the models page suggests that.. Looking at the linked slideshow there, it also seems that for proper analysis, you have to upscale to 4K.

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

Dear experts

i am faced with same problem while running libvmaf by means of ffmpeg:

.\ffmpeg.exe -hide_banner -r 60 -s 1920x1080 -i test60fps.yuv -r 60 -s 1920x1080 -i ref.yuv -vframes 2372 -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl:log_path=test30Mbps60fps.xml" -report -f null -

...
frame= 81 fps= 19 q=-0.0 size=N/A time=00:00:03.24 bitrate=N/A speed=0.761x
frame= 89 fps= 19 q=-0.0 size=N/A time=00:00:03.56 bitrate=N/A speed=0.748x
frame= 98 fps= 19 q=-0.0 size=N/A time=00:00:03.92 bitrate=N/A speed=0.74x
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.

If i downscale raw video resolution to 720x480 the ffmpeg succeeds to compute vmaf scores for all frames.

Task Manager does not reveal any problem (the memory usage does not exceed 5%).

How can i resolve this issue?