VMAF - Video Multi-Method Assessment Fusion

HolyWu · 7th November 2018, 11:41

https://github.com/HomeOfVapourSynth...ourSynth-VMAF/

poisondeathray · 7th November 2018, 16:11

Thanks

lansing · 7th November 2018, 16:37

I'm confused, what is this filter supposed to do?

ChaosKing · 7th November 2018, 16:52

It calculates a score.... You have 2 clips, the score tells you by how much the the seconds clips differs from the first one. The VMAF algo tries take the human perception into account. https://en.wikipedia.org/wiki/Video_...essment_Fusion

Selur · 7th November 2018, 20:07

Nice thanks!

ChaosKing · 8th November 2018, 11:11

Also available via vsrepo now

ifb · 9th November 2018, 02:56

Very timely for me. Thanks.

I've had mixed results getting yuv422p10le to work correctly with vmafossexec and/or ffmpeg builds with libvmaf. That's probably my fault for trying on Windows, but I was too lazy to try on a Linux VM at the time.

poisondeathray · 13th November 2018, 18:23

Request: is it possible to print out the aggregate psnr, ssim, ms_ssim scores ? (Currently it's only aggregate vmaf)

HolyWu · 14th November 2018, 12:21

Update r2.

Scale 10-bit pixel values to 8-bit range for correct score calculation.
Use stricter linear frame request since VMAF score will change if frame order is different.
Report aggregate PSNR, SSIM, and MS-SSIM scores in addition to VMAF score.

edcrfv94 · 14th November 2018, 15:09

Is it possible to registered to frame props like mvsfunc PlaneStatistics?

ChaosKing · 14th November 2018, 15:51

Quote:

Originally Posted by edcrfv94

Is it possible to registered to frame props like mvsfunc PlaneStatistics?

This would be awesome.

HolyWu · 15th November 2018, 03:43

Quote:

Originally Posted by edcrfv94

Is it possible to registered to frame props like mvsfunc PlaneStatistics?

Not possible with the exposed API of libvmaf. Only when all frames are deliverd and processed does the library print the aggregate scores and optionally write the log file. There is no way to access the library's internal data to get the per-frame score and add that to the frame property.

lansing · 19th January 2019, 09:30

Can you add an "average score" for each metric at the end of the log?

HolyWu · 19th January 2019, 15:42

Quote:

Originally Posted by lansing

Can you add an "average score" for each metric at the end of the log?

It's already there, albeit not at the end. For example a xml log:

Code:

<?xml version="1.0" encoding="ISO-8859-1"?>
<VMAF version="1.3.7">
<params subsample="1" scaledHeight="1080" scaledWidth="1920" model=""/>
<fyi execFps="2.3972" aggregateMS_SSIM="0.997629" aggregateSSIM="0.998861" aggregatePSNR="48.5527" aggregateVMAF="98.452" numOfFrames="6"/>
<frames>
<frame vmaf="95.1147" vif_scale3="0.999067" vif_scale2="0.997998" vif_scale1="0.986572" vif_scale0="0.711273" ssim="0.998833" psnr="48.1752" ms_ssim="0.997502" motion2="0" adm2="0.990135" frameNum="0"/>
<frame vmaf="98.0908" vif_scale3="0.999122" vif_scale2="0.998097" vif_scale1="0.987361" vif_scale0="0.725121" ssim="0.998876" psnr="48.5654" ms_ssim="0.997635" motion2="2.08641" adm2="0.991341" frameNum="1"/>
<frame vmaf="98.6133" vif_scale3="0.999151" vif_scale2="0.998111" vif_scale1="0.987686" vif_scale0="0.737886" ssim="0.998892" psnr="48.9502" ms_ssim="0.997769" motion2="2.4176" adm2="0.991782" frameNum="2"/>
<frame vmaf="99.0617" vif_scale3="0.999089" vif_scale2="0.998038" vif_scale1="0.98654" vif_scale0="0.71651" ssim="0.99883" psnr="48.3169" ms_ssim="0.997532" motion2="2.96657" adm2="0.990675" frameNum="3"/>
<frame vmaf="100" vif_scale3="0.999108" vif_scale2="0.998094" vif_scale1="0.987838" vif_scale0="0.738668" ssim="0.998885" psnr="48.9485" ms_ssim="0.997766" motion2="3.60763" adm2="0.991847" frameNum="4"/>
<frame vmaf="100" vif_scale3="0.999115" vif_scale2="0.99804" vif_scale1="0.986751" vif_scale0="0.720485" ssim="0.99885" psnr="48.371" ms_ssim="0.997572" motion2="4.2859" adm2="0.991102" frameNum="5"/>
</frames>
</VMAF>

lansing · 19th January 2019, 16:04

Oh, I was using the json format and it doesn't have it

lansing · 20th January 2019, 05:50

So I finally have time to play with this plugin, it's very good tool to help find optimal encode settings for my contents.

My test clip is a 1000 frames of 1440x1080 anime with no grain and very little motion. Here are the comparison between x264 and x265 I made with vmaf :

Code:

                                   Aggregate VMAF Percentage
source                                   98.8374

x264 300 kb/s medium animation           79.9185
x265 300 kb/s medium                     85.6454
x265 300 kb/s slower                     86.9443

x264 2500 kb/s medium animation          95.1616
x265 2500 kb/s medium                    94.9675

x264 CRF 25 medium animation             91.3914
x265 CRF 25 medium                       91.1363

x264 CRF 18 medium animation             96.1513
x265 CRF 21 medium                       93.6972

x264 CRF 18 medium animation             96.1513
x264 CRF 18 slower animation             96.3279

According to the FAQ, comparing the source clip to itself won't gives 100% score, so 98.8% here is the highest quality for this clip.

For encoding in low bitrate, x265 clearly wins. But for high/transparent encoding, it seems that x264 is still better if encode in same bitrate or same CRF. I don't have good 4k contents to see how that goes.

I have heard people said that x264 crf 18 is equivalent to x265 crf 21 so I did a comparison on those too, and the score shows that there is a difference.

On the last comparison I did it between two different presets. So with high bitrate encoding for my content, using slower preset is just a waste of time with insignificant amount of gains. It only make sense to use it on low bitrate encodes.

Boulder · 20th January 2019, 21:23

Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself.

gonca · 20th January 2019, 22:48

Quote:

Originally Posted by Boulder

Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself.

I think it is the screen, otherwise the --phone-model switch makes no sense

Boulder · 21st January 2019, 04:46

Quote:

Originally Posted by gonca

I think it is the screen, otherwise the --phone-model switch makes no sense

Yes, looking at the models page suggests that.. Looking at the linked slideshow there, it also seems that for proper analysis, you have to upscale to 4K.

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Code:

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

HolyWu · 21st January 2019, 06:45

Quote:

Originally Posted by Boulder

EDIT: with video upscaled to 4K, it runs out of memory quite often. Even with core.max_cache_size = 1024, the usage jumps to over 8GB quite fast and then the errors appear. I have 16GB on the machine so it's really not using all the memory.

Code:

Start calculating VMAF score...
Script exceeded memory limit. Consider raising cache size.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
error: aligned_malloc failed for data_buf.
Exec FPS: 3.574859
VMAF score (harmonic_mean) = nan

Can't reproduce. Provide the exact script you used.

7th November 2018, 11:41	#1 \| Link
HolyWu Registered User Join Date: Aug 2006 Location: Taiwan Posts: 391	VMAF - Video Multi-Method Assessment Fusion https://github.com/HomeOfVapourSynth...ourSynth-VMAF/

7th November 2018, 16:52	#4 \| Link
ChaosKing Registered User Join Date: Dec 2005 Location: Germany Posts: 1,751	It calculates a score.... You have 2 clips, the score tells you by how much the the seconds clips differs from the first one. The VMAF algo tries take the human perception into account. https://en.wikipedia.org/wiki/Video_...essment_Fusion __________________ AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK \|\| VapourSynth Database \|\| https://github.com/avisynth-repository

7th November 2018, 20:07	#5 \| Link
Selur Registered User Join Date: Oct 2001 Location: Germany Posts: 7,034	Nice thanks! __________________ Hybrid here in the forum, homepage

8th November 2018, 11:11	#6 \| Link
ChaosKing Registered User Join Date: Dec 2005 Location: Germany Posts: 1,751	Also available via vsrepo now __________________ AVSRepoGUI // VSRepoGUI - Package Manager for AviSynth // VapourSynth VapourSynth Portable FATPACK \|\| VapourSynth Database \|\| https://github.com/avisynth-repository

14th November 2018, 12:21	#9 \| Link
HolyWu Registered User Join Date: Aug 2006 Location: Taiwan Posts: 391	Update r2. Scale 10-bit pixel values to 8-bit range for correct score calculation. Use stricter linear frame request since VMAF score will change if frame order is different. Report aggregate PSNR, SSIM, and MS-SSIM scores in addition to VMAF score.

7th November 2018, 16:11	#2 \| Link
poisondeathray Registered User Join Date: Sep 2007 Posts: 5,221	Thanks

7th November 2018, 16:37	#3 \| Link
lansing Registered User Join Date: Sep 2006 Posts: 1,645	I'm confused, what is this filter supposed to do?

9th November 2018, 02:56	#7 \| Link
ifb Registered User Join Date: Dec 2009 Posts: 72	Very timely for me. Thanks. I've had mixed results getting yuv422p10le to work correctly with vmafossexec and/or ffmpeg builds with libvmaf. That's probably my fault for trying on Windows, but I was too lazy to try on a Linux VM at the time.

13th November 2018, 18:23	#8 \| Link
poisondeathray Registered User Join Date: Sep 2007 Posts: 5,221	Request: is it possible to print out the aggregate psnr, ssim, ms_ssim scores ? (Currently it's only aggregate vmaf)

14th November 2018, 15:09	#10 \| Link
edcrfv94 Registered User Join Date: Apr 2015 Posts: 84	Is it possible to registered to frame props like mvsfunc PlaneStatistics?

19th January 2019, 09:30	#13 \| Link
lansing Registered User Join Date: Sep 2006 Posts: 1,645	Can you add an "average score" for each metric at the end of the log?

19th January 2019, 16:04	#15 \| Link
lansing Registered User Join Date: Sep 2006 Posts: 1,645	Oh, I was using the json format and it doesn't have it

20th January 2019, 05:50	#16 \| Link
lansing Registered User Join Date: Sep 2006 Posts: 1,645	So I finally have time to play with this plugin, it's very good tool to help find optimal encode settings for my contents. My test clip is a 1000 frames of 1440x1080 anime with no grain and very little motion. Here are the comparison between x264 and x265 I made with vmaf : Code: Aggregate VMAF Percentage source 98.8374 x264 300 kb/s medium animation 79.9185 x265 300 kb/s medium 85.6454 x265 300 kb/s slower 86.9443 x264 2500 kb/s medium animation 95.1616 x265 2500 kb/s medium 94.9675 x264 CRF 25 medium animation 91.3914 x265 CRF 25 medium 91.1363 x264 CRF 18 medium animation 96.1513 x265 CRF 21 medium 93.6972 x264 CRF 18 medium animation 96.1513 x264 CRF 18 slower animation 96.3279 According to the FAQ, comparing the source clip to itself won't gives 100% score, so 98.8% here is the highest quality for this clip. For encoding in low bitrate, x265 clearly wins. But for high/transparent encoding, it seems that x264 is still better if encode in same bitrate or same CRF. I don't have good 4k contents to see how that goes. I have heard people said that x264 crf 18 is equivalent to x265 crf 21 so I did a comparison on those too, and the score shows that there is a difference. On the last comparison I did it between two different presets. So with high bitrate encoding for my content, using slower preset is just a waste of time with insignificant amount of gains. It only make sense to use it on low bitrate encodes.

20th January 2019, 21:23	#17 \| Link
Boulder Pig on the wing Join Date: Mar 2002 Location: Finland Posts: 5,610	Did I understand correctly that model 0 should be used if the tested material is 1080p or less? At first I thought that it meant the viewing device but after reading the FAQ, it looks like it's the content itself. __________________ And if the band you're in starts playing different tunes I'll see you on the dark side of the Moon...

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode