FFMpeg VMAF filter

[poisondeathray]

https://ffmpeg.org/ffmpeg-filters.html#libvmaf
https://github.com/Netflix/vmaf


(8/12/17) VMAF is now included as a filter in FFmpeg main branch, and can be configured using: ./configure --enable-libvmaf.

Anyone compile or use this yet in ffmpeg?

Is it "complete" yet ?

Experiences /observations / comments ?

[Jamaika]

Super. Will there be any examples for correct usage for all images and video codecs?
Is the filter VMAF only linux like neftix?

[WorBry]

I'd like to check out this FFMPEG filter, ideally on Windows 10; I'm running Zeranoe Static Windows 64-bit nightly builds currently.

If this is possible, how do you install the pre-requisite libvmaf library ?

I see there is also now a VapourSynth VMAF plugin:

https://forum.doom9.org/showthread.php?t=175862&highlight=VMAF+vapoursynth

But I would prefer to use commandline FFMPEG.

[WorBry]

Thanks, but I still need to install and point to the libvmaf library (model) do I not ?

http://i.imgur.com/cdMFuec.png (https://imgur.com/cdMFuec)

How to do that ?

[Wolfberry]

Place the model file in the same folder and try
ffmpeg -i main -i ref -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl" -f null -

[WorBry]

Ah, OK, so no need to compile/install anything - I just copy the vmaf_v0.6.1.pkl file from here ?:

https://github.com/Netflix/vmaf/tree/master/model

Tried that, but still:

http://i.imgur.com/S6JmiOh.png (https://imgur.com/S6JmiOh)

[Wolfberry]

I believe you need *.pkl and *.pkl.model in the same path as ffmpeg, not just the pkl file.

[WorBry]

Nope, it's still not reading vmaf_v0.6.1.pkl.

I've got the VapourSynth vmaf filter working though so I'll use that for now.

I really just wanted to see how the VMAF and SSIM measures compare for x264 encoded over a range of CRF values, using a high quality uncompressed source as reference. And also how the LibVMAF derived SSIM 'aggregate' score tallies with the standard FFMPEG SSIM metric 'average' score. This series shows quite a difference:

https://github.com/Netflix/vmaf/issues/22

Cheers.

[Wolfberry]

It works for me using a relative path.
I also tried libvmaf=model_path=vmaf_b_v0.6.3.pkl:enable_conf_interval=1 and that also works.
Something must be wrong on your side, but still good to know that the vapoursynth filter works.

[WorBry]

Thanks. I'll look at again once I've done testing with the VapourSynth filter.

[utack]

I think the "phone model" is not working for me. Am I using this incorrectly?
ffmpeg -i HDsource -i lowresencode -filter_complex [0]scale=960:512,libvmaf=model_path=/usr/share/model/vmaf_v0.6.1.pkl:phone_model -f null -
Both with and without "phone model" resulted in the same score

[Beelzebubu]

I think the "phone model" is not working for me. Am I using this incorrectly?

Both with and without "phone model" resulted in the same score

IIRC in FFmpeg syntax you need to specify the boolean value explicitly, e.g. "phone_model=1".

[utack]

IIRC in FFmpeg syntax you need to specify the boolean value explicitly, e.g. "phone_model=1".

Yep, that worked. My bad

[Chrisputer]

It works for me using a relative path.
I also tried libvmaf=model_path=vmaf_b_v0.6.3.pkl:enable_conf_interval=1 and that also works.
Something must be wrong on your side, but still good to know that the vapoursynth filter works.


:stupid:
Hey Wolfberry,
I'm actually have the exact same issue. I have to punch in the path *nix style to get it work. But I'm also getting a VMAF Score = nan for some reason... :'(

[Iron_Mike]

cannot get this to work under Windows w/ latest Zeranoe ffmpeg build 4.1.1... copied both "vmaf_v0.6.1.pkl" and "vmaf_v0.6.1.pkl.model" into same folder as ffmpeg.exe... running this command:

ffmpeg -i main.mpg -i ref.mpg -lavfi libvmaf="model_path=vmaf_v0.6.1.pkl:psnr=1:log_fmt=json" -f null -

throws this error:

No such filter: 'libvmaf'

docs state that ffmpeg requires to be configured with --enable-libvmaf but the Zeranoe Windiows build does not come with that...

any advice ?

from the posts above I assumed that simply copying the .pkl files into the correct folder would work...

Thanks.

[Wolfberry]

You can get my custom build of ffmpeg with libvmaf here (https://drive.google.com/open?id=12BFBGGO2YEjvnBQVrP3_0dzvwPXmTIOJ)

[benwaggoner]

You can get my custom build of ffmpeg with libvmaf here (https://drive.google.com/open?id=1fWFXoOSljIU4dVyRLo19uhZcNNoQrIs9)



Cool. Anything special/awesome about your build?


Sent from my iPhone using Tapatalk

[Iron_Mike]

You can get my custom build of ffmpeg with libvmaf here (https://drive.google.com/open?id=1fWFXoOSljIU4dVyRLo19uhZcNNoQrIs9)

Hi,

get the following error trying to run VMAF filter using your build:

---------------------------
ffmpeg.exe - System Error
---------------------------
The program can't start because VSScript.dll is missing from your computer. Try reinstalling the program to fix this problem.
---------------------------

what are the dependencies for this build ?

Thanks.

[Wolfberry]

ffmpeg-4.2-N-93255-g4571c7c05d (https://drive.google.com/open?id=12BFBGGO2YEjvnBQVrP3_0dzvwPXmTIOJ) built with gcc 8.3.1 (G41) 20190222

The vapoursynth (https://github.com/vapoursynth/vapoursynth/releases) dependencies is now included.

The build can also be found in my signature, future updates will be there.


configuration: --disable-autodetect --disable-debug --disable-w32threads --enable-amf --enable-avisynth --enable-bzlib --enable-cuda --enable-d3d11va --enable-ffnvcodec --enable-iconv --enable-libaom --enable-libass
--enable-libbluray --enable-libdav1d --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libmfx --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus
--enable-librubberband --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtesseract --enable-libvidstab --enable-libvmaf --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2
--enable-libzimg --enable-lzma --enable-mbedtls --enable-nvdec --enable-nvenc--enable-opencl --enable-pthreads --enable-sdl2 --enable-vapoursynth --enable-zlib --enable-gpl --enable-version3

libavutil 56. 26.100 / 56. 26.100
libavcodec 58. 47.102 / 58. 47.102
libavformat 58. 26.101 / 58. 26.101
libavdevice 58. 6.101 / 58. 6.101
libavfilter 7. 48.100 / 7. 48.100
libswscale 5. 4.100 / 5. 4.100
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100

Comparison with Zeranoe builds:
w/ libfribidi liblensfun librubberband libsrt libssh libtesseract libvmaf mbedtls opencl vapoursynth
w/o gmp gnutls libmp3lame libopencore-amrnb libopencore-amrwb libshine libsnappy libspeex libtheora libtwolame libvorbis libvo-amrwbenc libwavpack libxvid

[Iron_Mike]

Wolfberry,

thank you !

In addition to the VMAF filter, your build also support 12bit encoding !

Do you have a website or where can I follow you for future releases ?

Thanks.

[Iron_Mike]

It works for me using a relative path.
I also tried libvmaf=model_path=vmaf_b_v0.6.3.pkl:enable_conf_interval=1 and that also works.
Something must be wrong on your side, but still good to know that the vapoursynth filter works.

did you test the relative path on Windows ?

I cannot get the .pkl path to work on your ffmpeg Win build... I tried relative path (file in same folder as ffmpeg.exe), full Windows path or Unix style path... it throws the Caught VmafException: Error loading model (.pkl): error...

[Wolfberry]

ffmpeg -i 1.mkv -i 1.mkv -lavfi libvmaf=model_path=vmaf_v0.6.1.pkl:log_path=vmaf.log:log_fmt=json:psnr=1:ssim=1:ms_ssim=1 -f null -
vmaf log (https://0x0.st/z-yV.log) / ffmpeg log (https://0x0.st/z-y4.log) I'm using Windows 10 18346.1 Education

[excellentswordfight]

did you test the relative path on Windows ?

I cannot get the .pkl path to work on your ffmpeg Win build... I tried relative path (file in same folder as ffmpeg.exe), full Windows path or Unix style path... it throws the error...
Same issue here, tried all the variants i could think of and tried both 0.6.1 and 0.6.3

Caught VmafException: Error loading model (.pkl): Trouble reading the file:vmaf_v0.6.3.pkl
or
Input model at vmaf_v0.6.1.pkl cannot be read successfully.
Caught VmafException: Error loading model (.pkl): Unknown token

[Iron_Mike]

okay, made some progress...

if you call "ffmpeg" from CLI because you're either already in that folder or location (of that ffmpeg version) is on Windows PATH, then the relative path to VMAF will work !

but if you call ffmpeg with a fully qualified path (e.g. "d:/path/to/ffmpeg.exe" -i test.mkv.... etc ) then the relative path to the vmaf.pkl will not work... and for whatever reason I cannot seem to get fully qualified paths to the .pkl files working...

[Wolfberry]

Yes, exactly that. Future updates will be in my signature, update frequency will probably be every other day.

[poisondeathray]

Thanks Wolfberry, (and for vpy support!)

For people having syntax issues - For Windoze, the issue with ffmpeg and filters is escape characters . To enter full paths, for every colon , you need to enter a preceding backslash. This is true for all filters syntax in windows ffmpeg with full paths

eg.
(I marked in red in this example)


-lavfi "libvmaf=model_path='C\:/Program Files (x86)/VapourSynth/plugins64/VMAF-r3/plugins64/model/vmaf_v0.6.1.pkl':log_path='F\:/TEST/vmaf.log':log_fmt=json:psnr=1:ssim=1:ms_ssim=1"

[Iron_Mike]

Thanks Wolfberry, (and for vpy support!)

For people having syntax issues - For Windoze, the issue with ffmpeg and filters is escape characters . To enter full paths, for every colon , you need to enter a preceding backslash. This is true for all filters syntax in windows ffmpeg with full paths

eg.
(I marked in red in this example)


-lavfi "libvmaf=model_path='C\:/Program Files (x86)/VapourSynth/plugins64/VMAF-r3/plugins64/model/vmaf_v0.6.1.pkl':log_path='F\:/TEST/vmaf.log':log_fmt=json:psnr=1:ssim=1:ms_ssim=1"


Thanks for that, I tried that before (same escape syntax as specifying subtitle files) but it didn't work...

but got it to work now by using your exact syntax of wrapping the full VMAF filter command in double quotes, then wrapping the .pkl path in single quotes (that was t6he missing part), and then escaping the colon...

Thanks !

[Iron_Mike]

follow up question in regards to using VMAF:

I understand that one needs to upscale a video to the source resolution in order to compare via VMAF, but what about bit depth ?

so if I compare a 16-bit EXR sequence (full RGB) with a 12-bit 444 encode (yuv444p12le) - both in same resolution - do I need to adjust anything for the VMAF comparison to be as accurate as possible ?

read the VMAF FAQs but it does not mention bit depth differences...

[Beelzebubu]

If the encoded video and the source video have a different bitdepth, then yes, you need to choose what to measure it at. I would recommend to use the evaluation bitdepth used by the display device, and make sure to use a model tuned to that bitdepth (and resolution) also.

[Iron_Mike]

If the encoded video and the source video have a different bitdepth, then yes, you need to choose what to measure it at. I would recommend to use the evaluation bitdepth used by the display device, and make sure to use a model tuned to that bitdepth (and resolution) also.

could you provide an example of the ffmpeg command ?

I've tried passing in different pixel formats (for both inputs), VMAF score never changed...

both vids are HD res, so I used model 0.6.1...

lmk what parameters to change and I'll run tests...

Thanks.

[foxyshadis]

follow up question in regards to using VMAF:

I understand that one needs to upscale a video to the source resolution in order to compare via VMAF, but what about bit depth ?

so if I compare a 16-bit EXR sequence (full RGB) with a 12-bit 444 encode (yuv444p12le) - both in same resolution - do I need to adjust anything for the VMAF comparison to be as accurate as possible ?

read the VMAF FAQs but it does not mention bit depth differences...

VMAF only goes up to 10-bit, so ffmpeg's either converting both to 10-bit or to 8-bit before passing them on to VMAF.

[rco133]

Hi.

I think I need some help with libvmaf.

No matter what I do, after ffmpeg has scanned the files it ends up with VMAF nan

The version of ffmpeg used is the most recent downloaded from Wolfberry.

My command line is:

ffmpeg -i encoded.mkv -i lossless.mkv -lavfi libvmaf=vmaf_4k_v0.6.1.pkl -f null -

The ouput of ffmpeg is:

Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608 [SAR 1:1 DAR 160:67], 23.98 fps, 23.98 tbr, 1200k tbn, 23.98 tbc

Stream #1:0: Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1608, 23.98 tbr, 1200k tbn, 23.98 tbc

Stream mapping:
Stream #0:0 (hevc) -> libvmaf:main
Stream #1:0 (hevc) -> libvmaf:reference
libvmaf -> Stream #0:0 (wrapped_avframe)

frame= 0 fps=0.0 q=0.0 size=N/A time=-577014:32:22.77 bitrate=N/A speed=N/A Start calculating VMAF score...

Output #0, null, to 'pipe:':
Metadata:
encoder : Lavf58.31.104
Stream #0:0: Video: wrapped_avframe, yuv420p10le, 3840x1608 [SAR 1:1 DAR 160:67], q=2-31, 200 kb/s, 23.98 fps, 23.98 tbn, 23.98 tbc (default)
Metadata:
encoder : Lavc58.55.101 wrapped_avframe

And then it takes quite a while before the ouput comes. The files have a bit more than 6000 frames.

After scanning all the frames the output I get is this:

frame= 6151 fps=8.3 q=-0.0 Lsize=N/A time=00:04:16.54 bitrate=N/A speed=0.348x
video:3220kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exec FPS: 8.323916
VMAF score = nan
[libvmaf @ 00000284353da5c0] VMAF score: nan

I have managed to get vmafossexec.exe to output some numbers, but it only supports raw yuv files, and I would like to doublecheck the numbers as the PSNR and SSIM numbers from vmafossexec.exe, doesn't match what I have previously gotten from ffmpeg.

Any help will be most appreciated.

rco133

[benwaggoner]

VMAF only goes up to 10-bit, so ffmpeg's either converting both to 10-bit or to 8-bit before passing them on to VMAF.
If you are comparing encodes made from a source that gets color space converted, you need to compare with the actual frames that the encoder is taking as input. So the scaled 8/10-bit Y'CbCr 4:2:0.

If you can do a lossless encode using the same tool, that would work. like if using ffmpeg, pipe out to .y4m.

[sashaItats]

A second note on the filter usage:
- as in all my last filters, I make use of random selections tha plasma applied is randomly selected, the pattern also
- hence, if you do not like one outcome just relaunch the filter and you will get a different one, even with the same parameters

[orion44]

ffmpeg-4.2-N-93255-g4571c7c05d (https://drive.google.com/open?id=12BFBGGO2YEjvnBQVrP3_0dzvwPXmTIOJ) built with gcc 8.3.1 (G41) 20190222

The vapoursynth (https://github.com/vapoursynth/vapoursynth/releases) dependencies is now included.

The build can also be found in my signature, future updates will be there.


configuration: --disable-autodetect --disable-debug --disable-w32threads --enable-amf --enable-avisynth --enable-bzlib --enable-cuda --enable-d3d11va --enable-ffnvcodec --enable-iconv --enable-libaom --enable-libass
--enable-libbluray --enable-libdav1d --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libmfx --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus
--enable-librubberband --enable-libsoxr --enable-libsrt --enable-libssh --enable-libtesseract --enable-libvidstab --enable-libvmaf --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2
--enable-libzimg --enable-lzma --enable-mbedtls --enable-nvdec --enable-nvenc--enable-opencl --enable-pthreads --enable-sdl2 --enable-vapoursynth --enable-zlib --enable-gpl --enable-version3

libavutil 56. 26.100 / 56. 26.100
libavcodec 58. 47.102 / 58. 47.102
libavformat 58. 26.101 / 58. 26.101
libavdevice 58. 6.101 / 58. 6.101
libavfilter 7. 48.100 / 7. 48.100
libswscale 5. 4.100 / 5. 4.100
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100

Comparison with Zeranoe builds:
w/ libfribidi liblensfun librubberband libsrt libssh libtesseract libvmaf mbedtls opencl vapoursynth
w/o gmp gnutls libmp3lame libopencore-amrnb libopencore-amrwb libshine libsnappy libspeex libtheora libtwolame libvorbis libvo-amrwbenc libwavpack libxvid


The link for FFmpeg is dead. I really want to try out the VMAF filter.

[Vincent Vega]

is there a way to encode to say x264(software) or h264(nvidia hardware) and to simultanuously pipe/tee input and output frames to libvmaf for comparison?
i.e to do encode and vmaf calculation with single ffmpeg command?

[benwaggoner]

is there a way to encode to say x264(software) or h264(nvidia hardware) and to simultanuously pipe/tee input and output frames to libvmaf for comparison?
i.e to do encode and vmaf calculation with single ffmpeg command?
That should be feasible, and it's a lot faster to calculate the base metrics while the source and output frames are both in RAM together. So many erroneous VMAF results come from not quite getting frames to be apples-to-apples when decoding the source again, using suboptimal scaling, etcetera. Comparing the post-processed 4:2:0 frame that's going into the encoder, and the reconstructed frame that has been encoding, is a lot harder to mess up.

There's no reason the ML calculations need to happen while encoding; generating the base metrics is the slow part, and the VMAF calculations can be done later.

I like that approach as it allows for running different VMAF models without having to redo those calculations.

[tormento]

Is possible to have a recent version of ffmpeg for windows x64 with vmafcuda support?

It seems really hard to find.