jdawn1
7th September 2020, 02:48
I discovered VMAF and thought it was a really cool concept. Nothing could replace human manual quality checking, until now?
I spent way too much time playing around with it, so I thought I'd share my results.
Here are the raw files use.
I tested with a 4k source file (encoded with CRF 12 slow from the original because I couldn't get proper results using VMAF on the original stream)
For all encodes I used ffmpeg, bicubic resizing, slow preset,
with
1080p: CRF 12, 16, 18-36
1440p: CRF 20-42
2160p: CRF 24-44
To calculate vmaf, I made a quick 40 second clip of walking showing some landscape video. the input source was always upscaled to 4k and checked against the -CRF 12 reference.
I got nonsense results when scaling both to 1080.
here are the results.
https://i.imgur.com/KDVE0Ib.jpg
https://i.imgur.com/Xfqeq8v.jpg
My observations:
--Both models heavily favor 4k encodes over actual perceived quality(IMHO). It seems place extremely high priority on specific fine details (American flag in distance, trees on skyline, small house details, over ALL other details)
--There is NO way, anyone on this forum, will agree the 4k version at ~3.5 megabytes will agree looks superior to the 1080p version at ~3.5megabytes, despite what the VMAF reports
--IMHO, CRF 24 1080p (17.53MB) came out approx about the same quality as CRF 32 at 4k (16.07MB)
Anything smaller then this file size, I would favor the 1080p encode.
This indicates my crossover point was ~15 megabytes, vs ~3megabytes (LOL!) for VMAF. HUGE difference.
I understand the results are specific to the content, but I did some quick checks on different content and I found very similar results....
Comparing VMAF vs VMAF 4k model: The 4k model just favors the 4k encode even moreso then the regular VMAF model. The original model was already bad in this respect, it makes the 4k model even worse.
--it makes sense the 1440p encode maxes out at ~93, while the 1080 one maxes out at ~87, relative to 100 for the 4k model.
--I would expect the 2k encode to be closer to the 4k encode relative to the 1080p encode, but it's not.
You would think they would cross over at some point, and the they do, around 3MB (300kbit/sec for this video) , whereas my frame by frame picking would choose the crossover at ~15MB)
I didn't learn much by doing this, unfortunately.
I did make one small change, instead of encoding at 1080p CRF 24 for balanced archiving, I now do 1440p CRF 26.
I'll share the raw data and the video files here once uploaded.
https://1drv.ms/u/s!AuqwBZb5n5Jqh5BoI13POa9ylZrAVQ?e=mU3k0h
Feel free to download/use, i'll take it down in a few days.
I was just going to delete the folder and be done with it, but since I spent so much time, I thought I would share. Go easy on me.
I spent way too much time playing around with it, so I thought I'd share my results.
Here are the raw files use.
I tested with a 4k source file (encoded with CRF 12 slow from the original because I couldn't get proper results using VMAF on the original stream)
For all encodes I used ffmpeg, bicubic resizing, slow preset,
with
1080p: CRF 12, 16, 18-36
1440p: CRF 20-42
2160p: CRF 24-44
To calculate vmaf, I made a quick 40 second clip of walking showing some landscape video. the input source was always upscaled to 4k and checked against the -CRF 12 reference.
I got nonsense results when scaling both to 1080.
here are the results.
https://i.imgur.com/KDVE0Ib.jpg
https://i.imgur.com/Xfqeq8v.jpg
My observations:
--Both models heavily favor 4k encodes over actual perceived quality(IMHO). It seems place extremely high priority on specific fine details (American flag in distance, trees on skyline, small house details, over ALL other details)
--There is NO way, anyone on this forum, will agree the 4k version at ~3.5 megabytes will agree looks superior to the 1080p version at ~3.5megabytes, despite what the VMAF reports
--IMHO, CRF 24 1080p (17.53MB) came out approx about the same quality as CRF 32 at 4k (16.07MB)
Anything smaller then this file size, I would favor the 1080p encode.
This indicates my crossover point was ~15 megabytes, vs ~3megabytes (LOL!) for VMAF. HUGE difference.
I understand the results are specific to the content, but I did some quick checks on different content and I found very similar results....
Comparing VMAF vs VMAF 4k model: The 4k model just favors the 4k encode even moreso then the regular VMAF model. The original model was already bad in this respect, it makes the 4k model even worse.
--it makes sense the 1440p encode maxes out at ~93, while the 1080 one maxes out at ~87, relative to 100 for the 4k model.
--I would expect the 2k encode to be closer to the 4k encode relative to the 1080p encode, but it's not.
You would think they would cross over at some point, and the they do, around 3MB (300kbit/sec for this video) , whereas my frame by frame picking would choose the crossover at ~15MB)
I didn't learn much by doing this, unfortunately.
I did make one small change, instead of encoding at 1080p CRF 24 for balanced archiving, I now do 1440p CRF 26.
I'll share the raw data and the video files here once uploaded.
https://1drv.ms/u/s!AuqwBZb5n5Jqh5BoI13POa9ylZrAVQ?e=mU3k0h
Feel free to download/use, i'll take it down in a few days.
I was just going to delete the folder and be done with it, but since I spent so much time, I thought I would share. Go easy on me.