4k encode super small

[jriker1]

I have a 4k HDR10/DolbyVision IMAX of Top Gun Maverick. Want to put it on my media server. I ran the following against the video:

ffmpeg -i Top.GunMaverick.mkv -sn -an -f yuv4mpegpipe -strict -1 -pix_fmt yuv420p10le - | x265-10b - --input-depth 10 --output-depth 10 --y4m --preset slow --hdr10 --hdr10-opt --high-tier --repeat-headers --crf 20 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --max-cll "1000,401" --chromaloc 2 --no-sao --range limited --keyint 24 --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --dolby-vision-rpu rpu.bin --dolby-vision-profile 8.1 --vbv-bufsize 160000 --vbv-maxrate 160000 TopGunMaverick-output.hevc


The whole time it was encoding was complaining about

Skipping NAL Unit 63

Which I understand is some layer or something from DolbyVision that ffmpeg can't handle?

The final output of the video was 3.8GB from an extracted source of 78GB. Outside of watch it and see as I'm never good with that checking quality does that sound wrong?

[excellentswordfight]

I have a 4k HDR10/DolbyVision IMAX of Top Gun Maverick. Want to put it on my media server. I ran the following against the video:



The whole time it was encoding was complaining about

Skipping NAL Unit 63

Which I understand is some layer or something from DolbyVision that ffmpeg can't handle?

The final output of the video was 3.8GB from an extracted source of 78GB. Outside of watch it and see as I'm never good with that checking quality does that sound wrong?
I have encoded Maverick as well, and given that its a very clean movie with very little noise, im not supprised that it turned out that small at crf20. Ive seen clean good looking uhd sources re-encoded bellow 10Mbps at crf 15 (and close to visually lossless as well mind you), when not having to compress noise and grain its a bit suprising of how great the compression you can actually achieve, even though it might seem ”wrong” at first.

And it might be worth to mention that hdr usually get allocated fewer bits than sdr at the same crf,

[Boulder]

And it might be worth to mention that hdr usually get allocated fewer bits than sdr at the same crf,

This. Due to bits being allocated in a bit (:devil:) different way with --hdr10-opt, you should use a slightly lower CRF value than with SDR sources. I've ended up looking at the final QP value if it is in a sane range like 21-24 to make sure my CRF selection is valid.

EDIT: see here: https://forum.doom9.org/showthread.php?p=1977257#post1977257.

[jriker1]

Thanks for the replies. I'm starting to think it's related to the NAL unit 63 messages but maybe it's what it is supposed to be. DolbyVision videos that don't output that message over and over are encoding at a more understandable size and those that do output the 63 message seem unusually small. MediaInfo shows I guess the average bitrate of the source at 67MB/s and the one I create at 4.

Running it thru FFMetrics if relevant I get:

PSNR: 41.5829
SSIM: 0.9932
VMAF: 93.0939

Is a CRF of 20 to (never sure how to word this) high of a number and should go lower? Reading Code Calamity articles they came down to a CRF value in general for 4k of 20. Seemed like a knowledgeable person in the space but glad to adjust it. used to use CRF 13 for stuff I really liked and CRF 16 for stuff I was just keeping to keep. I also though was limiting maxrate to 17,500 and 15,00 respectively when doing that. Without those limits frequently the final video would be twice the size of the source.

Note also used to encode with Slower instead of Slow befor ereading the Code Calamity articles.

[excellentswordfight]

Thanks for the replies. I'm starting to think it's related to the NAL unit 63 messages but maybe it's what it is supposed to be. DolbyVision videos that don't output that message over and over are encoding at a more understandable size and those that do output the 63 message seem unusually small. MediaInfo shows I guess the average bitrate of the source at 67MB/s and the one I create at 4.

Running it thru FFMetrics if relevant I get:

PSNR: 41.5829
SSIM: 0.9932
VMAF: 93.0939

Is a CRF of 20 to (never sure how to word this) high of a number and should go lower? Reading Code Calamity articles they came down to a CRF value in general for 4k of 20. Seemed like a knowledgeable person in the space but glad to adjust it. used to use CRF 13 for stuff I really liked and CRF 16 for stuff I was just keeping to keep. I also though was limiting maxrate to 17,500 and 15,00 respectively when doing that. Without those limits frequently the final video would be twice the size of the source.
Well, it hard for someone else to tell others what is the correct value to use, its not us that are going to watch it, and we have no idea what your ideal trade off is between compression and quality loss.

That said, 20 is usually the starting point for good quality encodes, which your numbers also confirms (vmaf above 93 and psnr above 40 which are good indicators). And down towards 15-16 is were you would start to see excellent quality, were transparency is starting to be achieved.

Ive done some sanity checks on titles were crf 15-16 resualted in very low bitrate files, and I could not tell it apart from source in a/b testing (even when swopping between stills), i see no reason to go down further.

What you used to do with crf 13 and vbv-limits is a pretty strange strategy, what you get is pretty much constant bitrate encoding at that point if you got such huge files without it, you gain pretty much nothing from the low crf value.

It should also be noted that crf values cannot be compared between different settings, and is not a fixed quality level, this is extra apperent sense the faster presets usually give you smaller files ar the same crf than the slowest presets. The values mentioned above is for slow/slower at somewhat ordinary settings. Cause for example, changing aq-mode or tweaking psy can have huge impact on bitrate at given crf value, but that doenst mean that it offers better or worse compression.

[jriker1]

Yeah I think my logic was to give it a lot of bits to work with but cap it from getting out of control.

Note I've never done CRF 20 or for that matter preset of Slow until now. Always done like CRF 13 - 16 and always a preset of Slower. I'm encoding Creed III right now which I wasn't' thinking the original would be stellar quality and also giving the 63 dolby vision skipping messages. I haven't looked at the video quality physically but 40 minutes of show content encoded and file is 1Gb in size right now. So again going to be super small than what I'm used to but maybe my videos have been bigger than they need to for years now who knows. Thing will be smaller than the original audio which I do not usually modify.

[excellentswordfight]

Yeah I think my logic was to give it a lot of bits to work with but cap it from getting out of control.
Yes, and that can work for some cases, its just that for UHD 17Mbps cap is very low for crf 13, as that usually produce way higher bitrate than that. In a lot of cases it will probably go beyond the source, so you will actually get some side effects instead (more inefficient bit distribution as you wont get much variable bitrate gains) as you can pretty much be hitting the vbr wall constantly. A encode with 17Mbps average that will fluctuate between a few Mbps an close to a hundred, will look better than a encode with the same average bitrate but with a more or less constant bitrate. So using a higher crf value without the cap would look better.

...or for that matter preset of Slow until now.
It is ofc "better" to use preset 'slower', there is however not a huge difference between those presets when it comes to the actual output (like there are between some other presets), not enough to warrant the speed penalty imo (not for home use at least). But if you think the speed is usable with slower, 'slow' is not "better".

So again going to be super small than what I'm used to but maybe my videos have been bigger than they need to for years now who knows. Thing will be smaller than the original audio which I do not usually modify.
Well, just cause crf 20 will start to produce some good looking enocodes, doesnt mean that there is no gain of going lower, even though going as low as crf 13 will for most cases be overkill. You will just have to do some tests an evaluate what your sweetspot is for size/quality in that ~15-20 range. But if you were using crf 13 before, that for sure would create some "unnecessary" big files, even though you capped it.

And FYI, Im pretty sure that the NAL 63 warning is related to the EL layer of DV files. I think its just reporting that its skipping the EL layer.

[benwaggoner]

This. Due to bits being allocated in a bit (:devil:) different way with --hdr10-opt, you should use a slightly lower CRF value than with SDR sources. I've ended up looking at the final QP value if it is in a sane range like 21-24 to make sure my CRF selection is valid.

EDIT: see here: https://forum.doom9.org/showthread.php?p=1977257#post1977257.
And for some kinds of content, like scrolling credits, it can look visually lossless at 40+ QP if SAO is on.

[jriker1]

if you were using crf 13 before, that for sure would create some "unnecessary" big files, even though you capped it.


Thanks for the replies. Never can get an answer to this fully and know it's variable, however if you have a super clean source or an animated movie say that is 2 hours long, what should I be expecting? Is 3GB for the video output something that immediately says that's a bad looking encode or could be acceptable? Note I'm not super concerned with file size over quality but also don't want to waste space for no reason. Think I've been doing that for a long time now so having a hard time wrapping my arms around if a 3GB encode is acceptable for a 4k HDR10/Dolby Vision video where in the past I would always be lookin at 15GB - 35GB depending on grain.

[jriker1]

So this is confusing. All the same except the CRF and Preset values, getting weird results.

CRF20 Slow returns:

PSNR: 44.0456
SSIM: 0.9889
VMAF: 95.6835
File Size: 3.1GB

CRF20 Slower returns:

PSNR: 43.9303
SSIM: 0.9881
VMAF: 95.7374
File Size: 3.3GB

OK that seems about right I guess.

I decided to do an encode at CRF 19 and preset slow:

PSNR: 32.4574
SSIM: 0.9676
VMAF: 67.3648
File Size: 3.9GB

Only part of this that maybe seems right is the file size difference.

What am I missing?

Didn't see an option to cross out anything so marked my crossout in red. Apparently I ran the check against an MKV source and the HEVC output. Adding it into an MKV container returned better results. New output:

PSNR: 44.0941
SSIM: 0.9891
VMAF: 96.0053
File Size: 4.0GB

Note with all videos I'm skipping the first minute in the above scan and letting it check 10 minutes of video.

Changing my comment slightly, and I ran this multiple times so know it's clean what it's checking at least within the 10 minutes. Why is the slower version come back with a slightly lower SSIM and a lower PSNR than even it's "slow" counterpart.

[benwaggoner]

That big drop at CRF 19 suggests a metrics analysis problem, like you're comparing frames with a 1 frame offset or something. PSNR is particularly sensitive to that sort of thing.

How do those values compare to the PSNR AND SSIM that x265 logs directly (I like to use --csv-log-level 2 for later analysis).