Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > High Efficiency Video Coding (HEVC)

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th July 2023, 13:37   #1  |  Link
jriker1
Registered User
 
Join Date: Dec 2003
Posts: 487
4k encode super small

I have a 4k HDR10/DolbyVision IMAX of Top Gun Maverick. Want to put it on my media server. I ran the following against the video:

Quote:
ffmpeg -i Top.GunMaverick.mkv -sn -an -f yuv4mpegpipe -strict -1 -pix_fmt yuv420p10le - | x265-10b - --input-depth 10 --output-depth 10 --y4m --preset slow --hdr10 --hdr10-opt --high-tier --repeat-headers --crf 20 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50)" --max-cll "1000,401" --chromaloc 2 --no-sao --range limited --keyint 24 --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --dolby-vision-rpu rpu.bin --dolby-vision-profile 8.1 --vbv-bufsize 160000 --vbv-maxrate 160000 TopGunMaverick-output.hevc
The whole time it was encoding was complaining about

Skipping NAL Unit 63

Which I understand is some layer or something from DolbyVision that ffmpeg can't handle?

The final output of the video was 3.8GB from an extracted source of 78GB. Outside of watch it and see as I'm never good with that checking quality does that sound wrong?
jriker1 is offline   Reply With Quote
Old 29th July 2023, 14:44   #2  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 332
Quote:
Originally Posted by jriker1 View Post
I have a 4k HDR10/DolbyVision IMAX of Top Gun Maverick. Want to put it on my media server. I ran the following against the video:



The whole time it was encoding was complaining about

Skipping NAL Unit 63

Which I understand is some layer or something from DolbyVision that ffmpeg can't handle?

The final output of the video was 3.8GB from an extracted source of 78GB. Outside of watch it and see as I'm never good with that checking quality does that sound wrong?
I have encoded Maverick as well, and given that its a very clean movie with very little noise, im not supprised that it turned out that small at crf20. Ive seen clean good looking uhd sources re-encoded bellow 10Mbps at crf 15 (and close to visually lossless as well mind you), when not having to compress noise and grain its a bit suprising of how great the compression you can actually achieve, even though it might seem ”wrong” at first.

And it might be worth to mention that hdr usually get allocated fewer bits than sdr at the same crf,

Last edited by excellentswordfight; 29th July 2023 at 15:20.
excellentswordfight is offline   Reply With Quote
Old 29th July 2023, 15:48   #3  |  Link
Boulder
Pig on the wing
 
Boulder's Avatar
 
Join Date: Mar 2002
Location: Finland
Posts: 5,770
Quote:
Originally Posted by excellentswordfight View Post
And it might be worth to mention that hdr usually get allocated fewer bits than sdr at the same crf,
This. Due to bits being allocated in a bit () different way with --hdr10-opt, you should use a slightly lower CRF value than with SDR sources. I've ended up looking at the final QP value if it is in a sane range like 21-24 to make sure my CRF selection is valid.

EDIT: see here: https://forum.doom9.org/showthread.p...57#post1977257.
__________________
And if the band you're in starts playing different tunes
I'll see you on the dark side of the Moon...

Last edited by Boulder; 29th July 2023 at 15:51.
Boulder is offline   Reply With Quote
Old 29th July 2023, 18:09   #4  |  Link
jriker1
Registered User
 
Join Date: Dec 2003
Posts: 487
Thanks for the replies. I'm starting to think it's related to the NAL unit 63 messages but maybe it's what it is supposed to be. DolbyVision videos that don't output that message over and over are encoding at a more understandable size and those that do output the 63 message seem unusually small. MediaInfo shows I guess the average bitrate of the source at 67MB/s and the one I create at 4.

Running it thru FFMetrics if relevant I get:

PSNR: 41.5829
SSIM: 0.9932
VMAF: 93.0939

Is a CRF of 20 to (never sure how to word this) high of a number and should go lower? Reading Code Calamity articles they came down to a CRF value in general for 4k of 20. Seemed like a knowledgeable person in the space but glad to adjust it. used to use CRF 13 for stuff I really liked and CRF 16 for stuff I was just keeping to keep. I also though was limiting maxrate to 17,500 and 15,00 respectively when doing that. Without those limits frequently the final video would be twice the size of the source.

Note also used to encode with Slower instead of Slow befor ereading the Code Calamity articles.

Last edited by jriker1; 29th July 2023 at 18:20.
jriker1 is offline   Reply With Quote
Old 29th July 2023, 19:49   #5  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 332
Quote:
Originally Posted by jriker1 View Post
Thanks for the replies. I'm starting to think it's related to the NAL unit 63 messages but maybe it's what it is supposed to be. DolbyVision videos that don't output that message over and over are encoding at a more understandable size and those that do output the 63 message seem unusually small. MediaInfo shows I guess the average bitrate of the source at 67MB/s and the one I create at 4.

Running it thru FFMetrics if relevant I get:

PSNR: 41.5829
SSIM: 0.9932
VMAF: 93.0939

Is a CRF of 20 to (never sure how to word this) high of a number and should go lower? Reading Code Calamity articles they came down to a CRF value in general for 4k of 20. Seemed like a knowledgeable person in the space but glad to adjust it. used to use CRF 13 for stuff I really liked and CRF 16 for stuff I was just keeping to keep. I also though was limiting maxrate to 17,500 and 15,00 respectively when doing that. Without those limits frequently the final video would be twice the size of the source.
Well, it hard for someone else to tell others what is the correct value to use, its not us that are going to watch it, and we have no idea what your ideal trade off is between compression and quality loss.

That said, 20 is usually the starting point for good quality encodes, which your numbers also confirms (vmaf above 93 and psnr above 40 which are good indicators). And down towards 15-16 is were you would start to see excellent quality, were transparency is starting to be achieved.

Ive done some sanity checks on titles were crf 15-16 resualted in very low bitrate files, and I could not tell it apart from source in a/b testing (even when swopping between stills), i see no reason to go down further.

What you used to do with crf 13 and vbv-limits is a pretty strange strategy, what you get is pretty much constant bitrate encoding at that point if you got such huge files without it, you gain pretty much nothing from the low crf value.

It should also be noted that crf values cannot be compared between different settings, and is not a fixed quality level, this is extra apperent sense the faster presets usually give you smaller files ar the same crf than the slowest presets. The values mentioned above is for slow/slower at somewhat ordinary settings. Cause for example, changing aq-mode or tweaking psy can have huge impact on bitrate at given crf value, but that doenst mean that it offers better or worse compression.

Last edited by excellentswordfight; 29th July 2023 at 20:13.
excellentswordfight is offline   Reply With Quote
Old 30th July 2023, 21:48   #6  |  Link
jriker1
Registered User
 
Join Date: Dec 2003
Posts: 487
Yeah I think my logic was to give it a lot of bits to work with but cap it from getting out of control.

Note I've never done CRF 20 or for that matter preset of Slow until now. Always done like CRF 13 - 16 and always a preset of Slower. I'm encoding Creed III right now which I wasn't' thinking the original would be stellar quality and also giving the 63 dolby vision skipping messages. I haven't looked at the video quality physically but 40 minutes of show content encoded and file is 1Gb in size right now. So again going to be super small than what I'm used to but maybe my videos have been bigger than they need to for years now who knows. Thing will be smaller than the original audio which I do not usually modify.
jriker1 is offline   Reply With Quote
Old 30th July 2023, 23:38   #7  |  Link
excellentswordfight
Lost my old account :(
 
Join Date: Jul 2017
Posts: 332
Quote:
Originally Posted by jriker1 View Post
Yeah I think my logic was to give it a lot of bits to work with but cap it from getting out of control.
Yes, and that can work for some cases, its just that for UHD 17Mbps cap is very low for crf 13, as that usually produce way higher bitrate than that. In a lot of cases it will probably go beyond the source, so you will actually get some side effects instead (more inefficient bit distribution as you wont get much variable bitrate gains) as you can pretty much be hitting the vbr wall constantly. A encode with 17Mbps average that will fluctuate between a few Mbps an close to a hundred, will look better than a encode with the same average bitrate but with a more or less constant bitrate. So using a higher crf value without the cap would look better.

Quote:
...or for that matter preset of Slow until now.
It is ofc "better" to use preset 'slower', there is however not a huge difference between those presets when it comes to the actual output (like there are between some other presets), not enough to warrant the speed penalty imo (not for home use at least). But if you think the speed is usable with slower, 'slow' is not "better".

Quote:
So again going to be super small than what I'm used to but maybe my videos have been bigger than they need to for years now who knows. Thing will be smaller than the original audio which I do not usually modify.
Well, just cause crf 20 will start to produce some good looking enocodes, doesnt mean that there is no gain of going lower, even though going as low as crf 13 will for most cases be overkill. You will just have to do some tests an evaluate what your sweetspot is for size/quality in that ~15-20 range. But if you were using crf 13 before, that for sure would create some "unnecessary" big files, even though you capped it.

And FYI, Im pretty sure that the NAL 63 warning is related to the EL layer of DV files. I think its just reporting that its skipping the EL layer.

Last edited by excellentswordfight; 31st July 2023 at 00:06.
excellentswordfight is offline   Reply With Quote
Old 31st July 2023, 06:42   #8  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,829
Quote:
Originally Posted by Boulder View Post
This. Due to bits being allocated in a bit () different way with --hdr10-opt, you should use a slightly lower CRF value than with SDR sources. I've ended up looking at the final QP value if it is in a sane range like 21-24 to make sure my CRF selection is valid.

EDIT: see here: https://forum.doom9.org/showthread.p...57#post1977257.
And for some kinds of content, like scrolling credits, it can look visually lossless at 40+ QP if SAO is on.
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Old 31st July 2023, 16:55   #9  |  Link
jriker1
Registered User
 
Join Date: Dec 2003
Posts: 487
Quote:
Originally Posted by excellentswordfight View Post
if you were using crf 13 before, that for sure would create some "unnecessary" big files, even though you capped it.
Thanks for the replies. Never can get an answer to this fully and know it's variable, however if you have a super clean source or an animated movie say that is 2 hours long, what should I be expecting? Is 3GB for the video output something that immediately says that's a bad looking encode or could be acceptable? Note I'm not super concerned with file size over quality but also don't want to waste space for no reason. Think I've been doing that for a long time now so having a hard time wrapping my arms around if a 3GB encode is acceptable for a 4k HDR10/Dolby Vision video where in the past I would always be lookin at 15GB - 35GB depending on grain.
jriker1 is offline   Reply With Quote
Old 1st August 2023, 15:32   #10  |  Link
jriker1
Registered User
 
Join Date: Dec 2003
Posts: 487
So this is confusing. All the same except the CRF and Preset values, getting weird results.

CRF20 Slow returns:

PSNR: 44.0456
SSIM: 0.9889
VMAF: 95.6835
File Size: 3.1GB

CRF20 Slower returns:

PSNR: 43.9303
SSIM: 0.9881
VMAF: 95.7374
File Size: 3.3GB

OK that seems about right I guess.

I decided to do an encode at CRF 19 and preset slow:

PSNR: 32.4574
SSIM: 0.9676
VMAF: 67.3648
File Size: 3.9GB


Only part of this that maybe seems right is the file size difference.

What am I missing?

Didn't see an option to cross out anything so marked my crossout in red. Apparently I ran the check against an MKV source and the HEVC output. Adding it into an MKV container returned better results. New output:

PSNR: 44.0941
SSIM: 0.9891
VMAF: 96.0053
File Size: 4.0GB

Note with all videos I'm skipping the first minute in the above scan and letting it check 10 minutes of video.

Changing my comment slightly, and I ran this multiple times so know it's clean what it's checking at least within the 10 minutes. Why is the slower version come back with a slightly lower SSIM and a lower PSNR than even it's "slow" counterpart.

Last edited by jriker1; 1st August 2023 at 16:41.
jriker1 is offline   Reply With Quote
Old 2nd August 2023, 19:53   #11  |  Link
benwaggoner
Moderator
 
Join Date: Jan 2006
Location: Portland, OR
Posts: 4,829
That big drop at CRF 19 suggests a metrics analysis problem, like you're comparing frames with a 1 frame offset or something. PSNR is particularly sensitive to that sort of thing.

How do those values compare to the PSNR AND SSIM that x265 logs directly (I like to use --csv-log-level 2 for later analysis).
__________________
Ben Waggoner
Principal Video Specialist, Amazon Prime Video

My Compression Book
benwaggoner is offline   Reply With Quote
Reply

Tags
ffmpeg, x265

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 00:29.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.