Optimizing ffmpeg x265 for UHD collection

[speedy]

I've historically backed up my entire UHD Blu-ray collection to my NAS as full AVCHD backups (via MakeMKV) that I play on my Oppo 203, but I recently have gotten into using Plex and would like to encode my whole collection for playback via Infuse (via Plex) on my Apple TV 4K Gen 2.

I wanted to do a sanity check on my workflow and ffmpeg x265 settings before I'm too far down the rabbit hole.

My biggest area of concern (given that it's the most time consuming) is optimization of my compression settings. I've decided to encode the majority of my UHD collection with the Slow preset between CRF 16-18. With that said, I'll end up encoding some of my favorite films with Slower and some of the stuff I'm not sure that I'll ever watch again on Medium (up to CRF 20).

Given that, I'd like to keep a list of optimal settings for certain types of files:

No-grain films
Low-grain films
Medium-grain films
High-grain films
Low luminance films
Anime with grain
Anime without grain
Hand-drawn animation
CG animation
Anything else I'm not thinking of...


The only advanced options I've set that stray from encoder defaults are:
deblock=-1,-1 / selective-sao=2
...but I'm honestly not sure if these are optimal in most cases or how to test given the hundreds of UHD discs I'll be encoding.

I don't know enough about x265 to produce a list of optimal settings for each of these given my goals. With that said, my goals are to optimize quality over storage enough that streaming over the internet isn't difficult when I'm on a good internet connection.

As an aside, here's my workflow currently:

Backup movie to NAS as AVCHD folder with MakeMKV (already done)
Create uncompressed "Original" MKV file with DoVi Profile 7 video, lossless audio, and all English subtitles.
Encode with video ffmpeg x265 10-bit passing through HDR metadata
Encode lossless 7.1 audio formats to EAC3 7.1 with Plex's EasyAudioEncoder or encode lossless 5.1 audio formats to AAC 5.1 with QAAC TVBR 127.
If DoVi, strip DoVi RPU from original MKV and interleave RPU back into compressed HEVC video with dovi_tool as Profile 8.1
Remux compressed video, compressed audio, and English subtitles labeled as Subtitles, SDH, and Translations with the
I also strip any commentary/duplicate subtitles while remuxing and set my optimal MKV flags for everything. 'Default = off' for all subtitles, 'hearing impaired = on' for SDH, and 'forced = on' for Translations). I also set 'original = on' for everything that's in the original language.
Side note, I have a slightly different workflow for foreign films to ensure I capture the additional audio and occasionally multiple subtitle formats (dubtitles, original language subtitles, SDH, forced translations, etc).


Any advice/recommendations are appreciated. Thank you!

[RanmaCanada]

Personally I would recommend you get rid of your Apple POS and move to a Shield as it's far more user friendly in regards to playing your own content. Apple devices are great if you want to stay in their ecosystem, not so great if you actually want to use the hardware outside of it.

[benwaggoner]

This is all HDR? HDR and SDR need a variety of different parameters.

At CRF 16-18 most things should look pretty good, so it's more a matter of tuning for encoding speed and file size.

deblock=-1,-1 is much less needed in x265 than x264.
--ctu 32 is a good default for live action
--hdr10-opt is pretty essential for HDR-10 encoding
The more grain, the higher --psy-rd, --psy-rdoq, --nr-intra and --nr-inter values you'll want to use. --rd 4 can outperform --rd 6 with high grain as well.

--rskip 2 is better than the default mode.

For animation, I like to use --tskip, --tskip-fast, --tu-intra-depth 4, --tu-inter-depth 4

--csv-log-level 2 can be great for advanced users in diagnosing quality issues.

Do NOT use --tune grain, which is horribly old and terrible.

Assuming you are doing SDR and HDR across 8 genres, that's 16 different parameter sets. If you want more than just general advice, you should do some test runs, then share your command lines and more specific questions.

I have no idea what counts as a "low luminance film" - can you give an example?

[GodzilaAvenger]

This is all HDR? HDR and SDR need a variety of different parameters.

At CRF 16-18 most things should look pretty good, so it's more a matter of tuning for encoding speed and file size.

deblock=-1,-1 is much less needed in x265 than x264.
--ctu 32 is a good default for live action
--hdr10-opt is pretty essential for HDR-10 encoding
The more grain, the higher --psy-rd, --psy-rdoq, --nr-intra and --nr-inter values you'll want to use. --rd 4 can outperform --rd 6 with high grain as well.

--rskip 2 is better than the default mode.

For animation, I like to use --tskip, --tskip-fast, --tu-intra-depth 4, --tu-inter-depth 4

--csv-log-level 2 can be great for advanced users in diagnosing quality issues.

Do NOT use --tune grain, which is horribly old and terrible.

Assuming you are doing SDR and HDR across 8 genres, that's 16 different parameter sets. If you want more than just general advice, you should do some test runs, then share your command lines and more specific questions.

I have no idea what counts as a "low luminance film" - can you give an example?

Any recommendations for encoding UHD HDR Blu-rays at CRF 22 for the three categories (no grain/grain/animation)? I usually use the Slow preset with aq-mode 3, the rest default settings. Here's a sample of the setting I use:

cpuid=1111039 / frame-threads=4 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x2160 / interlace=0 / total-frames=0 /
level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop /
min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 /
hist-scenecut=0 / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 /
rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 /
limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao /
no-sao-non-deblock / rd=4 / selective-sao=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 /
psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=22.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 /
pbratio=1.30 / aq-mode=3 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 /
videoformat=5 / range=0 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=1 / chromaloc-top=2 / chromaloc-bottom=2 / display-window=0 /
min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps /
scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / hdr10 / hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 /
analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao /
ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field /
qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass

(writing library: x265 3.5+1-f0c1022b6:[Linux][GCC 11.2.0][64 bit] 10bit)

[benwaggoner]

I don't have much experience encoding for UHD Blu-ray.

Could you share your actual command line? Those are easier to parse.

For HDR, --cbqpoffs and --crqpoffs can be better at -1 or -2.
--selective-sao 2 is a free way to get a bit of a speedup without hurting quality.
A number of these settings aren't Blu-ray compatible. Did you set. --uhd-bd.

The constraints of --uhd-bd might give you some perf back that could allow you to try --preset slower.

[GodzilaAvenger]

I misspoke perhaps, I want to re-encode my personal UHD HDR Blu-ray collection like the OP. Here's the FFmpeg CLI options I use (for video):

-max_muxing_queue_size 1024 -map 0:0
-c:v libx265 -pix_fmt yuv420p10le
-x265-params "aq-mode=3:repeat-headers=1:strong-intra-smoothing=1:bframes=4:b-adapt=2:frame-threads=0
:colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:hdr10_opt=1:hdr10=1:chromaloc=2"
-crf:v 22 -preset:v slow

[benwaggoner]

So you're encoding to general .mp4 HEVC from UHD-BD sources? Not making UHD-BD compatible files?

[GodzilaAvenger]

Yes, encoding from a Blu-ray source, though usually to a .mkv file.

[benwaggoner]

Ah, okay. That makes a lot more sense. Have you tried my recommendations?

[GodzilaAvenger]

Yes. I've been testing out different options on my copy of The Matrix Resurrections as a start, and because I can't seem to tell the difference visually and don't like to go pixel-peeping, I've also been calculating the VMAF score (along with CAMBI, SSIM, and PSNR) for the first 10 minutes to try and see which options work better.

I saw a small speed improvement with selective-sao=2 (from 6.09 fps to 6.11 fps) without hurting the quality. no-sao resulted in a higher VMAF score (99.07 compared to 99.02), though weirdly the speed was the same as selective-sao=4 (i.e. 6.09 fps). I'm not sure if at CRF 22 the bitrate is low enough that I should keep SAO to prevent ringing.

I tested different b-frame values and ended up settling on the default 4 for the Slow preset, since larger ones seem to only increase encoding time without any improvement in quality. I have not changed the psy-rd and psy-rdoq values because there is no film grain, but I'm testing ctu=32 now to see if there is any improvement.

Any recommendations before I move on to more grainy stuff?

[Boulder]

--rskip 2 --rskip-edge-threshold 2 is recommended for grain or detail retention. It will also help with the onion artifacts that sometimes tend to appear in flat areas.

[benwaggoner]

Yes. I've been testing out different options on my copy of The Matrix Resurrections as a start, and because I can't seem to tell the difference visually and don't like to go pixel-peeping, I've also been calculating the VMAF score (along with CAMBI, SSIM, and PSNR) for the first 10 minutes to try and see which options work better.
The error bars on VMAF are pretty big. A difference of less than 3 VMAF is only visible 75% of the time IIRC. And VMAF is a lot less accurate for HDR than SDR, unfortunately. It's useful for fine tuning, but there's really no way around pixel-peeping if you want to know how good something looks at low-moderate bitrates.

I saw a small speed improvement with selective-sao=2 (from 6.09 fps to 6.11 fps) without hurting the quality. no-sao resulted in a higher VMAF score (99.07 compared to 99.02), though weirdly the speed was the same as selective-sao=4 (i.e. 6.09 fps). I'm not sure if at CRF 22 the bitrate is low enough that I should keep SAO to prevent ringing.
I consider SAO beneficial overall at CRF 22.

A VMAF difference of 0.05 with HDR doesn't tell you anything, alas. You'll need to eyeball SAO versus non-SAO if you want to know which is better.

I tested different b-frame values and ended up settling on the default 4 for the Slow preset, since larger ones seem to only increase encoding time without any improvement in quality. I have not changed the psy-rd and psy-rdoq values because there is no film grain, but I'm testing ctu=32 now to see if there is any improvement.
More B-frames help improve compression efficiency with low noise content, particularly animation. You'd expect to see reduced file size more than a visual quality change with higher values.

ctu=32 will speed up multithreaded performance if you have a lot of cores, and can help film grain quality a bit. It will reduce compression efficiency for cleaner content, however.

Any recommendations before I move on to more grainy stuff?
The main thing is that you don't have any objective metrics that have strong subjective correlation for HDR content.
Also, for longer-form content, the average of any metric is even less useful. The quality impact of the worst looking shots is a lot higher than the best looking shots, so the variability in metrics is more important. For example, visually inspect shots with the the lowest values for harmonic mean 2 second rolling averages. A lot of my encoding tuning time is spent making the worst 1% of frames look better.

[GodzilaAvenger]

This is really great advice, thanks!

[lucasrj2002]

Hello, thanks for starting this thread, I'm also interested in doing high quality encodes of UHD HDR movies/shows as well.

Using x265 v3.5, under the -preset slow setting, the aq-mode=2 is the default. Have you deviated from this and tried aq-mode=4 and aq-strength=0.8 for your HDR videos?

Just wondering if aq-mode=4 with aq-strength=0.8 would be better that the aq-mode=2 default setting in your opinion.

Thanks!

[benwaggoner]

Hello, thanks for starting this thread, I'm also interested in doing high quality encodes of UHD HDR movies/shows as well.

Using x265 v3.5, under the -preset slow setting, the aq-mode=2 is the default. Have you deviated from this and tried aq-mode=4 and aq-strength=0.8 for your HDR videos?

Just wondering if aq-mode=4 with aq-strength=0.8 would be better that the aq-mode=2 default setting in your opinion.
I've found --aq-mode 2 to be superior to --aq-mode 4 with HDR content in my testing, and given a whole lot of other tuning.