Advice on optimizing encode for grainy animation [sample provided]

[aletaziar]

Looking for some help optimizing an encode with ffmpeg/x265. It's a 4k animation that has grain and HDR. I'm trying to find the right flags for a transparent encode at a reasonable size but it's proving difficult.

The test encodes I've made have *mostly* worked, but at 00:10.427 (frame 249, 23.976) in the sample scene provided, the left figure has pink blotches in the waist sash in the encodes. The right figure has blockier skin in the encodes, especially in the chest/shoulder and around the nose.

These issues don't seem noticeable when played in MPC-BE via MadVR, but they do show up when I compare the encode with the original using video-compare (https://github.com/pixop/video-compare), which uses ffplay without HDR. Is MadVR hiding these issues with the HDR tone-mapping? Maybe ffplay is kind of "inventing" issues because of the lack of HDR tone-mapping? I would think the blockiness I see is an issue regardless of whether the colors are accurate in ffplay.

I'm trying to reduce both those issues but I don't want to drop my crf anymore since I'm already at 12. Any suggestions would be helpful, this is my first foray into HEVC encoding. I've been doing a lot of reading but figuring this out is proving difficult. The three encode variations I tried all look pretty much the same to me.

samples:
one 15s original quality sample
three 15s encode variations
https://app.box.com/s/6atm9dfchj1nzjzhqaxjusaxchqcqz34

video-compare command, with padding for encode:
video-compare -w x980 -r pad=iw:ih+80:0:80/2 test.mkv test.x265_crf12-nosao-aqmode4.mkv
video-compare command, with cropping for original:
video-compare -w x980 -l crop=iw:ih-80 test.mkv test.x265_crf12-nosao-aqmode4.mkv

encode base:
ff -v level+panic -i test.mkv -max_muxing_queue_size 8192 \
-map 0:v -vf "crop=iw:ih-80" -pix_fmt yuv420p10le -f yuv4mpegpipe \
-strict -1 - | x265 - --y4m --profile main10 --preset slower --crf 12 \
--vbv-bufsize=160000 --vbv-maxrate=160000 --colorprim "bt2020" \
--transfer "smpte2084" --colormatrix "bt2020nc" --range "limited" \
--chromaloc=2 --master-display \
"G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,50)" \
--max-cll "1000,140" --hdr10 --hdr10-opt --repeat-headers --aud \
--hrd --no-strong-intra-smoothing --limit-refs 3 --rd 4 --dynamic-rd 3
crf12:
--selective-sao 2 test.x265_crf12.hevc
crf12, aqmode=4:
--selective-sao 2 --aq-mode 4 test.x265_crf12-aqmode4.hevc
crf12, aqmode=4, no-sao:
--no-sao --aq-mode 4 test.x265_crf12-aqmode4-nosao.hevc

test.mkv mediainfo:
General
Unique ID : 333647730702598027554471434130870826641 (0xFB02372DDF2EA7B8816759A9A0D4F691)
Complete name : R:\test.mkv
Format : Matroska
Format version : Version 4
File size : 110 MiB
Duration : 15 s 683 ms
Overall bit rate : 58.6 Mb/s
Frame rate : 23.720 FPS
Encoded date : 2023-12-09 05:01:16 UTC
Writing application : mkvmerge v81.0 ('Milliontown') 64-bit
Writing library : libebml v1.4.4 + libmatroska v1.7.1

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@High
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : V_MPEGH/ISO/HEVC
Duration : 15 s 683 ms
Bit rate : 58.6 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Variable
Frame rate : 23.720 FPS
Original frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.298
Time code of first frame : 01:13:14:22
Stream size : 110 MiB (100%)
Title : MPEG-H HEVC Video / 63607 kbps / 2160p / 23.976 fps / 16:9 / Main 10 @ Level 5.1 @ High / 10 bits / HDR10 / BT.2020
Writing library : ATEME Titan File 3.9.6 (4.9.6.2)
Language : zxx
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level : 1000 cd/m2
Maximum Frame-Average Light Level : 140 cd/m2

[benwaggoner]

Reducing your chroma qp offsets to -1 can help with the sort of color blotching you describe.

[aletaziar]

Anyone? Here are a couple of screenshots to show the pink blotches in the waist sash I'm seeing:

slowpoke compare: https://slow.pics/c/g4qzZQm2

original:
https://i.imgur.com/H9N1aDw.png

encode:
https://i.imgur.com/sBZxBhS.png

[rwill]

Anyone?

All I can suggest is to raise CRF a little to lower rate first and then use --cbqpoffs and --crqpoffs with both set to -4 or -6. Some BT.2020 HDR sequences need that if the encoder has no concept of chroma complexity.

[benwaggoner]

I'd remove --chromaloc=2 unless you know your source is that.

Also remove --no-strong-intra-smoothing unless you have some evidence it is helpful for this content. It generally isn't

Remove --dynamic-rd until you have quality tuned in. Changing --rd during the encode could cause some weird results, and it probably isn't doing anything given your very high VBV

Are you seeing quality degradations at full-speed playback? Those subtle differences may not be perceptible. Presenting HDR as non-HDR can also show "problems" not perceivable in HDR. Playing back at speed on a HDR display is really the only way to know what you're looking at.

As rwill said, try reducing --cbqpoffs and --crqpoffs. I'd start with -1 and incrementally increase. Too negative a value can starve luma of bitrate, and I'm not sure how well tuned and tested high values are.

Some --nr can help a lot with fine grain. --nr-inter low-pass filters out high temporal and spatial frequency noise. Since it doesn't impact intra frames/blocks, the texture of grain is generally preserved but it doesn't swirl around as much and is easier to encode. Something like --nr-intra 100 --nr-inter 400 could be a good starting point to tweak from.

I don't think --no-sao would help with content like this where the fine detail is almost entirely noise.

[aletaziar]

I'd remove --chromaloc=2 unless you know your source is that.

Also remove --no-strong-intra-smoothing unless you have some evidence it is helpful for this content. It generally isn't

Remove --dynamic-rd until you have quality tuned in. Changing --rd during the encode could cause some weird results, and it probably isn't doing anything given your very high VBV

Are you seeing quality degradations at full-speed playback? Those subtle differences may not be perceptible. Presenting HDR as non-HDR can also show "problems" not perceivable in HDR. Playing back at speed on a HDR display is really the only way to know what you're looking at.

As rwill said, try reducing --cbqpoffs and --crqpoffs. I'd start with -1 and incrementally increase. Too negative a value can starve luma of bitrate, and I'm not sure how well tuned and tested high values are.

Some --nr can help a lot with fine grain. --nr-inter low-pass filters out high temporal and spatial frequency noise. Since it doesn't impact intra frames/blocks, the texture of grain is generally preserved but it doesn't swirl around as much and is easier to encode. Something like --nr-intra 100 --nr-inter 400 could be a good starting point to tweak from.

I don't think --no-sao would help with content like this where the fine detail is almost entirely noise.

The source in MediaInfo shows Chroma subsampling : 4:2:0 (Type 2) so I think I do need chromaloc for this particular file. I will check out the other settings you and rwill mentioned, and avoid --no-sao and --no-strong-intra-smoothing.

Is --aq-mode 4 something I should leave in as well?

[benwaggoner]

The source in MediaInfo shows Chroma subsampling : 4:2:0 (Type 2) so I think I do need chromaloc for this particular file.

Yeah, if the source has it at 2, you'll want to keep that going forward.

I will check out the other settings you and rwill mentioned, and avoid --no-sao and --no-strong-intra-smoothing.

Is --aq-mode 4 something I should leave in as well?
--aq-mode 4 is generally fine. However, grainy animation is an unusual optimization target. I imagine you'll have to play around some with --psy-rd, --psy-rdoq, and --aq-strength to find optimal values. Generally they go higher with grain and lower with animation, so I don't have clear intuition about what might work. I suggest --rskip 2 and maybe --rskip-threshold 3 to start with. The default --rskip can have issues with grain, particularly in HDR.

You could also try --aq-mode 2 if 4 isn't working. 1 and 3 are unlikely to be helpful.