Minimizing/eliminating banding in 10-bit encodes of flat'ish subtle gradient areas

[Stereodude]

I have a 1920x816 10-bit 4:2:0 source that I'm attempting to encode with x265. The problem is the 10-bit output from x265 has banding/posterization in the subtle gradients of certain areas of the clip.

Aside from the --aq strength parameter what other command line tweaks can minimize banding / posterization of subtle gradients in 10-bit encodes without blowing up the bitrate? --lossless isn't a viable option. :p

The banding is not in the source going into x265 and the banding in the output is only visible if you have a full 10-bit (or greater) end to end playback chain. It's totally invisible when the encode is played back dithered to 8-bits by madVR.

I started out with these command line options for the encode:

--crf 21.0 -p veryslow --vbv-maxrate 50000 --vbv-bufsize 50000 --level 4.1 --keyint 120 --open-gop -D 10 --colorprim "bt709" --transfer "bt709" --colormatrix "bt709" --sar 1:1 -o output_g1.0_21_10b.265 --input - --y4m
This happens:
avs2pipemod[info]: writing 12706 frames of 24000/1001 fps, 1920x816,
sar 0:0, YUV-420-planar-10bit progressive video.
y4m [info]: 1920x816 fps 24000/1001 i420p10 sar 1:1 unknown frame count
raw [info]: output file: output_g1.0_21_10b.265
x265 [info]: HEVC encoder version 2.7+3-9086c8a3e76d
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit+8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-4.1 (High tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices : 1
x265 [info]: frame threads / pool features : 4 / wpp(13 rows)
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 3 inter / 3 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 4 / 4
x265 [info]: Keyframe min / max / scenecut / bias: 12 / 120 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt : 40 / 8 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 1
x265 [info]: References / ref-limit cu / depth : 5 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : CRF-21.0 / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init : 50000 / 50000 / 0.900
x265 [info]: tools: rect amp limit-modes rd=6 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: rskip limit-tu=4 signhide tmvp b-intra
x265 [info]: tools: strong-intra-smoothing deblock sao
avs2pipemod[info]: finished, wrote 12706 frames [100%].
avs2pipemod[info]: total elapsed time is 7246.633 sec.
x265 [info]: frame I: 160, Avg QP:17.40 kb/s: 5027.14
x265 [info]: frame P: 2199, Avg QP:18.95 kb/s: 2655.17
x265 [info]: frame B: 10347, Avg QP:25.57 kb/s: 438.57
x265 [info]: Weighted P-Frames: Y:1.5% UV:1.0%
x265 [info]: Weighted B-Frames: Y:0.8% UV:0.4%
x265 [info]: consecutive B-frames: 6.4% 1.1% 3.3% 21.8% 14.5% 29.6% 8.7% 9.8% 4.8%

encoded 12706 frames in 7253.52s (1.75 fps), 879.97 kb/s, Avg QP:24.32

Aside from the ugly banding in the gradients in a few spots it's not visually objectionable and the output is only 55.6MB.

Increasing the CRF to 18.0 helps some, but doesn't eliminate it. The encode grows to 91.3MB.

Increasing the aq strength also helps, but continues to push the bitrate up. Keeping the CRF of 18.0 and somewhere around an aq strength of 1.5 the banding is pretty minimal and hard to notice unless you're really looking for it but the encode has now grown to 229MB.

An 8-bit ( -D 8 ) x265 CRF 18.0 encode with f3kdb(range=20, grainY=32, grainC=24, sample_mode=2, dither_algo=3, dynamic_grain=true, keep_tv_range=false) as the last AVS+ script step looks great and is only 81.1MB. I keep reading about how 10-bit encoding is more efficient, but I'm struggling to find those settings to realize similar visual quality even accepting a size penalty (which I supposedly shouldn't have to).

Also, I have confirmed the banding is not in the source being fed to x265 by playing the raw y4m output back through the same playback chain (it's clean). The banding also isn't an artifact of HEVC 10-bit playback through my playback chain (like a decoder bug) which I've confirmed by using a lossless x265 encode of it (that is banding free).

Any other defaults to change or encoding parameters to add?

[poisondeathray]

880kbps for 1920x816p23.976 doesn't seem to be a lot of bitrate to me. I would expect banding (along with other problems) for a typical source. So it looks better with lower CRF (no really?! :P :D) and no banding with lossless - that pretty much confirms you're not using enough bitrate.

If there are specific scenes or frames that are problems, you might try zones to affect those only areas

[foxyshadis]

Set --aq-mode 3 and bump up --aq-strength to prevent low-bitrate blocks from decimating down to DC-only, which is automatic banding in any bit-depth.

[Stereodude]

880kbps for 1920x816p23.976 doesn't seem to be a lot of bitrate to me. I would expect banding (along with other problems) for a typical source. So it looks better with lower CRF (no really?! :P :D) and no banding with lossless - that pretty much confirms you're not using enough bitrate.

If there are specific scenes or frames that are problems, you might try zones to affect those only areas
Yes, I get all that, but why do I need >3x the bitrate for a 10-bit encode vs a dithered 8-bit encode to minimize the banding?

[sneaker_ger]

why do I need >3x the bitrate for a 10-bit encode
Do you? Your tests haven't shown this. CRF is arbitrary.

[Stereodude]

Do you? Your tests haven't shown this. CRF is arbitrary.
Well, that's kind of the root of my question/post... My test so far show a 10-bit encode that still has some minor banding is ~3x the size of an 8-bit encode that doesn't have banding.

Adding dithering to an 8-bit source to prevent banding increases bitrate. AFAIK, 10-bit encoding is supposed to be more efficient due to reduced quantizer errors. So, it seems like in this particular case a 10-bit encode should be smaller due to the lack of 8-bit dithering and due to 10-bit encoder efficiency. However, using my limited knowledge of x265 I can't even get close to relative visual parity (on a full end to end >=10-bit playback chain) between the 10-bit and 8-bit encode at a file similar size/bitrate.

Sure, I haven't run every possible permutation of command line options. I'm here looking for suggestions / pointers because my results so far are not at all what I expected and I'm no x265 expert.

Here's a 6 second lossless HEVC snippet of 3 scenes, each 2 seconds long, from the source that show banding after x265 10-bit encoding if someone wants to try their hand at it.

Here's the troublesome areas highlighted for each scene.
First:
https://i.imgur.com/6xMfWDI.jpg

Second:
https://i.imgur.com/vYVkgC5.jpg

Third:
https://i.imgur.com/sWMFKXT.jpg

[poisondeathray]

The compression efficiency difference for 10bit vs. 8bit is larger for x264 compared to x265

Is your source native 10bit, and is it clean ? What is your script ?

You said "as the last AVS+ script" - If you're using the same f3kdb script, are you sure you even need that for 10bit ? You're just adding noise and that's going to require more bitrate

[poisondeathray]

Ah I didn't see your last post - If those screenshots are of the source, it's a garbage source - you can see banding and quantization artifacts already

[Stereodude]

The compression efficiency difference for 10bit vs. 8bit is larger for x264 compared to x265

Is your source native 10bit, and is it clean ? What is your script ?

You said "as the last AVS+ script" - If you're using the same f3kdb script, are you sure you even need that for 10bit ? You're just adding noise and that's going to require more bitrate
No, the original source isn't native 10-bit. It's a clip that had the levels of the luma channel shifted by 16. I corrected the video levels in 12-bits and dithered down to 10-bits.

I'm not using f3kdb for the 10-bit encode.

10-bit script:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")
MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNVx86.dll")

DGSource("source.dgi").crop(0,132,-0,-132)
### prefetch: 10, 5

### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\MVtools\mvtools2.dll")
Import("C:\HDTV Tools\MCTD\MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="low", radius=3)
### prefetch: 8, 4
### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\flash3kyuu_deband\flash3kyuu_deband-x86.dll")
ConvertBits(12)
ColorYUV(conditional=true,gamma_y=0.0, levels="TV", analyze=false)
ConditionalReader("AC_levels_t.txt", "coloryuv_off_y", false)
ConditionalReader("AC_gain_t.txt", "coloryuv_gain_y", false)
ConvertBits(10, dither = 1)

### prefetch: 8, 4
### ###

""")

8-bit script:
LoadPlugin("C:\HDTV Tools\MP_Pipeline\x64\MP_Pipeline.dll")
MP_Pipeline("""

### platform: win32
# this part of script will be run in a 32 bit process
LoadPlugin("C:\HDTV Tools\DGDecNV\DGDecodeNVx86.dll")

DGSource("source.dgi").crop(0,132,-0,-132)

ConvertBits(12)
ColorYUV(conditional=true,gamma_y=0.0, levels="TV", analyze=false)
ConditionalReader("AC_levels_t.txt", "coloryuv_off_y", false)
ConditionalReader("AC_gain_t.txt", "coloryuv_gain_y", false)
ConvertBits(8, dither = 1)
### prefetch: 10, 5

### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\MVtools\mvtools2.dll")
Import("C:\HDTV Tools\MCTD\MCTemporalDenoise.v1.4.20.avsi")

SetMemoryMax(1280)
MCTemporalDenoise(settings="low", radius=3)
### prefetch: 8, 4
### ###
### platform: win32
LoadPlugin("C:\HDTV Tools\flash3kyuu_deband\flash3kyuu_deband-x86.dll")

f3kdb(range=20, grainY=32, grainC=24, sample_mode=2, dither_algo=3, dynamic_grain=true, keep_tv_range=false)

### prefetch: 8, 4
### ###

""")

[Stereodude]

Ah I didn't see your last post - If those screenshots are of the source, it's a garbage source - you can see banding and quantization artifacts already
Well, they're not 16-bit PNG's. They're quick and dirty half res JPEGs that were converted to 8-bits however VD2 renders P010 to 8-bits for display and then copied to the clipboard where I resized them in paint.net and saved them to JPEG.

I'm not claiming the source is great, but after processing it's a whole lot better looking than the 10-bit x265 encode of it (at similar bitrates to an 8-bit x265 encode).

[poisondeathray]

If you look at your 10bit "source" , there are splotches/patches and banding problems already there that will predispose you to worse problems after encoding. If you can't see them, boost the gamma to visualize it temporarily.

The 8bit encode "covers up" the problems with f3kdb; since your "source" has problems you probably need to do something as well for the 10bit version. In order for x265 (or x264) to retain grain or dither, you have to use appropriate settings too.

If you take a fairly clean 8bit source, but prone to banding with a 2nd generation encode eg. (blue sky outdoor shot). That is where 10bit encoding helps immensely. You don't even have to add additional grain, or filters like f3kdb, nor do you need any special encoding settings

[foxyshadis]

Did you even try --aq-mode 3?

Your source is kinda shit, but you aren't highlighting the worst source areas. x265 can definitely do better.

[Stereodude]

Did you even try --aq-mode 3?

Your source is kinda shit, but you aren't highlighting the worst source areas. x265 can definitely do better.
I did. I didn't see a whole lot of improvement. It was a little better.

I tried using mClean which is high bitdepth capable NR instead of MCTD (which is limited to 8 bits) and it makes a better looking histogram and gave quite a bit better looking x265 output with less banding with similar command lines, so I figured that the non smooth / contiguous nature of the chroma and luma channels with narrow intermittent spikes was messing with x265 in my original processing method.

I decided to take that to the logical conclusion and work in 16 bits and use fk3db to convert/dither/deband down to 10-bits. The histograms look very smooth and continuous without narrow spikes, but the x265 output looked as bad as what I started with, even when set to encode lossless. I was convinced there was somehow a mistake in the script or something else, but came up empty handed figuring out how the even smoother looking histograms gave banding on playback. I gave up in frustration for the day on Sunday and haven't revisited it yet.

From a given sample video frame:

https://i.imgur.com/Y2OFlFd.png

Part of my problem is that I only have one 10-bit capable playback system (my HTPC + TV) so it's hard to visualize what I'm doing since I'm doing all the fiddling with AVIsynth+ on my desktop where they basically all look the same in 8-bits. I have to work somewhat blind with no real preview ability and then encode and go watch it in the other room to see what I got.

[Stereodude]

Okay, so I went back and took another look at it last night. The HEVC encodes from the source after being processed by mClean without fk3db are some of the best looking ones at low'ish bitrates. They still have some banding, but's not as visibly bad with the same command line as other processing/filter combinations in AVIsynth+. However, with the mClean script increasing the bitrate (by decreasing the CRF and or increasing the AQ strength) doesn't make the encodes appreciably better.

If I compare the raw uncompressed 10-bit outputs from AVIsynth+, the 10-bit output from a script using mClean without fk3db is the worst looking of them (in terms of banding). The best looking raw AVIsynth+ 10-bit output for banding is actually MCTD + fk3db (working to 10 bits). However, that best looking AVIsynth+ 10-bit output gives the worst looking x265 encodes with the greatest visible banding (with the same command line options). Go figure...

I'm at a point of frustration where it seems I should just stick with 8-bits and consider 10-bits a total failure for this. Either what's coming out of AVIsynth+ has some underlying fundamental issue(s) with this original source when processed and turned into 10-bit output that causes it to not to give visually pleasing results at 10-bits or my expectations of 10-bit are totally unrealistic (maybe both).

[foxyshadis]

I think you should investigate whether your playback chain -- decoder, renderer (resizer and color converter), monitor -- is even feeding you 10-bit at all, at least up to the RGB conversion. If anything in your playback chain is converting to 8-bit, or truncating to 8-bit while converting to RGB, then of course you're going to have enormous frustrations trying to find the difference. I'd use MPDN or MPC-BE+MadVR and enable their info panes to see what's actually happening behind the scenes, especially with any driver-based hardware acceleration.

[Stereodude]

I think you should investigate whether your playback chain -- decoder, renderer (resizer and color converter), monitor -- is even feeding you 10-bit at all, at least up to the RGB conversion. If anything in your playback chain is converting to 8-bit, or truncating to 8-bit while converting to RGB, then of course you're going to have enormous frustrations trying to find the difference. I'd use MPDN or MPC-BE+MadVR and enable their info panes to see what's actually happening behind the scenes, especially with any driver-based hardware acceleration.
I've done all that already. I am absolutely certain I have one setup with end to end 10-bit playback capability. I have Windows 8.1 using MPC-HC w/ MadVR FSE DX11 mode for 10-bit output on a GTX 1080. The video card is set to output 1920x1080 12-bit over HDMI. MadVR OSD says it's getting P10. The Denon receiver says it's doing 12-bit in and 12-bit out. The Samsung TV accepts and displays >8-bit. The best looking raw uncompressed YCbCr output from the AVIsynth+ script played with MPC-HC on the same computer and playback chain looks considerably better than what I can get out of x265 for this clip.

[sneaker_ger]

Okay, so I went back and took another look at it last night. The HEVC encodes from the source after being processed by mClean without fk3db are some of the best looking ones at low'ish bitrates. They still have some banding, but's not as visibly bad with the same command line as other processing/filter combinations in AVIsynth+. However, with the mClean script increasing the bitrate (by decreasing the CRF and or increasing the AQ strength) doesn't make the encodes appreciably better.

If I compare the raw uncompressed 10-bit outputs from AVIsynth+, the 10-bit output from a script using mClean without fk3db is the worst looking of them (in terms of banding). The best looking raw AVIsynth+ 10-bit output for banding is actually MCTD + fk3db (working to 10 bits).
Have you tried different dither algos/grain strenths of f3kdb?

[poisondeathray]

Do you want to upload your problematic 8bit source ?

The 10bit "source" you uploaded already has problems, so even if you encode lossless you will still have problems

[Stereodude]

Have you tried different dither algos/grain strenths of f3kdb?
I didn't try the other algorithms. I did dabble with grain strengths.

[benwaggoner]

Did you even try --aq-mode 3?

Your source is kinda shit, but you aren't highlighting the worst source areas. x265 can definitely do better.
-aq-mode 2 doesn't do as well in low luma, but can net out as more efficient with some content since the x265 implementation seems to lower QP in low luma more than is necessary. At a fixed bitrate, I'd expect aq-mode 2 to look better in some cases as the improved efficiency would yield lower QPs overall, including in low luma.

It's hard to derive a clear rule of thumb here.

[K.i.N.G]

As already mentioned use aqmode 3... i found that raising the qcomp also helps

[Krautmaster]

start /low /wait %workingdir%/ffmpeg -y -i "%%f" -ss 00:25:00 -t 00:03:00 -map 0:v -map 0:a -map 0:s? -c:v libx265 -x265-params "rc-lookahead=35:ssim-rd=1:ctu=32:crf=!CQ!:pbratio=1.22:subme=3:aq-mode=3:aq-strength=1.2:qcomp=0.64:no-sao=1:rd=4:psy-rd=2.5:psy-rdoq=3.5:rdoq-level=2:bframes=8:no-strong-intra-smoothing=1:weightb=1:b-intra=1:rect=1:limit-modes=1" !audio_param_0! !audio_param_1! -c:s copy "%workingdir%\%%~nf_sample.mkv"

try these values for example

aq-mode=3:aq-strength=1.2