Color banding when downscaling from UHD HDR10 to FHD - bug in scaler?

[Woodstock]

I would like to downscale a video from 2160p to 1080p, and want to encode it in HEVC with HDR10, same as the source. I already found in the past days that this is not a trivial thing to do if you want to avoid video encoding tools that only use 8 bit internally. I only found and tried two working ways so far: Decode and downscale with either AviSynth+ or FFmpeg, and pipe it to x265 (10bit version).

Unfortunately I found color banding in the reencoded video, which looks much worse than encoding it with SDR. I tried to find out where the banding happens. I piped the FFmpeg output to y4m files and found the color banding already there, so x265 was innocent. I skipped the downscaling and the banding was gone, so I suspected the scaler did something. I confirmed that the scaler has matching input and output pixel formats (yuv420p10le), so that there should be no conversion. I checked different scaling algorithms (bicubic, bilinear, lanczos, neighbor), but all show the banding, even nearest neighbor! I checked different output video sizes and found that 2176x1224 and below shows banding, but 2240x1260 and above did not. This happens with both AviSynth+ and FFmpeg! I can extract video frames with banding and post them here, if it would be useful for someone to find an explanation.

One of my test conversion lines:
ffmpeg.exe -i input.mp4 -strict -1 -pix_fmt yuv420p10le
-vf "scale=iw/2:-1:out_color_matrix=bt2020:out_h_chr_pos=0:out_v_chr_pos=0:flags=print_info"
-map 0:v:0 -f yuv4mpegpipe - |
x265.exe --preset slow --input - --y4m --output-depth 10 --profile main10 --level-idc 5.1 --crf 0
--vbv-bufsize 160000 --vbv-maxrate 160000 --range limited
--colorprim bt2020 --transfer smpte2084 --colormatrix bt2020nc
--master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)"
--max-cll "1000,400" --output output.hevc

I would appreciate any help to find out where the banding comes from, if it is a bug, if there is a workaround or another program, etc.

[Motenai Yoda]

try to add -sws_dither bayer in the ffmpeg command line

[Woodstock]

try to add -sws_dither bayer in the ffmpeg command line

Thank you for the suggestion. Dithering is of course a solution to fix color banding, but I am looking for a solution on how to avoid it. The decoding and encoding is not responsible for it, so there must be something unwanted happening in the scaling. I fear that the scaling happens in an 8 bit color space, which would explain where the banding comes from.

[sneaker_ger]

Dithering is banding avoidance. Don't confuse it with debanding.

If scale filter doesn't work try zscale. (Also with dither.)

[Motenai Yoda]

I fear that the scaling happens in an 8 bit color space, which would explain where the banding comes from.
what I suspect is that scaler doesn't dither its output or at least doesn't do it in a compression-proof way.
HDR contents dynamic range is so wide that 10bit shades are about the same as 8bit SDR ones.
So dithering is avoidable for 10bit SDR (IMHO not a good idea) but almost mandatory for 10bit HDR stuff.

(also I don't know if ffmpeg-scaler will process in float or higher bitdepth, maybe converting to yuv420p12le first can help)

[benwaggoner]

what I suspect is that scaler doesn't dither its output or at least doesn't do it in a compression-proof way.
HDR contents dynamic range is so wide that 10bit shades are about the same as 8bit SDR ones.
So dithering is avoidable for 10bit SDR (IMHO not a good idea) but almost mandatory for 10bit HDR stuff.

(also I don't know if ffmpeg-scaler will process in float or higher bitdepth, maybe converting to yuv420p12le first can help)
Last time I dove into the incredibly-undocumented weeds of ffmpeg, lots of the interesting scaling and dithering stuff didn't work with >8-bit.

[nevcairiel]

Try adding "accurate_rnd" to the scale flags, it can in such cases improve the precision.

[kolak]

Yes, dithering in ffmpeg is somehow "broken" and gives strange results (or most often no results).

[Woodstock]

Dithering is banding avoidance. Don't confuse it with debanding.

Thanks, let me rephrase: I want to avoid the operation which causes the banding. The decoded video shows no banding, and scaling should not cause banding in this case, especially with nearest neighbor algorithm. So if the banding is caused by some other process, e.g. color space conversion, I want to eliminate it. I also find it weird that the banding only shows for 2176x1224 and lower resolutions, but not higher. Does ffmpeg has different dithering defaults for different resolutions?

If scale filter doesn't work try zscale. (Also with dither.)

Thanks, I tried (without dither). Unfortunately the same result as scale, banding for 2176x1224 and lower.

[Woodstock]

Try adding "accurate_rnd" to the scale flags, it can in such cases improve the precision.

Thanks, I tried. Unfortunately no visible change. And I don't think that the scaling itself is responsible for the banding as it also happens with nearest neighbor algorithm.

[Asmodian]

Why would NN not cause banding? :confused:

[Woodstock]

Why would NN not cause banding? :confused:

Because NN only picks colors which already exist in the source. If the source shows no color banding, the destination should also not show it. NN should only cause aliasing effects, but no banding.

[poisondeathray]

Can you stream copy a few frames from the source mp4 and post it ?

[nevcairiel]

Because NN only picks colors which already exist in the source. If the source shows no color banding, the destination should also not show it. NN should only cause aliasing effects, but no banding.

Thats not quite true. If you have a smooth gradient and downscale it with NN (or realistically, any algorithm), you'll get banding because the resulting image is quite simply smaller, and the gradient has to be represented with less pixels - less pixels means perhaps not enough pixel to fully represent every step of the gradient, and missed steps cause banding.

[Asmodian]

And that is why you need dithering. :)

[Blue_MiSfit]

Does zscale not dither properly during >=10 bit scaling?

[Weyoun]

Be sure to compare your colors after you downscale and dither. It may give a tint to your video.

[Woodstock]

Thats not quite true. If you have a smooth gradient and downscale it with NN (or realistically, any algorithm), you'll get banding because the resulting image is quite simply smaller, and the gradient has to be represented with less pixels - less pixels means perhaps not enough pixel to fully represent every step of the gradient, and missed steps cause banding.

Is it possible that you mix it up? Let's start with a smooth gradient as you mentioned:
0123456789

Then downscale it with nearest neighbor:
02468

See what I mean? If you have a smooth gradient and reduce the pixels, how can you have not enough pixels afterwards? I guess you meant the opposite. You can get banding if you upscale it:

0000111122223333444455556666777788889999

Because you have not enough shades for each pixel.
Or if you upscale 02468 with nearest neighbor you get for example:

00002222444466668888

Also banding due to missing interpolation.

If is also possible to get banding from scaling algorithms like bilinear or bicubic as they can smooth away an existing dithering. But I don't see how downscaling with nearest neighbor can cause banding. I am always happy to learn more though. :)

[nevcairiel]

Is it possible that you mix it up? Let's start with a smooth gradient as you mentioned:
0123456789

Then downscale it with nearest neighbor:
02468


But this isn't quite smooth anymore, is it? Sure, its even steps, but still bigger steps then before. How many steps can you take out before you see visual banding?

[Woodstock]

But this isn't quite smooth anymore, is it? Sure, its even steps, but still bigger steps then before. How many steps can you take out before you see visual banding?

If each pixel has another shade it is perfect: the gradient can't be smoother than that on the specific display. Color banding by definition requires the same color on multiple pixels, so something like 00112233445566778899 would be the minimum to call it banding.

[Woodstock]

Can you stream copy a few frames from the source mp4 and post it ?

Thank you, it was your request that brought me one step forward. I didn't want to copy the full UHD frames, so I cropped out a part which significantly shows the banding after down scaling. Surprisingly it now showed the banding without any scaling, just with cropping.

After a few tests I found that ffmpeg is innocent: The player causes the color banding! So I have to do some research and then maybe post in the Software players forum.

Just some info already here: I use PotPlayer with madVR 92.10. For unknown reasons it shows color banding if the source video is 2176x1224 or smaller, but not if it is 2240x1260 or bigger. The output video size doesn't matter. The madVR debug OSD shows the same correct values for both. According to my knowledge (MediaInfo) both test videos are identical apart from the size. Both videos were created with the same ffmpeg and x265 settings from the same UHD source. Both videos look the same in VLC, but VLC shows washed out colors as it doesn't properly support HDR.

[sneaker_ger]

Are you using DXVA native? Is dithering in madvr disabled?

[Woodstock]

Are you using DXVA native? Is dithering in madvr disabled?

I already tried many settings, including with and without DXVA, and different dithering algorithms. They all showed the banding.