Tilling when encoding with x265 10bits HDR

[rabbithood]

[Solved - SAO disabled] Hello everyone,

When I encode Blade Runner (1982) in HEVC from the UHD Bluray, I have background tilling. You can see it in the picture below:
https://i.goopics.net/wtfyoa.jpg

It's much stronger in real life and I don't know where it comes from.
I'm on linux and I tried with very various settings (resolution, bitrate, number of pass, mono thread, multithread, deblock, etc.) and with the following encoders :
- Handbrake stock.
- Handbrake on debian backports.
- Handbrake flatpak.
- Fastflix
- FFmpeg stock.
- FFmpeg static builds 5.0, 5.1.1.

I can't get rid of it. Thanks for your help !

[benwaggoner]

it's more a matter of what parameters you're using than what build you use. Are you specifying a preset and tune?

I've seen --ctu 32 help with that sort of artifact before.

[rabbithood]

Thanks for the reply !

I've tried --ctu 32. No changes whatsoever.

This is the command line (I stripped filenames) :

-metadata title="Blade Runner: The Final Cut" -max_muxing_queue_size 1024 -filter_complex "[0:0]scale=1920:-8:flags=neighbor[v]" -map "[v]" -c:v libx265 -pix_fmt yuv420p10le -x265-params "ctu=32:aq-mode=2:repeat-headers=1:strong-intra-smoothing=1:bframes=4:b-adapt=2:frame-threads=0:colorprim=bt2020:transfer=smpte2084:colormatrix=bt2020nc:hdr10_opt=1:master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(40000000,50):max-cll=1655,117:hdr10=1:chromaloc=2" -crf:v 18 -preset:v slow -map_metadata -1 -map_chapters 0 -default_mode infer_no_subs

I've not only tried with different encoders. I've tried with no multithreading. Same results.

Thanks for your help !

[Boulder]

Try with aq-mode 1. Also --rskip 2 --rskip-edge-threshold 2 will help with any grainy/noisy video.

[rwill]

This should be caused by the sample adaptive offset implementation in x265. Disable SAO and try again ?

[rabbithood]

Yes ! It was sao. :thanks: everybody.

[rwill]

I have given a reproduction case of this x265 SAO behavior here:

https://forum.doom9.org/showthread.php?p=1949376#post1949376

But no one cared, MCW even less I assume because they have not fixed it yet.

[benwaggoner]

I have given a reproduction case of this x265 SAO behavior here:

https://forum.doom9.org/showthread.php?p=1949376#post1949376

But no one cared, MCW even less I assume because they have not fixed it yet.
I hope they'll get to it. We finally got a good bunch of commits last week; the most progress we've seen with x265 in a year or two.

[rwill]

I hope they'll get to it. We finally got a good bunch of commits last week; the most progress we've seen with x265 in a year or two.

The commits were just for 2 features. One temporal filter where I don't think its any good in an encoder and one that is switching AQ modes more or less like a headless chicken. If MCW wants to be even more productive I recommend to split each of their feature patches into 3-5 lines and commit these separately. That way they get even more commits and the simple folk would be even more impressed.

[jpsdr]

@rwill
Do you have any idea (by any chance), between sbrc and aq-mode 5 if one seems better than the other ?

Edit :
According your comment, it made me remember there is indeed a risk with sbrc (and so also aq-mode 5), for the few i understood, comparing to an issue i've encountered looong time ago in one of my filter experiment, with a frame by frame decision with a threshold, and when there is noticeable difference result between the results : FLICKERING !!!
If you're unlucky... no... when you be unlucky... :D, you have, for the same scene (so a succession of frames almost identical), the computed value just at the threshold value, and with just a little of noise in your pictures, you have frame N under threshold, N+1 above, N+2 under, N+3 above, etc... (the more the noise, the more the effect will happend with computed value not just at the threshold), when, in theory, as it's the same scene, so for the scene, the decision should stay the same.
For things like that, i think, it's better to use instead, rough exemple, if computed value < 30 filter value = 1.0 else filter value = 0.0, use a ramp with % over threshold, and doing instead : if computed value < 30 - 20% (or 25% or ..) my filter value = 1.0 else if computed value > 30 + 20% (or...) filter value = 0.0 else filter value = linear ramp 1.0 to 0.0 according computed value.

[Stereodude]

The commits were just for 2 features. One temporal filter where I don't think its any good in an encoder and one that is switching AQ modes more or less like a headless chicken.
Why do you say it switches AQ modes like a headless chicken?

[jpsdr]

For each frame, the filter compute edgeIntensity and brightnessIntensity, and choose AQ-MODE according.
Maybe there is the risk i described i my post just above, and so the AQ-MODE can greatly change frame by frame.
I think the idea is very interesting, but implemented with "fuzzy logic" would probably avoid the issue i'm afraid of...


/* AQ mode switch */
if (edgeIntensity < FRAME_EDGE_THRESHOLD)
preFrame->m_frameSegment = brightnessIntensity > FRAME_BRIGHTNESS_THRESHOLD? X265_AQ_AUTO_VARIANCE : X265_AQ_AUTO_VARIANCE_BIASED;
else
preFrame->m_frameSegment = brightnessIntensity > FRAME_BRIGHTNESS_THRESHOLD? X265_AQ_EDGE : X265_AQ_EDGE_BIASED;

[benwaggoner]

@rwill
Do you have any idea (by any chance), between sbrc and aq-mode 5 if one seems better than the other ?

Edit :
According your comment, it made me remember there is indeed a risk with sbrc (and so also aq-mode 5), for the few i understood, comparing to an issue i've encountered looong time ago in one of my filter experiment, with a frame by frame decision with a threshold, and when there is noticeable difference result between the results : FLICKERING !!!
Boolean thresholds like that are often problematic in video encoding, as you get threshold conditions like you describe. When something straddles the threshold, all kinds of unpleasant and suboptimal things can happen. Some kind of a lookahead with a deadzone so it picks the optimal mode per-shot is preferable. It's rare and only in weird corner cases where it would be beneficial to change aq-mode during a contiguous series of frames.

[benwaggoner]

The commits were just for 2 features. One temporal filter where I don't think its any good in an encoder
Are you saying you don't think that temporal filters are good in general, or that this particular implementation isn't good.

There's a ton of implicit temporal filtering in encoding already just by having interframe prediction, so I don't think we can make any general statement about temporal filtering being good or bad. --cutree is a temporal filter, for example, biasing towards CUs that future frames make predictions from.

It is, like so many things, down to the implementation. I've certainly seen MCTF filters improve quality at a given bitrate, and lower ABR when using --crf. That required a whole lot of tuning and encoder integration to make it reliably "do no harm."

--aq-motion is an example of temporal filtering that wasn't refined enough to be safe to use by default.

[jpsdr]

It's rare and only in weird corner cases where it would be beneficial to change aq-mode during a contiguous series of frames.

Not so weird, much more specific... A typical scene who be some kind of dancing party with flashing light. Despite being the same scene, you'll encounter successive frames with big enough different "average light", which probably may benefit to change aq-mode.

About mctf, as it requires disabling multi-threading, i may be wrong, but i don't think a lot of people will use it unless they have insane CPU power or quantity of time...

[rwill]

Are you saying you don't think that temporal filters are good in general, or that this particular implementation isn't good.

From my experience, putting a filter before the prediction step in a video encoder is a bad idea most of the time.

I also looked at the code of the specific x265 patches and now have some doubts about runtime and threading performance.

[Boulder]

I would personally never allow the encoder itself to do any filtering. There are much better alternatives for that since each and every source requires at least validation of the parameters and their visual effect.

[benwaggoner]

Not so weird, much more specific... A typical scene who be some kind of dancing party with flashing light. Despite being the same scene, you'll encounter successive frames with big enough different "average light", which probably may benefit to change aq-mode.

About mctf, as it requires disabling multi-threading, i may be wrong, but i don't think a lot of people will use it unless they have insane CPU power or quantity of time...
Or they're doing high resolutions without so many cores. Doing 4K on my MacBook Pro, all eight performance cores are saturated even without frame threading.

There are other ways to accelerate with lots of cores besides frame threading, like --pmode, --rc-lookahead, --lookahead-slices, etc.

[ShortKatz]

@rwill
Do you have any idea (by any chance), between sbrc and aq-mode 5 if one seems better than the other ?

There is no aq-mode 5 with the official x265 (see /doc/reST/cli.rst). Only with the mods on GitHub.