Possible bug with cutree in new version of x265 included with HandBrake

[Hellboy.]

Possible bug with cutree in the new version of x265 included with HandBrake v1.10.0. Image looks distorted. I think the problem is the new x265 because v1.9.2 don't have this bug.

Advanced Options
repeat-headers=1:aud=1:hrd=1:strong-intra-smoothing=0:rect=0:selective-sao=2:aq-mode=1:rd=3:psy-rd=0.75:psy-rdoq=0.0:rdoq-level=1:rskip=2:high-tier=1:vbv-maxrate=160000:vbv-bufsize=160000

When i use (not at the same time)"cutree=0, strong-intra-smoothing=1 or qcomp=0.70", the bug don't occurs.

When i try (not at the same time)"openGog=0, aq-mode=2, limitTu=1 or selective-sao=4", the bug still occurs.

EDIT:
Sorry, i forgot to say that i use
All filters off
FPS - Same as source - Variable Framerate
crf - 17
Encoder Preset Slow
Encoder Tune - None
Encoder Profile - Main10
Encoder Level - 5.1

*The sample i share is the original video without the bug, so anyone can convert the file and try the bug themself.

Clean Sample
https://drive.google.com/file/d/1g7C4ORrlr8ECSaCINab6NWLXPpfonTjH/view?usp=sharing

Bug happens from minute 1:13 to 1:23

[Z2697]

There are too many dials in HandBrake, are we supposed to figure it out?

[Ritsuka]

HandBrake v1.10.0 is using the latest x265 master plus a patch from the x265-devel mailing list to fix 2 pass (https://bitbucket.org/multicoreware/x265_git/issues/1001/poor-2-pass-performance-in-recent-41), I don't doubt something can go wrong.

Unfortunately there isn't much activity on x265 lately.

[Z2697]

HandBrake v1.10.0 is using the latest x265 master plus a patch from the x265-devel mailing list to fix 2 pass (https://bitbucket.org/multicoreware/x265_git/issues/1001/poor-2-pass-performance-in-recent-41), I don't doubt something can go wrong.

Unfortunately there isn't much activity on x265 lately.

That patch is not merged yet.
Maybe you mean this one: https://bitbucket.org/multicoreware/x265_git/commits/c8ceb6b8a7d1b8a8ceed66e58a7536163bd0e420

[Z2697]

This looks similar to https://forum.doom9.org/showthread.php?t=186393
Maybe if you can upload the encoded samples we'll be able to identify it easily.

[Ritsuka]

That patch is not merged yet.
Maybe you mean this one: https://bitbucket.org/multicoreware/x265_git/commits/c8ceb6b8a7d1b8a8ceed66e58a7536163bd0e420

I know it's not merged in x265 master branch, but I added it to HandBrake build system, and it's applied when building HandBrake, because without it 2 pass is seriously broken: https://github.com/HandBrake/HandBrake/tree/master/contrib/x265

[Hellboy.]

Sorry, i forgot to say that i use
All filters off
FPS - Same as source - Variable Framerate
crf - 17
Encoder Preset Slow
Encoder Tune - None
Encoder Profile - Main10
Encoder Level - 5.1

*The sample i share is the original video without the bug, so anyone can convert the file and try the bug themself.

[Z2697]

Another MVD overflow confirmed
This has nothing to do with CUTree, or any of the parameter mentioned, if you want to mitigate this, disable temporal-mvp.
EDIT: well I read some code again and the parameter temporal-mvp only controls the temporal part of the AMVP, the spatial AMVP is always on, it seems, so there're chances that the MVD can still overflow even with temporal-mvp disabled.

For HandBrake developers, since you are OK to incorporate un-merged patches, this patch should fix it: https://mailman.videolan.org/pipermail/x265-devel/2025-July/014451.html

[Hellboy.]

Thanks. That works.
temporal-mvp improve quality or speed?

[Ritsuka]

@Hellboy.: try a snapshot, https://github.com/HandBrake/HandBrake-snapshots , I added the patch above.

[Hellboy.]

Try the snapshot (HandBrake-20250814-ae6b0f35d-x86_64-Win_GUI)
and it worked great. Thanks.