Corrupted playback when using Hardware decoding on a file created using x265

[SquallMX]

Hi. I'm having a issue when decoding a file created by x265 3.6:

https://1drv.ms/v/s!AiCgDguh7E_Ogc4L...SHuxg?e=lgoOQf

When playing with HW acceleration of any kind there is a corrupted set of frames at 1818-1822, when decoding using LWLibavVideoSource or FFMPEG it plays perfectly.

HW Decoding:


SW Decoding


The file was encoded using x265 3.6+13+1-726aec126:[Windows][GCC 14.1.0][64 bit] using the following settings:

--colorprim bt2020 --transfer smpte2084 --colormatrix bt2020nc --master-display "G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(40000000,50)" --crf 19 --fps 24000/1001 --sar 1:1 --profile main10 --output-depth 10 --preset slow --no-strong-intra-smoothing --no-rect --aud --limit-modes --aq-mode 2 --hdr10 --no-opt-ref-list-length-pps --hdr10-opt --no-opt-qp-pps --limit-sao --selective-sao 2 --sao-non-deblock --chromaloc 2 --keyint 60 --min-keyint 6 --log2-max-poc-lsb 8 --dynamic-refine --rskip 2 --rskip-edge-threshold 3 --rdoq-level 2 --rc-lookahead 24 --level-idc 5 --psy-rd 2.00 --psy-rdoq 1.00 --vbv-maxrate 54000 --vbv-bufsize 46000 --deblock -1 --nr-intra 0 --nr-inter 0 --no-info --ctu 64 --merange 57 --y4m

If I use a older 3.5 build the resulting file plays perfectly on both HW and SW decoding:

https://1drv.ms/v/s!AiCgDguh7E_Ogc4N...FjxTg?e=5OSEVM

[benwaggoner]

Huh.

I'd start by stripping out a lot of the less essential parameters and see if that changes anything. If so, you can mix and match to see what is doing it.

There's a lot of oddball stuff in there that could cause an edge case but isn't going to make any material difference at real-world bitrates. Like --no-opt-ref-list-length-pps, -log2-max-poc-lsb 8. And stuff like --no-strong-intra-smoothing, --deblock -1, -limit-sao, --sao-non-deblock I'm unaware of ever helping real-world content that I recall, while degrading compression efficiency in general. --deblock also gets two parameters; I'm not sure what happens if only one is used as in here. Lowering it was a good idea in x264, but HEVC's design is different, and the default is generally quite optimal.

--chromaloc 2 is only appropriate if your source and/or output is Blu-ray, or is otherwise known to use --chromaloc 2.

That said, I've seen some odd stuff with --rskip 2 lately; you might try with --rskip 0 or 1 and see if that helps.

[SquallMX]

Originally Posted by benwaggoner

That said, I've seen some odd stuff with --rskip 2 lately; you might try with --rskip 0 or 1 and see if that helps.

You were right, culprit was --rskip 2, using 1 or 0 creates a file that plays properly on HW decoding, unfortunately this decreases encoding speed by 50%, so I will continue to use the old 3.5 build for the time being.

-limit-sao, --sao-non-deblock

Unfortunately disabling SAO is the only way to get decent quality on some grainy content.

[Hellboy.]

Maybe you can try with the official 3.6
I use Handbrake so i don't know much about x265 but i think this is the official page
https://bitbucket.org/multicoreware/x265_git/downloads/

[excellentswordfight]

Originally Posted by SquallMX

Unfortunately disabling SAO is the only way to get decent quality on some grainy content.

Well, just use --no-sao instead of --limit-sao --selective-sao 2 --sao-non-deblock --no-strong-intra-smoothing

And do not disable strong-intra-smoothing, it might sound bad, but I have never seen it affect any perceivable difference in details or sharpness, but disabling it creates issues with blocking. As can be seen here it can even result in more bluring with disabling it. And why limit-sao? If you wanna keep some sao (selective-sao 2), doesnt that just make the SAO filter worse (I thought this limits the analysis of the filter, and not the "effect" per se of SAO)?

And just a note on --rskip 2, I was going a to create a thread on this a long time ago, but when I found this setting I was also impressed with the speed increase. But I found that it had an odd side-effect were it would create pixel-shifting. So if you toggled stills back and forth (input->output) you could see that some "pixels" was not in the same positions as the input, so it "moved" things, like strands of hair etc. So I stopped using it. Anyone else noticed this?

[SquallMX]

Originally Posted by Hellboy.

Maybe you can try with the official 3.6
I use Handbrake so i don't know much about x265 but i think this is the official page
https://bitbucket.org/multicoreware/x265_git/downloads/

Thanks, I will try the official build.

Originally Posted by excellentswordfight

Well, just use --no-sao instead of --limit-sao --selective-sao 2 --sao-non-deblock --no-strong-intra-smoothing

And do not disable strong-intra-smoothing, it might sound bad, but I have never seen it affect any perceivable difference in details or sharpness, but disabling it creates issues with blocking. As can be seen here it can even result in more bluring with disabling it. And why limit-sao? If you wanna keep some sao (selective-sao 2), doesnt that just make the SAO filter worse (I thought this limits the analysis of the filter, and not the "effect" per se of SAO)?

Yes, for grainy content I use --no-sao and --aq 3, some movies like Bill and Ted 2 looks like sh** otherwise. But for more mainstream movies --limit-sao looks better to me while keeping bitrates low, obviously YMMV.

[Asmodian]

Originally Posted by SquallMX

But for more mainstream movies --limit-sao looks better to me while keeping bitrates low, obviously YMMV.

But --limit-sao is a speed v.s. quality tradeoff, it is not limiting the effect of SAO (it is faster, but lower quality).

Do you really like the look of --limit-sao more than the default SAO analysis?

[SquallMX]

Originally Posted by Asmodian

But --limit-sao is a speed v.s. quality tradeoff, it is not limiting the effect of SAO (it is faster, but lower quality).

Do you really like the look of --limit-sao more than the default SAO analysis?

Thats a good question, last time I made a comparative --limit-sao vs --sao, I preferred the --limit-sao encoding, but I guess it can vary from video to video.

[benwaggoner]

Originally Posted by SquallMX

Thats a good question, last time I made a comparative --limit-sao vs --sao, I preferred the --limit-sao encoding, but I guess it can vary from video to video.

--limit-sao is just a speed-over-quality optimization for --sao. Are you sure you weren't comparing to --no-sao?

[hello_hello]

Originally Posted by excellentswordfight

And do not disable strong-intra-smoothing, it might sound bad, but I have never seen it affect any perceivable difference in details or sharpness, but disabling it creates issues with blocking?

I do so like to argue.

I rarely encode anything that hasn't been run through GradFun3 or one of it's variants at the end of a script, so maybe that's why I'm not having banding issues, but --no-strong-intra-smoothing seemed to make a difference. Unless for the example below --no-sao made the difference and I'm mistaken, but I'm pretty sure I tested those two options independently at some stage.
Maybe liking --no-strong-intra-smoothing comes down to personal taste, and maybe it depends on other settings, but here's the only results I still have from my initial testing. I've probably fiddled with some of the settings for my own encodes since (I no longer include --no-rect), but....

The source is included for completeness even though the video was lightly denoised and dithered on the way through, so none of the encodes would be expected to look exactly like the original. I also ran the video by x264, just because....

Source:
https://gifyu.com/image/SiQqW

Encode A: 1756 kb/s
https://gifyu.com/image/SiQqg
x265 --crf 20.0 --profile main10 --preset slow --deblock=-1:-1 --no-open-gop --rc-lookahead 80 --lookahead-slices 0 --psy-rdoq 1.7 --aq-strength 1.3 --no-strong-intra-smoothing --no-rect --no-sao --selective-sao 0

Encode B: 2516 kb/s
https://gifyu.com/image/SiQqu
x264 --crf 20.0 --level 4.1 --preset slow --tune film --b-adapt 2 --me umh --subme 9 --partitions all --no-fast-pskip --stitchable

Encode C: 1605 kb/s
https://gifyu.com/image/SiQqc
x265 --crf 20.0 --profile main10 --preset slow --deblock=-1:-1 --no-open-gop --rc-lookahead 80 --lookahead-slices 0 --psy-rdoq 1.7 --aq-strength 1.3

Encode D: 1753 kb/s
https://gifyu.com/image/SiQqQ
x265 --crf 18.0 --profile main10 --preset slow

The encodes above are ranked according to how closely they resembled the source to me, obsessing over the background in this case, as impossible as it was for them to match the source.... my logic at the time was to look at the results based on the filtering I'd normally use rather than something different.

[excellentswordfight]

Originally Posted by hello_hello

I do so like to argue.

I rarely encode anything that hasn't been run through GradFun3 or one of it's variants at the end of a script, so maybe that's why I'm not having banding issues, but --no-strong-intra-smoothing seemed to make a difference. Unless for the example below --no-sao made the difference and I'm mistaken, but I'm pretty sure I tested those two options independently at some stage.

It would be better, that instead of arguing, show examples were the only difference in settings is turning off strong intra smoothing, and thereby also validating your claim. Cause that you maybe tried it isnt really much of an argument I think no one that has used x265 and used no-sao can argue against that there are plenty of cases were its beneficial to turn off SAO, and that turning it off definitely has a impact on the image.

Personally I use no-sao for most of my encodes, selective-sao 1 or 2 for cartoon and anime, and full sao for low-bitrate/trytocompressasmuchaspossible stuff.

[tormento]

Give a try to VMAF too.

Of course, you don't have to apply filters to compare apples to apples.

[benwaggoner]

Originally Posted by excellentswordfight

Personally I use no-sao for most of my encodes, selective-sao 1 or 2 for cartoon and anime, and full sao for low-bitrate/trytocompressasmuchaspossible stuff.

If you are using SAO, --selective-sao 2 is pretty much identical to full SAO in quality, but is a bit faster (more of a benefit with faster presets). SAO is really designed for forward prediction only, so it really only does anything with I and P frames. --selective-sao 2 really should be the default, and I imagine it would be if MCW ever updates the presets.

That said, x265 3.6 has support for more layers of B-frames, and I've not experimented to see if that makes --selective-sao 3 offer any marginal benefit now. SAO won't ever do anything for non-reference b frames, which are the majority of frames in most distribution encodes.

[benwaggoner]

Originally Posted by tormento

Give a try to VMAF too.

Of course, you don't have to apply filters to compare apples to apples.

VMAF is definitely our least-bad generally available full reference metric.

Still, VMAF is luma only, so won't catch chroma issues and is not great at comparing different adaptive quantization methods

And as a per-frame metric, it is up to people to figure out how to aggregate those metrics to estimate overall quality. Saying "mean VMAF of 80" for a two hour movie could indicate a movie that is consistently mediocre, or one that oscillates between pristine and terrible quality.

[tormento]

Originally Posted by benwaggoner

VMAF is definitely our least-bad generally available full reference metric.

Still, VMAF is luma only, so won't catch chroma issues and is not great at comparing different adaptive quantization methods

And as a per-frame metric, it is up to people to figure out how to aggregate those metrics to estimate overall quality. Saying "mean VMAF of 80" for a two hour movie could indicate a movie that is consistently mediocre, or one that oscillates between pristine and terrible quality.

That’s why I use the output graphs, feeding source as primary stream, not numbers.

[benwaggoner]

Originally Posted by tormento

That’s why I use the output graphs, feeding source as primary stream, not numbers.

Yeah, that is a much better approach. I've yet to find one I find truly satisfying. More R&D is needed.

Harmonic mean of scores in a GOP seems less bad than others.

Choice of a metric is also a challenge. VMAF was a big step forward, but still doesn't have that high a correlation with subjective quality ratings.

[excellentswordfight]

Originally Posted by benwaggoner

And as a per-frame metric, it is up to people to figure out how to aggregate those metrics to estimate overall quality. Saying "mean VMAF of 80" for a two hour movie could indicate a movie that is consistently mediocre, or one that oscillates between pristine and terrible quality.

Originally Posted by tormento

That’s why I use the output graphs, feeding source as primary stream, not numbers.

Yes, I agree, output it as a graph is necessity!

For people who dont really understand how big of an issue this is here is an example:

In a test AMD:s VCN encoder recieved a VMAF of 95.39 and Nvidias Nvenc 96.13, so one would think that they are pretty much the same, right? Well the graph tells a completely different story:



That an encoder can dip that much on big bunny is one thing , but the dips really demonstrate the flaw of trying to simple down quality to one number. Its a bit like the percentiles discussion that happened in gaming benchmark a few years ago, were now it standard to look at 1% and 5% lows aswell as the avrage.

[benwaggoner]

Originally Posted by excellentswordfight

Yes, I agree, output it as a graph is necessity!

For people who dont really understand how big of an issue this is here is an example:

Wow, that is a great example. Our sense of how good a video looks is much more about how good the worst parts of the video looked than the average quality.