x265 HEVC Encoder

[GeoffreyA]

Makes sense. Its somewhat different tools and ranges yield different-looking artefacts as a side effect, and x265's implementation of these tools may not be the gold standard. qcomp was helping, but as you said, just shifting around or pumping up the bits.

SVT-AV1 can give a better picture at the frame level, but going in the temporal direction, particularly with grain, we're walking on shaky ground. The HDR fork does better, but changes the appearance of the grain, making it coarser.

[benwaggoner]

Makes sense. Its somewhat different tools and ranges yield different-looking artefacts as a side effect, and x265's implementation of these tools may not be the gold standard. qcomp was helping, but as you said, just shifting around or pumping up the bits.

SVT-AV1 can give a better picture at the frame level, but going in the temporal direction, particularly with grain, we're walking on shaky ground. The HDR fork does better, but changes the appearance of the grain, making it coarser.
And, as always, with a video codec, how it looks temporally is really the only thing that matters. Making still images look good in isolation is a much simpler problem than getting a coherent, consistent experience over time. And good psychovisually compression efficiency can involve doing things that can make individual frames paused look not great, but which are invisible during actual playback. Temporal masking is a powerful tool.

[Z2697]

I found that x265's SAO performs better or less bad in 12bit mode some time ago... maybe worth take a look.
But I cannot really understand that part of the code, and since AV1 has better filter(s) and better open source implementation(s), I'm not very motivated :o

[GeoffreyA]

And, as always, with a video codec, how it looks temporally is really the only thing that matters. Making still images look good in isolation is a much simpler problem than getting a coherent, consistent experience over time. And good psychovisually compression efficiency can involve doing things that can make individual frames paused look not great, but which are invisible during actual playback. Temporal masking is a powerful tool.

Yes, we tend to fixate on comparing single frames, but at the end of the day, it's the video. x264/5 seem mostly fine in this regard, the lesser-quality frames feeling consistent.

I found that x265's SAO performs better or less bad in 12bit mode some time ago... maybe worth take a look.
But I cannot really understand that part of the code, and since AV1 has better filter(s) and better open source implementation(s), I'm not very motivated :o

Well, 12-bit might be challenging to play on some hardware, and I've made peace parting with SAO for life! :)
Jokes aside, I've found some success with aq-mode 4 these past few days. So far, it's working well.

[Z2697]

Well, 12-bit might be challenging to play on some hardware, and I've made peace parting with SAO for life! :)
Jokes aside, I've found some success with aq-mode 4 these past few days. So far, it's working well.

The problem is it shouldn't perform distinctively worse in 8 and 10 bits mode, hence there may be some bug in the code.
I mean the SAO mode decision, not the fps perform, you know.

[GeoffreyA]

The problem is it shouldn't perform distinctively worse in 8 and 10 bits mode, hence there may be some bug in the code.
I mean the SAO mode decision, not the fps perform, you know.

So, a precision sort of bug? Perhaps an overflow somewhere in the calculation?

[rwill]

Well you could give my gradient_with_noise sequence a try, with 10 and 12 bit, and check if its no longer blocking things up.

https://forum.doom9.net/showthread.php?p=1949376#post1949376

But I guess nothing did change. Last time I checked x265 had no real "mode decision" for SAO but only went by what it understands as "distortion". I remember debugging some large offenders and the numbers checked out, no bug there it seems.

[GeoffreyA]

Does x265's SAO use the same implementation as HM, the reference encoder?

[rwill]

Does x265's SAO use the same implementation as HM, the reference encoder?

Well I guess they did not come up with something novel on their own....

[Z2697]

Well you could give my gradient_with_noise sequence a try, with 10 and 12 bit, and check if its no longer blocking things up.

https://forum.doom9.net/showthread.php?p=1949376#post1949376

But I guess nothing did change. Last time I checked x265 had no real "mode decision" for SAO but only went by what it understands as "distortion". I remember debugging some large offenders and the numbers checked out, no bug there it seems.

Least "distortion" (and seems calculated in it's own way) is still some kind of mode decision... right?

I tested this same clip before, 12bit does not create same "blocky columns" as 10bit in the test, that's actually what I mean less bad.

But this kind of content should use "no operation" mode I think? 12bit still doesn't do that.

[rwill]

Least "distortion" (and seems calculated in it's own way) is still some kind of mode decision... right?

I tested this same clip before, 12bit does not create same "blocky columns" as 10bit in the test, that's actually what I mean less bad.

But this kind of content should use "no operation" mode I think? 12bit still doesn't do that.

Yes, picking a mode based on least distortion is a mode decision. I meant something like a full RD one with normal SSD and PSY distortion, like the CU mode decision.

That the clip does not exhibit the blocking at 12bit is interesting and probably worth investigating. Its strange that 8 and 10 bit pick the 'offset' mode and 12 bit does not. As for the optimal mode, it would be better if it were selected by real RD costs, that is, recon against source distortion with the (PSY-)RD distortion metric used everywere else.

[benwaggoner]

Does x265's SAO use the same implementation as HM, the reference encoder?
That was my understanding back in the day, and I do not recall that ever changing.

[hellgauss]

Perhaps related to RD, I found an unwanted behavior in rd-refine option. The following screen is from Vinland Saga Season 1 Episode 16, source is blu-ray @34MBps

https://imgur.com/iwfhscB

NOTE: the screen is taken via VirtualDub2 and the artifacts are present in software decoder, so probably it is not the same MV issue I described a few months ago in another thread.

The flickering fire leads to fast moving gradient in a dark background, which is somewhat a worst case scenario for video encoding. There are a lot of similar random macroblocks in the whole scene, each last for 1 frames.

Bat file used (Windows):

set bf=6
set subme=4

for %%i in (0,1) do (
for %%j in (10,12) do (

rem i=rd-refine j=bit

start /low ffmpeg -threads 1 -y -hide_banner -nostdin -stats_period 33 -bitexact -i "Vinland-Saga_S1E16.mkv" -map 0:v -ss 206.580 -t 17.977 -profile:v main%%j -vf "removegrain=1:2:2,removegrain=0:2:2,format=yuv420p%%jle" -sws_flags accurate_rnd -c:v libx265 -preset veryslow -crf 17.2 -x265-params "deblock=-1,-1:no-sao=1:no-strong-intra-smoothing=1:tu-intra-depth=4:tu-inter-depth=4:merange=58:ref=6:rc-lookahead=99:bframes=6:rd=6:rd-refine=%%i:subme=4:vbv-maxrate=20000:vbv-bufsize=24924:frame-threads=1:pools=none:no-wpp=1:level-idc=51:high-tier=1:colorprim=1:colormatrix=1:transfer=1:range=limited" -bitexact "Vinland-Saga_S1E16_r%%i_b%%j.mp4"
))


Writing library : x265 4.1+211-9e551a994:[Windows][GCC 15.2.0][64 bit]

If rd-refine=1, artifacts appears both in 10 and 12 bit. With rd-refine=0 there are no wrong macroblocks. However rd-refine seems to give nice results in other scenarios.

[Z2697]

That's a known issue, among some people.
https://github.com/Mr-Z-2697/x265-experimental/commit/69ea7c75e8c245e3323fda336f2a4df2c730ad7a

However my test result didn't favor rd-refine even if it's supposed to be fixed (if I'm lucky).
However yet again I didn't test the ultimate single thread encoding like you do.

I'll port that branch into a latest x265 and compile a test build for anyone that might be interested.

https://pixeldrain.com/u/z9TjFxsh
or
https://workupload.com/file/kcWyHpU2rX3

[hellgauss]

An update: it seems that the rd-refine problem is also related to the limit-tu option.

Test script used (all encodes with rd-refine=1, loop through limit-tu=%%j)

set rdr=1
set bf=5
set subme=3

for %%j in (0,1,2,3,4) do (

rem j=limit-tu

start /low ffmpeg -threads 1 -y -hide_banner -nostdin -stats_period 33 -bitexact -i "Vinland-Saga_S1E16-rdr-scene.mkv" -map 0:v -profile:v main10 -vf "removegrain=1:2:2,removegrain=0:2:2,format=yuv420p10le" -sws_flags accurate_rnd -c:v libx265 -preset slower -crf 17.2 -x265-params "limit-tu=%%j:deblock=-1,-1:no-sao=1:no-strong-intra-smoothing=1:rc-lookahead=99:bframes=%bf%:rd=6:rd-refine=%rdr%:subme=%subme%:vbv-maxrate=20000:vbv-bufsize=24924:frame-threads=1:pools=none:no-wpp=1:level-idc=51:high-tier=1:colorprim=1:colormatrix=1:transfer=1:range=limited" -bitexact "Vinland-Saga_S1E16_[rdr_%rdr%][ltu_%%j].mp4"
)

Results:
Macroblocks are evident with limit-tu =0 and 1. Less evident with 2, absent with limit-tu=3 or 4. Replacing with rd-refine=0, all tests present no "macroblocks" artifacts.

EDIT: thanks Z2697 for your build, I'll give it a try as soon as I setup a proper framework (I'm used to ffmpeg and not familiar with x265.exe, but I guess it is not difficult)

[jpsdr]

@Z2697
I took a look on your experimental repo, and see in your experiments the remove of distorstion chroma only for 444. Did test/check and finaly found that it produces better results ?

[Z2697]

An update: it seems that the rd-refine problem is also related to the limit-tu option.

This bug is quite random, it may disappear here but can happen at other places.

[Z2697]

@Z2697
I took a look on your experimental repo, and see in your experiments the remove of distorstion chroma only for 444. Did test/check and finaly found that it produces better results ?

See more detail in https://forum.doom9.org/showthread.php?t=186790
In theory, and in the way I assess chroma quality, it's better.

[GeoffreyA]

It seems to be more in line with x264, undoing that commit.

[benwaggoner]

Gosh, I hadn't thought about --rd-refine in years. I never found any examples where it provided a material difference. Have people found it helpful in some situations?

[rwill]

Gosh, I hadn't thought about --rd-refine in years. I never found any examples where it provided a material difference. Have people found it helpful in some situations?

Its a small BD-Rate optimization at a somewhat moderate speed cost when tuning for PSNR. It catches the case were a CU is sometimes better coded with a different QP in quantization than the originaly chosen RD QP (lambda).

So mainly helpful if you want to really push the objective metric the encoder is optimizing for in RD, not really helpful otherwise.

That is, if the lambdas for QPs are tuned so one would get just a narrow normal distribution of delta QPs around the originaly chosen QP when using that refine, which hopefully should be the case.

[Z2697]

We have seen such description in x264's subme option. (somehow)
subme is one of the more puzzling options in x264 that does a mixture of things...
I didn't find out exactly what's under the hood of the "RD refinement" here and the connection with x265's rd-refine.
If we search the word "rd_refine" in x264 code we can only find "intra_rd_refine" function.

-m, --subme <integer> Subpixel motion estimation and mode decision [7]
- 0: fullpel only (not recommended)
- 1: SAD mode decision, one qpel iteration
- 2: SATD mode decision
- 3-5: Progressively more qpel
- 6: RD mode decision for I/P-frames
- 7: RD mode decision for all frames
- 8: RD refinement for I/P-frames
- 9: RD refinement for all frames
- 10: QP-RD - requires trellis=2, aq-mode>0
- 11: Full RD: disable all early terminations

[hellgauss]

@Z2697

I tested your build in the same scene, with same results. Square artifacts with rd-refine=1 and limit-tu=0, 1. With 2 they are less evident. With 3 and 4 no square artifacts.

Perhaps they are in general less evident with respect to the official build.

With rd-refine=0 no artifacts whatever limit-tu.

[Z2697]

Well because I have changed the option.
1 is indeed the same.

H0(" --rd-refine <0|1|2|3> Enable QP-based RD refinement for rd levels 5 and 6.\n"
" - 0: disabled.\n"
" - 1: original behavior.\n"
" - 2: proposed fix.\n"
" - 3: try not to code skip CU.\n"
" Default %d\n", param->rdRefine);

[hellgauss]

Same results again with rd-refine 2 or 3 - blocks evident with limit-tu=0, 1 , less evident with limit-tu=2, no blocks with limit-tu=3, 4. It really seems that it is related to limit-tu also.

[Z2697]

Then it's beyond fix. (more like skill issues, dumb dumb Z-guy)
Not that it really matters though, it was a nothing burger (or maybe a positron burger) to begin with.

[Z2697]

We have seen such description in x264's subme option. (somehow)
subme is one of the more puzzling options in x264 that does a mixture of things...
I didn't find out exactly what's under the hood of the "RD refinement" here and the connection with x265's rd-refine.
If we search the word "rd_refine" in x264 code we can only find "intra_rd_refine" function.

-m, --subme <integer> Subpixel motion estimation and mode decision [7]
- 0: fullpel only (not recommended)
- 1: SAD mode decision, one qpel iteration
- 2: SATD mode decision
- 3-5: Progressively more qpel
- 6: RD mode decision for I/P-frames
- 7: RD mode decision for all frames
- 8: RD refinement for I/P-frames
- 9: RD refinement for all frames
- 10: QP-RD - requires trellis=2, aq-mode>0
- 11: Full RD: disable all early terminations


It looks like the "RD refinement" in x264 refers to refining the motion estimation, where x265 rd-refine... is just "FAFO" with different QP for each block...
(I might be wrong)

[Barough]

x265 v4.1+222-afa0028
https://www.mediafire.com/file/wqtdn8iwhgby53m
Win x86/x64 binaries, GCC

[benwaggoner]

Its a small BD-Rate optimization at a somewhat moderate speed cost when tuning for PSNR. It catches the case were a CU is sometimes better coded with a different QP in quantization than the originaly chosen RD QP (lambda).

So mainly helpful if you want to really push the objective metric the encoder is optimizing for in RD, not really helpful otherwise.
Yeah, and I don't care about objective metric improvements at all, only subjective ones. Objective metrics are a very useful tool in defining subjectively-tuned encodes, but are never the goal in and of themselves.

[benwaggoner]

We have seen such description in x264's subme option. (somehow)
subme is one of the more puzzling options in x264 that does a mixture of things...
One of my favorite felicities of x265 was unpacking that mess of options and just making things that should be independent parameters into those.

[nhw_pulsar]

Yeah, and I don't care about objective metric improvements at all, only subjective ones. Objective metrics are a very useful tool in defining subjectively-tuned encodes, but are never the goal in and of themselves.

Yes, I always thought that visual subjective improvement was a lot more important than objective metrics improvement, as in my codec very very often PSNR was worse and visual quality was better, and reversely PSNR was better but visual quality was worse. But on the opposite, MPEG develops for PSNR and surprisingly (for me) it seems to work, as I have tested for image compression, and visual quality follows PSNR: JPEG XR<x264 intra<x265 intra<VVC intra<ECM intra, so always for me, MPEG seems to improve PSNR and in the same time subjective visual quality....

[benwaggoner]

Yes, I always thought that visual subjective improvement was a lot more important than objective metrics improvement, as in my codec very very often PSNR was worse and visual quality was better, and reversely PSNR was better but visual quality was worse. But on the opposite, MPEG develops for PSNR and surprisingly (for me) it seems to work, as I have tested for image compression, and visual quality follows PSNR: JPEG XR<x264 intra<x265 intra<VVC intra<ECM intra, so always for me, MPEG seems to improve PSNR and in the same time subjective visual quality....
This is a very interesting philosophical discussion.

PSNR is a simple metric, so easy to calculate in a consistent way, and being able to make pixels mathematically similar to how they started is a decent first order approximation for quality. And a whole lot of bitstream features are about how to get that kind of mathematical match with as few bits as possible. I think of PSNR as like the individual shard of glass that make up a mosaic.

Now, past that low level of "is this number like it was before" comes subjective optimization. And a codec/bitstream standard needs to have tools that allow for an encoder to do psychovisual optimizations. The psychovisually preferable in-loop deblocking might not be the one that gives the best PSNR. But that's all outside of the normative codec spec. Actual encoding video that looks good and is deliverable within particular constraints is like the artist making the mosaic. Even starting with the same pieces of glass, a great artist can make a much better looking mosaic than a bad one, or some algorithm that scans a picture and assigned a piece of glass to each square. A less mathematically accurate mosaic can still look like the thing it is of a lot more accurately and esthetically.

[nhw_pulsar]

This is a very interesting philosophical discussion.

PSNR is a simple metric, so easy to calculate in a consistent way, and being able to make pixels mathematically similar to how they started is a decent first order approximation for quality. And a whole lot of bitstream features are about how to get that kind of mathematical match with as few bits as possible. I think of PSNR as like the individual shard of glass that make up a mosaic.

Now, past that low level of "is this number like it was before" comes subjective optimization. And a codec/bitstream standard needs to have tools that allow for an encoder to do psychovisual optimizations. The psychovisually preferable in-loop deblocking might not be the one that gives the best PSNR. But that's all outside of the normative codec spec. Actual encoding video that looks good and is deliverable within particular constraints is like the artist making the mosaic. Even starting with the same pieces of glass, a great artist can make a much better looking mosaic than a bad one, or some algorithm that scans a picture and assigned a piece of glass to each square. A less mathematically accurate mosaic can still look like the thing it is of a lot more accurately and esthetically.

Yes, I am not an expert like you, but as you say, it seems that standard bodies like MPEG release a new standard test model with a generational PSNR leap, and then you have to psychovisually tune this base model, independently of PSNR.

This is a unique way of development, proper to MPEG, in my little experience, my codec (NHW) since 2007 has not improved in PSNR, but visual quality has a lot improved...

Cheers,
Raphael

[benwaggoner]

Yes, I am not an expert like you, but as you say, it seems that standard bodies like MPEG release a new standard test model with a generational PSNR leap, and then you have to psychovisually tune this base model, independently of PSNR.
I concur it seems there should be a way to do a psychovisual-forward model of development. And subjective video tests are integrated earlier and earlier in each generation of codec development, and a lot more thought goes into making good syntax for those kinds of optimizations. Both MPEG-4 ASP and VC-1 suffered from inefficient signaling in adaptive quantization, for example, which was a big challenge in psychovisual optimization. Even though the baseline numbers published focus on PSNR, other testing gets done.

One potential downside of this is that non-optiional bitstream features that would improve psychovisual quality while not improving or even reducing PSNR will tend to get ignored. But features that have a bigger psychovisual improvement than PSNR improvement can get more attention than the PSNR improvement alone would merit.

This is a unique way of development, proper to MPEG, in my little experience, my codec (NHW) since 2007 has not improved in PSNR, but visual quality has a lot improved..
Yep, that is psychovisual optimization.

And it's not just MPEG. AOM has done AV1/2 development with a similar model, JXL. Really, all codec development for decades has had PSNR at the core of it.

[nhw_pulsar]

I concur it seems there should be a way to do a psychovisual-forward model of development. And subjective video tests are integrated earlier and earlier in each generation of codec development, and a lot more thought goes into making good syntax for those kinds of optimizations. Both MPEG-4 ASP and VC-1 suffered from inefficient signaling in adaptive quantization, for example, which was a big challenge in psychovisual optimization. Even though the baseline numbers published focus on PSNR, other testing gets done.

One potential downside of this is that non-optiional bitstream features that would improve psychovisual quality while not improving or even reducing PSNR will tend to get ignored. But features that have a bigger psychovisual improvement than PSNR improvement can get more attention than the PSNR improvement alone would merit.


Yep, that is psychovisual optimization.

And it's not just MPEG. AOM has done AV1/2 development with a similar model, JXL. Really, all codec development for decades has had PSNR at the core of it.

Ok, just a very quick off-topic, with my non-expert criterias and eyes, I always found that MPEG and AOM common technology (DCT+inter/intra prediction) lacked of neatness, at least on still-image...

Very curious to see if the next AV2 and ECM will correct this, despite a colossal computation time, and especially at decoding...

[Z2697]

Yes, I always thought that visual subjective improvement was a lot more important than objective metrics improvement, as in my codec very very often PSNR was worse and visual quality was better, and reversely PSNR was better but visual quality was worse. But on the opposite, MPEG develops for PSNR and surprisingly (for me) it seems to work, as I have tested for image compression, and visual quality follows PSNR: JPEG XR<x264 intra<x265 intra<VVC intra<ECM intra, so always for me, MPEG seems to improve PSNR and in the same time subjective visual quality....

They are different generations of codec after all.
How about you compare options within one codec?

[nhw_pulsar]

They are different generations of codec after all.
How about you compare options within one codec?


Actually I did not change/vary options within one codec for still image, I just used the default options (but always YUV420 if I remember correctly)...

That is right for JPEG XR, for x264 intra I used UCI codec default options, for x265 intra I used BPG default options (but often -m 1 fastest speed that has also very good quality), for VVC intra I used Jamaika binaries (VTM 12.3) provided on this forum with default command (slowest preset).

That's right I must admit I did not vary the options to have the best psychovisual results... but on the custom binaries for still image I have, I also don't think there are all the options as in the video codec...

[Z2697]

Actually I did not change/vary options within one codec for still image, I just used the default options (but always YUV420 if I remember correctly)...

That is right for JPEG XR, for x264 intra I used UCI codec default options, for x265 intra I used BPG default options (but often -m 1 fastest speed that has also very good quality), for VVC intra I used Jamaika binaries (VTM 12.3) provided on this forum with default command (slowest preset).

That's right I must admit I did not vary the options to have the best psychovisual results... but on the custom binaries for still image I have, I also don't think there are all the options as in the video codec...

The problem is the more tools available in later codecs will improve quality regardless of how the encoder is tuned (hopefully). Especially for Intra frame.
It's not PSNR works for them, it's the tools that works.
Maybe they are chosen based on PSNR merit, but doesn't stop them from acting as a visual improvement.

[benwaggoner]

Ok, just a very quick off-topic, with my non-expert criterias and eyes, I always found that MPEG and AOM common technology (DCT+inter/intra prediction) lacked of neatness, at least on still-image...

Very curious to see if the next AV2 and ECM will correct this, despite a colossal computation time, and especially at decoding...
Bear in mind that still images aren't the main goal of a video codec. Moving image and thus interframe compression is the essential features. All sorts of interesting transforms work well for still image, like wavelets. But no one has come up with a better way to handle interprediction than block based transforms. And block-based interprediction works very well with discreet sinusoidal transformations. And if you're using those for motion, doing your intra coding that way is lower complexity and a lot easier to integrate.

No one has come up with any inter frame encoding scheme better than using power of 2 sized rectangles with a DCT-ish basic transform.

[nhw_pulsar]

Bear in mind that still images aren't the main goal of a video codec. Moving image and thus interframe compression is the essential features. All sorts of interesting transforms work well for still image, like wavelets. But no one has come up with a better way to handle interprediction than block based transforms. And block-based interprediction works very well with discreet sinusoidal transformations. And if you're using those for motion, doing your intra coding that way is lower complexity and a lot easier to integrate.

No one has come up with any inter frame encoding scheme better than using power of 2 sized rectangles with a DCT-ish basic transform.

Yes, it is however possible to use power of 2 sized rectangles with wavelet transform but I think the main problem will be blocking artifacts.I must admit that in my codec, my wavelet 5/3 and 9/7 transform design introduces some border blocking artifacts that I think, are more pronounced than MPEG and AOM DCT transform ones...

My non-expert thinking makes me suppose that we should apply then the wavelet transform/compression on the whole residual image of the inter coding, but experts seem nearly sure that it won't be as good as x265, AV1, VVC...

[Patman]

I found the discussion about the --rd-refine parameter of x265 and its effects here quite interesting and took a look at the code myself. I have provided three test bins here (https://github.com/Patman86/x265-Mod-by-Patman/issues/49#issuecomment-3848765443) that you are welcome to try out. Maybe there is a version that works quite well with your parameters.

[Z2697]

I found the discussion about the --rd-refine parameter of x265 and its effects here quite interesting and took a look at the code myself. I have provided three test bins here (https://github.com/Patman86/x265-Mod-by-Patman/issues/49#issuecomment-3848765443) that you are welcome to try out. Maybe there is a version that works quite well with your parameters.

The test1 and test2 produce same result (still with glitches), and test3 produces result as if the rd-refine were not enabled.

[Patman]

The test1 and test2 produce same result (still with glitches), and test3 produces result as if the rd-refine were not enabled.
Thank you for your feedback. I have uploaded new files, perhaps one of them will be (partially) successful. Same link as before...

[Z2697]

Thank you for your feedback. I have uploaded new files, perhaps one of them will be (partially) successful. Same link as before...

What did you change with the overhaul?

[Patman]

What did you change with the overhaul?

void Analysis::qprdRefine(const CUData& parentCTU, const CUGeom& cuGeom, int32_t qp, int32_t lqp)
{
- uint32_t depth = cuGeom.depth;
- ModeDepth& md = m_modeDepth[depth];
- md.bestMode = NULL;
+ uint32_t depth = cuGeom.depth;
+ ModeDepth &md = m_modeDepth[depth];
+ md.bestMode = NULL;

bool bDecidedDepth = parentCTU.m_cuDepth[cuGeom.absPartIdx] == depth;

- int bestCUQP = qp;
- int lambdaQP = lqp;
+ int bestCUQP = qp;
+ int lambdaQP = lqp;
+
bool doQPRefine = (bDecidedDepth && depth <= m_slice->m_pps->maxCuDQPDepth) || (!bDecidedDepth && depth == m_slice->m_pps->maxCuDQPDepth);
+
if (m_param->analysisLoadReuseLevel >= 7)
doQPRefine = false;
- if (doQPRefine)
+
+ if (!doQPRefine)
+ {
+ recodeCU(parentCTU, cuGeom, bestCUQP, lambdaQP);
+ md.bestMode->cu.copyToPic(depth);
+ md.bestMode->reconYuv.copyToPicYuv(*m_frame->m_reconPic[0], parentCTU.m_cuAddr, cuGeom.absPartIdx);
+ return;
+ }
+
+ if (!md.bestMode || md.bestMode->cu.isSkipped(0) || cuGeom.log2CUSize <= 2)
{
- uint64_t bestCUCost, origCUCost, cuCost, cuPrevCost;
+ recodeCU(parentCTU, cuGeom, bestCUQP, lambdaQP);
+ md.bestMode->cu.copyToPic(depth);
+ md.bestMode->reconYuv.copyToPicYuv(*m_frame->m_reconPic[0], parentCTU.m_cuAddr, cuGeom.absPartIdx);
+ return;
+ }

- int cuIdx = (cuGeom.childOffset - 1) / 3;
- bestCUCost = origCUCost = cacheCost[cuIdx];
+ uint64_t bestCUCost, origCUCost, cuCost, cuPrevCost;
+
+ int cuIdx = (cuGeom.childOffset - 1) / 3;
+ bestCUCost = origCUCost = cacheCost[cuIdx];

- int direction = m_param->bOptCUDeltaQP ? 1 : 2;
+ int direction = m_param->bOptCUDeltaQP ? 1 : 1;

- for (int dir = direction; dir >= -direction; dir -= (direction * 2))
+ int maxDeltaQP = 2;
+
+ if ((m_limitTU & X265_TU_LIMIT_NEIGH) && cuGeom.log2CUSize >= 4)
+ maxDeltaQP = 1;
+
+ bool strongPsy = (m_param->psyRd > 0.5);
+
+ if (strongPsy && maxDeltaQP > 1)
+ maxDeltaQP = 1;
+
+ int minQP = X265_MAX(m_param->rc.qpMin, qp - maxDeltaQP);
+ int maxQP = X265_MIN(QP_MAX_SPEC, qp + maxDeltaQP);
+
+ int prevQP = (int)parentCTU.m_meanQP;
+ int dirToPrev = (prevQP > qp) ? +1 : (prevQP < qp ? -1 : 0);
+
+ for (int dirSign = 0; dirSign < 2; dirSign++)
+ {
+ int dir = (dirSign == 0) ? direction : -direction;
+
+ if (m_param->bOptCUDeltaQP && dir != 1)
+ continue;
+
+ if (m_param->bOptCUDeltaQP && dir == 1 && (qp + 3) >= prevQP)
+ break;
+
+ int baseThresh = (maxDeltaQP <= 1) ? 0 : 1;
+ int threshold = baseThresh;
+ if (dirToPrev != 0 && dir == dirToPrev && maxDeltaQP > 1)
+ threshold++;
+
+ int failure = 0;
+ cuPrevCost = origCUCost;
+
+ int modCUQP = qp + dir;
+ while (modCUQP >= minQP && modCUQP <= maxQP)
{
- if (m_param->bOptCUDeltaQP && ((dir != 1) || ((qp + 3) >= (int32_t)parentCTU.m_meanQP)))
+ if (m_param->bOptCUDeltaQP && modCUQP > prevQP)
break;

- int threshold = 1;
- int failure = 0;
- cuPrevCost = origCUCost;
+ recodeCU(parentCTU, cuGeom, modCUQP, qp);
+ cuCost = md.bestMode->rdCost;

- int modCUQP = qp + dir;
- while (modCUQP >= m_param->rc.qpMin && modCUQP <= QP_MAX_SPEC)
- {
- if (m_param->bOptCUDeltaQP && modCUQP > (int32_t)parentCTU.m_meanQP)
- break;
-
- recodeCU(parentCTU, cuGeom, modCUQP, qp);
- cuCost = md.bestMode->rdCost;
+ COPY2_IF_LT(bestCUCost, cuCost, bestCUQP, modCUQP);

- COPY2_IF_LT(bestCUCost, cuCost, bestCUQP, modCUQP);
- if (cuCost < cuPrevCost)
- failure = 0;
- else if (cuCost > cuPrevCost)
- failure++;
+ if (cuCost < cuPrevCost)
+ failure = 0;
+ else
+ failure++;

- if (failure > threshold)
- break;
+ if (failure > threshold)
+ break;

- cuPrevCost = cuCost;
- modCUQP += dir;
- }
+ cuPrevCost = cuCost;
+ modCUQP += dir;
}
- lambdaQP = bestCUQP;
}

- recodeCU(parentCTU, cuGeom, bestCUQP, lambdaQP);
+ lambdaQP = bestCUQP;
+ setLambdaFromQP(parentCTU, lambdaQP);

- /* Copy best data to encData CTU and recon */
+ recodeCU(parentCTU, cuGeom, bestCUQP, lambdaQP);
md.bestMode->cu.copyToPic(depth);
md.bestMode->reconYuv.copyToPicYuv(*m_frame->m_reconPic[0], parentCTU.m_cuAddr, cuGeom.absPartIdx);
}

[Z2697]

The most significant change being the QP adjustment range is limited to plus or minus 1 or 2, right?
Technically that would work, but then there's one less reason to use rd-refine.
Too narrow of a range, then the effect is reduced to virtually nothing, too wide of a range, then the potential of glitch is still present.

None of the rd-refine, original or with patch, improves BD rate in my test though. (except the one with less skip, but it's even slower)
Modifying QP on a per block basis without knowing basically anything else is not a good thing, it seems.

Limit-tu is not the problem, nor the fix.

[hellgauss]

Test results for second set of @Patman builds.

Script used. Test A (30 encodes)

set dir=patman
set bf=6
set subme=3

for %%b in (1,2,3) do (
for %%r in (rd-refine,no-rd-refine) do (
for %%j in (0,1,2,3,4) do (

rem b,r,j = build, rd-refine, limit-tu
rem build 1 = m_limit_tu
rem build 2 = overhaul
rem build 3 = overhaul + intrardrefine

ffmpeg.exe -loglevel level+error -i "Vinland-Saga_S1E16-rdr-scene.mkv" -map 0:v -vf "removegrain=1:2:2,removegrain=0:2:2,format=yuv420p10le" -sws_flags accurate_rnd -strict -1 -f yuv4mpegpipe - | %dir%\x265_%%b.exe --y4m -D 10 --input-res 1920x1080 --input-depth 10 --fps 24000/1001 --preset slower --crf 17.2 --%%r --limit-tu %%j --deblock=-1:-1 --no-sao --no-strong-intra-smoothing --rc-lookahead 99 --bframes %bf% --rd 6 --subme %subme% --vbv-maxrate 20000 --vbv-bufsize 24924 --frame-threads 1 --pools none --no-wpp --level-idc 51 --high-tier --colorprim 1 --colormatrix 1 --transfer 1 --range limited --input - --output output.hevc

mkvmerge.exe --deterministic 1 -o "Vinland-Saga_S1E16_ [build_%%b][%%r][ltu_%%j].mkv" -B -T -M output.hevc

)))

Test B (6 encodes, same as A but with preset medium and rd=6)
set dir=patman
set bf=6
set subme=3

for %%b in (1,2,3) do (
for %%r in (rd-refine,no-rd-refine) do (

rem b,r = build, rd-refine
rem build 1 = m_limit_tu
rem build 2 = overhaul
rem build 3 = overhaul + intrardrefine

ffmpeg.exe -loglevel level+error -i "Vinland-Saga_S1E16-rdr-scene.mkv" -map 0:v -vf "removegrain=1:2:2,removegrain=0:2:2,format=yuv420p10le" -sws_flags accurate_rnd -strict -1 -f yuv4mpegpipe - | %dir%\x265_%%b.exe --y4m -D 10 --input-res 1920x1080 --input-depth 10 --fps 24000/1001 --crf 17.2 --%%r --rd 6 --deblock=-1:-1 --no-sao --no-strong-intra-smoothing --rc-lookahead 99 --bframes %bf% --rd 6 --subme %subme% --vbv-maxrate 20000 --vbv-bufsize 24924 --frame-threads 1 --pools none --no-wpp --level-idc 51 --high-tier --colorprim 1 --colormatrix 1 --transfer 1 --range limited --input - --output output.hevc

mkvmerge.exe --deterministic 1 -o "Vinland-Saga_S1E16_ [build_%%b][%%r].mkv" -B -T -M output.hevc

))

In both tests, build 2 and 3 are the same, they give bit-equals output. If rd-refine=0 outputs do not depend on build.

--------------

Artifacts:
evident in test A, build 1, rd-refine and limit-tu=0 or 1, less evident with limit-tu=2. No artifact (or I did not notice them) in other tests.

[Z2697]

Is x265 SAO same as HM? This question was asked some where that I can't recall now.

I'd say it's based on HM but worse.
The structure does have the resemblance, but the core logic is different.
HM uses source pixels to calculate the real cost.
x265 use reconstructed pixels to estimate the cost: SAO NOP = 0 distortion + lambda * signal bits, and other SAO modes actually have "negative" distortion + lambda * some more bits.
So basically x265's SAO loses the "in-loop" attribute, and becomes some awkward game of balancing between 2 "random" things.
This could be an optmization choice... but at the cost of obliterating quality, is it really worth it? Is it effective? Because it still loops over all modes.
Perhaps I'll try bring real cost back and see how things will be.

[benwaggoner]

Is x265 SAO same as HM? This question was asked some where that I can't recall now.

I'd say it's based on HM but worse.
The structure does have the resemblance, but the core logic is different.
HM uses source pixels to calculate the real cost.
x265 use reconstructed pixels to estimate the cost: SAO NOP = 0 distortion + lambda * signal bits, and other SAO modes actually have "negative" distortion + lambda * some more bits.
So basically x265's SAO loses the "in-loop" attribute, and becomes some awkward game of balancing between 2 "random" things.
This could be an optmization choice... but at the cost of obliterating quality, is it really worth it? Is it effective? Because it still loops over all modes.
Perhaps I'll try bring real cost back and see how things will be.
Yeah, give the source and reconstructed frames should both be in L3 cache at that point, I wouldn't think the overhead would be that much. It might make sense as a quality/perf flag. Like:

--sao-mode-full (for proposed behavior)
--sao-mode-fast (for existing behavior)

[Z2697]

Uh, I was wrong, the recon (pre-SAO) to source distortion is stored in a variable during initilization.
So I guess it's on par with HM, which I haven't really used.
This precalculated dist won't tell how the post-SAO distortion is compared to the pre-SAO distortion, but HM and x265 SAO look similar if I squint my eyes, I count that as they are doing the same.
HM passes the source YUV to the init function in the "main structure", while x265 accesses source YUV directly in the init function, that's where I stumbled.
Which means both are not good?

But this SAO-specific distortion calculation does result in negative value A LOT, even like it's the majority.
I'm still under the impression that this is what causes it to be so bad.

I don't know what I'm talking about at this point, really.
Quit.