Handling reddish, chroma-heavy scenes

[GeoffreyA]

Good morning,

Generally speaking, apart from lowering CRF, how would one tackle the rare, red, chroma-heavy scene where the quality plummets noticeably, particularly with blocking, relative to the rest of the encode? Setting a negative Cr offset in x265 doesn't seem to help much, and yes, it's naturally harder because one is dealing with a half-resolution plane and a colour quantised more heavily than others. I found that denoising the chroma does help, but that's another tradeoff in terms of softening.

Also, in general and as a default, is it worthwhile setting an offset of -3 to the Cb and Cr channels, as x264 does? If yes, wouldn't an asymmetric setting—say, -4 for Cr and -2 for Cb—be more logical?

Thank you.

[Columbo]

Use Tweak() to reduce saturation for red hues (defined by startHue/endHue).

[GeoffreyA]

Thank you, sir. Is that an AviSynth function? But good idea.

[Z2697]

What is it that you are encoding?

[GeoffreyA]

What is it that you are encoding?

"Last Night in Soho," which has many stylised scenes bathed in red light, for dream-like, surreal imagery. There are other scenes flickering with blue or coloured light.

[microchip8]

I use -4 for both Cr and Cb when doing HDR. I have not come across a red scene that looks blocky.

[Columbo]

Thank you, sir. Is that an AviSynth function? But good idea. Yes, also Vapoursynth has an implementation. Look at the theory: Cb and Cr don't mean what you may think.

[GeoffreyA]

I use -4 for both Cr and Cb when doing HDR. I have not come across a red scene that looks blocky.

I've changed my script to use -3 for both, and am only encoding SDR at present.

Yes, also Vapoursynth has an implementation. Look at the theory: Cb and Cr don't mean what you may think.

For that film, I would be hesitant to touch the saturation, it being very much part of the style, and the grading spot on. In other cases, toning down the saturation might be useful, even aesthetically, considering the fact that many a film today is oversaturated in the name of "vibrancy."

What I've been doing is using VapourSynth's ShufflePlanes() to get a view; when in YUV, these colourful scenes look bright and pinkish; otherwise, if luma is prevalent, it looks dull. I do find the theory fascinating and am trying to learn what I can, especially how it relates to our eyes.

[microchip8]

I've changed my script to use -3 for both, and am only encoding SDR at present.

For SDR content, the lowest you should go is -1. Going any lower will oversaturate colors. HDR is different as it has a much wider color gamut that benefits from much lower values. I've seen encodes that use the lowest -6 but I find that extreme.

[Z2697]

Oversaturate in terms of bits or what?
It's not possible to oversaturate the color by lowering the QP.

[Z2697]

What I've been doing is using VapourSynth's ShufflePlanes() to get a view; when in YUV, these colourful scenes look bright and pinkish; otherwise, if luma is prevalent, it looks dull. I do find the theory fascinating and am trying to learn what I can, especially how it relates to our eyes.

It's just the algorithmic method we currently use can't perfectly separate luminance and color -- we should have such method(s) already, I assume, but for the sake of performance we use simpler method.
(Not even BT2020CL can do that and it's already not used due to complexity I guess)

[rwill]

With x265 there is the problem that, if I remember correctly, they apply psy-rd to the chroma channels as well. So this favors blocking there too if it is enabled and there is some chroma detail.

Have you tried to disable psy-rd?

[microchip8]

Oversaturate in terms of bits or what?
It's not possible to oversaturate the color by lowering the QP.

Maybe oversaturated is the wrong word, but the lower you go on Cb/Cr QP, the less compression there is, thus making the colors more intense/stand out more strongly.

[Z2697]

With x265 there is the problem that, if I remember correctly, they apply psy-rd to the chroma channels as well. So this favors blocking there too if it is enabled and there is some chroma detail.

Have you tried to disable psy-rd?

But there's no separate QP control for chroma besides the fixed offset? So the effect shouldn't be that much right?
x264 does have default chroma QP offset tied to the psy-rd and psy-trellis though.

[Z2697]

Maybe oversaturated is the wrong word, but the lower you go on Cb/Cr QP, the less compression there is, thus making the colors more intense/stand out more strongly.

I'm not sure what you are talking about.
It's not possible for it to go beyond the source in an abnormal way.
The offset does not alter the pixel value, but of course the result of different compression will influence pixel value.
I hope you get what I mean.

The only way for it to make something close to such difference is well, the difference in the compression level.
In which case you should also note that the chroma are usually subsampled and have lower "contrast", so it's not unreasonable to have a different compression level.

I think what you should keep in mind (for what's wrong in you concept, including the "subme sharpening" thing)
is that the compression is about reconstruction not alteration.
At least it should be, and have been for many encoders, including the x264 and x265 (minus the buggy features like SAO).

[GeoffreyA]

For SDR content, the lowest you should go is -1. Going any lower will oversaturate colors. HDR is different as it has a much wider color gamut that benefits from much lower values. I've seen encodes that use the lowest -6 but I find that extreme.

Thanks, microchip. -3 is probably overkill, and truth be told, it's hard to see differences between these settings. I noticed that Beatrice-Raws, a celebrated anime release group, uses -2 for both Cb and Cr, in both their recent scripts and those from years ago, and even that would be more of a placebo approach.

It's just the algorithmic method we currently use can't perfectly separate luminance and color -- we should have such method(s) already, I assume, but for the sake of performance we use simpler method.
(Not even BT2020CL can do that and it's already not used due to complexity I guess)

Yes, the entire field of encoding is one trade-off after another. It frustrates, but that's perhaps why we enjoy the challenge.

With x265 there is the problem that, if I remember correctly, they apply psy-rd to the chroma channels as well. So this favors blocking there too if it is enabled and there is some chroma detail.

In x264, both psy rdo and psy trellis hurt chroma (https://code.videolan.org/videolan/x264/-/blob/master/encoder/encoder.c?ref_type=heads#L1224), so the offset mitigates that. x265 probably does the same, but interestingly, there's no offset by default. Perhaps the developers thought it looks the same.

[Z2697]

Chroma quality is supposed to be hard to distinguish (in a majority of cases), for better understanding, I suggest you post the sample clip and the whole script and settings you used to encode it.

[rwill]

In x264, both psy rdo and psy trellis hurt chroma (https://code.videolan.org/videolan/x264/-/blob/master/encoder/encoder.c?ref_type=heads#L1224), so the offset mitigates that. x265 probably does the same, but interestingly, there's no offset by default. Perhaps the developers thought it looks the same.

x265 does more than that. In certain (sub) mode decisions they apply the psy-rd metric to chroma as well. This leads to x265 having a certain bias when it can decide between a squeaky clean low energy and a fucked up high energy chroma block.

[Z2697]

There seems to be something wrong with the x265 chroma qp offset.
It has less effect on bitrate (in a pure red test clip) than x264 and hevc_nvenc. Even in CQP mode.
I don't know how that's possible as the offset is a fixed value and coded into PPS. (which x265 does correctly)

update It's coding too much skip blocks I think
If a block is skipped then there's no even a quantization step

[GeoffreyA]

rwill and Z2697, very interesting indeed. I'll post some clips and the script tomorrow. To be sure, with the last encoding, the quality is not bad and a lower CRF would make it transparent; but this is an interesting investigation, especially in the light of Z2697's findings.

[Z2697]

Even if I let x265 use only inter/intra mode it still not coding TUs for many blocks, which is equivalent to skip mode (but slower :o)

By the way, for "pure red test clip" I use
ffmpeg -i something -vf mergeplanes=0x000000:gbrp10le,lut=c1=0:c2=0,zscale=m=709:f=spline64,format=yuv420p10le -map v:0 -c:v ffv1 something_else
(probably should converted the input to full range first but... yeah *wave hand*)
(I don't know why lut acts like it's getting RGB when I give it GBRPxx -- which is the only planar RGB formats in FFmpeg and mergeplanes require planar format -- so red is the "c0")
As a bonus you can see how subsampling murders such scenes.

The Hiss has invaded our planes.

[GeoffreyA]

Here are the main clip and a multi-chroma bonus. (https://workupload.com/archive/jQQPLtwGYt) The scenes are flickering, so those sensitive to that effect, do take note.

ffmpeg -colorspace bt709 -color_trc bt709 -color_primaries bt709 -color_range tv
-i "%src%" -map 0:v:0 -map_metadata -1 -map_chapters -1 -disposition:v:0 default

-vf zscale=dither=none,format=yuv420p10le,crop=%crop%:exact=true

-c:v libx265 -preset slow -crf 21
-x265-params level-idc=41:no-sao=1:deblock=-1,-1:aq-mode=1:cbqpoffs=-3:crqpoffs=-3:
ctu=32:tu-intra-depth=3:tu-inter-depth=3:limit-tu=4:rd=4:rskip=2:rskip-edge-threshold=1:
ref=4:limit-refs=3:subme=4:max-merge=4:keyint=240:min-keyint=24:rc-lookahead=240:
bframes=8:b-intra=0:weightb=1:fades=1:no-amp=1 video.hevc

(The map -1 options are from when I used to output to MP4. Using only the bitstream now, they can be removed.)

[GeoffreyA]

By the way, for "pure red test clip" I use
ffmpeg -i something -vf mergeplanes=0x000000:gbrp10le,lut=c1=0:c2=0,zscale=m=709:f=spline64,format=yuv420p10le -map v:0 -c:v ffv1 something_else


Useful script; I changed the Matrix from green to red :)

https://i.imgur.com/wepcXLw.jpeg

[Z2697]

Positive offset works likely as aniticipated (and 0 offset of course), it's the negative offset is way off the track...
Maybe something is wiping out the residue buffer?
Or some steps are skipped based solely on luma CBF?
Sigh... I have skill issues.

[GeoffreyA]

I noticed that a negative offset increases the bitrate but less than x264. Say, for example, -4 might match x264's -3.

Do you think this is a potential bug, or an effect of changed algorithms? In the source code, in class Quant, it looks as if the offset is added straight without anything fishy going on. But elsewhere, the calculations look more complicated.

[Z2697]

I persoanlly think this is a bug...
I'm out, I can't find it.
I'll head back to play Control until Control Resonant come out

[GeoffreyA]

Well, enjoy Control. I hope it's better than Remedy's Alan Wake 2 :)

[microchip8]

What if you increase AQ1's strength to 1.1? The blockiness in the red image appears on the wall which is very flat. Increasing the strength a bit will relocate bits to uniform/flat areas of the picture like smooth walls or skies.

[GeoffreyA]

What if you increase AQ1's strength to 1.1? The blockiness in the red image appears on the wall which is very flat. Increasing the strength a bit will relocate bits to uniform/flat areas of the picture like smooth walls or skies.

Good idea because, in Soho, most of the red, nightmarish scenes seem to be large swathes of flat colour. I tried some tests earlier, and at a strength of 1.1, it's hard to see a difference. At 2, perhaps, but that might hurt normal scenes.

[Z2697]

Should have used git bisect right away
I wasted so many hours trying to understand x265 code
Git bisect only took less than one

2d01af69c4757b5087274e36e27b660b08244010 is the first bad commit
commit 2d01af69c4757b5087274e36e27b660b08244010 (HEAD)
Author: Deepthi Nandakumar <deepthi@multicorewareinc.com>
Date: Mon Jan 11 12:24:47 2016 +0530

rdcost: weight chroma distortion only for 444 formats

--HG--
branch : stable

source/encoder/rdcost.h | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)


Actually, I did find this part of the code
But I just thought, ah it's some special treatment for 444, what can possibly go wrong?
And I didn't look further!
Turns out it's the opposite way around: it's a special non-treatment for non-444!
Why did they come up with this thing???

[GeoffreyA]

In setQP()?

Yesterday, I just searched for chromaqpoffset and similar wording. Usually, it seems that they add the two offset values, the one from the slice struct and one referenced from the pps member, and some other things that seem quite cryptic.

[Z2697]

In setQP()?

yep.
:)

[Z2697]

In setQP()?

Yesterday, I just searched for chromaqpoffset and similar wording. Usually, it seems that they add the two offset values, the one from the slice struct and one referenced from the pps member, and some other things that seem quite cryptic.

The slice level offsets are only used internally for hdr10-opt in x265, to improve chroma quality, because the PQ transfer + BT2020 primaries really makes chroma channels "dull".
Which also didn't work because of the... bug thingy.
The global/PPS one is what the exposed offset parameters set.

Yeah, big brain moment, I know. (To me who went into elaborate mode decision code and didn't realize the lambda offset thing is the problem)

[GeoffreyA]

The slice level offsets are only used internally for hdr10-opt in x265, to improve chroma quality, because the PQ transfer + BT2020 primaries really makes chroma channels "dull".
Which also didn't work because of the... bug thingy.
The global/PPS one is what the exposed offset parameters set.

Yeah, big brain moment, I know. (To me who went into elaborate mode decision code and didn't realize the lambda offset thing is the problem)

I see. So, hdr10-opt was acting as a placebo all along, with respect to chroma? The whole setup they've got doesn't seem very elegant to me.

So AQ problems, chroma problems: bloody hell, I wonder what other mice are lurking in the codebase.

Well done, Mr. Z.

[Z2697]

I see. So, hdr10-opt was acting as a placebo all along, with respect to chroma?

Sort of...
If the overall offset is in the range of "slightly negative" to +12 then it probably will have some effect.
But if it's more negative then it will not have the anticipated effect.

(slightly negative: I'd say around -2 or something)

On the extreme end, it will have adversarial effect due to the RDO formula is skewed too much. (cost = distortion + lambda * bits, lambda is what that code is affecting)
update: Well, NO... it's scaling the "distortion" part. :o (and since "skip" is not costing bits, this makes sense?) But nonetheless, it's affecting the RDO formula.

[Z2697]

So AQ problems, chroma problems: bloody hell, I wonder what other mice are lurking in the codebase.

Don't forget about the SAO :devil: (heavy metal background music)

[GeoffreyA]

Don't forget about the SAO :devil: (heavy metal background music)

How could we forget x265's celebrated SAO, renowned for its excellence and detail!

[Z2697]

Well at least I can rest in peace, for now :) (until I decide to takle the SAO?)
Have you played Control? Very red, hmm.

[Z2697]

Here's the patched version ("vanilla" otherwise)
You can test if it fixes your problem with chroma qp offset instead of a global crf adjust.
https://workupload.com/file/akdHXxzwGgt

[rwill]

I am confused.

Isn't this m_chromaDistWeight getting smaller for smaller chroma QP values? The values are taken from the x265_chroma_lambda2_offset_tab table so it seems so. Then somewhat later chroma distortion is scaled by this value, making the distortion smaller than it really is.

Consider the following simple thought experiment:

chromaDistWeight = 0.5 (128), Lambda = 5

Mode0:
Distortion0 = 8
Bits0 = 1
Cost0_Weight = ( 8 * 0.5 ) + ( 1 * 5 ) = 9
Cost0_NoWeight = 8 + ( 1 * 5 ) = 13

Mode1:
Distortion1 = 2
Bits1 = 2
Cost1_Weight = ( 2 * 0.5 ) + ( 2 * 5 ) = 11
Cost1_NoWeight = 2 + ( 2 * 5 ) = 12

Weighted the one with the higher distortion would be chosen, Unweighted the one with more bits but the lower distortion. So I guess your fix wont resolve the problem of high distortion in chroma planes.

I would recommend to experiment with setting outCost.energy to 0 in codeIntraChromaQt and codeIntraChromaTSkip, set nonZeroEnergyC and zeroEnergyC to 0 in estimateResidualQT when related to chroma. That is, pretend m_rdCost.psyCost() returns 0 when it is used on chroma blocks.

[GeoffreyA]

Well at least I can rest in peace, for now :) (until I decide to takle the SAO?)
Have you played Control? Very red, hmm.

You have earned a long rest. Luckily, the Amulet of NoSAO holds off the Sample-Adaptive Dragon for now. How much longer the sharpness may hold, we know not :)

I haven't played Control but know it's famous. Well, in October, I gave Alan Wake 2 a try, having high hopes for it and yearning for more surreal, psychological horror after finishing Silent Hill 2 Remake, which was outstanding and moving. AW2's premise was great, but I felt the game was bogged down by clumsy mechanics and dreadful level design where it was hard to get a sense of direction. After much frustration, I stopped playing.

Here's the patched version ("vanilla" otherwise)
You can test if it fixes your problem with chroma qp offset instead of a global crf adjust.
https://workupload.com/file/akdHXxzwGgt

Many thanks. I'll give it a go tomorrow.

[Z2697]

I am confused.

Isn't this m_chromaDistWeight getting smaller for smaller chroma QP values? The values are taken from the x265_chroma_lambda2_offset_tab table so it seems so. Then somewhat later chroma distortion is scaled by this value, making the distortion smaller than it really is.

Consider the following simple thought experiment:

chromaDistWeight = 0.5 (128), Lambda = 5

Mode0:
Distortion0 = 8
Bits0 = 1
Cost0_Weight = ( 8 * 0.5 ) + ( 1 * 5 ) = 9
Cost0_NoWeight = 8 + ( 1 * 5 ) = 13

Mode1:
Distortion1 = 2
Bits1 = 2
Cost1_Weight = ( 2 * 0.5 ) + ( 2 * 5 ) = 11
Cost1_NoWeight = 2 + ( 2 * 5 ) = 12

Weighted the one with the higher distortion would be chosen, Unweighted the one with more bits but the lower distortion. So I guess your fix wont resolve the problem of high distortion in chroma planes.

I would recommend to experiment with setting outCost.energy to 0 in codeIntraChromaQt and codeIntraChromaTSkip, set nonZeroEnergyC and zeroEnergyC to 0 in estimateResidualQT when related to chroma. That is, pretend m_rdCost.psyCost() returns 0 when it is used on chroma blocks.

That's quite puzzling, but it's actually the x264 way of doing it.

Like I said, it feels like a fruitless hunt that wasted my time... (I did try modifying the other functions you mentioned)
It looks like some rdcost are calculated without the scale.
But I still don't understand it.
But but, it has worked, for x264, for all these years... it must be good :o

[Z2697]

You have earned a long rest. Luckily, the Amulet of NoSAO holds off the Sample-Adaptive Dragon for now. How much longer the sharpness may hold, we know not :)

I haven't played Control but know it's famous. Well, in October, I gave Alan Wake 2 a try, having high hopes for it and yearning for more surreal, psychological horror after finishing Silent Hill 2 Remake, which was outstanding and moving. AW2's premise was great, but I felt the game was bogged down by clumsy mechanics and dreadful level design where it was hard to get a sense of direction. After much frustration, I stopped playing.



Many thanks. I'll give it a go tomorrow.

Yeah AW2 is more of a fan service than an overall good game.
Even though I enjoyed it, I don't want to replay it (new game +) that much.

[GeoffreyA]

As a rough test, using high CRFs and chroma offsets of -12, there is a big difference.

x265cw --input - --y4m --input-depth 8 --preset slow --crf 27 --aq-mode 1 --no-sao --deblock -1:-1 --cbqpoffs x --crqpoff x

https://workupload.com/archive/G2cxVpUDQW

Weighted the one with the higher distortion would be chosen, Unweighted the one with more bits but the lower distortion. So I guess your fix wont resolve the problem of high distortion in chroma planes.

That's quite puzzling, but it's actually the x264 way of doing it.

Like I said, it feels like a fruitless hunt that wasted my time... (I did try modifying the other functions you mentioned)
It looks like some rdcost are calculated without the scale.
But I still don't understand it.
But but, it has worked, for x264, for all these years... it must be good :o

In theory, I think it makes sense: make the more-distorted chroma look attractive in terms of RD cost, so it is chosen, saving bits to go elsewhere.

Clearly, though, their code is unsound:


int qpCb, qpCr;
if (slice.m_sps->chromaFormatIdc == X265_CSP_I420)
{
qpCb = (int)g_chromaScale[x265_clip3(QP_MIN, QP_MAX_MAX, qp + slice.m_pps->chromaQpOffset[0] + slice.m_chromaQpOffset[0])];
qpCr = (int)g_chromaScale[x265_clip3(QP_MIN, QP_MAX_MAX, qp + slice.m_pps->chromaQpOffset[1] + slice.m_chromaQpOffset[1])];
}
else
{
qpCb = x265_clip3(QP_MIN, QP_MAX_SPEC, qp + slice.m_pps->chromaQpOffset[0] + slice.m_chromaQpOffset[0]);
qpCr = x265_clip3(QP_MIN, QP_MAX_SPEC, qp + slice.m_pps->chromaQpOffset[1] + slice.m_chromaQpOffset[1]);
}

if (slice.m_sps->chromaFormatIdc == X265_CSP_I444)
{
int chroma_offset_idx = X265_MIN(qp - qpCb + 12, MAX_CHROMA_LAMBDA_OFFSET);
uint16_t lambdaOffset = m_psyRd ? x265_chroma_lambda2_offset_tab[chroma_offset_idx] : 256;
m_chromaDistWeight[0] = lambdaOffset;

chroma_offset_idx = X265_MIN(qp - qpCr + 12, MAX_CHROMA_LAMBDA_OFFSET);
lambdaOffset = m_psyRd ? x265_chroma_lambda2_offset_tab[chroma_offset_idx] : 256;
m_chromaDistWeight[1] = lambdaOffset;
}
else
m_chromaDistWeight[0] = m_chromaDistWeight[1] = 256;

qpCb and qpCr are calculated for 420 and other cases, using the offsets. Then, these two variables are used only in the 444 case to calculate the distortion weight.

Being an error, will MCW take the patch? Also, as this goes back to 2016, it will change behaviour that people are used to. What about that?

[Z2697]

For 444 everything should remain the same, for other subsampling, the default offset will be the same, except one thing: the qpCb/Cr for 420 is scaled when it's originally > 29, so the default crf 28 will be slightly affected, lower ones are less likely.

[Z2697]

In theory, I think it makes sense: make the more-distorted chroma look attractive in terms of RD cost, so it is chosen, saving bits to go elsewhere.

It's a bit, erm, backwards isn't it?
With negative offset we want (and indeed) spend more bits on chroma.

[GeoffreyA]

It's a bit, erm, backwards isn't it?
With negative offset we want (and indeed) spend more bits on chroma.

In my limited understanding, I reckon the overarching system was first developed in x264, under the philosophy of chroma bad, luma good, which is correct most of the time. Perhaps the offset was later added as a hack to mitigate the damage. x265 inherited all that.

So, yes, it is a bit backwards. An ideal system would alter the chroma depending on its importance to the frame.

[Z2697]

I am confused.

Isn't this m_chromaDistWeight getting smaller for smaller chroma QP values? The values are taken from the x265_chroma_lambda2_offset_tab table so it seems so. Then somewhat later chroma distortion is scaled by this value, making the distortion smaller than it really is.

Consider the following simple thought experiment:

chromaDistWeight = 0.5 (128), Lambda = 5

Mode0:
Distortion0 = 8
Bits0 = 1
Cost0_Weight = ( 8 * 0.5 ) + ( 1 * 5 ) = 9
Cost0_NoWeight = 8 + ( 1 * 5 ) = 13

Mode1:
Distortion1 = 2
Bits1 = 2
Cost1_Weight = ( 2 * 0.5 ) + ( 2 * 5 ) = 11
Cost1_NoWeight = 2 + ( 2 * 5 ) = 12

Weighted the one with the higher distortion would be chosen, Unweighted the one with more bits but the lower distortion. So I guess your fix wont resolve the problem of high distortion in chroma planes.

I would recommend to experiment with setting outCost.energy to 0 in codeIntraChromaQt and codeIntraChromaTSkip, set nonZeroEnergyC and zeroEnergyC to 0 in estimateResidualQT when related to chroma. That is, pretend m_rdCost.psyCost() returns 0 when it is used on chroma blocks.

Oh, you got it the opposite way around, the qp - qpC + 12 will result in higher value for negative offsets.
I did realize this at first but get confused after reading you reply.

(Guess my brain is worse than LLM, I completely forgotten I have understood it already :o)
(This is what you get from playing too much video game instead of having a giant datacenter to host your consciousness)

[rwill]

Ah so I made a sign error, how typical.

[GeoffreyA]

For 444 everything should remain the same, for other subsampling, the default offset will be the same, except one thing: the qpCb/Cr for 420 is scaled when it's originally > 29, so the default crf 28 will be slightly affected, lower ones are less likely.

There should be no negatives—only increased bitrate—and it seems to fix an important function of the encoder. I find that, for a value of -3, it goes a bit higher than x264 in terms of bitrate: a few per cent. This is certainly not a negative for those wanting to tune the chroma. Are you going to submit your patch and findings to x265?

(This is what you get from playing too much video game instead of having a giant datacenter to host your consciousness)

Uploaded and distributed consciousness!