x265 psy-rd and psy-rdoq confusion

[valnar]

Depending on what AI chat program I use, and what day or phase of the moon it is, I get wildly different recommendations on what to set psy-rd and psy-rdoq to for a particular encode, even when referencing the same 1980's grainy movie.

Some say around --psy-rd=2.0 and --psy-rdoq=1.0 is good. Some say the opposite, --psy-rd=1.0 and --psy-rdoq=2.0. Sometimes it is both between 1.2 and 1.6. Other times I ask it might be something wildly different like psy-rdoq 4.0.

I realize my goals for that encode should dictate what I use, but my goals are always the same. "I'd like to reduce the file size while targeting a decent encode."

Occasionally I may ask what to change if I want to reduce grain, but that rarely changes their x265 parameters much and tend to move to Vapoursynth filters (which is fine - I use those too on occasion). I'm not a big grain junkie.

Is there something universally accepted with these two parameters? Or perhaps a couple of options, depending on source?

eg.
1) Grainy 1980's movie on crappy film, like Star Trek IV or A Fish Called Wanda.
2) Modern post-2000 movie shot digital.

Visually, I can't tell much difference between psy-rdoq 1.2 and 2.0, for example, but I can tell one might be throwing bits into the file size for no reason.

[hellgauss]

For anime content and 10bit encoding:

- I feel better with low psy-rd. In x264 it was useful to avoid banding, but with x265 it is not a problem. The suggested value psy-rd=0.4 for anime perhaps is still very low, I usually set psy-rd=0.5-0.9. Perhaps the default value 2.0 is good for movies, especially if they have grain.

- I think the default 1.0 value of psy-rdoq is quite good. Rise it a little bit if you want to keep grain and lower it a little bit if the source is clean.

Note: if you do not render the image as 10 bit using proper HW and SW, banding is still present even in x265 10bit. For 8bit display or player use dithering.

[microchip8]

I use psy-rd=2.5 and psy-rdoq=3.0 for all live action content and don't regret it. I don't do anime. Make sure to use rdoq-level=1 instead of 2 as this is when psy-rdoq becomes most effective as it has more stuff to work with compared to level 2 which cuts corners and makes psy-rdoq less effective.

[excellentswordfight]

Depending on what AI chat program I use, and what day or phase of the moon it is, I get wildly different recommendations on what to set psy-rd and psy-rdoq to for a particular encode, even when referencing the same 1980's grainy movie.

Some say around --psy-rd=2.0 and --psy-rdoq=1.0 is good. Some say the opposite, --psy-rd=1.0 and --psy-rdoq=2.0. Sometimes it is both between 1.2 and 1.6. Other times I ask it might be something wildly different like psy-rdoq 4.0.

I realize my goals for that encode should dictate what I use, but my goals are always the same. "I'd like to reduce the file size while targeting a decent encode."

Occasionally I may ask what to change if I want to reduce grain, but that rarely changes their x265 parameters much and tend to move to Vapoursynth filters (which is fine - I use those too on occasion). I'm not a big grain junkie.

Is there something universally accepted with these two parameters? Or perhaps a couple of options, depending on source?

eg.
1) Grainy 1980's movie on crappy film, like Star Trek IV or A Fish Called Wanda.
2) Modern post-2000 movie shot digital.

Visually, I can't tell much difference between psy-rdoq 1.2 and 2.0, for example, but I can tell one might be throwing bits into the file size for no reason.
For high values, I did quite recently a test with only --preset slow & --no-sao --bitrate 20000 (2pass vbr) with maybe 10 encodes of different psy-rd and psy-rdoq combinations (with values higher then defaults) on a very grainy 1080p source, subjective preference aside, all tests with higher values were less transparent from the source (change in grain structure, amplifying grain etc) so I stopped using high values, even when tuning for grain. So default values seems to quite reasonable, at least for live action.

[microchip8]

For high values, I did quite recently a test with only --preset slow & --no-sao --bitrate 20000 (2pass vbr) with maybe 10 encodes of different psy-rd and psy-rdoq combinations (with values higher then defaults) on a very grainy 1080p source, subjective preference aside, all tests with higher values were less transparent from the source (change in grain structure, amplifying grain etc) so I stopped using high values, even when tuning for grain. So default values seems to quite reasonable, at least for live action.

Define "high values". In the past, when x265 was very active in development, people used psy-rdoq of 15 (me including) but this is no longer recommended.

Higher values preserve texture and fine detail better than lower ones, and should in theory better preserve grain and produce less smoothing of complex textures. But the higher you go, the more artifacts it tends to produce like ringing and other artifacts. It also goes in the diminishing returns. Higher values lead to better visual perceptual experience at the cost of less compression and being less mathematically "correct" .

[Cmptr]

I use psy-rd=2.5 and psy-rdoq=3.0 for all live action content and don't regret it. I don't do anime. Make sure to use rdoq-level=1 instead of 2 as this is when psy-rdoq becomes most effective as it has more stuff to work with compared to level 2 which cuts corners and makes psy-rdoq less effective.

Using rdoq-level 1 over 2 will yield a much larger file though.

I used to use psy-rd 2 and psy-rdoq 2, but I noticed the video is more noisy. I changed it to the defaults that the slower presets have. Psy-rd 2 psy-rdoq 1 and the video looks very transparent. There is a bit more noise if you really look for it. Other than that, it has done pretty well. Probably my new settings to be honest. Note that I also disable cutree, aq-mode, aq-strength, sao, strong intra smoothing and deblock. I also uses a pretty low crf as well. Right now for season one of Snowpiercer, for 1080p straight from bluray I am using crf 14.5. I'm saving a minimum of 2gb per episode. Each episode is around 9gb. It may not sound like much, but for transparent, near visually lossless, anything smaller is better (Within reason). That is a minimum of 20gb that I would save for each season. It adds up.

[microchip8]

Using rdoq-level 1 over 2 will yield a much larger file though.

I used to use psy-rd 2 and psy-rdoq 2, but I noticed the video is more noisy. I changed it to the defaults that the slower presets have. Psy-rd 2 psy-rdoq 1 and the video looks very transparent. There is a bit more noise if you really look for it. Other than that, it has done pretty well. Probably my new settings to be honest. Note that I also disable cutree, aq-mode, aq-strength, sao, strong intra smoothing and deblock. I also uses a pretty low crf as well. Right now for season one of Snowpiercer, for 1080p straight from bluray I am using crf 14.5. I'm saving a minimum of 2gb per episode. Each episode is around 9gb. It may not sound like much, but for transparent, near visually lossless, anything smaller is better (Within reason). That is a minimum of 20gb that I would save for each season. It adds up.

Level 1 does not cut corners like level 2 so psy-rdoq goes deeper in optimizing for detail and texture preservation. It may increase ratecontrol a bit but not by huge amounts so it won't blow file sizes by a huge amount. It also depends which CRF you use as these two (crf + psy-rdoq and level) play together.

Your priorities are different from mine. I don't target near-lossless visual quality. I target "stunning" quality at a very good compression level using crf 20 for HDR 1080p encodes as I scale down UHD as my PC is too slow to encode it with these settings. My encodes vary from 2 GiB (very clean sources) up to 9 GiB (very noisy source). I'm happy with these sizes and particularly with the visual quality.

Also, using CRF of 14.5 is totally insane. If I was you, I'd just keep the original.

[excellentswordfight]

Define "high values". In the past, when x265 was very active in development, people used psy-rdoq of 15 (me including) but this is no longer recommended.

Higher values preserve texture and fine detail better than lower ones, and should in theory better preserve grain and produce less smoothing of complex textures. But the higher you go, the more artifacts it tends to produce like ringing and other artifacts. It also goes in the diminishing returns. Higher values lead to better visual perceptual experience at the cost of less compression and being less mathematically "correct" .
I went step by step for both values from default up to 5. Default values had the most similarity to a very grainy source when doing vbr encoding at sufficent bitrate.

How this effects perceived image quality at different crf values is another question though... Im not making any statements If high values combined with higher crf values produces overall a preferable encode to lower values with lower crf-values for a broad range of titles (if we say that output bitrate will be somewhat comparable). Im just sharing my findings, as I was a bit surprised as thought thought that I would need to go much higher to find this (10-15 as you mention). I assume that if you want to achieve high compression, while prioritizing details and retaining noise, its probably totally reasonable of doing like you do, and use high crf in combination with high psy values.

[Cmptr]

Level 1 does not cut corners like level 2 so psy-rdoq goes deeper in optimizing for detail and texture preservation.

I know rdoq-level 1 makes psy-rdoq more effective, but also, rdoq-level 2 performs a more in depth search and tries to retain more detail, does it not? That's what has always conflicted me. Level 2 is more in depth, but level 1 makes psy-rdoq more effective. My guess is because level 2 goes deeper, the whole block isnt seen as one, and can further be broken into smaller sections, which is why it can retain detail, why it can save space, and why it takes longer. Level 1 just acts on the whole block. Sure there is more info to work with, but at the same time, if the encoder thinks it should blur it, it's a bigger area to blur. If the encoder thinks it shouldn't, it gives the whole block more bits, even when there are flat areas that doesnt need them that would have been broken down and have a different qp set to in level 2.

I could be wrong though, that is just what I think. I would need to look at the code.

Level 2 works on 4x4 coding groups, so it does split the CU further. Level 1 tries retain energy in the residual. Yes level 2 makes psy-rdoq less effective to preserve energy because it works on smaller blocks. But it would give bits to a smaller group that needs them, take them away from a smaller group that doesnt need them, whereas level 1 would make a decision on the whole block, essentially wasting bits.

I would need clarification that this is how it works, because again, I could be wrong. It's just what I'm interpreting from the docs.

[valnar]

I've been doing a lot of psy-rd 1.5 and psy-rdoq 1.2 or so and I like the result, but it wasn't through exhaustive analysis of values between 0 and 3.0 or anything. It was a wet finger in the air after averaging many recommendations posted on the Interwebs and averaging some of those results.

I can notice that using 0 for either makes the image look flat.

[microchip8]

I know rdoq-level 1 makes psy-rdoq more effective, but also, rdoq-level 2 performs a more in depth search and tries to retain more detail, does it not? That's what has always conflicted me. Level 2 is more in depth, but level 1 makes psy-rdoq more effective. My guess is because level 2 goes deeper, the whole block isnt seen as one, and can further be broken into smaller sections, which is why it can retain detail, why it can save space, and why it takes longer. Level 1 just acts on the whole block. Sure there is more info to work with, but at the same time, if the encoder thinks it should blur it, it's a bigger area to blur. If the encoder thinks it shouldn't, it gives the whole block more bits, even when there are flat areas that doesnt need them that would have been broken down and have a different qp set to in level 2.

I could be wrong though, that is just what I think. I would need to look at the code.

Level 2 works on 4x4 coding groups, so it does split the CU further. Level 1 tries retain energy in the residual. Yes level 2 makes psy-rdoq less effective to preserve energy because it works on smaller blocks. But it would give bits to a smaller group that needs them, take them away from a smaller group that doesnt need them, whereas level 1 would make a decision on the whole block, essentially wasting bits.

I would need clarification that this is how it works, because again, I could be wrong. It's just what I'm interpreting from the docs.

Level 1 performs RDOQ mainly on the trasform coefficients.

Level 2 adds more exhaustive RDOQ decisions, including additional coefficient-level optimization/trellis work/etc

in Level 1, RDOQ evaluates decisions with a rate-distortion cost, but the decisions interact with later stages such as CABAC context modeling, transform selection, and mode decisions. A more exhaustive coefficient search can choose coefficients that look better according to its local cost while producing a slightly worse final encode.

Level 2 is substantially more computational for what may look worse subjectively

In short, Level 1 often produces subjectively visually better results than Level 2 and is faster.

I've said it before on these forums (it was in a thread about which AQ is better)... Sometimes the more simple algorithm can produce better (subjective) visual results. Many people focus too much on metrics like SSIM, VMAF, etc and think that if it's higher, it should translate to subjectively better image. This is not always true

[Cmptr]

Level 1 performs RDOQ mainly on the trasform coefficients.

Level 2 adds more exhaustive RDOQ decisions, including additional coefficient-level optimization/trellis work/etc

in Level 1, RDOQ evaluates decisions with a rate-distortion cost, but the decisions interact with later stages such as CABAC context modeling, transform selection, and mode decisions. A more exhaustive coefficient search can choose coefficients that look better according to its local cost while producing a slightly worse final encode.

Level 2 is substantially more computational for what may look worse subjectively

In short, Level 1 often produces subjectively visually better results than Level 2 and is faster.

I've said it before on these forums (it was in a thread about which AQ is better)... Sometimes the more simple algorithm can produce better (subjective) visual results. Many people focus too much on metrics like SSIM, VMAF, etc and think that if it's higher, it should translate to subjectively better image. This is not always true

I dont wanna hijack this thread, I do have questions. You do make a really good point in that sometimes a more simple algorithm can produce better quality. Gives me an idea for a new thread. I agree with what you say as well about the metrics.

[microchip8]

I dont wanna hijack this thread, I do have questions. You do make a really good point in that sometimes a more simple algorithm can produce better quality. Gives me an idea for a new thread. I agree with what you say as well about the metrics.

Just try it. I already convinced a few people who insisted Level 2 should be better. After trying Level 1, both agreed that it looks visually better than Level 2 (not by huge margins, but often noticeable enough)

[Cmptr]

Just try it. I already convinced a few people who insisted Level 2 should be better. After trying Level 1, both agreed that it looks visually better than Level 2 (not by huge margins, but often noticeable enough)

I see what it does now. After what level 1 does to the coefficients, level 2 decides if it's worth to keep the whole 4×4 group, or just zero it out.

[benwaggoner]

Just try it. I already convinced a few people who insisted Level 2 should be better. After trying Level 1, both agreed that it looks visually better than Level 2 (not by huge margins, but often noticeable enough)
I've seen some cases where Level 2 was better, but it definitely shouldn't be the default. I never figured out a great rubric to predict where it would be better either.

[microchip8]

I've seen some cases where Level 2 was better, but it definitely shouldn't be the default. I never figured out a great rubric to predict where it would be better either.

That's possible, but overall level 1 generally provides better image quality and is slightly faster :)

[benwaggoner]

That's possible, but overall level 1 generally provides better image quality and is slightly faster :)
I concur that level 1 is the appropriate default.

[Cmptr]

That's possible, but overall level 1 generally provides better image quality and is slightly faster :)

Maybe it's just how I use a low crf (Though that has changed as I have noticed that aq-mode 3 really helps in some dark areas, whereas no other mode does. This has required me to increase crf), but I do not notice a difference using rdoq-level 1 from 2, other than file size and speed. I'll run more test, but I've been using rdoq-level 2, and so far it doesnt look any worse or better than 1. Just a smaller file size. I dont notice any bland cu's either.


If level 1 gives a bigger file size, and better quality at say crf 18, if I use level 2 but lower the crf to match the size that of what level 1 produced... Would level 2 not have better quality in that case? (Yes I know I can try, I'm just encoding a movie right now)

[microchip8]

Maybe it's just how I use a low crf (Though that has changed as I have noticed that aq-mode 3 really helps in some dark areas, whereas no other mode does. This has required me to increase crf), but I do not notice a difference using rdoq-level 1 from 2, other than file size and speed. I'll run more test, but I've been using rdoq-level 2, and so far it doesnt look any worse or better than 1. Just a smaller file size. I dont notice any bland cu's either.


If level 1 gives a bigger file size, and better quality at say crf 18, if I use level 2 but lower the crf to match the size that of what level 1 produced... Would level 2 not have better quality in that case? (Yes I know I can try, I'm just encoding a movie right now)

rdoq depends a lot on psy-rd's strength, which does the heavy lifting and rdoq is the "support". I do not know what result you'll get if you change crf again, but overall rdoq-level 1 should provide subtle improvements over level 2. I am talking about visual quality, not compression/file sizes

[Cmptr]

rdoq depends a lot on psy-rd's strength, which does the heavy lifting and rdoq is the "support". I do not know what result you'll get if you change crf again, but overall rdoq-level 1 should provide subtle improvements over level 2. I am talking about visual quality, not compression/file sizes

Any way you could show me some examples? Ive tired a few and cant seem to see anything. I'm really curious, because I must be missing it. I do agree with you though that level 1 would provide more detail over level 2, because level 2 does what level 1 does, but also decides if the whole 4x4 group is worth it or not, and may zero out the whole group. Unless it also does make further decisions after level 1, prior to deciding if the block is worth it.

[Z2697]

if (dstCoeff[blkPos] > 1 || m_rdoqLevel == 1)
{
foundLast = true;
break;
}

[microchip8]

Any way you could show me some examples? Ive tired a few and cant seem to see anything. I'm really curious, because I must be missing it. I do agree with you though that level 1 would provide more detail over level 2, because level 2 does what level 1 does, but also decides if the whole 4x4 group is worth it or not, and may zero out the whole group. Unless it also does make further decisions after level 1, prior to deciding if the block is worth it.

Not at hand, but in the past I've seen slightly better retention of fine noise (think mosquito noise like in Blade Runner 2049) and regularly appears to the eye as if more detail is preserved. Level 1 stops searching if the coefficient is greater than 1 while level 2 goes deeper and may mistakenly zero out a 4x4 group (to my understanding). Plus you get the small benefit with level 1 that's it's a bit faster

However, there's an important distinction: rdoq level doesn't simply imply "higher = more detail." rdoq optimizes coefficients according to the distortion/rate cost. At a fixed CRF or bitrate, rdoq-level 2 can produce better overall fidelity, but it may also quantize some visually insignificant coefficients more aggressively while retaining more important ones.

Pick your poison, I'd say :)

[Cmptr]

if (dstCoeff[blkPos] > 1 || m_rdoqLevel == 1)
{
foundLast = true;
break;
}

This doesnt tell me much with this conditional expression other than it is in some loop, inside of some class, inside of some non static function, and if value in the array dstCoeff at index blkPos is > 1 or if m_rdoqLevel is 1, assign true to foundLast, then break out of the loop.

Not at hand, but in the past I've seen slightly better retention of fine noise (think mosquito noise like in Blade Runner 2049) and regularly appears to the eye as if more detail is preserved. Level 1 stops searching if the coefficient is greater than 1 while level 2 goes deeper and may mistakenly zero out a 4x4 group (to my understanding). Plus you get the small benefit with level 1 that's it's a bit faster

However, there's an important distinction: rdoq level doesn't simply imply "higher = more detail." rdoq optimizes coefficients according to the distortion/rate cost. At a fixed CRF or bitrate, rdoq-level 2 can produce better overall fidelity, but it may also quantize some visually insignificant coefficients more aggressively while retaining more important ones.

Pick your poison, I'd say :)

I took a look at the eyes, and there does seem to be a bit of a difference, I just cant choose which one is better, and closer to original. I looked at other frames as well and zoomed. They definitely arent the same, but neither is worse or better than the other. So my decision is that, the noticble extra bits rdoq level 1 uses over level 2, is not worth it, even if it does take a tiny bit more time. Id rather save the extra 2 - 3GB.

I havent tested it that much yet though either, so there may still be time for me to be added to the list of changing over. :)