What is the RDOQ algorithm in HM?

[xkfz007]

I have learned something about the RDOQ. And I have read the RDOQ source codes in X264. However, I find the RDOQ algorithm in HM differs a lot. Do anyone have some materials about the RDOQ? Are there any proposals on RDOQ in JCT-VC?

[LoRd_MuldeR]

RDO = Rate Distortion Optimization

Basically RDO means that if the encoder has a number of choices, it will not simply pick the one that results in the smallest distortion (i.e. smallest difference between original and encoded frame), like a non-RDO optimizer would do. Instead, it also takes into account how many bits each possible choice will cost in the end. In other words, RDO picks the choice that gives the "best" compromise between distortion and bit cost (the "rate"). For example, it may be reasonable to pick a choice that causes a somewhat stronger distortion, when that choice saves many bits. At the same time, the choice that would give the smallest distortion might be overly expensive, in terms of bits. Of course implementation details of RDO can differ a lot.

Expressed as a formula, RDO tries to minimize D + λ*R, where D is the distortion (e.g. as Mean Squared Error), R is the rate (e.g. cost in bits) and λ is some weighting factor.

[xkfz007]

RDO = Rate Distortion Optimization

Basically RDO means that if the encoder has a number of choices, it will not simply pick the one that results in the smallest distortion (i.e. smallest difference between original and encoded frame), like a non-RDO optimizer would do. Instead, it also takes into account how many bits each possible choice will cost in the end. In other words, RDO picks the choice that gives the "best" compromise between distortion and bit cost (the "rate"). For example, it may be reasonable to pick a choice that causes a somewhat stronger distortion, when that choice saves many bits. At the same time, the choice that would give the smallest distortion might be overly expensive, in terms of bits. Of course implementation details of RDO can differ a lot.

Expressed as a formula, RDO tries to minimize D + λ*R, where D is the distortion (e.g. as Mean Squared Error), R is the rate (e.g. cost in bits) and λ is some weighting factor.

RDO is used to decide the prediction mode. But RDOQ is used to quantize the transform coefficients. I know the RDO process and knows a little about RDOQ.

[LoRd_MuldeR]

RDO is used to decide the prediction mode. But RDOQ is used to quantize the transform coefficients. I know the RDO process and knows a little about RDOQ.

I would rather say that RDO is a general approach, which can be applied on various optimization problems - may it be selecting the prediction mode or selecting the quantized transform coefficients or something else :)

Now, in the quantization step, we can choose (at least) between rounding down the coefficient, round up the coefficient and forcing the coefficient to zero.

And again the idea of RDO can help to decide which of these choices would give the best "distortion vs. bit cost" compromise - rather than using "dead zones" or just selecting the one that would minimize the distortion (regardless of bit cost).

Actually, I think that is exactly what Trellis quantization (http://akuvian.org/src/x264/trellis.txt) does in x264 ;)

[xkfz007]

I would rather say that RDO is a general approach, which can be applied on various optimization problems - may it be selecting the prediction mode or selecting the quantized transform coefficients or something else :)

Now, in the quantization step, we can choose (at least) between rounding down the coefficient, round up the coefficient and forcing the coefficient to zero.

And again the idea of RDO can help to decide which of these choices would give the best "distortion vs. bit cost" compromise - rather than using "dead zones" or just selecting the one that would minimize the distortion (regardless of bit cost).

Actually, I think that is exactly what Trellis quantization (http://akuvian.org/src/x264/trellis.txt) does in x264 ;)

Thanks！
After a couple days of study of HM RDOQ, I have known the main process of the algorithm. The HM RDOQ is different from the Trellis RDOQ in X264. However, as you said, the main principle is based on the RDO idea. And RDO is used to calculate the cost of the residual syntax elements.