Resolve 16.2 mangles chroma

[Stereodude]

Is there a way to get Resolve to not mangle the chroma of a video with non square pixels? (I'm guessing the non square pixels are the problem)

I have a NTSC 16:9 enhanced DVD (720x480), I converted it to a .mov containing 8-bit 420 H.264 (crf 1) with VD2 so I can mess with the grade in Resolve. I put it into Resolve into a 720x480 NTSC DV 16:9 timeline, do some color grading and then output it as uncompressed 10bit 422 YUV in a .mov and I get this crap. :(

Here's the input (scaled to 2x by madVR)
https://i.imgur.com/QsiGbH1.jpg

Here's the output (scaled to 2x by madVR)
https://i.imgur.com/DTalCx8.jpg

I've tried other output formats, like Cineform and got the same thing. I tried a custom 720x480 timeline, but that doesn't seem to stick. It keeps reverting back to 720x480 NTSC DV 16:9.

Are there some settings I'm missing (I'm very far from a Resolve guru) or is Resolve just broken?

[Stereodude]

So I did a little more testing. I'm not sure if it is Resolve's 420 -> 422 conversion that's the problem or perhaps an issue with H.264, but if I convert the DVD to ProRes 422 instead of near lossless H.264 and feed that into Resolve the chroma in the output is fine.

[poisondeathray]

So I did a little more testing. I'm not sure if it is Resolve's 420 -> 422 conversion that's the problem or perhaps an issue with H.264, but if I convert the DVD to ProRes 422 instead of near lossless H.264 and feed that into Resolve the chroma in the output is fine.


I've tried other output formats, like Cineform and got the same thing


If that's the case why would cineform 422 be affected as well ?

[Stereodude]

If that's the case why would cineform 422 be affected as well ?
Because the CineForm 422 was an output format I tried with the same near lossless 420 H.264 input. When I changed the input format to ProRes 422 or uncompressed v210 the problem goes away in the output.

I can't narrow down the issue on the input side any more because Resolve won't take uncompressed 8-bit or 10-bit 420 video in a .mov container as a source file. So I'm not sure if it's a H.264 420 specific issue or a 420 specific issue. Resolve didn't want to take a 4:2:2 H.264 input file either.

They desperately need to support VFW AVI codecs.

[poisondeathray]

Because the CineForm 422 was an output format I tried with the same near lossless 420 H.264 input.


I misunderstood, I though you meant you tested cineform as the input and output (yes, even though it said output... )


They desperately need to support VFW AVI codecs.

This will never happen

[Cary Knoop]

Did you verify it is not an interlacing thing instead of a chroma thing?

[Stereodude]

Did you verify it is not an interlacing thing instead of a chroma thing?
The source is progressive 30/1.001.

[poisondeathray]

It looks like point sampled (nearest neighbor) chroma . It's just duplicating samples up to 4:4:4 then RGB, then back down to 4:2:2 when exporting a 4:2:2 format

So if you export some 4:2:2 format, you can downsample back to 4:2:0 using chromaresample="point" and it will look close to what you started with

#exported 4:2:2 format like prores
ConvertBits(8)
ConvertToYV12(chromaresample="point")

Or you can use avsresize eg. for 10bit , but 4:2:0
z_ConvertFormat(pixel_type="YUV420P10", resample_filter="point")

(I tested this on some patterns in Resolve Studio 15, similar conditions 720x480 x264 4:2:0 import, 720x480 timeline settings, 422 export)


https://i.postimg.cc/yxrFWv8g/point.png (https://postimages.org/)

[Stereodude]

It looks like point sampled (nearest neighbor) chroma . It's just duplicating samples up to 4:4:4 then RGB, then back down to 4:2:2 when exporting a 4:2:2 format
I'll take really poor algorithm choices for $1000 Alex.

BTW, thanks for figuring this out! :thanks:

[poisondeathray]

BTW that bottom graphic is labelled in correctly it should be avsresize point 10bit420, not 10bit422 - (ie. 422=>420 just discarding the samples that were duplicated)

[Stereodude]

So if you export some 4:2:2 format, you can downsample back to 4:2:0 using chromaresample="point" and it will look close to what you started with

#exported 4:2:2 format like prores
ConvertBits(8)
ConvertToYV12(chromaresample="point")

Or you can use avsresize eg. for 10bit , but 4:2:0
z_ConvertFormat(pixel_type="YUV420P10", resample_filter="point")
So I decided to measure the PNSR of this. The numbers aren't stellar. I can't speak to whether the results are visually apparent or not. I didn't attempt to examine that aspect.

This is lossless:
Convert YV12 source to v210 in Avisynth -> uncompressed v210 .mov (VD2) -> Resolve -> uncompressed v210 .mov -> Avisynth convert back to YV12

This is not even close:
Compress the YV12 source to near lossless (qp 1 / keyint 1) H264 -> Resolve -> v210 .mov -> Avisynth convert back to YV12

The near lossless H264 compression is good for -~70-75dB

The Resolve -> YV12 in Avisynth step degrades it quite a bit further dipping all the way down to ~46dB for a section on my 5000 frame test clip.

https://i.imgur.com/MIsDYG6.png

My guess is that it has something to do with the chroma placement and the transformation matrix not aligning between the 4:2:0 -> 4:2:2 conversion in Resolve and 4:2:2 -> 4:2:0 back in Avisynth.

Edit: It seems like this is potentially a significant issue for anyone who uses 4:2:0 video from their cameras directly in Resolve and then exports from Resolve in a lossless or with a "light touch" codec to encode in something else. You can visually improve the chroma with a point resample of it back to 4:2:0, but you can't actually reverse the operation.


Also for a point of reference while I was at it I measured Cineform's performance (forcing a re-encode in Resolve) using the same flow vs. v210 uncompressed and got this:

https://i.imgur.com/kBlbp9i.png

Convert YV12 source to v210 in Avisynth -> CineForm FS3 v210 .mov (VD2) -> Resolve -> CineForm Best v210 .mov -> Avisynth convert back to YV12

[poisondeathray]

My guess is that it has something to do with the chroma placement and the transformation matrix not aligning between the 4:2:0 -> 4:2:2 conversion in Resolve and 4:2:2 -> 4:2:0 back in Avisynth.

There is an intermediate RGB step in Resolve too. If there wasn't, and Resolve could work in YUV, you would expect Y to be the same since you're measuring Y psnr in those graphs, not Y, U, V or aggregate YUV PSNR. For programs that are YUV capable, there is no alteration of the Y plane . v210 gets special treatment - it actually gets special treatment in other NLE's too, even ones that work internally in RGB . It's a native format that gets passthrough . For some Windows based NLE's, you can use IYUV (8bit 4:2:0, with fourcc IYUV, or 8bit 4:2:2 with fourcc UYVY) and those gets passthrough , but Resolve is really devloped from a Mac environment , there is no way to get it to work with uncompressed 8bit 4:2:0

Was this internal avs function conversion, or z_convert_format?

Recall there is a chroma shifting issue in the internal avs resizers . It's not a true point resize. There are workarounds such as resize8, but z_convert_format works properly

But it would partially depend on how resolve is defining the chroma placement too.

[poisondeathray]

On a synthetic test clip (the one above, but there is animation, rotation, but it's just simple text motion)

Same thing 720x480 YV12 source => Resolve => Export v210 => Convert back to YV12

ConvertBits(8).ConvertToYV12(chromaresample="point")
[Parsed_psnr_4 @ 000000702506d280] PSNR y:71.482031 u:48.201779 v:48.396939 average:53.027946 min:52.964134 max:53.075222

Z_ConvertFormat(pixel_type="YV12", resample_filter="point")
[Parsed_psnr_4 @ 000000a5d07f7800] PSNR y:68.920585 u:61.097875 v:73.214861 average:66.519582 min:64.670853 max:67.972861

The U, V scores are much lower for the internal avs as expected with the known shifting

I'll try some real clips when I get a chance , but with synthetic test patterns it might be possible to figure out how it's interpeting the chroma and reverse it more closely

[poisondeathray]

UHD YV12 1152 frame , fairly high scene complexity with foliage, people, movement

Spline16Resize => 720x480p23.976 . Did not compensate for colormatrix or anything else. This is the new source "orig"

Encoded x264 --qp 1 --keyint 1 --tune psnr

Import qp1_intra version into Resolve, matched all import/export settings. Check marked retain sub black and super white on export (this clip has some since it was a native camera shot)

Calculated per frame metrics, but I didn't plot it out, I'm just posting the averages . FFmpeg psnr

self test check orig vs. orig
lossless

x264_qp1_intra vs. orig
PSNR y:68.087383 u:67.117369 v:67.088122 average:67.733770 min:67.343042 max:68.028963

ConvertBits(8).ConvertToYV12(chromaresample="point") vs. orig
PSNR y:67.782155 u:65.525216 v:66.421621 average:67.082155 min:65.246296 max:67.911099

ConvertToYUV420(chromaresample="point").ConvertBits(8) vs. orig
#In theory 10bit downsample before converting to 8 is better, but it doesn't matter when you're just dropping samples with "point", it should with other kernals
PSNR y:67.782155 u:65.525216 v:66.421621 average:67.082155 min:65.246296 max:67.911099

z_convertformat(pixel_type="YV12", resample_filter="point", resample_filter_uv="point") vs. orig
PSNR y:53.656829 u:66.398999 v:66.530673 average:55.305442 min:53.734979 max:57.186774


There is some loss , but PSNR Y,U,V > 65 is pretty good . Minus 1 or 2 db per channel vs. the qp1_keyint1 input into Resolve . You expect some loss because of the internal YUV=>RGB=>YUV conversion in resolve, in addition to, and separate from the subsampling/upsampling

Using long GOP, you expect that up/down pattern in those charts, especially with b-frames using default I/P/B ratios. Maybe try repeating with intra

Maybe a big problem with zimg Y plane 10bit => 8bit conversion. A 12-13db PSNR delta is massive. I have a feeling user error somewhere but I repeated results with vapoursynth version , same values. I'll recheck everything, but on that colored text clip it was lower than internal AVS too.

[Stereodude]

There is an intermediate RGB step in Resolve too. If there wasn't, and Resolve could work in YUV, you would expect Y to be the same since you're measuring Y psnr in those graphs, not Y, U, V or aggregate YUV PSNR. For programs that are YUV capable, there is no alteration of the Y plane . v210 gets special treatment - it actually gets special treatment in other NLE's too, even ones that work internally in RGB . It's a native format that gets passthrough . For some Windows based NLE's, you can use IYUV (8bit 4:2:0, with fourcc IYUV, or 8bit 4:2:2 with fourcc UYVY) and those gets passthrough , but Resolve is really devloped from a Mac environment , there is no way to get it to work with uncompressed 8bit 4:2:0
I guess I can't speak to what resolve is doing internally. I presumed it converts to RGB all the time, but maybe not. I unchecked the box for "Bypass re-encode when possible". Obviously I couldn't make any changes to the video or I'd lose my ability to do a PNSR. I guess I could have use two inline grade nodes that cancel each other out. I didn't try that.

I also didn't realize I was only looking at the Y plane. Oops. I'm brand new to using the MSU VQMT tool.

Was this internal avs function conversion, or z_convert_format?

Recall there is a chroma shifting issue in the internal avs resizers . It's not a true point resize. There are workarounds such as resize8, but z_convert_format works properly

But it would partially depend on how resolve is defining the chroma placement too.
I tried both internal avs function and z_convert. I need to retest using YUV PNSR.

How are you measuring PNSR with ffmpeg? The free version of MSU VQMT can't do 10-bit which makes this a big pain.

[Stereodude]

Checking with YUV PNSR, z_lib (avsresize) is lossless 4:2:0 -> 4:2:2 -> 4:2:0 using point resize. The internal Avisynth functions are not lossless.

A quick test of a few other z_lib resize kernels show they are are not lossless.

[poisondeathray]

How are you measuring PNSR with ffmpeg?


https://ffmpeg.org/ffmpeg-filters.html#psnr



The free version of MSU VQMT can't do 10-bit which makes this a big pain.

Nor can it do >720p :( . But the GUI and plots are a nice touch and it's user friendly

[Stereodude]

I used a MPEG-2 DVD source with non-square pixels since that's what I've been working on in Resolve. Maybe that has something to do with the numbers.

I compressed to H264 using --qp 1 --keyint 1 --tune psnr using the x264 engine in the latest VD2. (it's the upper set of lines on the graphs)

I redid my tests using YUV

I converted the v210 422 Quicktime back to 8-bit 420 with
z_ConvertFormat(pixel_type="YV12", resample_filter="point", colorspace_op="170m:601:170m:f=>170m:601:170m:f")
https://i.imgur.com/z9x2wDJ.png

This gave different results
ConvertToYUV420(matrix="PC.601", chromaresample="point").ConvertBits(8, dither=0)
https://i.imgur.com/1ScNVI6.png

[poisondeathray]

Seems a bit low for some of those sections. Is something special about them visually?

I tried vqmt and got slightly different values (maybe slightly different psnr calcs) , but the trend was the same, average in the mid sixties. Nothing dropping that low, maybe 59.something was the lowest. Nothing to suggest a significant difference between ffmpeg psnr and msu vqmt's psnr

Mine was oversampled non square pixel , 720x480 . It was encoded with --sar 32:27 too but I doubt that would make a difference

Can you upload small clip test section that includes a peak and valley ?

[kolak]

Resolve handling of 420/422 was far from correct.
It was enough to export v210 5x back to v210 (which was always going through RGB) and you had red channel shifted few pixels! Then they improved it a bit, but it's still not very good. They will also now pass v210 (and not only) without any processing if you do just cut editing.

[Stereodude]

Resolve handling of 420/422 was far from correct.
It was enough to export v210 5x back to v210 (which was always going through RGB) and you had red channel shifted few pixels! Then they improved it a bit, but it's still not very good. They will also now pass v210 (and not only) without any processing if you do just cut editing.
:confused: So is there any format that it handles better or a workaround?

[poisondeathray]

I mentioned it to you in a previous thread - there is a completely lossless workflow , but it involves EXR sequences and vapoursynth . There were issues before, but Resolve handles EXR float properly now, since v12 or 13 I think . Definitely with v15 it's ok.

Float enables you to convert YUV<=>RGB losslessly when done properly. If avs+ gets a float importer (import EXR that doesn't clip to 16bit int, like ffmpeg) , then it should possible in avs+ too since it can handle float now too.

[Stereodude]

I've never messed with vapoursynth. Not sure of the learning curve of it vs. avisynth. Maybe I'll take a look at it later.

However, I suspect I'm missing the forest for the trees getting lost in the minutia here. The less than idea PNSR result doesn't mean mean it looks bad or has visible artifacts. I didn't do a super exhaustive comparison, but in my quick look at a few frames, I couldn't see any quickly apparent difference in the before and after frame despite the sub 50dB PNSR of the worst spots (using uncompressed video formats).

[poisondeathray]

However, I suspect I'm missing the forest for the trees getting lost in the minutia here. The less than idea PNSR result doesn't mean mean it looks bad or has visible artifacts. I didn't do a super exhaustive comparison, but in my quick look at a few frames, I couldn't see any quickly apparent difference in the before and after frame despite the sub 50dB PNSR of the worst spots (using uncompressed video formats).

Of course, and 50dB is usually great for intermediates.

But it seems the min values are lower than expected. This is qp 1 intra. You should be getting results higher than prores xq or cineform fs2.

PSNR is good for is detecting lossless (or not), and indicating trends, and potential workflow problems

So I'm wondering if something else going on there

[Stereodude]

Of course, and 50dB is usually great for intermediates.

But it seems the min values are lower than expected. This is qp 1 intra. You should be getting results higher than prores xq or cineform fs2.

PSNR is good for is detecting lossless (or not), and indicating trends, and potential workflow problems

So I'm wondering if something else going on there
I did find two issues.

1) The "retain sub-black and super-white data" in Resolve's export has to be checked.
2) You have to also use z.lib explicitly set to full range, the matrix and transfer coefficients seem to have no effect, but the range certainly does.

I also found that if you're using the internal avisynth functions converting to 8 bit first then to YV12 gives a slightly better PNSR score. I was not able to find a matrix for the internal function that gave similar results.

Here's the best z.lib/avsresize result I was able to get:
https://i.imgur.com/NEg4w70.png

Here's the best internal function result I could get:
https://i.imgur.com/Bl8ARte.png

[poisondeathray]

2) You have to also use z.lib explicitly set to full range, the matrix and transfer coefficients seem to have no effect, but the range certainly does.


Nice, that fixes the earlier observations about low Y PSNR values for avsresize/zimg for that conversion.

My thinking was input=output so it's a no-op, so I omitted the args. Lesson learned!


I also found that if you're using the internal avisynth functions converting to 8 bit first then to YV12 gives a slightly better PNSR score.

Interesting. You'd think higher precision @ 10bit operations would produce slightly better results


I was not able to find a matrix for the internal function that gave similar results.


Matrix only applies to YUV<=>RGB transforms , so it's not applicable here. That I'm sure of for the internal functions. (I'm questioning my understanding of how avsresize works.)



My results using the clip in avsresize/zimg. Not sure if it was the same clip because it was 5002 frames vs 5000. And the trends seem different w.r.t the frame ranges . My U plane seemed more problematic than V , but your V seemed to have more trouble than your U . Values seem slightly higher too , the min was 56.288 in vqmt , but you have clear V drops below 50
https://postimg.cc/QHcjV8V7

VQMT scales the U,V planes to Y plane size, that might introduce some errors. I tried unchecking that option , but it would crash. It might be a pro version feature. Anyways it's the trends that are important

FWIW this is ffmpeg PSNR , and the abs. min value is higher , and the U,V averages are higher than VQMT - maybe some of the difference is from the VQMT UV scaling, not sure. Often there are slightly different calcs too between programs
PSNR y:65.979368 u:64.950274 v:66.120163 average:65.812176 min:59.375502 max:88.556475

You can combine methods too. If a different method produces higher values in a plane you mix/match with combineplanes. But like you said this is splitting hairs these higher PSNR values. It just sucks to see drops like that in sections

[Stereodude]

Interesting. You'd think higher precision @ 10bit operations would produce slightly better results
Yes, it seems counter intuitive.

Matrix only applies to YUV<=>RGB transforms , so it's not applicable here. That I'm sure of for the internal functions. (I'm questioning my understanding of how avsresize works.)
There is talk about full vs. limited range in the matrix selection section (http://avisynth.nl/index.php/Convert#Parameter_Details), hence I thought it might have some impact, but apparently it's only used if going between RGB and YUV.

VQMT scales the U,V planes to Y plane size, that might introduce some errors. I tried unchecking that option , but it would crash. It might be a pro version feature. Anyways it's the trends that are important
I found the same thing. Insta-crash.

FWIW this is ffmpeg PSNR , and the abs. min value is higher , and the U,V averages are higher than VQMT - maybe some of the difference is from the VQMT UV scaling, not sure. Often there are slightly different calcs too between programs
PSNR y:65.979368 u:64.950274 v:66.120163 average:65.812176 min:59.375502 max:88.556475
I did the comparison in ffmpeg and got this: PSNR y:65.974546 u:57.684810 v:51.288692 average:57.718014 min:53.666230 max:88.556475

I also found that you want to make sure the two sources are exactly the same length or ffmpeg will give you erroneous numbers where as VQMT just uses the length of the shorter source for the comparison.

[poisondeathray]

I did the comparison in ffmpeg and got this: PSNR y:65.974546 u:57.684810 v:51.288692 average:57.718014 min:53.666230 max:88.556475

I also found that you want to make sure the two sources are exactly the same length or ffmpeg will give you erroneous numbers where as VQMT just uses the length of the shorter source for the comparison.


Yes, for all types of metrics, you also need frames to align exactly, and frame accuracy.

FFmpeg can tricky because it uses timestamps to guide it. There are some side cases that can still yield incorrect results if you're not careful. eg. If container timebases are off (e.g. MP4 vs. MKV express differently), or jitter in the timestamps, the results can be off. 23.976 is not the same thing as 24000/1001. There are ways to reset timebase and set framerate within ffmpeg to try to get it to align (there is an example in the docs with settb and setpts). But there are still other cases - ffmpeg psnr/ssim/vmaf calcs will drop negative timestamps or open GOP b-frames at the beginning in the calculation. But if you were to decode the same video to rawvideo (ES), and use that as input, it keeps those b-frames. So on one hand ffmpeg keeps the frames, but on the other it drops them. That inconsistency can cause a problem. For example , official vmafossexec (Netflix VMAF) takes rawvideo input . This can cause discrepancies in the reported values, even though they use the same library

If you feed it avs script or vpy script , you can check for alignment (e.g. in avspmod or vsedit) , and be certain the same frames are decoded the same way and have the same timestamps (info , or use AssumeFPS if they differ)

Did you use avs input for ffmpeg ssim ? That 5-6db delta between our avg U,V results is too large for my liking if it was the same clip. And the avg Y values are close, so it doesn't suggest some misalignment issue

Same with VQMT - it can't be exactly the same clip because I had 5002 frames vs. your 5000.

[Stereodude]

Did you use avs input for ffmpeg ssim ? That 5-6db delta between our avg U,V results is too large for my liking if it was the same clip. And the avg Y values are close, so it doesn't suggest some misalignment issue
Yes, I used two avs inputs into both VQMT and ffmpeg. What did you use?

Same with VQMT - it can't be exactly the same clip because I had 5002 frames vs. your 5000.
I had adjusted the avs script to 5002 frames to match the sample I sent you (which was cut on the MPEG-2 GOP), but Resolve apparently didn't notice that I lengthened the intra .h264 input file and only output 5000 frames, so I just kept the comparison to 5000 frames.

However, I just went back and redid the whole thing and made sure I got 5002 frames this time.

PSNR y:65.979373 u:57.686445 v:51.298844 average:57.726204 min:53.681992 max:88.556475

[Stereodude]

I also got curious and decided to try something. Simply changing the iDCT algorithm used by MPEG2Source can change the output quite a bit in PSNR terms. All using the same 5002 frame MPEG-2 source.

The three 32 bit routines compared to the IEEE-1180 Reference:
PSNR y:62.449197 u:62.938405 v:62.566806 average:62.546733 min:58.816064 max:90.653474

The 64-bit floating point compared to the IEEE-1180 Reference:
PSNR y:102.999039 u:104.085312 v:104.101176 average:103.334047 min:89.595436 max:inf

SSEMMX (Skal) compared to the IEEE-1180 Reference:
PSNR y:62.376757 u:62.547402 v:62.376608 average:62.404711 min:58.623051 max:90.653474

Simple MMX (XviD) compared to the IEEE-1180 Reference:
PSNR y:65.242412 u:67.925161 v:66.576349 average:65.801940 min:61.771485 max:90.653474

ffms2 vs MPEG2Source (IEEE-1180 Reference):
PSNR y:65.346402 u:66.217016 v:65.103104 average:65.437119 min:61.781376 max:inf

Does H264 have similarly vague iDCT specs that allows different decoders to do different things that give slightly different results?

[poisondeathray]

Yes, I used two avs inputs into both VQMT and ffmpeg. What did you use?



Yes, avs for everything. The "original" tested against in all cases is the same mpeg2source script you provided in PM



However, I just went back and redid the whole thing and made sure I got 5002 frames this time.

PSNR y:65.979373 u:57.686445 v:51.298844 average:57.726204 min:53.681992 max:88.556475

Something else is going on . That avg U,V delta between our results is too large.



Does H264 have similarly vague iDCT specs that allows different decoders to do different things that give slightly different results?


For h264 - I don't think so. LSmash/FFMS2 do not list those types of options. DGDecodeNV, the old DGIndexAVC do not either.

The only decoder side switch you might sometimes see is disable deblocking

[videoh]

poisondeathray is correct. All valid 264 decoders must produce bit identical output. The IDCT is defined in the specification. Disabling deblocking is nonsensical.

[Stereodude]

Something else is going on . That avg U,V delta between our results is too large.
I'd chalk it up to changes in Resolve or some setting I don't know I need to change in Resolve. I'm using 16.2 and you're using 15. We get the same PSNR results on the intra version of the source.

[Stereodude]

poisondeathray is correct. All valid 264 decoders must produce bit identical output. The IDCT is defined in the specification.
Good to know.

[Stereodude]

I haven't tried it, but nothing in their list of changes would lead me to believe they fixed it.