Understanding YUV 4:2:0 and denoising

[Atlantis]

I want to understand YUV 4:2:0 and how to denoise.

I watched and read some info but still don't understand the correspondence between 4:2:0 and the planes in YUV.

I understand what 4:2:0 subsampling is.
https://2ley7l42nt9s3jvzio2zneqa-wpengine.netdna-ssl.com/wp-content/uploads/2016/02/Common_chroma_subsampling_ratios-0216.jpg

Y plane is Luma.
What I don't get is what each chroma U and V plane correspond to? Is U the first line of 4 pixels? Is V the second line of the 4 pixels? Does that mean that in 4:2:0 the planes U and V are exactly the same?

After understanding this I want to know the amount of denoising you should apply to each plane of a 4:2:0 video.

For example using FFT3DFilter or DFTTest, you can apply different sigmas to each plane. Some of these by default denoise only Y plane or some all 3 planes.

After countless tests, I have found out that the best results are to apply to all 3 planes but only the half to U and V. For example if you apply a sigma 6 to Y, better to apply 3 to U and V. I suppose it makes sense since chroma planes are already compressed. I just don't know should I go x to Y, x/2 to U and x/4 to V or x/2 to V.

What are your opinions? How the 3 different planes should be touched in denoising a 4:2:0 video?

[Asmodian]

The U and V planes share the same pixel locations. You need three values to describe color for a single pixel, Y is one and U and V are the other two. With subsampled chroma pixels have their own Y value but share their neighbors U and V values.

That chart is horribly confusing because they wanted to get 4:1:1 on it. If you drop 4:1:1 and delete the second block of four pixels (the yellow block in 4:2:0) it makes more sense. You can also subsample by resizing the chroma planes with bicubic or something besides the nearest neighbor used in the chart, but nearest neighbor is the easiest to understand.

Your findings do make sense if you have video with a lot of chroma noise. It is common to not denoise the chroma planes. Usually chroma noise is not very visible and chroma is already low detail (half resolution and soft to begin with). If denoising it looks better it is very reasonable to denoise at a lower strength, but the exact ratio will be source dependent. It is common to have more or less (usually less) chroma noise compared to luma noise so do not assume the results from testing with one source apply to all sources.

[Atlantis]

Thank. Yes you are right. I can't assume I have to apply half Y to U V. Totally dependent on the source after looking at more sources. In some sources if you denoise chroma, the colors become a little washed out.

On a personal curiosity, I still don't understand what U and what V is. Is it correct to assume that they are not the subsampling? Meaning that the 2:0 part exists in both U and V? If that's the case, why not just have one plane for chroma? Should I assume I have to apply the same amount of denoise to both U and V?

[StainlessS]

http://web.archive.org/web/20140315154448/http://img245.imageshack.us/img245/7905/coloryuv.gif

https://forum.doom9.org/showthread.php?p=1859515
https://en.wikipedia.org/wiki/YUV
https://en.wikipedia.org/wiki/Chrominance
https://en.wikipedia.org/wiki/Luma_(video)
https://en.wikipedia.org/wiki/Chroma_subsampling
https://en.wikipedia.org/wiki/YCbCr

[Asmodian]

On a personal curiosity, I still don't understand what U and what V is. Is it correct to assume that they are not the subsampling? Meaning that the 2:0 part exists in both U and V? If that's the case, why not just have one plane for chroma? Should I assume I have to apply the same amount of denoise to both U and V?

You cannot have one plane for chroma, you need two values to mathematically combine with the Y to get distinct R, G and B values. U and V are not the subsampling. YUV 4:4:4 has no subsampling, it is full resolution chroma just like RGB. The subsampling is simply resizing the U and V planes to a smaller resolution. For example for 1920x1080 4:2:2 the Y plane is 1920x1080 but the U and V planes are 960x1080. For 4:2:0 the U and V planes are both 960x540. It is a ancient lossy compression method used to save bandwidth.

Y, U, and V are all mathematical transformations of the R, G, and B values that are the primary colors for human vision. Originally invented in analogue electronics to allow new color signals to be compatible with black and white TVs. Using this transform allowed embedding the color data into the already existing black and white signals in a backwards compatible way. U and V are a bit crazy if you want to understand what they actually represent. I simply think of them as the two numbers that when combined with the Y value and the correct matrix allow you to recover the R, G, and B values you need to display color information. Check the links from StainlessS for more. :)

Humans turn out to be less sensitive to lowering the bandwidth for U and V compared to Y so subsampling is visually better than simply using 2/3 (4:2:2) or 1/2 (4:2:0) of the bandwidth for all the planes. 1920x1080 YUV 4:2:0 looks better than 1358x764 RGB or YUV 4:4:4 but they take the same amount of bandwidth. Today turning up the quantizer for the chroma planes instead would result in better quality, but even UHD bluray still simply resizes both the chroma planes to 1920x1080. :(

I have no idea why they call it 4:2:0 because there is always the same amount of U and V data but the fact that there are three numbers and channels makes you think the 4 is for Y, the 2 is for U and the 0 is for V. This is not the case. The whole 4:x:x format isn't really useful to understand what each is, just think of it as a label.

[Atlantis]

Thank you so much for the explanation and the numbers. Now I understand better. So did a little math

Y 1920 x 1080 = 2,073,600
UV 960 x 540 = 518,400

Y 3840 x 2160 = 8,294,400
UV 1920 x 1080 = 2,073,600

So in actuality UV has 1/4 of the resolution of Y. Now I'm thinking to add 1/4 of the Y sigma to UV planes for starting point of denoising!

Why you said 1/2 (4:2:0) of the bandwidth for all the planes? UV is 1/4 of the Y resolution.

[StainlessS]

I have no idea why they call it 4:2:0 because there is always the same amount of U and V data but the fact that there are three numbers and channels makes you think the 4 is for Y, the 2 is for U and the 0 is for V. This is not the case. The whole 4:x:x format isn't really useful to understand what each is, just think of it as a label.
No idea, Me neither, I spent many an hour trying to figure out what the 4:2:0 referred to, finally decided who cares, it just is [a label].

Why you said 1/2 (4:2:0) of the bandwidth for all the planes? UV is 1/4 of the Y resolution.
Yep, but there are two of them.

EDIT: When I use Y,U,V for some kind of measurement/metric, I generally weight U+V combined as 1/2 Y weight, or 1/4 each. [EDIT: So Y=0.666 : (U+V)=0.333]

Yep, but there are two of them.
YV12 4:2:0 is considered 12 bit, so for 4x4 pixel block, 4 * Y @ 8 bit each = 32 bit, + 8 bit for U and 8 bit for V = 48 bit for 4 pixels, or 48/4 for single pixel average.

[Atlantis]

Actually I know why they call it 4:2:0. It's easy. 4 means 4 Luma pixels (I think it should actually be 8 instead!), 2 means 2 chroma pixels in the first line, 0 means no new pixels in the second line.

[StainlessS]

I'm not thinkin' bout it any more, but Asmodian probably likes it. :)

Note Edit in prev post, ie
I generally weight U+V combined as 1/2 Y weight, or 1/4 each. [EDIT: So Y=0.666 : (U+V)=0.333]
or ChromaWeight=0.333, YWeight = 1.0 - ChromaWeight : TotWeighted = Y * YWeight + ((U + V)/2.0) * ChromaWeight

EDIT: Oops, above added in RED.

[Asmodian]

Actually I know why they call it 4:2:0. It's easy. 4 means 4 Luma pixels (I think it should actually be 8 instead!), 2 means 2 chroma pixels in the first line, 0 means no new pixels in the second line.

It still seems weird. The second ":" is for the next line but the first one is to separate luma from chroma? I could see how it developed but it doesn't make it a good description. Edit: Good job working it out though, it does fit really well. :)

Also, why 8? There are only 4 total luma samples, and then up to 4 chroma samples for the two pixels on the first and second lines.
Edit: I see, for 4:1:1, but it really is 1 chroma per 4 luma samples because they end up sharing 1 U and V sample across 4 luma samples for each line. The collection of 8:8:8, 8:4:4, 8:4:0, and 8:2:2 is just begging to be divided by two.

Why you said 1/2 (4:2:0) of the bandwidth for all the planes? UV is 1/4 of the Y resolution.
Because that is the total bandwidth difference. Y is full bandwidth but each U and V plane is 1/4 of the bandwidth, so we only have to transmit a total of 1.5 the number of samples we had in one plane before subsampling, or half the total number of samples. This is why it is a compression scheme, you can send an apparently higher resolution image with the same amount of bandwidth.

I'm not thinkin' bout it any more, but Asmodian probably likes it. :)


lol I do oddly enjoy thinking about things like this, even though it is rather pointless. :o

[Sharc]

http://web.archive.org/web/20061005110348/http:/members.aol.com/ajaynejr/vidbug2.htm
4:2:0 means that for every four luminance pixels across there are two color pixels retained for odd scan lines and none for even scan lines. (The actual color should be averaged for each pair of scan lines and shared as we described above.) We have every 2x2 block of luminance pixels sharing the same color. In practice, half of the color scan lines in 4:2:0 encoding are derived from and packaged with the even fields for interlaced video.

4:2:2 means that for every four luminance pixels across, there are two color pixels for odd scan lines and also two color pixels for even scan lines, in other words each scan line has its own color.

According to this the numbers relate to sampling frequency ratios rather than pixels:

http://web.archive.org/web/20040817164459/http://www.quantel.com/domisphere/infopool.nsf/HTML/dfb444?OpenDocument
http://web.archive.org/web/20040817165029/http://www.quantel.com/domisphere/infopool.nsf/HTML/dfb422?OpenDocument
http://web.archive.org/web/20051105145052/http://www.quantel.com/domisphere/infopool.nsf/html/BDB6BA8DBF70BE9380256C7E003BC117
http://web.archive.org/web/20051031143750/http://www.quantel.com/domisphere/infopool.nsf/html/EF467C39664A407580256C7E003B8BD4

[Asmodian]

Ah, the idea of odd and even scan lines does make the notation less crazy. :)

Thanks for find the links.


According to this the numbers relate to sampling frequency ratios rather than pixels:

This was all invented in analog so that has to be true, pixels are not an analog concept.

[Atlantis]

Also, why 8?
Using pixels for simplicity. We have 8 original data (pixels) of luma, 2 lines of 4.

So I think the subsampling names would make more sense if it was 8:4:4. 8:2:2 and 8:2:0.

[wonkey_monkey]

So I think the subsampling names would make more sense if it was 8:4:4. 8:2:2 and 8:2:0.

Not sure I understand the logic there. Why not reduce the ratios as far as you can, to 4:2:2, 4:1:1, and 4:2:0?

If you ask me the whole system's overly complicated and it'd be far easier if we just specified the shape of the luma samples covered by one chroma sample, e.g. 2x1, 4x1, 2x2.

Info2 (https://forum.doom9.org/showthread.php?t=176563) draws a little glyph next to the standard subsampling numbers.

[Atlantis]

The system is based on a rectangle of 8 pixels 2 x 4.

https://ianbauters.be/blog/wp-content/uploads/2018/04/subsampling-1024x576.png

[wonkey_monkey]

But then the second number only refers to the first line of pixels anyway. There's nothing particularly logical about any of it.

It makes no more or less sense to use 8:2:2 instead of 4:2:2.

[Sharc]

Using pixels for simplicity. We have 8 original data (pixels) of luma, 2 lines of 4.

So I think the subsampling names would make more sense if it was 8:4:4. 8:2:2 and 8:2:0.
Keep in mind that these 'subsampling names' were defined in analogue TV times and the later mapping to pixels is a bit artificial and confusing, IMHO.

[wonkey_monkey]

Keep in mind that these 'subsampling names' were defined in analogue TV times

Were they definitely though? In analogue TV times there was no sampling.

[Sharc]

Were they definitely though? In analogue TV times there was no sampling.
Hmm... true. I should have said that the numbers were originally defined as bandwidth ratios for Y, Cb, Cr components for the transmission of color TV over bandwidth limitied analog TV broadcast channels, and later 'adopted' for the digital realm where the Cb and Cr signals could be subsampled without introducing additional losses thanks to their reduced analogue bandwidth (Nyquist), as I understand.

[_Al_]

The best approach perhaps is to think of video being in planes separately for Y,U and V and picturing those dimensions for those arrays. Because when filtering is it not almost stored that way anyway? Except packed RGB32 or YUY2 (are there any other practical packed formats used?).
Y plane is always full video dimension and for chroma planes (U and V):
4:4:4 chroma planes have full dimensions for x and full for y (same as for any planar RGB)
4:2:2 chroma planes have half dimension for x and full for y
4:2:0 chroma planes have half dimension for x and half for y
4:1:1 chroma planes have 1/4 dimension for x and full for y

So it is kind of simple if working with 4:2:0, then knowing those chroma planes have always half dimensions for both x and y as oppose to Y plane.This gives visual and practical sense in this digital era.

[Asmodian]

So I think the subsampling names would make more sense if it was 8:4:4. 8:2:2 and 8:2:0.

But that is wrong. e.g. There are 8 chroma samples for each line for 4:4:4, not 4. You are combining U and V.

So it is kind of simple if working with 4:2:0, then knowing those chroma planes have always half dimensions for both x and y as oppose to Y plane.This gives visual and practical sense in this digital era.

If in the context of filtering don't forget chroma placement! We cannot let it seem simple. ;)

[Atlantis]

It makes no more or less sense to use 8:2:2 instead of 4:2:2.
It makes sense to me, because the resolution is correct this way.

8:2:2 would say we have 8 Luma and 4 Chroma, which is correct instead of 4:2:2.

We have double the Luma. With 4:2:2, someone could think 4 Luma and 4 chroma which is not correct.

[Atlantis]

But that is wrong. e.g. There are 8 chroma samples for each line for 4:4:4, not 4. You are combining U and V.
I do not understand this sentence. 8 chroma samples for each line? No, you have only 4 chroma per line in 4:4:4. We should not mix subsampling with U V planes. They are 2 different things.

UV should be considered only 1 plane in our discussion of subsampling. in subsampling there are only 2 in the end, Luma and Chroma. In the theory and the naming of the thing.

[Cary Knoop]

With 4:2:2, someone could think 4 Luma and 4 chroma which is not correct.
If they think so they are obviously confused about the meaning of those numbers.

[tormento]

I will get a ride on the train to better understand video rendering on a PC.

I am currently using MPC-BE with LAV filters and MPC Video Renderer or MadVR.

Looking at statistics, when I play a x264 encoded video (usually from BD), I always get TV range (16-235) instead of full PC range.

Is this correct? How does the range works in a encoding job?

[Sharc]

I will get a ride on the train to better understand video rendering on a PC.

I am currently using MPC-BE with LAV filters and MPC Video Renderer or MadVR.

Looking at statistics, when I play a x264 encoded video (usually from BD), I always get TV range (16-235) instead of full PC range.

Is this correct? How does the range works in a encoding job?
In MPC-HC you can select PC (0-255) or TV (16-235) range.
With x264 encoder you have the option '--range x' with x=auto, tv, or pc

[crystalfunky]

So bottom line: What settings do we need for filters like neo_fft3d or neo_dftest if the denoising effect should be optimal with as little quality loss as possible?

[StainlessS]

So bottom line: What settings do we need for filters like neo_fft3d or neo_dftest if the denoising effect should be optimal with as little quality loss as possible?
Tweak settings until you are happy [repeating "AbraCadabra"].

[Asmodian]

We should not mix subsampling with U V planes. They are 2 different things.

But they aren't! The reason for this notation is to reference how much bandwidth is used, not the resolution of the image. Remember this is all from the analog domain.

This is why I said I ignore what it represents and think of it as a label, today this notation really isn't a useful way to think about chroma. We do not care about odd or even lines anymore either but at this point we are simply stuck with the historical conventions.

Tweak settings until you are happy [repeating "AbraCadabra"].

We all wish there was a better way.

My personal rule of thumb, leave chroma alone unless it seems to need it and then tweak settings until it looks as good as it can. I don't necessarily repeat AbraCadabra, unless it is in really terrible condition.

[tormento]

In MPC-HC you can select PC (0-255) or TV (16-235) range. With x264 encoder you have the option '--range x' with x=auto, tv, or pc
I knew the first, not the second.

How can I tell the range of a source file, i.e. a m2ts? Sometimes I can see the color space but nothing more than that.

[Boulder]

I think it's safe to assume that DVD and BD content is in TV range.

[StainlessS]

How can I tell the range of a source file
As boulder said, but,

Well you could just inspect Histogram(Mode="Levels"),
or maybe programmatically using eg RT_QueryLumaMinMax(). [8 bit only].

From RT_Stats plug,

RT_QueryLumaMinMax(clip c,int "Samples"=40,float "Ignore"=0.4,String "Prefix"="QLMM",bool "DEBUG"=false,int "X"=0,int "Y"=0,int "W"=0,int "H"=0, \
int "Matrix"=(Width>1100||Height>600?3:2),int "Start"=Undefined,int "End"=Undefined)
Prescan function to get Luma Min,Max for clip scanning Samples frames area x,y,w,h.
Returns luma min/max as string Default eg "QLMMMin=25 QLMMMax=244", use eg "Eval(RT_QueryLumaMinMax())" to set variables for use in script.

Samples:=40=frames to sample from clip, limited to Framecount.
Samples = 0 converted to FrameCount and Auto Credits skipping disabled.

Ignore:=0.4=Percentage of extreme pixels to ignore (noise) when getting luma min/max (As Threshold arg for YPlaneMin/Max).

Prefix:="QLMM"=Prefix for return string values.

DEBUG:=false= dont show. True=debug info. Need DebugView: http://technet.microsoft.com/en-gb/sysinternals/bb545027

X=Y=W=H=0=Full frame, as crop, area to examine.

Matrix:, For conversion of RGB to YUV-Y, 0 = Rec601, 1 = Rec709, 2 = PC601, 3 = PC709
Default for RGB is 3(PC709) if Width > 1100 || Height > 600 Else 2(PC601) : YUV not used
For RGB, it probably does not make sense to use anything other than PC levels.

Start: Default Undefined. Start frame of scan area. The Start arg overrides Auto Intro credits skipping.

End: Default Undefined. End frame for scan area. Overrides Auto End credits skipping. 0 (or less) will be converted to Framecount - 1.


If user supplied Start and End frame numbers given then that marks the range of frames from which to select the Samples frames.
If neither Start nor End given, the function tries to avoid sampling artificial black/white in Intro and End Credits sequences,
for Auto Intro and End Credits Skipping to be set to 5% (of FrameCount for Intro Skipping) and 90% (for End Skipping), the number of
frames between them must be greater or equal to 250 frames, and MUST also be greater than Samples, otherwise Auto skipping ignored and
the Start and End frame numbers are set to 0 and Framecount - 1.
If a user supplies eg a Start frame number ONLY, then End Skipping has to comply with the same above conditions, range between End Skip
frame and user supplied Start has to be at least 250 frames and greater than Samples, otherwise End frame set to FrameCount - 1. The
same conditions apply if only a user supplied End frame only.
After either user supplied Start/End, or via Auto Credits skipping, or defaulted to 0 & FrameCount -1, we have a sample scan range.
Setting Debug=True, will send debug info including resulting sample frame range to DebugView window.

Usage: Clip global Auto levels using Levels() filter (auto contrast).
AUTOLEVEL_STRENGTH = 0.5 # 0.0 -> 1.0
Eval(RT_QueryLumaMinMax()) # using default values, sets QLMMMin and QLMMMax.
if(IsRGB()) { # Requires GScript
CSMin = 0
CSMax = 255
} else {
CSMin = 16
CSMax = 235
}
ALMin = Int(CSMin - ((CSMin - QLMMMin) * AUTOLEVEL_STRENGTH) + 0.5) # Round Up
ALMax = Int(CSMax - ((CSMax - QLMMMax) * AUTOLEVEL_STRENGTH)) # Round down
Levels(ALMin,1.0,ALMax,CSMin,CSMax,Coring=False) # DO NOT use Coring


Maybe something like (Untested)

AviSource("...")

SAMPLES = 40 # Frames sampled, equally spread across entire clip
IGNORE = 100.0/256 # Ignore extreme pixels noise %, Same as Threshold for YPlaneMin, YPlaneMax, YPlaneMinMax.
X = 16 # Ignore some border noise around clip
Y = X
W =-X
H =-X
Start = Undefined # Set to start sample frame number, skip intro where artificial titles may be outside of TV range
End = Undefined # End Sample FrameNo

Eval(RT_QueryLumaMinMax(samples=SAMPLES,ignore=IGNORE,x=X,y=Y,w=W,h=H,start=START,end=END)) # sets QLMMMin=minYVal, QLMMMax=maxYVal.

IsTVLevels = (QLMMMin >= (0+16)/2 && QLMMMax <= (235+255)/2)


EDIT: Oops, IGNORE = 100.0/256 NOT, IGNORE = 1.0/256

[Sharc]

How can I tell the range of a source file, i.e. a m2ts? Sometimes I can see the color space but nothing more than that.
In Avisynth:
colorYUV(analyze=true)
Histogram("levels")
http://avisynth.nl/index.php/ColorYUV
http://avisynth.nl/index.php/Histogram

In Vdub2:
Filter Histogram

[wonkey_monkey]

Remember this is all from the analog domain.

But doesn't it only apply to sampling analogue video, in other words digit(al)ising it? It's subsampling, and pure analogue video isn't *sampled* at all, it's just a continuous signal.

[Sharc]

Perhaps this paper demystifies the 3-digit notation and its historic development and rationale.
http://scanline.ca/ycbcr/Chroma_subsampling_notation.pdf

[feisty2]

But doesn't it only apply to sampling analogue video, in other words digit(al)ising it? It's subsampling, and pure analogue video isn't *sampled* at all, it's just a continuous signal.

analog videos are indeed sampled, but are not quantized, so each sample is a random point in R instead of an integer between [0, 255].

film is an instance of a motion picture medium that is not spatially sampled, it is still temporally sampled tho (therefore there's the concept of frames per sec), but film is usually not regarded as a form of "video".

[wonkey_monkey]

analog videos are indeed sampled

On Laserdisc that might be the case, but not videotape?

[Sharc]

film is an instance of a motion picture medium that is not spatially sampled, it is still temporally sampled tho (therefore there's the concept of frames per sec), but film is usually not regarded as a form of "video".
Film seems to fall somehow in between. Random spatial quantization is given by the size and distribution of the crystals and particles of the emulsion, and the temporal quantization or sampling precision (aperture) is limited by the shutter speed and shutter movement leading to 'continuous' motion blur and similar.
At the end, light comes in quants if we believe in quantum theory. Just splitting hairs .....

[Sharc]

On Laserdisc that might be the case, but not videotape?
Didn't the CCD sensors/imagers which were widely used in legacy 'analog' VHS videocameras already introduce spatial and temporal quantization, before the signal was converted to a continuous anologue (modulated) wave which was suitable for being recorded on a VHS tape?