Banding artifact of h.264???

[drpaulng]

I have read about the banding artifact of h.264. When compared head to head, mpeg2 is much better than h.246. Is that a prejudice or fact?

[LoRd_MuldeR]

It's nonsense.

[drpaulng]

http://www.hd.club.tw/thread-22908-5-1.html

Direct comparison here.

http://img25.imageshack.us/img25/971/200906206900d1d43180225.jpg

[drpaulng]

http://img188.imageshack.us/img188/5707/20090620543e94457440b3d.jpg

[J_Darnley]

Really? Where?

[LoRd_MuldeR]

I can't read the page you linked, because I don't understand that language.

But be careful: There is not "the MPEG-2" and there's not "the H.264". There is a great number of different MPEG-2 and H.264 encoders that may produce quite different results!

Also a comparison is only valid if both clips were encoded at the same bitrate !!!

For example: If you compare a high-bitrate MPEG-2 stream that was encoded with a reasonable good encoder to a low-bitrate H.264 that was encoded with a crappy encoder, then the MPEG-2 will look better for obvious reasons. But that doesn't tell you anything about MPEG-2 vs. H.264. At the same bitrate H.264 should always look better (or at least equal) to MPEG-2...

Have a look here:
http://mirror05.x264.nl/Dark/website/compare.html

--[EDIT]--

I see the pictures you posted ;)

But these pictures are useless without the required information:
* What encoders were used to create those samples?
* What were the encoding parameters, especially the target bitrates?
* What pre-processing has been used?

[drpaulng]

The original picture link is problematic (might be some protection scheme)...so I uploaded it to somewhere.
I know banding is due to 8bit 4:2:0 scheme...but why mpeg2 is immune to it is unknown. I personally doubt it myself too.
Maybe LoRd_MuldeR can prove it wrong with head to head comparison of a color-gradient picture like that.

Edit:
Suppose we do not use pre-processing such as dithering on both encoders, why should mpeg2 be immune from banding while h.264 suffers?

[LoRd_MuldeR]

If you use the 8-Bit 4:2:0 color format, then there is no reason why H.264 should show more banding than MPEG-2 in general.

Still you didn't tell us how your samples were encoded and at what bitrate. So they don't mean anything!

What you see probably is the result of a bad (or wrongly configured) H.264 encoder -or- the result of giving the H.264 encoder too few bits.

Try x264 as your H.264 encoder. Give it a reasonable bitrate (or CRF value) and enable VAQ as well as Psy-RDO and Psy-Trellis ;)

[drpaulng]

I don't know. The thread I quoted from talked about a new BD on sale in Taiwan. The production team seemed to be in favour of mpeg2 rather than h.264. It has the highest total flowrate of 80Mbps with 7.1 audio. I personally doubt that. It just does not make sense to add stringent test against the BD players. Maybe it is intended to be so unreasonable... that people can sell their superior high-end product that passes the test.

By the way, it is the blu-ray standard that uses 8bit 4:2:0. It is not up to you. We are restricted to it. You do not have other choices if you intend to publish your work (commercially or not) to the public (consumers, all of us). My understanding to the 8bit is that it is actually limited to 16-253 color gradient (totally 219 instead of 256). Banding occurs because of the analog world uses such limited standard for broadcasting (only 219 color gradient) The new xvColor scheme uses the same 8bit but allocated the bits differently (not just direct expansion to 0-255 gradient of computer graphics). PS3 is supposed to be able to play AVCHD from the latest models of HD camcorder with xvColor. So, 8bit might not be so bad...it is just "mis-used" by the broadcasting world when technology is limited some 10-20 years ago. I don't know if xvColor is directly sent through a live camcorder or can be retrieved from the MTS (h.264) files stored inside of the camcorder. I suppose the xvColor scheme is backward compatible with old 219 color gradient...the extra bits can be extracted by new equipment and expands the color range to 170% more. I think 10bit color scheme would be the next generation product because all blu-ray production is squeezed into 8bit-219 scheme at the moment.

[LoRd_MuldeR]

I don't know. The thread I quoted from talked about a new BD on sale in Taiwan. The production team seemed to be in favour of mpeg2 rather than h.264. It has the highest total flowrate of 80Mbps with 7.1 audio. I personally doubt that. It just does not make sense to add stringent test against the BD players. Maybe it is intended to be so unreasonable... that people can sell their superior high-end product that passes the test.

...which only proves that the "production team" either uses a crappy H.264 encoder or they are too incompetent to choose the right encoder settings :rolleyes:

Do your own test: Encode form the original uncompressed source and use a good H.264 encoder, such as x264!

(It's a known fact that many BluRay discs on the market are encoded horribly. But you can't conclude anything about H.264 vs. MPEG-2 from that)

By the way, it is the blu-ray standard that uses 8bit 4:2:0, It is not up to you. We are restricted to it. You do not have other choices if you intend to publish your work (commercially or not) to the public (consumers, all of us).

So you can't blame H.264 for that limitation...

[CruNcher]

it's very interesting never the less if that is the same bitrate and the mpeg-2 encoder was needing less cpu time then the h.264 encoder :) i mean Psy-RD as the (RD) implies in the name is very power hungry compared to the Mpeg-2 Encoder result here on the other side we are in the time that most of the H.264 encoders by now are very good optimized, so as LoRd_MuldeR already said without knowing both encoders and the setup of this it's hard to say anything.
Visually of course you can say immediately that the H.264 result here is very bad and that is also the kind of banding you saw with x264 before Psy-RD came but still Vaq used.
Dark-Shikari made once a very very good test scene for this case with Vendeta degraining it to the bone and compressing it to 700 kbps (720p) it would be nice if someone could redo this with the current state of x264 with Psy and without :)

Another problem with this kinds of tests though is also your Viewing device especially on 8 bit LCD some manufactures use other dithering techniques then others so what i see is not that what you might see in terms of how the banding looks (more advanced lcs also have own socs with post processing features ala ffdshow built in) :P

[nm]

it's very interesting never the less if that is the same bitrate and the mpeg-2 encoder was needing less cpu time then the h.264 encoder :) i mean Psy-RD as the (RD) implies in the name is very power hungry compared to the Mpeg-2 Encoder result
Encoding time is not that important in BD authoring. Secondly, x264's psy-rd doesn't slow encoding down significantly.

[Lyris]

It has the highest total flowrate of 80Mbps with 7.1 audio. I personally doubt that.
80mbps is WAY above the highest bit rate allowed on BD (54mbps).

As LoRd_MuldeR says: anyone who favors MPEG2 quality over AVC hasn't seen what AVC can do. MPEG2 is often used in BD production when speed is essential: to encode trailers or other material that's arrived at the last minute (or if production capacity is under stress).

[LoRd_MuldeR]

it's very interesting never the less if that is the same bitrate and the mpeg-2 encoder was needing less cpu time then the h.264 encoder :)

If both encodes used the same bitrate, but the H.264 encode looks clearly worse than the MPEG-2 encode (as shown in the samples above), this only proves how crappy their H.264 encoder is :rolleyes:

Also comparing encodes of the same bitrate, but using an ultra-high bitrate where both encodes are expected to look transparent (he talks about 80 MBit/s !!!), is absolutely useless!

If they had used a good H.264 encoder and if they had used the lowest bitrate, where the H.264 encoder still looks transparent, the MPEG-2 encode would have looked clearly worse at that bitrate :p

Secondly, x264's psy-rd doesn't slow encoding down significantly.

We have to admit that one of the "slower" subme modes (6 at least!) must be used to make Psy-RDO take effect. Subme 9 is needed to make Psy-RDO effect all frame types.

[Soulhunter]

Also comparing encodes of the same bitrate, but using an ultra-high bitrate where both encodes are expected to look transparent (he talks about 80 MBit/s !!!), is absolutely useless
Except for proving that a encoder isn't able to reach transparent results! Actually a lot early h.264 implementations had (some still have...) this problem, especially in regard to banding and loss of fine details. Probably they just used some crappy encoder (thinking about it, most encoders except x264 would fit this criteria) and/or suboptimal settings... Btw, the samples on Dark Shikaris page show nothing (in regard to banding) as the bitrate is way too low (goal -> compression, not transparent results) to see banding (not caused by insufficient bitrate but a flaw in the encoder), you only see blocking and ringing etc. ;D

[LoRd_MuldeR]

I only pointed to these samples to illustrate that H.264 actually does look a lot better than MPEG-2 when compared at a reasonable bitrate - given that good encoders are used ;)

And I agree: If their H.264 encoder fails to produce transparent quality for "real life" 1080p content at 80 Mbit/s, either that encoder has a serious problem or it was misconfigured badly.

In their special situation - bitrate isn't really a limitation and their H.264 encoder sucks, in contrast to their MPEG-2 encoder - the obvious conclusion is to go with with MPEG-2.

But to make it clear once again: We can't conclude anything about MPEG-2 -vs- H.264 from that...

[Soulhunter]

True, we cant conclude anything from this! Tho Id like to see a proper transparency comparison with some worst case scenarios -> Smooth gradient, animated gradient, gradient with other stuff moving in the foreground, dunno... Someone else wants to do it, since I retired from my job here as serial tester long ago? >.>
For the "reasonable bitrate" thing... For one person its reasonable to spend 50k$ on a car, someone else isn't even willing (tho able) to spend 10k$ and gets a used one instead! ;)

[LoRd_MuldeR]

With "reasonable" I meant "reasonable for a comparison" ;)

For a useful comparison we need to lower the bitrate to the point, where at least one of the encodes that we are comparing starts to look non-transparent.

Of course we must not lower the bitrate too much. If both encodes just look "horrible", we can't conclude anything either...

[Soulhunter]

For a useful comparison we need to lower the bitrate to the point, where at least one of the encodes that we are comparing starts to look non-transparent. Yep, supposed all look transparent at min. quantizer to begin with. If, we should step down until one is distinguishable from the source... Now, we need someone willing to actually do it!

[LoRd_MuldeR]

Fist of all we'd need to agree on a source, should be available as uncompressed YUV or lossless HuffYUV.

Then: What MPEG-2 encoder should be used?

Is there are an OpenSource or at least a Freeware MPEG-2 encoder that can compete with the best commercial solutions?

[Soulhunter]

Fist of all we'd need to agree on a source, should be available as uncompressed YUV or lossless HuffYUV.
Suggestion:

1. Static greyscale vertical gradient (some frames + fadeout)
2. 1 + Rotating 360 degree in 200 frames
3. Static greyscale diagonal gradient (some frames + fadeout)
4. 3 + Rotating 360 degree in 200 frames
5. Scrolling greyscale diagonal gradient
6. 1-5 with red->blue gradient (subsampling and differences in chroma treatment comes into play)


Then: What MPEG-2 encoder should be used?

Is there are an OpenSource or at least a Freeware MPEG-2 encoder that can compete with the best commercial solutions?
Perhaps multiple ones, free and non free? Same for h.264? Also adding 1 or 2 MPEG4 ASP codecs for fun?

[poisondeathray]

Suggestion:

1. Static greyscale vertical gradient (some frames + fadeout)
2. 1 + Rotating 360 degree in 200 frames
3. Static greyscale diagonal gradient (some frames + fadeout)
4. 3 + Rotating 360 degree in 200 frames
5. Scrolling greyscale diagonal gradient
6. 1-5 with red->blue gradient (subsampling and differences in chroma treatment comes into play)



Here are a couple test gradient videos, RGB lagarith, 29.97, 640x480 1:1 square pixel, 10sec

1 sec fade in
1 sec hold
6 sec rotate 360 degrees
1 sec hold
1 sec fade out

Greyscale Rotating lagarith
http://www.megaupload.com/?d=CYVCBXTP

Red-Blue Rotating lagarith
http://www.megaupload.com/?d=7G5NFXII

I'm not sure if any of the other suggested test combinations will add any new information, but if you want different specs, or some other gradient let me know

[drpaulng]

OK, here are my test result that I authored with multiAVCHD:
3000kbps target bitrate
http://img5.imageshack.us/img5/3824/3000kbpsditheringwithgr.jpg

http://img29.imageshack.us/img29/4777/3000kbpsmoreditheringwi.jpg

http://img29.imageshack.us/img29/351/3000kbpsoriginalwithout.jpg

12500kbps target bitrate
http://img8.imageshack.us/img8/9029/125000kbpsditheringwith.jpg

http://img29.imageshack.us/img29/3361/125000kbpsmoredithering.jpg

http://img199.imageshack.us/img199/4126/125000kbpsoriginalwitho.jpg

mpeg2 25000kbps taget bitrate
http://img37.imageshack.us/img37/6323/mpeg2ditheringwithgrain.jpg

http://img218.imageshack.us/img218/4360/mpeg2moreditheringwithg.jpg

http://img13.imageshack.us/img13/6679/mpeg2originalwithoutdit.jpg

The original gradient background with a movie poster on top
http://img38.imageshack.us/img38/1192/passengersbg.jpg

[drpaulng]

x264 vs mpeg2 with no dithering
http://img37.imageshack.us/img37/3749/avcvsmpg2originalwithou.jpg

x264 vs mpeg2 with dithering
http://img23.imageshack.us/img23/8277/avcvsmpg2ditheringwithg.jpg

x264 vs mpeg2 with more dithering
http://img26.imageshack.us/img26/6564/avcvsmpg2moreditheringw.jpg

[drpaulng]

The test is simple: Gradient background + normal picture on top
The normal picture is immune from banding as expected while the gradient background suffers from banding.
x264 low bitrate and high bitrate are more or less the same.
x264 vs mpeg2 has bigger impact on color tune than gradient banding
both mpeg2 and x264 suffers from color banding on gradient background

CONCLUSION:
There must be some kind of preprocessing done to the previous HDClub samples. It is not the fault of AVC codec.
Color banding is due to the 8bit-219 scheme (broadcast, blu-ray standard), where as the original still picture I provided is 8bit-256 scheme (computer standard).
In order to compensate the downsampling problem, dithering is a must for gradient scene especially the darker scenes and anime with broad bands of homogeneous colors.
SONY announced their preprocessing successfully done on the new re-issue on EVANGELION: YOU ARE NOT ALONE.

Link:
http://www.lyris-lite.net/2009/06/19/sony-uses-noise-shaping-to-avoid-banding-on-anime-bd-title.html

[poisondeathray]

drpaulng - thanks for sharing your results, but there might be a few issues:

1) you used jpg compression = artifacts; consider using png instead

2) luminance issues with the "split screen" comparisons, especially evident in the black levels (the levels aren't consistent, it looks as if you used a different renderer eg. vrm9 renderless vs. overlay). How are you taking these screencaps?

3) is this a static menu? what is the actual bitrate and length of sequence? (e.g. you may have dialed in 25000kbps, but I doubt a static menu sequence would actually consume that much)

[drpaulng]

You may do the same experiment with my original picture.
I just used what I have on hand.
It shows that HDClub has some prejudice against AVC, maybe due to different workflow with their mpeg2 encoding.
Please download the original and put some noise on it. Use your own settings and bitrates. Do some experiments...go.

EDIT: clarifying
http://www.eclecticelectronics.net/news/sony-makes-your-blu-ray-flicks-look-even-better-with-super-bit-mapping-for-video-sbmv/
Most people think that they need SONY BD Player to play that Super Bit Mapping for Video. In fact it is not the hardware problem. No special decoding scheme is needed for the dithering preprocessing. It is the final result with "invisible" color banding that SONY has done to the video by using dithering. Just like the mpeg2 sample from HDClub...I never think AVC encoding is to blame.

[drpaulng]

drpaulng - thanks for sharing your results, but there might be a few issues:

1) you used jpg compression = artifacts; consider using png instead

2) luminance issues with the "split screen" comparisons, especially evident in the black levels (the levels aren't consistent, it looks as if you used a different renderer eg. vrm9 renderless vs. overlay). How are you taking these screencaps?

3) is this a static menu? what is the actual bitrate and length of sequence? (e.g. you may have dialed in 25000kbps, but I doubt a static menu sequence would actually consume that much)

Original PNG with no great compression like the previous 8bit-256 JPG. It is 24bit
http://img291.imageshack.us/img291/496/passengersbg.png

[Audionut]

Subme 9 is needed to make Psy-RDO effect all frame types.

Subme 7.

Subme 9 adds "refinement" for all frames.

[LoRd_MuldeR]

Subme 7.

Subme 9 adds "refinement" for all frames.

You are right :)

[Soulhunter]

Here are a couple test gradient videos, RGB lagarith, 29.97, 640x480 1:1 square pixel, 10sec

1 sec fade in
1 sec hold
6 sec rotate 360 degrees
1 sec hold
1 sec fade out

Greyscale Rotating lagarith
http://www.megaupload.com/?d=CYVCBXTP

Red-Blue Rotating lagarith
http://www.megaupload.com/?d=7G5NFXII

I'm not sure if any of the other suggested test combinations will add any new information, but if you want different specs, or some other gradient let me know

Thank you! :)

[Soulhunter]

For the comparison of drpaulng:

[EDIT:] See below...

Also, it seems there happens a level conversation somewhere in the chain (encoder, decoder, renderer) that has to be prevented!

[drpaulng]

PNG 32bit graphics yields the same banding artifact as from 8bit JPG (picture is not provided).
Since the final color grades are limited to 8bit 16-235, there is no way a 32bit graphics would yield better result than the slightly banded 8bit grahpics.
What we concern now is how to minimized banding artifact by introducing noise carefully. SONY is obviously very proud to announce their first BD with such noise-making technology. I don't think x264 or mpeg2 can do any better without the preprocessing of dithering. It is totally unrelated to the effectiveness of compression. For example, an uncompressed LPCM 48kHz/24bit wav is down-converted to 48kHz/16bit. There is no compression at all but the 24bit-to-16bit process needs dithering in order to preserve details for low volume signals.

[Soulhunter]

PNG 32bit? I was talking about the "The original gradient background with a movie poster on top" in JPEG (visible banding) -vs- the "Original PNG with no great compression like the previous 8bit-256 JPG" 24bit PNG (no real visible banding). Or did you use the 24bit PNG as input for the test?

[drpaulng]

Yes, whatever. 32bit, 16bit, 10bit, 8bit...all higher bits of data truncated at the 8bit-220 grading. So, that's the way it is. You may try to use 24bit PNG and down-converted to uncompressed 8bit avi. Data are truncated with bit-depth conversion - It is completely unrelated to the encoder. I did the experiment with 32bit PNG of course, picture not shown just because there is no such need - it is the same banding artifact.

http://en.wikipedia.org/wiki/Dithering

If people are still suspicious, I'll try not to use any video encoder but lossless avi. Just to test against the 24bit-still picture to 8bit-movie conversion.

[Soulhunter]

What I meant is that the input (YV12) should be free of banding to begin with... 24bit RGB with usual conversation to YV12 should already create banding, no?

[drpaulng]

Yes, you are right.
The JPG is already with bandings but is very subtle because it is 8bit 0-256 JPG format (256 grades), but the ITU 709 4:2:0 8bit 16-235 movie format (220 grades) makes every other higher formats truncated with visually offensive bandings on the gradient background.

[poisondeathray]

My understanding is the chroma subsampling is causing the banding, not the bits per channel

i.e. the conversion from RGB 4:4:4 to YV12 4:2:0 causes the banding, you can test this on different gradients that look fine but as soon as you use ConvertToYV12() banding occurs.

The 24-bit png is 8 bits per channel (8 red, 8 green, 8 blue), the 32-bit png just adds an alpha channel, so it will look the same in this case.

The MPEG2 and h.264 are 24-bit as well (8bpc), but 4:2:0 unlike the RGB image, or lagarith RGB videos

Also I think it's much preferable to use prefiltering (e.g. gradfun2db or one of it's derivatives), as opposed to dithering, because less damage to the picture, and you can limit it's use to specific frames or even use masks to specific sections of frames.

That is part A. Part B is the actual comparison between MPEG2 and h.264

If you want I can post the bunch of tests, but it's pretty easy to see for yourselves...

The first one is the YV12 converted image, notice the banding compared to the png original above; this is what the "encoder sees"
ImageSource("passengersbg.png",start=0,end=30,fps=29.97)
ConvertToYV12()
http://i39.tinypic.com/2gwvz3a.png

This is YV12 converted image with the debanding filter, gradfun2dbmod applied, again what the "encoder sees", screens taken through AvsP
ImageSource("passengersbg.png",start=0,end=30,fps=29.97)
ConvertToYV12()
GradFun2DBMod()
http://i39.tinypic.com/4lnm2r.png

I won't bother posting the lagarith samples, but the same thing happens with the RGB test gradient videos above. The ConvertToYV12() causes banding in the gradient, which can be improved with filtering

[Dark Shikari]

My understanding is the chroma subsampling is causing the banding, not the bits per channelYour understanding is wrong

[drpaulng]

CORRECTION:
Both the PNG and JPG are 24bit color Red 8bit, Green 8bit, Blue 8bit
The main difference is level of compression:
PNG 1.46MB (8bit 0-255 grades = 8bit sRGB lossless compression)
JPG 694KB (8bit 16-235 grades = 8bit YCbCr lossy compression)

It is very confusing as 8bit color may mean 8bit color Red 3bit, Green 3bit, Blue 2bit (Blue is 2bit instead of 3bit, because human vision tend to ignore blue color and blue color tend to soften image...a lot of movie use blue tone to represent night scene - actually bright light was shone on the scene during shooting, it is the post-processing that render it with blue tone).

I myself have it confused too, because of the 8bit color description of movie.
I think the computer world has a lot of confusing namings.
Sorry that I was wrong with improper description of the still pictures. 4:2:0 subsampling discards most of the colors that human eyes have low sensitivity at. It deceives us by displaying less color informations to our eyes. It's impact on color banding is not understood by me. Hope somebody can shine a light (...and thanks in advance).

[Dust Signs]

@drpaulng: IMHO the statements in your last post are completely wrong. Subsampling has nothing to do with the number of bits representing a color component. To be more precise, those two things are completely unrelated.
As the name indicates, subsampling is about the sampling rate, so 4:4:4 sampling uses more samples per picture area (macroblock or whatever region you look at) than 4:2:0 subsampling. Some more information about subsampling can be found p.e. here (http://en.wikipedia.org/wiki/Chroma_subsampling).
When talking about bit precision we talk about the number of possible values for a color component. When using 8 bit, there are 256 possible values for each color component. As I said this is totally unrelated to the chroma subsampling used.
The actual value range allowed for each color component is also unrelated to the other two (chroma subsampling and bit depth): the reserved values 0-15 and 236-255 are reserved when obeying ITU-R Rec. 601 (http://en.wikipedia.org/wiki/ITU-R_601). I hope I was able to explain the correct connection between the things you mixed up in your post.

Dust Signs

[poisondeathray]

Your understanding is wrong

Can you explain farther please, so we could learn?

In this case, I made the observation that the subsampling is contributing to the banding, just from the avisynth conversion. Is this not the case? or is there another contributing explanation?

I didn't say that higher bpc wouldn't help fix banding (it does, e.g. you can set a 16 or 32bpc project in after effects, and the higher bpc certainly helps with banding gradients, or footage shot with 10bpc AVC-Intra 100 certainly has less banding than that shot with 8bpc AVCHD)

[Manao]

Let's consider for a moment the following serie of numbers, and let's assume those are pixel values :0 1 2 3 4 5 6 7 8 9 10Reducing the bitdepth by 2 (equivalent to rounding to the nearest multiple of 4) would change the values into :0 0 4 4 4 4 8 8 8 8That is banding. Now, if you were to resize it instead, by a factor 2, you'd get :0 2 4 6 8 10That is not banding (and actually, when you upscale it, you get back the original values).

Following the same reasoning, 4:2:0 won't exhibit more banding than 4:4:4, even if it's 12 bpp instead of 24. There is a loss, but the loss occurs at the edges, and definitely not on flat areas (which are the areas where banding shows). However, reducing the actual bitdepth on which each values is written does create banding.

[poisondeathray]

Thanks for the explanation, manao

How do you explain the observations of banding from the RGB .png => YV12 conversion in avisynth ?

Has the bitdepth changed? The terminology I learned was bits per channel; are you using these terms interchangably?

http://en.wikipedia.org/wiki/Color_depth

"24-bit truecolor uses 8 bits to represent red, 8 bits to represent blue and 8 bits to represent green. 28 = 256 levels of each of these three colors can therefore be combined to give a total of 16,777,216 mixed colors (256 × 256 × 256). Twenty-four-bit color is referred to as "millions of colors" "

My understanding is the original PNG image was 8bpc or "24-bit" true color, and that the avisynth conversion to YV12 retails the same color depth (or bpc). Is this not the case? Maybe I've done something wrong in avisynth?

eg. the first is an image from the lagarith gradient video I posted above during the 6 second static section (screens taken in avsp)
ImageSource("greyscale.png")
http://i42.tinypic.com/124y7at.png

The 2nd adds converttoyv12()
ImageSource("greyscale.png")
ConvertToYV12()
http://i41.tinypic.com/ixto9w.png

I agree that the RGB=>YV12 conversion should see line shifting at the edges (e.g. when you do several colorspace conversions on a test chart or colorbars), and your explanation above accounts for that observation, but not the gradient banding

[Manao]

There are several reasons, all more or less occurring at the same time :
- you convert to yv12, so Y = round(aR + bG + cB). So you might end up with 0 1 1 3 4 5 7 7 instead of 0 1 2 3 4 5 6 7 for Y
- you might convert to yv12 [16-235] range : there's a slight reduction in dynamic that amounts to a small loss of bitdepth. Where you would have had 0 1 2 3 4 5 6 7 8, you'll get 0 1 1 3 4 5 6 6 8. There's a small loss, which is somewhat visible.
- in order to show it on your screen, you have to convert back to RGB. The first problem might occur again
- you screen has (most probably) a bitdepth of only 6. In such a case, it does some dithering in order to hide the loss of bitdepth. Usually, that works very well. But the previous rounding errors created banding that will mess up with the dithering algorithm.

[drpaulng]

Thanks Manao,
So, anything that alters the color values may cause differences. Only that the differences are "invisible" most of the time. However, if near-colors are in a row (color gradients), some algorithm tend to alter these values and group them together into homogeneous color values at a row and causes bandings. My understanding is that "bit depth down-conversion" has the most significant impact. I do not totally eliminated the possibility of "subsampling" role. I was perplexed by the saying that mpeg2 is better because it is banding-free...and thus this thread. Evidence from HDClub tells us the story but I tend to disbelieve it and prove with my tools on hand.

poisondeathray has his experience which shows somehow "subsampling" scheme might involves creation of banding. I learned from my research on the web about multiple factors contributing to seeing bands including displaying hardware monitor/video cards etc... not just the encoded picture itself. To me, I am only interested in how to fight against and avoid from it.

Now if bands are already encoded into a movie, does the 10bit upsampling of some new HDTV on the market helps smoothing them out?

[Manao]

No, if you already have bands, the only thing that can save you is adding noise in an intelligent manner, much like gradfun (avisynth filter) does. It will act like a ditherer, so it won't be as good as a real ditherer since the banding is already created.

As for mpeg2, it has no advantage over h264 concerning banding. The most likely causes for such a statement would be either a difference in bitrate (for example, somebody comparing 80 mbps mpeg2 against 40 mbps h264, saying it is fair because h264 is supposed to be twice more efficient than mpeg2) or a difference in encoder quality (and especially, the use of psychovisual schemes in those encoders).

[10L23r]

related questions: what does avisynth use to downsample chroma in convertoyv12()? and what do decoders use to upsample it for playback?

edit: No, if you already have bands, the only thing that can save you is adding noise in an intelligent manner, much like gradfun (avisynth filter) does. It will act like a ditherer, so it won't be as good as a real ditherer since the banding is already created.


when upsampling form 8 to 10, how about adding 10bit noise to 8bit pictures?

[Manao]

do decoders use to upsample it for playbackIt's usually not the decoder that upsamples, but the graphic card. The only way to know for sure what will be used is to use ffdshow and configure it to output RGB.

As for avisynth, the downsampler isn't good (yuy2 -> yv12 averages chroma lines together)

[10L23r]

It's usually not the decoder that upsamples, but the graphic card. The only way to know for sure what will be used is to use ffdshow and configure it to output RGB.

As for avisynth, the downsampler isn't good (yuy2 -> yv12 averages chroma lines together)

how do you enable yv12>rgb in ffdshow? i don't see an enable/disable box anywhere in the rgb conversion part of the video decoder configuration.

there is an option for "high quality yv12 to rgb conversion", but what does that do?

in avisynth, what about rgb to yv12?

[Manao]

You have to force ffdshow to output RGB (disable any other kind of output), and to enable "high quality yv12 to rgb conversion". In avisynth, chroma downsampling isn't good, whatever the conversion.

[10L23r]

k, i figured out the ffdshow thing, thx

so when converting rgb to 4:2:0, does it help to manually downsample the chroma? (i.e. converttoyv24, resize chroma, mergechroma...)

There are several reasons, all more or less occurring at the same time :
- you convert to yv12, so Y = round(aR + bG + cB). So you might end up with 0 1 1 3 4 5 7 7 instead of 0 1 2 3 4 5 6 7 for Y
- in order to show it on your screen, you have to convert back to RGB. The first problem might occur again


i think those are the two main reasons. if you open the second image in an image editor and compare the colors of adjacent pixels in different bands, the difference is two. and each band is not one specific color.

[drpaulng]

After reading a lot of materials about the 8bit down-conversion. I see that many authors wrote with different (mis-)concepts about the present broadcast standard 8bit 4:2:0 video format. I am trying not to make more confusion by stating the fact and conclude for this thread now.

According to HDTV standard ITU-R Recommendation BT.709, of the 8bit (full 0-255), Black and white values are limited to 16-235 (luma) while color values are limited to 16-240 (chroma), the same as the old SDTV standard ITU Rec. 601. However, the forumula for RGB-to-YCrCb 4:2:0 conversion has different correction constants for Rec. 601 (old TV set), and Rec. 709 (new HDTV set) with different gamma correction characteristics.

REFERENCE: http://en.wikipedia.org/wiki/YCbCr

The main reason of banding is caused by truncated values of bit-depth...just like 24bit music is down-converted to 16bit. When a lot of near-the-same values are grouped together - they tend to be flattened in groups (loss of sound or color details). This happens before further compression such as ac3/dts(audio) or mpeg2/avc(video) is applied.