How to minimize/eliminate banding

[spexman]

I am trying to encode MY Blu-ray disc of Ratatouille into XVID. I notice that regardless of bit rate(I have tried 6000 up to 10000), I get visible banding during certain scenes; I tried DIVX, but it had worse banding that a higher bit rate could not fix either.(the original BR scene shows NO such banding)(and I am using Fast Recompress to eliminate colour space issues)
I am a relative 'newbie' and have relied on these threads to gain knowledge in using such programs as AVISYNTH and VD.(which I thank you for)!
My question is(and my apologies if it is a dumb one, yet, I searched everywhere and can find no answer), can this banding be alleviated somehow, with certain settings, or is it simply the limitation of the XVID/Divx codec?(thus, the answer being I must live with banding?)
Thanks in advance for any help!

[Dark Shikari]

You can try using gradfun2db on playback, but generally avoiding banding is very difficult.

You could try x264 with psy RDO, which is generally able to eliminate banding even at lower bitrates.

[kandrey89]

I'm sorry to cut in but what is banding?
Could you post an image and point out the banding?
Thank You

[spexman]

Thank you Dark Shikari.
I have a 'hardware player' that does not support X264 and thus must stick to XVID or Divx. Are you thus saying that banding is impossible to eliminate in these 'older' codecs?
By banding(for kandrey 89), I mean posterization or false contouring; these usually happen with blue skies, blue oceans, midnight/moonlight skies, etc, where there is a large gradient of colour/intensity. Instead of being rendered smoothly, they come out as distinct patches or contours, like contour plots.
To see what I am referring to, see the Codec shootout where pix of underwater shots from Saving Private Ryan are; there you will see banding in the blue water, to the right of the bullet shot. This is the banding I am referring to.

[setarip_old]

@spexman

Hi!I am trying to encode MY Blu-ray disc of Ratatouille into XVID. ...I get visible banding during certain scenes...the original BR scene shows NO such bandingDoes your initial rip (before converting/compressing toXviD) exhibit the banding when played from your hard drive?

I have a 'hardware player' that does not support X264 and thus must stick to XVID or Divx.Are you referring to a standalone DVD player or part of your PC system - or yet something else?

[Dark Shikari]

Thank you Dark Shikari.
I have a 'hardware player' that does not support X264 and thus must stick to XVID or Divx. Are you thus saying that banding is impossible to eliminate in these 'older' codecs?Its quite possible in an older format; its just that neither Xvid nor DivX have the code to do so and I don't feel like porting the feature.

Maybe you might be able to use CQMs or something to try to replicate the effect.

[spexman]

The initial rip does NOT have the banding.
I am re-encoding to divx/xvid so I can watch on my Linkplayer(divx/xvid capable DVD/network player).

Dark Shikari, I appreciate your efforts; I am NOT familiar with CQMs; can you 'briefly' tell me what this is and how I can use it to TRY and alleviate the banding?
THANKS!

[setarip_old]

In light of the fact that you posted on May 18 about having the identical problem with OTA HD files, perhaps the problem lies with your hardware...

http://forum.doom9.org/showthread.php?p=1139052#post1139052

I am using VD to encode my MPEG-2 HD OTA files into DIVX-HD avi's.
I use 2 pass, extreme quality; bitrates about 6-7000. For the most part I am happy, but, do notice the one biggest artifact is false contouring, especially troublesome on blue sky shots.

[spexman]

Thanks for noticing that post; it got no response in the DIVX forum and I am grateful for some discussion here.
I am not clear as to why you would think it is hardware related. The fact that the original stream has NO false contouring but AFTER encoding does, I would pin the issue with the encoding codec. Especially since XVID seems to lessen the contouring versus DIVX. What 'hardware' do you think would be the culprit; please do note, that for virtually all other scenes, the results are stunning; it is just this contouring that I am noticing as an obvious artifact; alleviating this would be the cherry on top, so to speak! Thanks for your interest.

[setarip_old]

I am not clear as to why you would think it is hardware related.Simple speculation, because you've noted the IDENTICAL problem with two different types of original source material - On The Air HD and a BluRay disc...

[Sharc]

@spexman
If you can afford bitrates of 3500+ you may want to try mpeg-2 encoding for comparison.

[spexman]

Do you mean try MPEG2 in XVID's quantization type menu? If so, I did try it along with other 'custom' ones; while they slightly reduced the banding, they also added more 'blockiness' and noise.
IF you mean to encode using MPEG2 and not XVID, I would love to try but cannot see this option in Virtual Dub; I am willing to try if there is a way.

[Dark Shikari]

@spexman
If you can afford bitrates of 3500+ you may want to try mpeg-2 encoding for comparison.He said a Blu-ray disc; 3500 is nothing for HD encoding with Xvid.

[spexman]

At a bit rate of 5000, using XVID, apart from the one problematic scene where Remy is in the bottle over the river at night, the encode looks stunning, losing very little from the original BR stream.(I just finished viewing it and doing many A/Bs with the original) IF I can find a way to alleviate the banding from the one scene, it would be the 'cherry on top'.
Thanks again.

[Sharc]

I actually suggested to try an encode with an mpeg2 encoder (HC, CCE or similar) instead of using Xvid. Mpeg2 will however need a higher bitrate for a comparable quality.
I have found mpeg2 to be less prone to color banding - my impression from few tests - provided one has sufficient bitrate. If you can encode at 3500 or higher the quality should be reasonable with mpeg2. Mpeg2 shouldn't be a problem for playback on a SAP.

[raeltheimperialaerosolkid]

Spex, maybe you can try with a costant quantizer single pass (to say, Q2 o Q3) letting the codec decide what is the best bitrate scene by scene. The use of custom matrix is not suggested if you need to play in some stand alone player.
Another question, what is the final resolution of your encode? Could be some problem of bad resizing?

[spexman]

Sharc and rael, thanks for the interest!
I am not resizing at all or changing frame rates; just compressing as is. I am not familiar with 'quantizer' options, but, I WILL try it when I get home from work; thanks.
As for mpeg2, I suspect I will need a much higher bit rate, but, I am willing to try. I don't mind going a little higher, but, too high and it becomes pointless.(that is why mpeg2 didn't cross my mind, but yes, it will play in my SAP) I will try to find the programs you suggested and give it a try. Thanks again guys.

[setarip_old]

@spexman

Further to my earlier question regarding your 5/18 posting at:

http://forum.doom9.org/showthread.php?p=1139052#post1139052

1) What was the title, video format and resolution of your OTA download?

2) Regarding my speculation of a hardware-related problem, have you tried playing your XviD conversions of both your OTA download and "Ratatouille" BluRay disc on a different (more powerful) system, to see if the "banding" is still present?

[spexman]

The OTA mpeg2 file was an episode of 'ER', 1080i.
I have played the encoded XVID of Ratatouille on my computer and 2 DVD players, all producing this banding effect only at that one challenging scene; that is why I do not think it is hardware, but encoding codec.

[Sharc]

.....As for mpeg2, I suspect I will need a much higher bit rate, but, I am willing to try. I don't mind going a little higher, but, too high and it becomes pointless.(that is why mpeg2 didn't cross my mind, but yes, it will play in my SAP) I will try to find the programs you suggested and give it a try. Thanks again guys.
I was missing the point that you are going to re-encode from a Blu-Ray source. So yes, you would probably need a high bitrate for mpeg-2 if you want to keep the original resolution. My 3500 bitrate for good quality referred actually to a DVD (720x576).
Anyway here the link to a very good free mpeg-2 encoder:
http://www.bitburners.com/hc-encoder/
You might also want to try the VAQ variant of the Xvid encoder. It may help against the color banding, I didn't try it myself though. You can download it from here:
http://www.koepi.info/
Otherwise I second rael: try a single pass CQ encode with the qantizer set to 2 (XviD).

[spexman]

Sharc,
Don't mean to sound like a broken recored, but thanks for all your advice.
I tried HC; I need a bit rate of 10000 to minimize banding, but it is still visible; 5000 is ugly; therefore, I will pass on mpeg2. BUt, thanks for the link to the program; it will come in handy!
I tried XVID Q2; no improvement; out of curiosity, I tried Q1; this looked VERY good, the best yet, with just a hint of banding left; but, of course, it defeats the purpose of compression.
SO, I am left to try the XVID VAQ variant(first time I heard of it so thanks for the link) or anything else one may advise!
I will keep you posted on my results....

[Sharc]

Perhaps you have noticed that you can define *Zones* in Xvid. If the banding is limited to certain scenes (frames) you may want to define these as Zones and allocate a low quantizer (Q1) to them, leaving the rest alone. I have not tried it though.

[Blue_MiSfit]

Have you tried any CQMs?

~MiSfit

[spexman]

Sorry guys, was away for a while.
I just tried the VAQ variant of XVID; no help.
I would rather not have to sit and define zones; my goal is to 'blindly' convert many of my BR's to XVID; I do not have the time to 'assess' each one and define zones.
As for CQM's, if this means 'custom quantizer matrices', the answer is yes, I tried about 6 of the ones that come with XVID and none helped...if you/anyone has one that they feel is good against false contouring, please let me know...
I am beginning to 'conclude and accept' that this false contouring may indeed be a limitation of the codec(which is really too bad as it otherwise is very impressive!)

[Ranguvar]

I would run grad2fundb or one of its variants on the source, to make sure there's no (even extremely slight) banding. Of course, the best solution is to run it on playback, but that's not an option if you have a slower PC, or will be playing it on a standalone.

But other than VAQ coupled with high VHQ, VHQ on b-frames, chroma motion, and high motion search precision, there's nothing really you can do.

If you can use H.264/AVC, x264 with PsyRDO excels at not banding.

[Sharc]

@spexman:
You may want to try this matrix which helped me once in a similar situation with an mpeg-2 encode. Don't know how will behave with XviD:
08 08 08 09 11 13 14 17
08 08 09 11 13 13 14 17
08 08 11 12 13 14 17 94
09 11 13 13 14 17 17 94
11 11 13 13 14 17 94 94
13 13 14 16 17 20 94 94
13 13 14 17 94 94 94 94
13 14 17 94 94 94 94 94

12 12 13 14 15 16 22 26
12 13 14 15 16 22 26 32
13 14 15 16 22 26 32 41
14 15 16 22 26 32 41 53
15 16 22 26 32 41 53 94
16 22 26 32 41 53 70 94
22 26 32 41 53 70 94 94
26 32 41 53 94 94 94 94

What are your B-VOB parameters (Max. consecutive B-VOBs/Quantizer Ratio/Quantizer offset)?
You may want to try (2/1.65/-0.25), or even deselect B-VOBs.
Trial and error ....

[spexman]

Thanks guys for your continued interest!
I will try ALL of the above suggestions and hope one works well!
One last favour...can you tell me how I can load this matrix of yours in? I am not too 'savvy'...I suspect I need to save it as a .txt file, right? But how exactly should I do that such that it is recognized by XVID? Sorry for the 'newbie' question.
(2/1.5/1 are my current Max. consecutive B-VOBs/Quantizer Ratio/Quantizer offset)

[Sharc]

- You can enter the matrix coefficients manually in the XviD GUI selecting MPEG-Custom / Edit Matrix .....
- or you can load the matrix in the cqm or xcm format - not as text. Example attached.
Good luck.

[*.mp4 guy]

Here is what you can do.

1- Max all settings
2- set Xvid to allow q1 in the quantizer restriction menu
3-use the vaq build, enable vaq
4- disable trellis
5- disable post processing, it can cause banding
6- use hvs best quantization matrix
7- use gradfun2db(1.51) on the imput video before compression
8- 3 max b-frmaes, 1.41 multiplier, 0 offset
9- don't use an lcd to view the files, they introduce false banding quite often, I think this is likely in your case, as Xvid produces some of the most consistently banding free encodes of all the available mpeg family compressors.
10- check color conversion on playback options, check your default render, and try changing it to overlay, experiment to see which render does best with your system config, use ffdshow to decode the video, set it to use the high quality yv12->rgb24 conversion, set it to forced rgb output, check your display adapters color options, disable any "color enhancements" or "digital vibrancy" effects, set brighness, contrast, etc to their neutral values, check your screens color configuration, turn off any image enhancing features it may have (other then dithering) set "saturation" or "dynamic contrast" and other such options to to the value which alters the incoming video the least, make sure your color balance is calibrated correctly, make sure that your graphics card, and or display, and or codecs, and media player aplications are using the corrrect yv12 luminance range, multiple convertions between "tv" yv12 levels, and "pc" levels, will cause incorrect colors, including banding issues.

[spexman]

OK, guys, thanks.
I tried ALL possibilities/advice given above.
The 2 custom matrices along with 'maxing' settings do help; they lessen the contours, BUT, the contours do remain.
NOTHING I have tried alleviates them.
These contours are visible using playback on both my HTPC AND my stand alone ProHD JVC DVD player, so it is NOT settings on my computer that is the culprit on playback.
My display is a 9" CRT projector, so it is DEFINITELY not the culprit.
The original source IS contour free; if I use NO COMPRESSION on Virtual Dub Mod, then the result IS CONTOUR FREE as well!

...my conclusion; XVID simply CANNOT eliminate the banding, regardless of what settings one uses. IF someone has done a clean XVID run during this scene, I would LOVE to see it.

That being said, guys, the other 99.9 % of the movie DOES look stunning in XVID using your suggestions. Bottom line, I am content, and can live without 'the cherry on top'.

SO, THANKS for all your help!

Have a great weekend!