RGB->YV12 creates banding

[Lyris]

Can anyone help me solve this?

I have a source that was captured from DigiBeta at 10-bit (I think - different programs have told me different things). It's stored as a QuickTime movie in uncompressed RGB format. To get it into AVISynth, I run it through Sony Vegas Pro 8 and use DebugMode Frameserver output. This seems to work fine on its own (gamma manipulated to show problem):

http://lyris-lite.net/temp/rgb.png


But as soon as I add a ConvertTOYV12() command, it turns into this:

http://lyris-lite.net/temp/yv12.png

Can anyone explain why and how to fix it? The same happens when I work around Vegas and use another input method - I originally tried QTInput, but it gave me wacky levels.

I assume that the input into AviSynth can only be 8-bit - is that right?

[Blue_MiSfit]

Yes, AviSynth is 8 bit only. Try a debanding filter like gradfun2db

[Lyris]

I thought as much - what I don't understand though, is that if it's 8-bit only, why does the original RGB image look fine?

[Gavino]

You will get some banding on conversion of RGB to YUV because of the compressed range [0,255]->[16,235]. You can avoid this using matrix="PC.601" but you need to remember to use the same matrix when converting back to RGB for display (or for further processing).

[markanini]

Consider that in the orginal tones are represented by 1024 values from pure black to pure white, in 8-bits the number is down to 256.
In audio processing one applies low level noise before quantization to avoid rounding errors. Maybe a simmilar approach could be used here. I dunno...
The 'chromaresample' parameter might be worth looking in to. Or a different import method. Maybe the easiest way to experiement with that would be to try exporting your clip to a non-propertery format before importing to avisynth. This will require a lots of diskspace.

[markanini]

Some good advice above from Gavino.

[Lyris]

Got it. I understand the process, but I wasn't sure at what stage the truncation was occurring (it's obvious now). Thanks guys!