AVISynth poor YV12 Conversion

[Phil_L]

Hi

I've spotted an issue with AVISynth which causes banding in graduations when converting from RGB to YV12.

To test this I've used a grey gradient from Sony Vegas and exported it as Lagarith RGB, this shows a smooth gradient when opened via AVISynth with no color conversion, see image 'No color space conversion RGB.png'.

When I use ConvertToYV12 banding becomes evident and the gradient appears wrong with lighter shades appearing out of position. See 'AVISynth ConvertToYV12'. The same happens using ConvertToYUY2.

If I get Vegas to export using any of it's H264 templates (or to it's uncompressed YUV codec), so that now Vegas is doing a convert to YV12 I see something much much better, see 'Vegas to YV12.png'.

If I set Lagarith to work in YV12 and export to Lagarith from Vegas, I get exactly the same banding seen via AVISynth. This appears to be due to Vegas still exporting as RGB and Lagarith is doing the YV12 color conversion and on Lagarith's home page it states, "Other colorspace conversion routines were taken from AviSynth." Which explains it having the same issue I suspect.

Given AVISynth is used widely I'm surprised this bug exists without being spotted and corrected (I tried version 2.6 and that had the same issue). Anyone have any ideas how this might be fixed or how I could fix the Lagarith source code so it will be fixed when in YV12 mode?

Regards

Phil

[colours]

It's not a bug.

The inbuilt colourspace conversion functions use limited range BT.601 coefficients for conversion by default, where the luma has a slightly lower dynamic range than 8-bit greyscale. No dithering is used, which causes nonuniform rounding error.

This rounding error is exacerbated with the roundtrip conversion back to RGB for displaying on your monitor, which is most likely also done without dithering. (This depends on what program you're using to preview avs files.)

The solution to this is to use something that works at a higher precision internally and does dither, like Dither_convert_rgb_to_yuv() or Dither_convert_yuv_to_rgb(). Alternatively, use full range instead of limited range, though this may play with wrong colours on outdated setups.

[Phil_L]

Hi

Originally Posted by colours

It's not a bug.

The inbuilt colourspace conversion functions use limited range BT.601 coefficients for conversion by default, where the luma has a slightly lower dynamic range than 8-bit greyscale. No dithering is used, which causes nonuniform rounding error.

This rounding error is exacerbated with the roundtrip conversion back to RGB for displaying on your monitor, which is most likely also done without dithering. (This depends on what program you're using to preview avs files.)

The solution to this is to use something that works at a higher precision internally and does dither, like Dither_convert_rgb_to_yuv() or Dither_convert_yuv_to_rgb(). Alternatively, use full range instead of limited range, though this may play with wrong colours on outdated setups.

Many thanks, these tests were all done at full range.

I take it this means Sony Vegas is doing the YV12 conversion using a higher precision?

I'll give Dither_convert a go.

Regards

Phil

[Asmodian]

Originally Posted by Phil_L

these tests were all done at full range.

Did you use ConvertToYV12(matrix="PC.709") or "PC.601"? I assume Lagarith uses the Avisynth default of "Rec601" and it does not have an option for full range.

Originally Posted by Phil_L

I take it this means Sony Vegas is doing the YV12 conversion using a higher precision?

I don't know about the precision used by Vegas but it looks like the Avisynth ramp went through a levels compression. As colours stated a levels compression without dithering will necessarily cause those artifacts on a ramp (evenly spaced double wide bands). It looks like your screenshots have been resized as well?

[Phil_L]

Hi

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video, using a test chart the full-range remained but the banding appeared on the gradient test chart. When exporting from Vegas to AVC the full range remains without any of the banding, even compressing full-range to 16-235 from Vegas the banding was nowhere near as bad and didn't have the lighter shades seemingly out of order.

I've changed my workflow now after a bit of experimentation and export to Cineform from Vegas rather than to Lagarith, as Vegas feeds Cineform a YUV color space without any of the banding issues that are created when asking AVISynth to convert from RGB as it is in Lagarith. Trying to get Lagarith to work in YUV or YV12 didn't help because Lagarith still gets RGB from Vegas and then internally converts to YUV, and as it shares the same color space conversions as AVISynth, gave exactly the same problem.

Now it is already as YUV from Vegas in CineForm, the conversion to YV12 happens without the banding.

Regards

Phil

[feisty2]

try this, wouldn't have this banding problem
#rgb24/32 input#
dither_convert_rgb_to_yuv (lsb=true,output="yv12",chromak="spline64")
ditherpost (mode=6)

[Mounir]

Originally Posted by Phil_L

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video

Rec709 is 16-235 by default as far as i know

[Asmodian]

Originally Posted by Phil_L

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video, using a test chart the full-range remained but the banding appeared on the gradient test chart.

Mounir is correct, Rec709 is limited range. Rec = limited range, PC. = full range. RGB doesn't have limited range so if you were viewing the results in RGB (you must be) the beginning and end of a test chart would be in the same place but number of steps would be 235 instead of 255, causing banding.

I am glad you found a solution but I hope you do understand Cineform is not lossless (unless you are using uncompressed?), though I understand it doesn't suffer much from subsequent generations.

Simply using Lagarith in RGB mode and Avisynth's ConvertToYV12(matrix="PC.709") would give you a lossless route while keeping full range. Or you could use feisty2's suggestion (maybe using spline36 instead of 64 )

[Phil_L]

Hi

Originally Posted by Asmodian

Mounir is correct, Rec709 is limited range. Rec = limited range, PC. = full range. RGB doesn't have limited range so if you were viewing the results in RGB (you must be) the beginning and end of a test chart would be in the same place but number of steps would be 235 instead of 255, causing banding.

I am glad you found a solution but I hope you do understand Cineform is not lossless (unless you are using uncompressed?), though I understand it doesn't suffer much from subsequent generations.

Simply using Lagarith in RGB mode and Avisynth's ConvertToYV12(matrix="PC.709") or feisty2's suggestion (maybe using spline36 instead of 64 ) would give you a lossless route while keeping full range.

Yes definitely using full-range RGB AVI.

These are the AVISynth Tests I've tried and opening in Virtual dub for ease of viewing the results. Note I've over-ridden the graphics settings to always show video as full-range, so no level conversion happening on the screen.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()

This gives smooth gradient, no banding problems, and levels are unchanged as seen on a SMTPE test chart. I'm essentially seeing the original RGB output from Vegas.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="Rec709")

Now have YV12, this now has the banding on the gradient but levels are unchanged as seen on the test chart.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")

This gives the same banded gradient problem but worse as now the levels have been converted so I no longer have full-range, i.e. the SMTPE chart has lost black < 16 and white is now 255, so from the original 0-255 a range of 16-235 has been extracted and stretched back to 0-255, which is exactly how I understood PC.709 to work.

For CineForm I'm using the "Keying" setting which is around 500Mbps so while lossy it can't be too bad.

I would like to stick with the Lagarith AVI as an intermediate file to pass to x264 but the conversion to YV12 in AVISynth is not good, also I found letting x264 convert from RGB to YV12 also gives rise to a similar banding effect although not as bad.

Sony Vegas seems to have a superior RGB to YUV conversion.

Regards

Phil

[Wilbert]

Like other people say, it might be that Vegas uses higher precision or dithering.

Originally Posted by Phil_L

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="Rec709")

Now have YV12, this now has the banding on the gradient but levels are unchanged as seen on the test chart.

People explained to you why you have banding here (ie levels are converted from RGB [0,255] -> YV12 [16,235] -> RGB [0,255]). So you have two conversions that cause banding.

Originally Posted by Phil_L

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")

This gives the same banded gradient problem but worse as now the levels have been converted so I no longer have full-range, i.e. the SMTPE chart has lost black < 16 and white is now 255, so from the original 0-255 a range of 16-235 has been extracted and stretched back to 0-255, which is exactly how I understood PC.709 to work.

You are forgetting something. When viewing the script above, something is converting it back to RGB. Since you have a level problem i suspect it clamps the full range to [16,236] and stretches that to [0,255] upon conversion. That's not AviSynth's fault. You should use the following script to check for banding:

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")

or use a viewer/codec that converts the full range to RGB.

[Emulgator]

I dropped Largarith some time ago for the same reason.

Largarith used as intermediate codec exporting from DVDLabPro2 motion menu renderer gave banding and other encoding artifacts after encoding to MPEG-2.
Using Uncompressed and then encoding to MPEG-2 there were no artifacts.

BTW, Sony Vegas Pro composition engine can render in 32bit float if set so in project properties.

[Asmodian]

Were you using full range video as well?

[Phil_L]

Hi

Originally Posted by Wilbert

You are forgetting something. When viewing the script above, something is converting it back to RGB. Since you have a level problem i suspect it clamps the full range to [16,236] and stretches that to [0,255] upon conversion. That's not AviSynth's fault. You should use the following script to check for banding:

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")

Using PC.709 converts full range to 16-235 so that is of no use to keep 0-255, and it also creates worse banding, one lot introduced by the YV12 conversion and some more squashing 0-255 into 16-235.

There is no problem in the conversion to RGB on my PC, the graphics card is set to over-ride media settings and is video full-range. Output from Sony Vegas direct to YV12 as H264 so a YV12 conversion inside Sony Vegas equals full-range and no banding. Let AVISynth do a YV12 conversion using Rec709 and it is still full range as I still have 0-255 preserved, however the grey test gradient has banding.

Ask Lagarith to work with YV12, Sony still sends it RGB so Lagarith does the YV12 conversion, I have the exact same banding problem. On Lagarith's home page they say they use AVISynth color conversion code, so no surprise perhaps that it has the same issue.

I tried setting up the plug-ins to use dithering but couldn't get it work, so needs more time I can give it at the moment.

So the cure, exporting from Vegas to CineForm sees Vegas send YV12 it has converted with no banding, so avoids asking AVISynth to convert to YV12 and quality is maintained.

Not sure what makes Vegas so good at this YV12 conversion but it is pretty much indistinguishable from the original RGB.

I tried letting X264 do a color conversion, it is much better than AVISynth, but still has some mild banding.

Looks like with 4K moving in and ever better quality HD captured at higher bit-rates AVISynth is showing it's age.

Regards

Phil

[Wilbert]

Using PC.709 converts full range to 16-235 so that is of no use to keep 0-255, and it also creates worse banding, one lot introduced by the YV12 conversion and some more squashing 0-255 into 16-235.

No it doesn't, since it leaves the range intact. Did you try my script above?

[Phil_L]

Hi

Originally Posted by Wilbert

No it doesn't, since it leaves the range
intact. Did you try my script above?

If I do this ConvertToYV12(matrix="PC.709") I've lost full-range. In a media player or out via AVISynth to VirtualDub all I see is 16-235 stretched to 0-255. No matter what I adjust in terms of forcing full-range there is no <16 or > 235 there any more.

If I add a ConvertToRGB(matrix="PC.709") yes I seem to get full range back and banding is gone.

The question is, what is PC.709? Is it special to AVISynth? How can I get any media player to convert it back with the same matrix? If the matrix is special to AVISynth it isn't much use.

If I render as normal Rec709 from Sony Vegas or via CineForm the fullrange is playable in media players and no banding on gradients. Of course doing this via AVISynth causes banding.

The spec for Rec709 states, "Interface codes 0 and 255 are used for synchronization, and are prohibited from video data. Eight-bit codes between 1 and 15 provide footroom, and can be used to accommodate transient signal content such as filter undershoots. Eight-bit interface codes 236 through 254 provide headroom, and can be used to accommodate transient signal content such as filter overshoots."


So as I understand it Rec709 should be quite happy accommodating the extra data, and it indeed it does, just AVISynth gives a bad YV12 conversion.

It seems to me PC.709 isn't something widely supported and is perhaps special to AVISynth.

Regards

Phil

[Wilbert]

If I do this ConvertToYV12(matrix="PC.709") I've lost full-range. In a media player or out via AVISynth to VirtualDub all I see is 16-235 stretched to 0-255. No matter what I adjust in terms of forcing full-range there is no <16 or > 235 there any more.

No you haven't lost full-range after converting to YV12. You have lost full range when opening it in VirtualDub. Virtualdub converts YV12 to [0,255] RGB upon display. It assumes the input range is [16,235] for YCbCr by default.

If you want to keep the full range in Virtualdub, you need to go to Video Color Depth -> Decompression format: Other -> select the correct image format (Color space = YCbCR (Rec601/709) : Range = Full : Encoding = 4:2:0 in this case). I haven't played with this myself, but that should keeps the levels intact upon display.

The question is, what is PC.709? Is it special to AVISynth?

It might be convenient for you to read the manual instead of shouting something that AviSynth does NOT do. Yes the naming is special to AviSynth. The matrix setting only does something when converting YCbCr to RGB (or vice versa). PC601/PC709 keeps the fullrange, like you experienced above. Rec601/Rec709 squeezes the range.

[Phil_L]

Originally Posted by Wilbert

No you haven't lost full-range after converting to YV12.

If I can't see full-range than I have lost it. Sony Vegas will give me a Rec709 media file (RGB to Yv12) without losing full range that I can see and without banding.

You have lost full range when opening it in VirtualDub. Virtualdub converts YV12 to [0,255] RGB upon display. It assumes the input range is [16,235] for YCbCr by default.

So you don't think I haven't tried other media players, that I haven't changed the color space in Virtual Dub, or set fast recompress or no processing so what it shows is exactly what is delivered out of AVISynth? Also Sony Vegas to a YV12 AVI file as CineForm or its own uncompressed YUV codec can be put into an AVISynth, I can ConvertToYV12(matrix=Rec709), get full range and no banding in Virtual Dub. The difference this time is Sony Vegas has done the conversion from RGB to YUV, with AVISynth only doing the YV12 conversion which is pretty simple in comparison.

The problem isn't a setting issue, if it were, all YV12 from other sources would should the same problem, they don't.

If you want to keep the full range in Virtualdub, you need to go to Video Color Depth -> Decompression format: Other -> select the correct image format (Color space = YCbCR (Rec601/709) : Range = Full : Encoding = 4:2:0 in this case). I haven't played with this myself, but that should keeps the levels intact upon display.

It doesn't work or fix the problem.

It might be convenient for you to read the manual instead of shouting something that AviSynth does NOT do.

I have done, which is why I came here. If it is inconvenient for you to test your own advice, then perhaps you shouldn't provide it with such absolute authority? Simply try it yourself with a grey gradient.

Yes the naming is special to AviSynth. The matrix setting only does something when converting YCbCr to RGB (or vice versa). PC601/PC709 keeps the fullrange, like you experienced above. Rec601/Rec709 squeezes the range.

Rec709 DOES NOT squeeze the range.

PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing. AVISynth will preserve the full range in YV12 using Rec709, but creates very poor YV12 output (banding) over color gradients.

I've worked around it now and getting superior results by not letting AVISynth do a YUV or YV12 conversion. So I'll just accept its a bug with AVISynth and move on.

Regards

Phil

[poisondeathray]

Originally Posted by Phil_L

If I can't see full-range than I have lost it. Sony Vegas will give me a Rec709 media file (RGB to Yv12) without losing full range that I can see and without banding.

But you've shifted the range.

With respect, you're not really doing proper testing

Most YUV should be limited range, or mapped to Y'16-235, CbCr 16-240 . That's what media players, devices, web, etc.. are set to read. If you're encoding for web, blu-ray, DVD, anything basically. They also use Rec matrices to convert back to RGB for display . Your vegas test actually shifts the range , which is incorrect. On a normal display the YUV video should look like the RGB original, notice how your 3rd picture doesn't match up to the RGB original ?

In order to get vegas to handle "computer RGB" sources (it normally uses "studio RGB', which is similar to PC matrix in avisynth, even though avisynth also has a studioRGB function if you search) , you need to apply a computerRGB to studioRGB filter before exporting a h264/MP4 . That's the equivalent test of what avisynth is doing. That's the equivalent of using LEGAL range YUV . And guess what? You get the same degree of banding when viewed with a Rec matrix (e.g, open up in a media player, or vdub normally) one reason is because 8bit legal range (using the REC matrices) , 0,0,0-255,255,255 RGB is "mapped" to Y' 16-235, CbCr 16-240 or "squished" as Wilbert put it . (But there are more complex reasons, in that YUV and RGB models do not overlap - you're only viewing greyscale, but there are many colors not represented in the 8 bit RGB color model)

Long story short, you should read up on how vegas handles Studio RGB and Computer RGB for different sources. Basically it expects and converts some formats a certain way (native camera formats get studio RGB treatment) , and other formats another way

10bit Cineform, and 10bit YUV don't count in an 8bit apples to apples comparision, because they map 0-255 RGB to limited range 64-940, so there are more "slots" or range of expression, less banding even when converted back to 8bit 0-255 RGB for display

Rec709 DOES NOT squeeze the range.

Rec709 , when converting RGB to YUV, "maps" RGB 0,0,0 -255,255,255 to Y' 16-235, CbCr 16-240 . So yes it does "squeeze the range"

I think the problem is you're forgetting to view with a proper matrix. You're getting confused about what colorspace you're in. Remember you're not viewing YUV directly , you're viewing the RGB representation of YUV.

PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing. AVISynth will preserve the full range in YV12 using Rec709, but creates very poor YV12 output (banding) over color gradients.

PC709 does NOT squeeze the range . It "maps" RGB 0,0,0-255,255,255 to YUV 0-255 (and vice versa with the reverse transform). That is full range. That's very similar to Vegas' Studio RGB conversion (there are slight differences)


If you use search, many of the vegas quircks have been discussed before. Read up on the Glenn Chan articles for Vegas - it should be required reading for anyone more than a casual user

Cheers

[Asmodian]

Originally Posted by Phil_L

Rec709 DOES NOT squeeze the range.

PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing.

From the Avisynth wiki:

matrix: Default "Rec601", scaled to TV range [16,235].
"PC.601" : Uses Rec.601 coefficients, keep full range [0,255].
"Rec709" : Uses Rec.709 coefficients, scaled to TV range.
"PC.709" : Uses Rec.709 coefficients, keep full range.
"AVERAGE" : Uses averaged coefficients, keep full range (added in v2.60). (So the luma becomes the average of the RGB channels.)

I assume you never tried Wilbert's script which uses the full range matrix to convert back to RGB:

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")

If you play back a H.264 file compressed with x264 and the option "--input-range pc" with LAV filters and madVR the YUV->RGB will be done correctly. Sadly, as already mentioned, most playback methods will not respect the full range flag and always clip YUV to 16-235 and convert that to 0-255 RGB.

[Phil_L]

Hi

I've done some more tests.

Disregarding full-range for the moment, even using ConvertYV12 with Rec709 from RGB with AVISynth causes banding on a grey gradient, this time the gradient is from 16-235 and no level changes are taking affect.

So then I export from Sony Vegas direct to any of it's H264 codecs, where Sony Vegas is dealing with the RGB to YV12 conversion, no banding, it looks indistinguishable from the original. Also to CineForm YUV, no banding.

If I use AVISynth to convert from CineForm YUV to YV12, no banding.

If I take the CineForm file which is YUV with no banding and in AVISynth do a ConvertToRGB(matrix="Rec709"), all okay, no banding, but then if I convert back to YV12 using ConvertToYV12(matrix="Rec709"), banding is back.

Therefore this isn't a levels problem or an issue with using PC.709 on full-range video because I don't understand or haven't read the instructions as I have been accused off , but a bug or poor implementation of a RGB to YV12 conversion.

So using AVISynth to convert from RGB to YUV or YV12 will create visible banding in areas like sky or other scenes where this a big block of color with a subtle gradient on it.

Regards

Phil