AVISynth poor YV12 Conversion

[Phil_L]

Hi

I've spotted an issue with AVISynth which causes banding in graduations when converting from RGB to YV12.

To test this I've used a grey gradient from Sony Vegas and exported it as Lagarith RGB, this shows a smooth gradient when opened via AVISynth with no color conversion, see image 'No color space conversion RGB.png'.

When I use ConvertToYV12 banding becomes evident and the gradient appears wrong with lighter shades appearing out of position. See 'AVISynth ConvertToYV12'. The same happens using ConvertToYUY2.

If I get Vegas to export using any of it's H264 templates (or to it's uncompressed YUV codec), so that now Vegas is doing a convert to YV12 I see something much much better, see 'Vegas to YV12.png'.

If I set Lagarith to work in YV12 and export to Lagarith from Vegas, I get exactly the same banding seen via AVISynth. This appears to be due to Vegas still exporting as RGB and Lagarith is doing the YV12 color conversion and on Lagarith's home page it states, "Other colorspace conversion routines were taken from AviSynth." Which explains it having the same issue I suspect.

Given AVISynth is used widely I'm surprised this bug exists without being spotted and corrected (I tried version 2.6 and that had the same issue). Anyone have any ideas how this might be fixed or how I could fix the Lagarith source code so it will be fixed when in YV12 mode?

Regards

Phil

[colours]

It's not a bug.

The inbuilt colourspace conversion functions use limited range BT.601 coefficients for conversion by default, where the luma has a slightly lower dynamic range than 8-bit greyscale. No dithering is used, which causes nonuniform rounding error.

This rounding error is exacerbated with the roundtrip conversion back to RGB for displaying on your monitor, which is most likely also done without dithering. (This depends on what program you're using to preview avs files.)

The solution to this is to use something that works at a higher precision internally and does dither, like Dither_convert_rgb_to_yuv() or Dither_convert_yuv_to_rgb(). Alternatively, use full range instead of limited range, though this may play with wrong colours on outdated setups.

[Phil_L]

Hi

It's not a bug.

The inbuilt colourspace conversion functions use limited range BT.601 coefficients for conversion by default, where the luma has a slightly lower dynamic range than 8-bit greyscale. No dithering is used, which causes nonuniform rounding error.

This rounding error is exacerbated with the roundtrip conversion back to RGB for displaying on your monitor, which is most likely also done without dithering. (This depends on what program you're using to preview avs files.)

The solution to this is to use something that works at a higher precision internally and does dither, like Dither_convert_rgb_to_yuv() or Dither_convert_yuv_to_rgb(). Alternatively, use full range instead of limited range, though this may play with wrong colours on outdated setups.

Many thanks, these tests were all done at full range.

I take it this means Sony Vegas is doing the YV12 conversion using a higher precision?

I'll give Dither_convert a go.

Regards

Phil

[Asmodian]

these tests were all done at full range.

Did you use ConvertToYV12(matrix="PC.709") (http://avisynth.nl/index.php/Convert) or "PC.601"? I assume Lagarith uses the Avisynth default of "Rec601" and it does not have an option for full range.

I take it this means Sony Vegas is doing the YV12 conversion using a higher precision?

I don't know about the precision used by Vegas but it looks like the Avisynth ramp went through a levels compression. As colours stated a levels compression without dithering will necessarily cause those artifacts on a ramp (evenly spaced double wide bands). It looks like your screenshots have been resized as well?

[Phil_L]

Hi

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video, using a test chart the full-range remained but the banding appeared on the gradient test chart. When exporting from Vegas to AVC the full range remains without any of the banding, even compressing full-range to 16-235 from Vegas the banding was nowhere near as bad and didn't have the lighter shades seemingly out of order.

I've changed my workflow now after a bit of experimentation and export to Cineform from Vegas rather than to Lagarith, as Vegas feeds Cineform a YUV color space without any of the banding issues that are created when asking AVISynth to convert from RGB as it is in Lagarith. Trying to get Lagarith to work in YUV or YV12 didn't help because Lagarith still gets RGB from Vegas and then internally converts to YUV, and as it shares the same color space conversions as AVISynth, gave exactly the same problem.

Now it is already as YUV from Vegas in CineForm, the conversion to YV12 happens without the banding.

Regards

Phil

[feisty2]

try this, wouldn't have this banding problem
#rgb24/32 input#
dither_convert_rgb_to_yuv (lsb=true,output="yv12",chromak="spline64")
ditherpost (mode=6)

[Mounir]

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video

Rec709 is 16-235 by default as far as i know

[Asmodian]

Thank you for info. The matrix I was using was Rec709 as I wanted to keep the full range 0-255 video, using a test chart the full-range remained but the banding appeared on the gradient test chart.

Mounir is correct, Rec709 is limited range. Rec = limited range, PC. = full range. RGB doesn't have limited range so if you were viewing the results in RGB (you must be) the beginning and end of a test chart would be in the same place but number of steps would be 235 instead of 255, causing banding.

I am glad you found a solution but I hope you do understand Cineform is not lossless (unless you are using uncompressed?), though I understand it doesn't suffer much from subsequent generations.

Simply using Lagarith in RGB mode and Avisynth's ConvertToYV12(matrix="PC.709") would give you a lossless route while keeping full range. Or you could use feisty2's suggestion (maybe using spline36 instead of 64 ;))

[Phil_L]

Hi

Mounir is correct, Rec709 is limited range. Rec = limited range, PC. = full range. RGB doesn't have limited range so if you were viewing the results in RGB (you must be) the beginning and end of a test chart would be in the same place but number of steps would be 235 instead of 255, causing banding.

I am glad you found a solution but I hope you do understand Cineform is not lossless (unless you are using uncompressed?), though I understand it doesn't suffer much from subsequent generations.

Simply using Lagarith in RGB mode and Avisynth's ConvertToYV12(matrix="PC.709") or feisty2's suggestion (maybe using spline36 instead of 64 ;)) would give you a lossless route while keeping full range.

Yes definitely using full-range RGB AVI.

These are the AVISynth Tests I've tried and opening in Virtual dub for ease of viewing the results. Note I've over-ridden the graphics settings to always show video as full-range, so no level conversion happening on the screen.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()

This gives smooth gradient, no banding problems, and levels are unchanged as seen on a SMTPE test chart. I'm essentially seeing the original RGB output from Vegas.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="Rec709")

Now have YV12, this now has the banding on the gradient but levels are unchanged as seen on the test chart.

AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")

This gives the same banded gradient problem but worse as now the levels have been converted so I no longer have full-range, i.e. the SMTPE chart has lost black < 16 and white is now 255, so from the original 0-255 a range of 16-235 has been extracted and stretched back to 0-255, which is exactly how I understood PC.709 to work.

For CineForm I'm using the "Keying" setting which is around 500Mbps so while lossy it can't be too bad.

I would like to stick with the Lagarith AVI as an intermediate file to pass to x264 but the conversion to YV12 in AVISynth is not good, also I found letting x264 convert from RGB to YV12 also gives rise to a similar banding effect although not as bad.

Sony Vegas seems to have a superior RGB to YUV conversion.

Regards

Phil

[Wilbert]

Like other people say, it might be that Vegas uses higher precision or dithering.


AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="Rec709")

Now have YV12, this now has the banding on the gradient but levels are unchanged as seen on the test chart.
People explained to you why you have banding here (ie levels are converted from RGB [0,255] -> YV12 [16,235] -> RGB [0,255]). So you have two conversions that cause banding.



AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")

This gives the same banded gradient problem but worse as now the levels have been converted so I no longer have full-range, i.e. the SMTPE chart has lost black < 16 and white is now 255, so from the original 0-255 a range of 16-235 has been extracted and stretched back to 0-255, which is exactly how I understood PC.709 to work.
You are forgetting something. When viewing the script above, something is converting it back to RGB. Since you have a level problem i suspect it clamps the full range to [16,236] and stretches that to [0,255] upon conversion. That's not AviSynth's fault. You should use the following script to check for banding:


AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")


or use a viewer/codec that converts the full range to RGB.

[Emulgator]

I dropped Largarith some time ago for the same reason.

Largarith used as intermediate codec exporting from DVDLabPro2 motion menu renderer gave banding and other encoding artifacts after encoding to MPEG-2.
Using Uncompressed and then encoding to MPEG-2 there were no artifacts.

BTW, Sony Vegas Pro composition engine can render in 32bit float if set so in project properties.

[Asmodian]

Were you using full range video as well?

[Phil_L]

Hi


You are forgetting something. When viewing the script above, something is converting it back to RGB. Since you have a level problem i suspect it clamps the full range to [16,236] and stretches that to [0,255] upon conversion. That's not AviSynth's fault. You should use the following script to check for banding:


AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")


Using PC.709 converts full range to 16-235 so that is of no use to keep 0-255, and it also creates worse banding, one lot introduced by the YV12 conversion and some more squashing 0-255 into 16-235.

There is no problem in the conversion to RGB on my PC, the graphics card is set to over-ride media settings and is video full-range. Output from Sony Vegas direct to YV12 as H264 so a YV12 conversion inside Sony Vegas equals full-range and no banding. Let AVISynth do a YV12 conversion using Rec709 and it is still full range as I still have 0-255 preserved, however the grey test gradient has banding.

Ask Lagarith to work with YV12, Sony still sends it RGB so Lagarith does the YV12 conversion, I have the exact same banding problem. On Lagarith's home page they say they use AVISynth color conversion code, so no surprise perhaps that it has the same issue.

I tried setting up the plug-ins to use dithering but couldn't get it work, so needs more time I can give it at the moment.

So the cure, exporting from Vegas to CineForm sees Vegas send YV12 it has converted with no banding, so avoids asking AVISynth to convert to YV12 and quality is maintained.

Not sure what makes Vegas so good at this YV12 conversion but it is pretty much indistinguishable from the original RGB.

I tried letting X264 do a color conversion, it is much better than AVISynth, but still has some mild banding.

Looks like with 4K moving in and ever better quality HD captured at higher bit-rates AVISynth is showing it's age.

Regards

Phil

[Wilbert]

Using PC.709 converts full range to 16-235 so that is of no use to keep 0-255, and it also creates worse banding, one lot introduced by the YV12 conversion and some more squashing 0-255 into 16-235.
No it doesn't, since it leaves the range intact. Did you try my script above?

[Phil_L]

Hi

No it doesn't, since it leaves the range
intact. Did you try my script above?

If I do this ConvertToYV12(matrix="PC.709") I've lost full-range. In a media player or out via AVISynth to VirtualDub all I see is 16-235 stretched to 0-255. No matter what I adjust in terms of forcing full-range there is no <16 or > 235 there any more.

If I add a ConvertToRGB(matrix="PC.709") yes I seem to get full range back and banding is gone.

The question is, what is PC.709? Is it special to AVISynth? How can I get any media player to convert it back with the same matrix? If the matrix is special to AVISynth it isn't much use.

If I render as normal Rec709 from Sony Vegas or via CineForm the fullrange is playable in media players and no banding on gradients. Of course doing this via AVISynth causes banding.

The spec for Rec709 states, "Interface codes 0 and 255 are used for synchronization, and are prohibited from video data. Eight-bit codes between 1 and 15 provide footroom, and can be used to accommodate transient signal content such as filter undershoots. Eight-bit interface codes 236 through 254 provide headroom, and can be used to accommodate transient signal content such as filter overshoots."


So as I understand it Rec709 should be quite happy accommodating the extra data, and it indeed it does, just AVISynth gives a bad YV12 conversion.

It seems to me PC.709 isn't something widely supported and is perhaps special to AVISynth.

Regards

Phil

[Wilbert]

If I do this ConvertToYV12(matrix="PC.709") I've lost full-range. In a media player or out via AVISynth to VirtualDub all I see is 16-235 stretched to 0-255. No matter what I adjust in terms of forcing full-range there is no <16 or > 235 there any more.
No you haven't lost full-range after converting to YV12. You have lost full range when opening it in VirtualDub. Virtualdub converts YV12 to [0,255] RGB upon display. It assumes the input range is [16,235] for YCbCr by default.

If you want to keep the full range in Virtualdub, you need to go to Video Color Depth -> Decompression format: Other -> select the correct image format (Color space = YCbCR (Rec601/709) : Range = Full : Encoding = 4:2:0 in this case). I haven't played with this myself, but that should keeps the levels intact upon display.

The question is, what is PC.709? Is it special to AVISynth?
It might be convenient for you to read the manual instead of shouting something that AviSynth does NOT do. Yes the naming is special to AviSynth. The matrix setting only does something when converting YCbCr to RGB (or vice versa). PC601/PC709 keeps the fullrange, like you experienced above. Rec601/Rec709 squeezes the range.

[Phil_L]

No you haven't lost full-range after converting to YV12.

If I can't see full-range than I have lost it. Sony Vegas will give me a Rec709 media file (RGB to Yv12) without losing full range that I can see and without banding.

You have lost full range when opening it in VirtualDub. Virtualdub converts YV12 to [0,255] RGB upon display. It assumes the input range is [16,235] for YCbCr by default.

So you don't think I haven't tried other media players, that I haven't changed the color space in Virtual Dub, or set fast recompress or no processing so what it shows is exactly what is delivered out of AVISynth? Also Sony Vegas to a YV12 AVI file as CineForm or its own uncompressed YUV codec can be put into an AVISynth, I can ConvertToYV12(matrix=Rec709), get full range and no banding in Virtual Dub. The difference this time is Sony Vegas has done the conversion from RGB to YUV, with AVISynth only doing the YV12 conversion which is pretty simple in comparison.

The problem isn't a setting issue, if it were, all YV12 from other sources would should the same problem, they don't.


If you want to keep the full range in Virtualdub, you need to go to Video Color Depth -> Decompression format: Other -> select the correct image format (Color space = YCbCR (Rec601/709) : Range = Full : Encoding = 4:2:0 in this case). I haven't played with this myself, but that should keeps the levels intact upon display.

It doesn't work or fix the problem.


It might be convenient for you to read the manual instead of shouting something that AviSynth does NOT do.

I have done, which is why I came here. If it is inconvenient for you to test your own advice, then perhaps you shouldn't provide it with such absolute authority? Simply try it yourself with a grey gradient.


Yes the naming is special to AviSynth. The matrix setting only does something when converting YCbCr to RGB (or vice versa). PC601/PC709 keeps the fullrange, like you experienced above. Rec601/Rec709 squeezes the range.

Rec709 DOES NOT squeeze the range.

PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing. AVISynth will preserve the full range in YV12 using Rec709, but creates very poor YV12 output (banding) over color gradients.

I've worked around it now and getting superior results by not letting AVISynth do a YUV or YV12 conversion. So I'll just accept its a bug with AVISynth and move on.:thanks:

Regards

Phil

[poisondeathray]

If I can't see full-range than I have lost it. Sony Vegas will give me a Rec709 media file (RGB to Yv12) without losing full range that I can see and without banding.


But you've shifted the range.

With respect, you're not really doing proper testing

Most YUV should be limited range, or mapped to Y'16-235, CbCr 16-240 . That's what media players, devices, web, etc.. are set to read. If you're encoding for web, blu-ray, DVD, anything basically. They also use Rec matrices to convert back to RGB for display . Your vegas test actually shifts the range , which is incorrect. On a normal display the YUV video should look like the RGB original, notice how your 3rd picture doesn't match up to the RGB original ?

In order to get vegas to handle "computer RGB" sources (it normally uses "studio RGB', which is similar to PC matrix in avisynth, even though avisynth also has a studioRGB function if you search) , you need to apply a computerRGB to studioRGB filter before exporting a h264/MP4 . That's the equivalent test of what avisynth is doing. That's the equivalent of using LEGAL range YUV . And guess what? You get the same degree of banding when viewed with a Rec matrix (e.g, open up in a media player, or vdub normally) one reason is because 8bit legal range (using the REC matrices) , 0,0,0-255,255,255 RGB is "mapped" to Y' 16-235, CbCr 16-240 or "squished" as Wilbert put it . (But there are more complex reasons, in that YUV and RGB models do not overlap - you're only viewing greyscale, but there are many colors not represented in the 8 bit RGB color model)

Long story short, you should read up on how vegas handles Studio RGB and Computer RGB for different sources. Basically it expects and converts some formats a certain way (native camera formats get studio RGB treatment) , and other formats another way

10bit Cineform, and 10bit YUV don't count in an 8bit apples to apples comparision, because they map 0-255 RGB to limited range 64-940, so there are more "slots" or range of expression, less banding even when converted back to 8bit 0-255 RGB for display


Rec709 DOES NOT squeeze the range.


Rec709 , when converting RGB to YUV, "maps" RGB 0,0,0 -255,255,255 to Y' 16-235, CbCr 16-240 . So yes it does "squeeze the range"

I think the problem is you're forgetting to view with a proper matrix. You're getting confused about what colorspace you're in. Remember you're not viewing YUV directly , you're viewing the RGB representation of YUV.


PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing. AVISynth will preserve the full range in YV12 using Rec709, but creates very poor YV12 output (banding) over color gradients.




PC709 does NOT squeeze the range . It "maps" RGB 0,0,0-255,255,255 to YUV 0-255 (and vice versa with the reverse transform). That is full range. That's very similar to Vegas' Studio RGB conversion (there are slight differences)


If you use search, many of the vegas quircks have been discussed before. Read up on the Glenn Chan articles for Vegas - it should be required reading for anyone more than a casual user

Cheers

[Asmodian]

Rec709 DOES NOT squeeze the range.

PC709 does squeeze the range, it works fine if you want a level adjustment which is what the manual states it is for. I should be able to use Rec709 as the matrix without any level adjustments where it will preserve the full-range, as indeed Sony Vegas is more than happy doing.

From the Avisynth wiki (http://avisynth.nl/index.php/ConvertToYV12):
matrix: Default "Rec601", scaled to TV range [16,235].
"PC.601" : Uses Rec.601 coefficients, keep full range [0,255].
"Rec709" : Uses Rec.709 coefficients, scaled to TV range.
"PC.709" : Uses Rec.709 coefficients, keep full range.
"AVERAGE" : Uses averaged coefficients, keep full range (added in v2.60). (So the luma becomes the average of the RGB channels.)

I assume you never tried Wilbert's script which uses the full range matrix to convert back to RGB:
AviSource("C:\TestFullRange.avi", audio=false).AssumeFrameBased()
ConvertToYV12(matrix="PC.709")
ConvertToRGB(matrix="PC.709")

If you play back a H.264 file compressed with x264 and the option "--input-range pc" with LAV filters and madVR the YUV->RGB will be done correctly. Sadly, as already mentioned, most playback methods will not respect the full range flag and always clip YUV to 16-235 and convert that to 0-255 RGB.

[Phil_L]

Hi

I've done some more tests.

Disregarding full-range for the moment, even using ConvertYV12 with Rec709 from RGB with AVISynth causes banding on a grey gradient, this time the gradient is from 16-235 and no level changes are taking affect.

So then I export from Sony Vegas direct to any of it's H264 codecs, where Sony Vegas is dealing with the RGB to YV12 conversion, no banding, it looks indistinguishable from the original. Also to CineForm YUV, no banding.

If I use AVISynth to convert from CineForm YUV to YV12, no banding.

If I take the CineForm file which is YUV with no banding and in AVISynth do a ConvertToRGB(matrix="Rec709"), all okay, no banding, but then if I convert back to YV12 using ConvertToYV12(matrix="Rec709"), banding is back.

Therefore this isn't a levels problem or an issue with using PC.709 on full-range video because I don't understand or haven't read the instructions as I have been accused off :), but a bug or poor implementation of a RGB to YV12 conversion.

So using AVISynth to convert from RGB to YUV or YV12 will create visible banding in areas like sky or other scenes where this a big block of color with a subtle gradient on it.

Regards

Phil

[poisondeathray]

Disregarding full-range for the moment, even using ConvertYV12 with Rec709 from RGB with AVISynth causes banding on a grey gradient, this time the gradient is from 16-235 and no level changes are taking affect.


As expected, this is "normal" .

Think of it this way: It's mathematically impossible to express 256 "shades" (from 0-255) into 219 (from 16-235) slots in 8bit values properly . You would need 256 "slots" to express 256 shades, each with it's own "slot" .

This is why the others recommended dithering, to limit the visible banding effect

In full range , RGB 0,0,0 or black, gets "mapped" to Y=0 . RGB 1,1,1 gets "mapped" to Y=1, etc... so each value has it's own "slot" . In order to view full range correctly, it must be converted back to RGB for display with a full range matrix



So then I export from Sony Vegas direct to any of it's H264 codecs, where Sony Vegas is dealing with the RGB to YV12 conversion, no banding, it looks indistinguishable from the original.


I suspect you probably shifted the levels or didn't do the tests properly.

Sorry, I don't believe you. Post some samples, including the native 8bit YUV export (h264/mp4). I've done these tests before, as have many others. I can prove it to you.

I'll provide an 8bit RGB gradient test image. It goes from 0,0,0 to 255,255,255 . Black to White. 1280x720
https://www.mediafire.com/?7kevwyaz8kd9b98

You can definitely reduce the banding with dithering, which is what everyone does . Now the tests I've done were with an older version of vegas, is it possible the newest one adds dithering ? I don't think so , but your test will prove it





Also to CineForm YUV, no banding


If you've taken a 32bit gradient, and export 10bit YUV, then when displayed on a 8bit RGB display (0-255, black to white) , there will be no banding because white to black is 64-940 in limited range YUV (out of 0-1023 for 10bit full range vaules) . Think of it as all those "shades" from 0-255 can fit within 64-940, but 0-255 cannot "fit" within 16-235 very nicely . You can go from higher to lower precision, but the reverse is more problematic

(again that's only part of the explanation, you can still get banding with colors because of the RGB color model doesn't overlap completely with YUV color model)


If I use AVISynth to convert from CineForm YUV to YV12, no banding.

If I take the CineForm file which is YUV with no banding and in AVISynth do a ConvertToRGB(matrix="Rec709"), all okay, no banding, but then if I convert back to YV12 using ConvertToYV12(matrix="Rec709"), banding is back.


You take 10bit YUV, convert it to 8bit RGB, back to 8bit limited range YUV (16-235) => again that banding is expected



but a bug or poor implementation of a RGB to YV12 conversion.


No, you will get banding in 8bit limited range YUV. Sony Vegas included. Dithering can reduce the subjective appearance of the banding, but at an increased bitrate penalty (it's essentially adding noise to "cover up" the banding)


So using AVISynth to convert from RGB to YUV or YV12 will create visible banding in areas like sky or other scenes where this a big block of color with a subtle gradient on it.


The problem is math and limited range ITU Rec601/709, not limited to avisynth

[vivan]

Disregarding full-range for the moment, even using ConvertYV12 with Rec709 from RGB with AVISynth causes banding on a grey gradient, this time the gradient is from 16-235 and no level changes are taking affect.This is not true.
ConvertYV12 assumes that your RGB is full-range. So when you're performing that conversion:
0 -> 16 and 255 -> 235
16 -> 30 and 235 -> 219 (+-, I didn't bother to calculate using exact coefficients).
If you want to avoid range conversion - you should use PC.709.

[Sparktank]

try this, wouldn't have this banding problem
#rgb24/32 input#
dither_convert_rgb_to_yuv (lsb=true,output="yv12",chromak="spline64")
ditherpost (mode=6)

^ This would seem a most apt solution to using dither to reduce banding.

The dither package can be found in post #3 of this thread:
http://forum.doom9.org/showthread.php?p=1386559#post1386559

------

It's relatively straight-forward (as relative as it is to your own patience and competence).

The past dozen posts have all pointed out several things that show that it isn't a so-called "bug" in Avisynth.

I'm no developer or scientist or even mathematician but it the generic information readily-available on the internet about "banding" and "dithering" and the information provided by the many developers and users here convince me that it's not a bug in the Avisynth development.

Avisynth is open source, so if you were so inclined, you can dissect the program and see how it actually functions on its most simplistic nature.

All these other programs you use are closed-source, propriatary software. Unless you're part of the development team, you can't know entirely what they do but only draw a rational conclusion based on several (controlled) testings.

Given everyone's testings, data and explanations, it's easy for anyone to understand what's happening and what is a most acceptable solution (given the current limitations).

So far, you haven't once mentioned trying out the dither package by cretindesalpes. Unless that one generic statement where you said you have tried other methods to no avail is in regard to the dither package; but, you haven't posted your script for that dither package or produced enough data to verify your claim on its inefficiency.
It can be tricky to install the plugins in the appropriate places, but once it's installed and used at least once, it's relatively simple to use for future operations.

-------
Perhaps, you can remove Sony Vegas' (proprietary) encode engine from the scenario and export it via Debugmode FrameServer (if it's still compatible with recent version of Sony Vegas) to export your project to VirtualDub and use the open-source x264 do your own encode so that you know precisely what's going on during your encode (and are open to more settings/options)?

Debugmode FrameServer 2.14 (January 16, 2012)
http://www.videohelp.com/tools/Debugmode-FrameServer

[Phil_L]

Hi

I know what I'm testing. A gradient from 16-235 is what I tested with for Video levels, there is no problem converting 16-235 to 16-235, that doesn't introduce banding in itself.

Everyone just ignores the fact I can produce an 8 bit YV12 H264 encoded video from Sony Vegas without a trace of banding, both a full range gradient of 0-255 or 16-235 comes out perfectly from Sony Vegas. Why can't AVISynth do that?

As soon as AVISynth is used to do the YV12 or YUV conversion, no matter whether the PC709 or Rec709 is used, you can not produce an RGB to YUV or YV12 in AVISynth without terrible banding. Note that the levels have not been change, just banding introduced. Most people will only notice the poor conversion occasionally in solidish colors like sky and probably think it was introduced elsewhere.

Yes I could spend several hours playing with plugs in to do some form of dithering to cover up the flaws of AVISynth. Obviously someone else spotted the RGB to YV12 conversion issues to come up with the plugin in the first place.

What I don't understand with the AVISynth output is it isn't so much the banding, but the fact lighter shades are added where they shouldn't be, so shades sort of go 16,17,18,19,20,16,17,21,22,23,24,21,22,25,26,27 and so on. It is certainly a bug in the algorithm that is probably repeated in various open source similar programs because I lot of source is taken from AVISynth, (as it is in Lagarith codec, it produces identical banding when doing a YV12 conversion and admits to using AVISynth color conversion code), so perhaps everyone thinks it is normal. :(

I've found a new workflow now that avoids using AVISynth to do a YV12 conversion from RGB and the results are now free from banding and 100% better.

I'm a big fan of AVISynth but it hasn't really seen any development for years. As we move into ever better HD sources and now 4K, AVISynth is showing it's age and hasn't kept pace.

Regards

Phil

[poisondeathray]

Everyone just ignores the fact I can produce an 8 bit YV12 H264 encoded video from Sony Vegas without a trace of banding, both a full range gradient of 0-255 or 16-235 comes out perfectly from Sony Vegas. Why can't AVISynth do that?


Not ignoring, I just don't believe you, or maybe you didn't conduct the tests properly (perhaps you're not viewing it properly). Certainly the fact that you posted images with different levels in the 1st post suggests that is the case

I can't do it with my version of vegas. Maybe the newest version performs "magic" and defies math ? or maybe it just dithers heavily ?

I'm going to ask you to prove it. Take the RGB 0-255 test gradient I uploaded and produce an 8bit YV12 h264/mp4 , normal standard range YUV (ie. Rec709, 16-235)

You can definitely do it with full range. Avisynth can too




What I don't understand with the AVISynth output is it isn't so much the banding, but the fact lighter shades are added where they shouldn't be, so shades sort of go 16,17,18,19,20,16,17,21,22,23,24,21,22,25,26,27 and so on. It is certainly a bug in the algorithm that is probably repeated in various open source similar programs because I lot of source is taken from AVISynth, (as it is in Lagarith codec, it produces identical banding when doing a YV12 conversion and admits to using AVISynth color conversion code), so perhaps everyone thinks it is normal. :(


Did you just ignore the mathematical explanation above ? There aren't enough "slots" to express 0-255 in a range of 16-235 . Where do you put the "lighter shades?" Where do you put the overlaps ?

It occurs with Vegas too, at least older versions of Vegas, Premiere, After Effects, all professional software

[Asmodian]

Just incase this thread is found again here are some quick tests using poisondeathray's 8bit RGB gradient test image.

Original:
http://i1222.photobucket.com/albums/dd496/asmodian3/8bitRGBramp_zpsf822f732.png

ImageSource("8bit RGB ramp.png")
ConvertToYV12(matrix="PC.709")
http://i1222.photobucket.com/albums/dd496/asmodian3/YV12FullRange_zpscdc7f075.png

ImageSource("8bit RGB ramp.png")
ConvertToYV12(matrix="Rec709")
http://i1222.photobucket.com/albums/dd496/asmodian3/YV12LimitedRange_zpsddcfc513.png

try this, wouldn't have this banding problem
#rgb24/32 input#
dither_convert_rgb_to_yuv (lsb=true,output="yv12",chromak="spline64")
ditherpost (mode=6)
ImageSource ("8bit RGB ramp.png")
Dither_convert_rgb_to_yuv (lsb=true,output="yv12",chromak="spline64")
ditherpost (mode=6)
http://i1222.photobucket.com/albums/dd496/asmodian3/YV12LimitedRangeDithered_zps9438f143.png

dither isn't too hard to install
dithertools (http://forum.doom9.org/showthread.php?p=1386559)
masktools-v2.0a48.zip (http://manao4.free.fr/)
Put "avstp.dll", "dither.avsi", "dither.dll", "mt_masttools-26.dll", "mt_xxpand_multi.avsi", and "mvtools2.dll" in your plugins folder.
I am using Avisynth 2.6.0a5 (http://sourceforge.net/projects/avisynth2/files/AviSynth_Alpha_Releases/) , Avisynth 2.60 build:Sep 18 2013

These are played with Zoom Player v9.0.1, LAV 0.61.2, and madVR v0.87.10 so I can set full range YV12 -> RGB for the full range test. madVR's "reduce banding artifacts" and calibration was disabled.