measure luma variance within block (e.g. 8x8) of pixels?

[bennynihon]

I would like to more finely and more selectively add dither/grain to flat "blocks" in my source clip to avoid banding and noticeable macro blocks when encoding with x264 (note: for a great example of this, look at 007 Skyfall at 46:28 as Bond is opening the reflective door in a dark scene. Even a low crf like 18 or less looks like crap when encoding this scene).

Right now I'm using the dither plugin, which works wonders using Dither_add_grain16. However, I'm going with the shotgun approach and running it for all my movie encodes since I can't possibly check each and every result to ensure there is no banding, but this adds unnecessary complexity (slower encodes and larger files) to all scenes even though it's the exception to the rule that a scene exists like the Bond one I described above.

That's why I'm wondering if there's a way to measuring the amount of luma variation that exists within each block of pixels of some size (let's say 8x8) so that I can construct a mask that only adds dither to those blocks which have very little luma variation (and therefore little complexity). It looks as though there was a plugin called Blockbuster which did something similar, though I can't seem to get the .dll I found to work with any of the newer Avisynth builds. Thanks for any suggestions

[colours]

If you want to measure the amount of "variation" for each block of pixels, you could easily make something out of mt_inpand_multi/mt_expand_multi and mt_makediff.

However, for the problem you're facing, a somewhat commonly used solution is to just use a luma mask to add more noise to dark areas and less to bright areas, since that's where banding is most visible.

You could also try raising aq-strength or using aq-mode 3 in x264. If you're not already using 10-bit encoding, try that too if your use case supports that, though this would slow encodes down significantly.

[TurboPascal7]

The easiest somewhat correct "amount of variation for each block" you can get is using STD for each pixel and applying dctfilter to it (adjust the radius of std calculation area for possibly better results).
mt_luts(last, "std", mt_square(1), "y").mt_lut("x 15 *")
dctfilter(1,0,0,0,0,0,0,0).grayscale()
A reasonably close approximation with a lot better performance is an edgemask (which is close to mt_expand/mt_inpand + mt_makediff solution proposed by colors).
mt_edge("min/max", 0, 255, 0, 255).mt_lut("x 5 *")
dctfilter(1,0,0,0,0,0,0,0).grayscale()
But you really want to smooth it at the end because applying something based on blocks usually results in very strange output. The easiest way of smoothing is just avoid blocking it in the first place, i.e. removing the dctfilter call. And you get a simple edgemask, which is used by every sane debanding script ever written.

From my experience what colors suggested works okay. You can combine luma mask with reversed edge mask of course to avoid adding noise to dark lines if you have those, but it's rarely needed.

[johnmeyer]

I'd definitely take a look at the "RT_Stats" functions created by StanlessS.

[bennynihon]

A reasonably close approximation with a lot better performance is an edgemask

Thanks for the suggestions. I actually was already using this edgemask

mt_edge(mode="prewitt",thy1=0,thy2=32).mt_invert()

But this still would needlessly add grain to a movie which already contains a lot of grain (and would not result in any banding when encoding anyways). Hence the idea of more selectively applying grain/dither to only blocks that have very low luma variance (a threshold I can tweak to truly only target those most offending areas of a scene which result in a lot of banding). And colour's suggestion of a luma mask still doesn't eliminate the problem of unnecessarily adding grain to a dark scene that already contains a relatively high variance of luminance (such as a clip that already contains a lot of grain despite being dark).

After I wrote my original post, I realized I could use the same edge mask I was using but resize it down a great deal and then resize back up to create the blocking I was looking for. That, combined with a low threshold for mt_binarize does a pretty good job of approximating what I was after by targeting larger, flat portions with dither.

I found these settings to provide a great compromise between file size and quality. I also do use --aq-mode=3 which is found in kMod or tMod builds of x264, which colour also suggested.


vGrainMask=mt_edge(mode="prewitt",thy1=0,thy2=32).BilinearResize((width/16)*2,(height/16)*2).BilinearResize(width,height).grayscale().mt_binarize(threshold=3, upper=true)
Dither_convert_8_to_16()
vGrain=Dither_add_grain16(var=0.6,uvar=0,lsb_in=true)
Dither_merge16_8(vGrain, vGrainMask, u=2, v=2)
DitherPost(mode=3)

[StainlessS]

@JohnMeyer, the stuff I think you may be referring to was AvgLumaDif by Forensic (I coded it),
I also asked if it could be use for blocking detection but Forensic thought not.
I think you should find it via my sig, I never did add it the RT_stats, (which is a bit oversized as it is).
this lot should bring back memories: http://forum.doom9.org/search.php?searchid=6865543

[StainlessS]

Looks like I only posted a copy to Forensic, thought I had made public, anyway, here tis, but not I think want you want.
http://www.mediafire.com/download/1bzjorwg4lk14qb/AvgLumaDif.zip

[bennynihon]

vGrainMask=mt_edge(mode="prewitt",thy1=0,thy2=32).BilinearResize((width/16)*2,(height/16)*2).BilinearResize(width,height).grayscale().mt_binarize(threshold=3, upper=true)
Dither_convert_8_to_16()
vGrain=Dither_add_grain16(var=0.6,uvar=0,lsb_in=true)
Dither_merge16_8(vGrain, vGrainMask, u=2, v=2)
DitherPost(mode=3)

I can confirm that this works very well as a one-size-fits-all approach to adding grain to clips to avoid banding but only doing so where necessary. When used with a movie like Black Swan which already contains a ton of grain, the mask is essentially entirely black which prevents any more grain being added. For other movies or scenes where there are flat, clean areas with the potential for banding during encoding, the mask lets the added grain through.

Ultimately I wanted to archive my Blu-ray collection and encode them at an acceptable quality without banding. But since I couldn't possible manually determine which movies or scenes needed added grain to avoid banding during encoding, I was after an approach that works well for every case. And blindly adding grain to all movies without consideration to the amount already in the source was unacceptable, since that will result in unnecessarily large encoded files. This appears to do the trick.