SmoothD2 a deblocking and smoothing filter

[jconklin]

SmoothD2 is deblocking filter or at stronger settings, a smoothing filter.
It is a major rewrite of SmoothD. See http://forum.doom9.org/showthread.php?t=84355&highlight=SmoothD

I liked SmoothD both as a deblocker and as a smoother.
My problem was when I tried to use it with low quants for deblocking and later in the script with higher quants to produce a prefiltered clip for mvtools it would crash.
So I figured I would fix the crash bug.

Having restructured and cleaned up some of the code fixing the crash bug, I figured I might as well handle a minor 1/2 bit rounding problem.
I did that and I was hooked. Could I make it faster, yes 1.5 times!
Are the shift positions optimal, could I tell the difference if I changed them, yes, what is the difference and why, bang head, bang head, ponder, ponder then, 8 shifts on 8x8 squares, the 8 queens problem yes it worked great solution!!!
Let's add the other quant-dequant types so we now have the original mpeg_intra, as well as mpeg_inter, h263_intra, and h263_inter processing, nice.
How do matrices work? Better redo that code and try a bunch of them and see. Play, play, play ... Well I had fun doing that, sort of got an intuitive sense of what they do, got some you can try, but I don't know how they work, I had too much fun playing.

And the last problem, and the universe laughs,
can I control the filter strength within a frame, after all it would be nice to decrease the filtering strength as you move from the edge of a macro block toward its center.
Think, think, think, try this try that, duh. It's been right there in front of me.
The SmoothD zero_weight argument! It controls the filter strength by adding copies of the unshifted original image to the processed images.

There are now 5 Zero Weight arguments that provide filter strength control independently
at the whole frame level,
as a function of the position within a macro block,
at a per pixel level via a mask clip,
and as a function of the luma value of a pixel.

The end result of all of this is a completely reworked SmoothD with a new set of arguments and a new name.

SmoothD2(clip clip, int quant, int num_shift, int Matrix, int Qtype, int ZW, int ZWce, clip ZWmask, int ZWlmDark, int ZWlmBright)

This is the first release and I expect to add some more features and speed it up some more.

Try it and let me know what doesn't work, as well as how it could be better.

Main documentation page https://sites.google.com/site/jconklin754smoothd2/home
Full source code download page https://sites.google.com/site/jconklin754smoothd2/home/download

Thanks jconklin

Read a Tale of a Bug Squash. http://forum.doom9.org/showpost.php?p=1590560&postcount=32

Version A3 fixes a bug that would crash the filter producing a error message from Avisysnth on some users systems.
It also adds major speed improvments as well as mult cpu support.

Latest version A3 09-08-2012

[Zep]

I'll test it and let you know how things went :)

[TheProfileth]

Ooh I liked SmoothD, was a very promising filter. I look forward to this, I also hope for a speed increase but I guess I shall see.

[Mug Funky]

Best. First post. Ever.

[TheProfileth]

Best. First post. Ever.
Agreed :D
Edit: Mug Funky don't you mean "Very good" ;)

[jconklin]

Best. First post. Ever.

Thanks! I have been lurking long enough that I figured it was about time I gave something back.

I plan to run a set of speed tests against all the other deblockers I can find. Should have some results in a few days. Will post results when done.

I am getting SmoothD2 to deblock color. It works but requires too many additional arguments to get the color deblocking to be reasonable. I want to just give it a color deblock strength argument but haven't yet figured out how to make that work. Early days yet.

Bye the way, if "best" referred to my writing style, all of my teachers are going to fall down. :)

[ryrynz]

Hi! I'm after a filter that smoothes out color variations to achieve flat (clean) colors. ATM I'm using Minblur but I'm looking for a more specialized color cleaner for anime content. Once you perhaps have the color arguments sorted is there a possibility that your filter could achieve this? Or are you aware of something that perhaps already does it? Thanks.

[Yellow_]

Would your filter benefit from a 16bit mod for Dither Tools?

http://forum.doom9.org/showpost.php?p=1386559&postcount=3

[TheProfileth]

A nice 16bit deblocker would be very useful for heavy debanding

[jconklin]

SmoothD2 is an 8 bit luma deblocker/smoother. I am getting it to do color by wrapping 3 SmoothD2 calls in a .avs wrapper. Making it a 16 bit luma deblocker/smoother would require rewriting the xvid routines fdct, dequant, quant, and idct routines which are mmx optimized. It has been decades since I last wrote assembler so it is unlikly that I am going to take on that task. Its late and my eys are starting to close so this is probably nuts. If you can take a 16 bit luma image split it into 2, 8 bit clips with the high 8 bits in one clip and the low 8 bit clips in the other then run each through SmoothD2 and recombine them. If that works at all then there is a chance it can be made to work well.

For anime SmoothD2 with color might be useful. Give me a link to a 5 to 10 second clip of the worst anime you are trying to clean and I will use it in my testing.

[TheProfileth]

Btw regarding this section of the documentation
Qtype (1, 2, 3, 4): Quant Type. Selects the type of quant-dequant processing that is used in the filtering. They are ordered by their deblocking strength from least strong to strongest.

1. mpeg intra

2. h263 intra

3. mpeg inter

4. h263 inter


Do the Qtypes 3 and 4 use other frames? And by that I mean is there temporal filtering going on? It would be really cool if there was but I still would like to know what exactly the difference between the intra and the inter modes as they are never really explained in the documentation.

[Didée]

No temporal filtering. It's just different matrices. The basics (forum.doom9.org/showthread.php?t=54147).

[jconklin]

I knew I was going to need more documentation on Qtype.
In the XVID code that handles the mpeg and h263 processing only mpeg intra and inter allow the specification of a matrix. The h263 routines have the matrix built into their code. I assume they use what the h263 spec calls for. If using h263 intra or h263 inter the matrix argumenht is ignored.
I have not supplied "paired matricies" one for mpeg intra and one for mpeg inter. Any matrix, except 4 and 9, can be used with either mpeg intra or mpeg inter. 4 and 9 can only be used with mpeg intra. If there are enough requests I could add a couple of arguments to allow passing in a matrix as a string of 64 numbers.

[jconklin]

Version A2 is out
Fixed: A case where bad arguments were not caught.
New: Color deblocking using a script wrapper around SmoothD2
Improved:
1: Slightly better frame edge deblocking.
2: Improved documentation.


Didée
Thank you for the link to "Understanding a Quantization Matrix". [URL="http://forum.doom9.org/showthread.php?t=54147"]
A very good presentation. I have included the link in the documentation.

ryrynz
Thanks for the question about color. I got me to try color deblocking and smoothing. Version a2 has the color
deblock script SmoothD2c(). I expect the script to become more powerful and useful as people add to it. I suspect that if
you wrap SmoothD2c() in a temporal smoother such as MDegrain1() it will help your colors. I will download an anime clip
and add it to my testing.

TheProfileth
Thanks for the questions about Qtype. I have upgraded the documentation which I hope will make it a bit clearer.

[jconklin]

Here is the SmoothD2c() color deblock code. Some example of it at work https://sites.google.com/site/jconklin754smoothd2/home

#
# REQUIRED ROUTINES
#
# aWarpSharp2.dll avalable from http://sites.google.com/site/pavelsx2/
#

###
#
# This file contains the SmoothD2c() script routine.
#
# SmoothD2c() provides a wrapper around SmoothD2() that provides deblocking of color as well as luma.
# The arguments quant, num_shift, Matrix, Qtype, and ZW are the same as those in SmoothD2 and have the same defaults.
# The arguments cWarpToLuma and downSize are specific to SmoothD2c().
# cWarpToLuma (0...8) uses aWarpSharp2() to warp the chroma to the luma. Default 0 no warping 8 strong warping.
# downSize (1...4) Default 1 no downsizing. This specifies how much the image should be shrunk before deblocking.
# This can be useful if you have really bad chroma blocking. see examples at
# https://sites.google.com/site/jconklin754smoothd2/home
#
###



###
# This routine takes a clip that has already had the luma deblocked and uses SmoothD2 to deblock the chroma.
# Defaults SmoothD2c(quant=3, num_shift=3, Matrix=3, Qtype=1, ZW=1, cWarpToLuma=0, downSize=1)
Function SmoothD2c(clip clp, int "quant", int "num_shift", int "Matrix", int "Qtype", int "ZW", int "cWarpToLuma", int "downSize") {
quant = default( quant, 3 )
num_shift = default( num_shift, 3 )
Matrix = default( Matrix, 3 )
Qtype = default( Qtype, 1 )
ZW = default( ZW, 1 )
cWarpToLuma = default( cWarpToLuma, 0 )
downSize = default( downSize, 1 )


Assert( ( quant >= 1 && quant <= 31 ) ? true : false, chr(10) + "'quant' wrong value! [1-31]" + chr(10))
Assert( ( num_shift >= 1 && num_shift <= 4 ) ? true : false, chr(10) + "'num_shift' wrong value! [1-4]" + chr(10))
Assert( ( Matrix >= 1 && Matrix <= 110 ) ? true : false, chr(10) + "'Matrix' wrong value! [1-110]" + chr(10))
Assert( ( Qtype >= 0 && Qtype <= 4 ) ? true : false, chr(10) + "'Qtype' wrong value! [0-4]" + chr(10))
Assert( ( ZW >= 0 && ZW <= 16 ) ? true : false, chr(10) + "'ZW' wrong value! [1-16]" + chr(10))
Assert( ( cWarpToLuma >= 0 && cWarpToLuma <= 8 ) ? true : false, chr(10) + "'cWarpToLuma' wrong value! [0-8]" + chr(10))
Assert( ( downSize >= 1 && downSize <= 4 ) ? true : false, chr(10) + "'downSize' wrong value! [1-4]" + chr(10))

ow = clp.width
oh = clp.height
sd1 = clp

# sd1 = (downSize > 1) ? sd1.bilinearResize(m16(ow/downSize), m16(oh/downSize)) : sd1
sd1 = (downSize > 1) ? sd1.bicubicResize(m16(ow/downSize), m16(oh/downSize)) : sd1
sd1 = (cWarpToLuma > 0) ? sd1.colorWarpToLuma(strength=cWarpToLuma) : sd1

u2 = sd1.UToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)
v2 = sd1.VToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)

u2 = (downSize > 1) ? u2.bicubicResize(m16(ow/2), m16(oh/2)) : u2
v2 = (downSize > 1) ? v2.bicubicResize(m16(ow/2), m16(oh/2)) : v2

sd1 = YToUV(u2, v2).MergeLuma(clp)

return(sd1)
}


###
# m4 "mod-4" see "Crop restrictions" in http://avisynth.org/mediawiki/Crop
# m4 is used in may scripts that do resizing.
# m8 "mod-8" is used so that cropping will occure on luma block (8x8) boundaries
# and that resizing will produce an integer number of luma blocks.
# m16 "mod-16" is used so that cropping will occure on YV12 chroma block (16x16) boundaries
# and that resizing will produce an integer number of chroma blocks.
function m4(float x) {return(x<16?16:int(round(x/4.0) *4))}
function m8(float x) {return(x<16?16:int(round(x/8.0) *8))}
function m16(float x) {return(x<32?32:int(round(x/16.0)*16))}





###
#
# This routine moves the color information tward the edges in the luma
#
# Defaults colorWarpToLuma(strength=1)
function colorWarpToLuma(clip clp, int "strength") {
strength = default( strength, 1 )


thresh = (strength == 0) ? 1
\ : (strength == 1) ? 24
\ : (strength == 2) ? 32
\ : (strength == 3) ? 48
\ : (strength == 4) ? 64
\ : (strength == 5) ? 96
\ : (strength == 6) ? 128
\ : (strength == 7) ? 192
\ : 255

blur = strength + 2
clp
orig=clp
aWarpSharp2(thresh=thresh, blur=blur, type=1, depth=14, chroma=4)
mt_merge(orig, orig, last, y=2, u=5, v=5)

return(last)
}

[TheProfileth]

BTW jconklin
What exactly is the point of trying to "warp the chroma to the luma"? Obviously this is easier than rewriting code to allow for chroma denoising but is this nearly as effective as that would be?

[jconklin]

BTW jconklin
What exactly is the point of trying to "warp the chroma to the luma"? Obviously this is easier than rewriting code to allow for chroma denoising but is this nearly as effective as that would be?

Warp chroma to luma is different from deblocking or denoising chroma. What it does, using the warpsharp routine, is if there is a strong luma edge near a chroma edge it will move the chroma edge closer to the luma edge. This is occasionally useful. The downside is it can desaturate colors in dark areas. This is an option that if you have the right kind of damaged video it might be able to help.

The actual chroma deblocking is done by the calls to SmoothD2 that are made in the SmoothD2c routine.

I have added an example to https://sites.google.com/site/jconklin754smoothd2/examples/low_res_color_deblocking that shows the same image deblocked with and without using Warp chroma to luma.

[Schwartzvald]

Love this filter, especially the warpsharp routine in SmoothD2c.

However, I'm having trouble getting SmoothD2 to load in avs script on an older XP machine.

On a newer Windows 7 machine, it works fine. On the old XP box, Avisynth presents a "Plugin not found" error. If I try to force it with LoadPlugin(), it gives error code 0x36b1.

My best guess is I'm missing a dependency that comes stock with newer versions of Windows, like a VC runtime. Anyone know what that might be?

[Robert Martens]

Dependency Walker says it needs msvcr80.dll, which if I'm not mistaken corresponds to the Visual C++ 2005 Redistributable (http://www.microsoft.com/en-us/download/details.aspx?id=3387).

[Schwartzvald]

Yes, that got me on the right track. I'd never heard of Dependency Walker, either. Thanks for introducing me to it, and for the help.

It actually depends on this latest side-by-side of VC2005.:

https://www.microsoft.com/en-us/download/details.aspx?id=26347

Older versions won't do. I'm assuming that comes stock with Windows 7, but my XP versions were out of date.

[jconklin]

I will have to add that to the documentation. I am building and testing in Windows 7 using Visual C++ 2005.
I have been busy optimizing the code which looks like it will make it about 20% faster. I also have a first cut at makeing it multithreaded.
It is going to take me a while to get the optimizations and multithreading done and released.

[Schwartzvald]

Cool.

Regarding SmoothD2c.avs... I am not so great at writing scripts, so it took me a bit of staring at:


MergeLuma: Images must have same width and height!


and the script to figure out that I couldn't use the "downsize" parameter with video that wasn't mod16 at half resolution... which includes DVD video.

The following is probably incorrect, but it throwing me a message like this would have saved me a little time:


Assert( ( (m16(ow/2) != ow/2 || m16(oh/2) != oh/2) && downsize > 1 ) ? false : true, chr(10) + "'downsize' only works with mod32 resolutions!" + chr(10))


There's probably a better way to handle it--I'm a rather low-end Avisynth user. Forgive me if the syntax here is subpar or downright incorrect.

[jconklin]

Cool.

Regarding SmoothD2c.avs... I am not so great at writing scripts, so it took me a bit of staring at:


MergeLuma: Images must have same width and height!


and the script to figure out that I couldn't use the "downsize" parameter with video that wasn't mod16 at half resolution... which includes DVD video.

The following is probably incorrect, but it throwing me a message like this would have saved me a little time:


Assert( ( (m16(ow/2) != ow/2 || m16(oh/2) != oh/2) && downsize > 1 ) ? false : true, chr(10) + "'downsize' only works with mod32 resolutions!" + chr(10))


There's probably a better way to handle it--I'm a rather low-end Avisynth user. Forgive me if the syntax here is subpar or downright incorrect.


My appologies. I did not think about the code that resizes back to the original size. Since the routine knows that the original size works it doesn't need the m16(oh/2).
The fix is a few lines up from the bottom of the code.

Function SmoothD2c(clip clp, int "quant", int "num_shift", int "Matrix", int "Qtype", int "ZW", int "cWarpToLuma", int "downSize", int "ncpu") {
quant = default( quant, 3 )
num_shift = default( num_shift, 3 )
Matrix = default( Matrix, 3 )
Qtype = default( Qtype, 1 )
ZW = default( ZW, 1 )
cWarpToLuma = default( cWarpToLuma, 0 )
downSize = default( downSize, 1 )
ncpu = default( ncpu, 1 )

Assert( ( quant >= 1 && quant <= 31 ) ? true : false, chr(10) + "'quant' wrong value! [1-31]" + chr(10))
Assert( ( num_shift >= 1 && num_shift <= 4 ) ? true : false, chr(10) + "'num_shift' wrong value! [1-4]" + chr(10))
Assert( ( Matrix >= 1 && Matrix <= 110 ) ? true : false, chr(10) + "'Matrix' wrong value! [1-110]" + chr(10))
Assert( ( Qtype >= 0 && Qtype <= 4 ) ? true : false, chr(10) + "'Qtype' wrong value! [0-4]" + chr(10))
Assert( ( ZW >= 0 && ZW <= 16 ) ? true : false, chr(10) + "'ZW' wrong value! [1-16]" + chr(10))
Assert( ( cWarpToLuma >= 0 && cWarpToLuma <= 8 ) ? true : false, chr(10) + "'cWarpToLuma' wrong value! [0-8]" + chr(10))
Assert( ( downSize >= 1 && downSize <= 4 ) ? true : false, chr(10) + "'downSize' wrong value! [1-4]" + chr(10))

ow = clp.width
oh = clp.height
sd1 = clp

# sd1 = (downSize > 1) ? sd1.bilinearResize(m16(ow/downSize), m16(oh/downSize)) : sd1
sd1 = (downSize > 1) ? sd1.bicubicResize(m16(ow/downSize), m16(oh/downSize)) : sd1
sd1 = (cWarpToLuma > 0) ? sd1.colorWarpToLuma(strength=cWarpToLuma) : sd1

u2 = sd1.UToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW, ncpu=ncpu) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)
v2 = sd1.VToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW, ncpu=ncpu) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)

#u2 = (downSize > 1) ? u2.bicubicResize(m16(ow/2), m16(oh/2)) : u2
#v2 = (downSize > 1) ? v2.bicubicResize(m16(ow/2), m16(oh/2)) : v2
u2 = (downSize > 1) ? u2.bicubicResize(ow/2, oh/2) : u2
v2 = (downSize > 1) ? v2.bicubicResize(ow/2, oh/2) : v2

sd1 = YToUV(u2, v2).MergeLuma(clp)

return(sd1)
}

If this doesn't fix it let me know.
Thanks

[jconklin]

opps. Ignore ncpu it doesn't do anything yet. I think the smoothd2 version you have will accept it as an argument but will not do anything with it.

[jconklin]

And I realized that I forgot to say that smoothd2 works best at mod 8 for luma and mod 16 for chroma with the original block boundaries unshifted, it will work ok if the block boundaries have been shifted and mod 4 for luma and chroma. One of the things that drew me to smoothD and got me to do smoothD2 was working on a video that had been through enough encodes and decodes that there were frames where the block edge artifacs had been shifted 1 or 2 pixels out of there original position. Note that the "original" image in the block had not shifted, just the blocking artifact introduced at an early encoding stage. SmoothD managed to reduce it without destroying the rest of the image.

[Schwartzvald]

The code you posted works, but my version doesn't accept ncpu as an argument yet, so I just remarked it out for now. Thanks.

The nature by which downsize increases the strength of the filter is interesting. At high levels, it's like a chroma vampire.

I actually have some old videos in 320x240 like you describe with shifted blocks and other ugly noise that I keep around to throw filters at. I never used the original SmoothD, but SmoothD2 did a great job with them, because of all the different ways to control strength.

What I've really been using it for is for a really bad cartoon DVD set, which I believe was authored from tape. SmoothD2c killed the rainbows and chroma blocking when nothing else I could find or set up would...

[jconklin]

Schwartzvald - Thank you for letting me know that SmoothD2 is working so well for you.

I agree the chroma vampire can suck the life right out of the picture ;)
The few times I have tried to deal with it I have not had much luck. If I recall just increasing the saturation of the image blows out the color in flat areas and doesn’t bring it back enough where you want it. What I haven't tried is to take the difference between the color before SmoothD2 and the color after SmoothD2. Then use any simple smoother to smooth the difference. Then increase the saturation of the smoothed difference. Then add the result back to the image after SmoothD2 vampired the chroma.

A major improvement to SmoothD2 that I am thinking about how to implement is a dynamic strength control. SmoothD2 would determine the amount of blocking in each frame and use that to set the strength between max and min strength arguments. It should help with videos that get bit starved during 3-2 pull down. Other places in the same video need very little deblocking.

The term I use to think about the ability of a deblocker to deal effectively with both low and high levels of blocking in a video while maintaining a high amount of detail is the deblockers "dynamic range". If there is a standard term let me know so I can use it.

Many of the features I have put into SmoothD2 are there to increase its dynamic range though I admit that when I was adding them I was thinking along the lines of "I need this much deblocking, how do I get the detail back". Clearly it is easier to keep the detail if you don't lose it in the first place. So the need for SmoothD2 to change the strength it uses on a per frame basis.

If anyone has ideas on what inputs (variables, and how to compute them) should go into the dynamic strength control algorithm please let me know.

I am at the point were I need to build a decent regression test suit so if anyone has any test patterns that would be useful for testing deblockers let me know.

The next release should be out in a couple of weeks. I think I have cursed mmx enough so that it is starting to bend to my will. I have a feeling my wife would have a different perspective on that.

[satmonk]

I have Windows 7 but there is the annoying message: unable to load plugin...

I have msvcr80.dll (612kb) and put this file to all important folders (system32, syswow64, avisynth plugin folder)

But the error message remains :(

[jconklin]

Well 2 months ago I thought I would have the next release in a couple of weeks.
With luck I might actually get it out in a couple of weeks.
I also said that the single cpu speed might be as much as 25% faster.
Well it looks like it will be closer to twice as fast.
In the mean time I have a new version of SmoothD2c.avs that helps tame the chroma vampire.

###
# This version may have a mod8 width restriction. If it does the new version of SmoothD2 will reduce it to mod4
# This file contains the SmoothD2c() script routine.
#

###
# REQUIRED ROUTINES
#
# aWarpSharp2.dll available from http://sites.google.com/site/pavelsx2/
# Ylevels() and YlevelsS() see http://avisynth.org/mediawiki/Ylevels
# available from http://forum.doom9.org/showthread.php?t=79898
# MaskTools2.dll available from http://avisynth.org/mediawiki/MaskTools2

###
# RESTRICTIONS
#
# The Input clip must be evenly divisible by 4 in both height and width.

###
# ARGUMENTS
#
# SmoothD2c() provides a wrapper around SmoothD2() that takes a clip with the luma deblocked and uses SmoothD2 to deblock the chroma.
# The arguments quant, num_shift, Matrix, Qtype, and ZW are the same as those in SmoothD2 and have the same defaults.
# The arguments Cpr, cWarpToLuma and downSize are specific to SmoothD2c().
#
# Cpr (0..8) Chroma Protection Strength. Provides control over chroma desaturation at low luma levels.
# It does so by building a SmoothD2 Zero Weight Mask "ZWmask".
# Low luma pixels map to high Zero weights in ZWMASK.
# Default 0 no protection
# 8 strong protection.
#
# cWarpToLuma (0...8) uses aWarpSharp2() to warp the chroma to the luma. Default 0 no warping 8 strong warping.
# This will move the chroma from high luma areas toward strong low luma edges which is
# useful to reduce chroma bleeding.
# Places where the chroma has been moved from may be visibly desaturated.
# NOTE: The SmoothD2c() argument Cpr will have NO effect on chroma desaturation caused by
# cWarpToLuma.
#
# downSize (1...4) This specifies how much the image should be shrunk before deblocking.
# Default 1 no downsizing.
# 2 reduce image to 1/2 it's original size before processing.
# 3 reduce image to 1/3 it's original size before processing.
# 4 reduce image to 1/4 it's original size before processing.
# This can be useful if you have really bad chroma blocking. see examples at
# https://sites.google.com/site/jconklin754smoothd2/home
#

###
# GETTING STARTED
#
# First all parameters default to reasonable values so SmoothD2c() should work, though it will provide only very weak deblocking.
# The most important parameter is quant, which is the primary control of deblocking strength.
# If you can not get as much deblocking as desired then try using a different Matrix.
# Matrix 8, or Matrix 13-29
# If you still can't get enough deblocking then try Qtype=2 or 4.
# If you still can't get enough deblocking then increase downSize. For Qtype=1 you probably want a Matrix between 13-29.
# When you get enough deblocking find an area, usually in a darker part of the picture, that the chroma has washed out and start increasing Cpr
# until the chroma returns. Its a balancing act between chroma washout and chroma deblocking.
# When you have gotten this far and you have chroma bleeding around strong luma edges then you can try increasing cWarpToLuma
# to see if it will help reduce it. Note that cWarpToLuma especially at higher settings will tend to washout the chroma around the edges.
# This chroma washout cannot be corrected by increasing Cpr.


###
# This routine takes a clip that has already had the luma deblocked and uses SmoothD2 to deblock the chroma.
# Defaults SmoothD2c(quant=3, num_shift=3, Matrix=3, Qtype=1, ZW=1, Cpr=0, cWarpToLuma=0, downSize=1, ncpu=1)
Function SmoothD2c(clip clp, int "quant", int "num_shift", int "Matrix", int "Qtype", int "ZW", int "Cpr", int "cWarpToLuma", int "downSize") {
quant = default( quant, 3 )
num_shift = default( num_shift, 3 )
Matrix = default( Matrix, 3 )
Qtype = default( Qtype, 1 )
ZW = default( ZW, 1 )
Cpr = default( Cpr, 0 ) # Chroma Protection Strength aka Chroma Vampire Be Gone
cWarpToLuma = default( cWarpToLuma, 0 )
downSize = default( downSize, 1 )
#ncpu = default( ncpu, 1 )


Assert( ( quant >= 1 && quant <= 31 ) ? true : false, chr(10) + "'quant' wrong value! [1-31]" + chr(10))
Assert( ( num_shift >= 1 && num_shift <= 4 ) ? true : false, chr(10) + "'num_shift' wrong value! [1-4]" + chr(10))
Assert( ( Matrix >= 1 && Matrix <= 110 ) ? true : false, chr(10) + "'Matrix' wrong value! [1-110]" + chr(10))
Assert( ( Qtype >= 0 && Qtype <= 4 ) ? true : false, chr(10) + "'Qtype' wrong value! [0-4]" + chr(10))
Assert( ( ZW >= 0 && ZW <= 16 ) ? true : false, chr(10) + "'ZW' wrong value! [1-16]" + chr(10))
Assert( ( Cpr >= 0 && Cpr <= 6 ) ? true : false, chr(10) + "'Cpr' wrong value! [0-6]" + chr(10))
Assert( ( cWarpToLuma >= 0 && cWarpToLuma <= 8 ) ? true : false, chr(10) + "'cWarpToLuma' wrong value! [0-8]" + chr(10))
Assert( ( downSize >= 1 && downSize <= 4 ) ? true : false, chr(10) + "'downSize' wrong value! [1-4]" + chr(10))
#Assert( ( ncpu >= 1 && ncpu <= 4 ) ? true : false, chr(10) + "'ncpu' wrong value! [1-4]" + chr(10))
Assert( ( clp.width == m4(clp.width) ) ? true : false, chr(10) + "'clip width' must be a multiple of 4" + chr(10))
Assert( ( clp.height == m4(clp.height) ) ? true : false, chr(10) + "'clip height' must be a multiple of 4" + chr(10))

ow = clp.width
oh = clp.height

sd1 = clp

sd1 = (downSize > 1) ? sd1.Lanczos4Resize(m32(ow/downSize), m32(oh/downSize)) : sd1
sd1 = (cWarpToLuma > 0) ? sd1.colorWarpToLuma(strength=cWarpToLuma) : sd1

zw1 = (Cpr == 0) ? sd1.ylevels( 0, 1.0, 255, 0, 0) : sd1.buildZmask(cpr=cpr)


u2 = sd1.UToY()
v2 = sd1.VToY()

cw = u2.width
ch = u2.height

zwuv = zw1.Lanczos4Resize(cw, ch)

u2 = u2.SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW, ZWmask=zwuv)
v2 = v2.SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW, ZWmask=zwuv)

u2 = (downSize > 1) ? u2.Lanczos4Resize(ow/2, oh/2) : u2
v2 = (downSize > 1) ? v2.Lanczos4Resize(ow/2, oh/2) : v2

# u2 = (downSize > 1) ? u2.bicubicResize(ow/2, oh/2) : u2
# v2 = (downSize > 1) ? v2.bicubicResize(ow/2, oh/2) : v2

sd1 = YToUV(u2, v2).MergeLuma(clp)

return(sd1)
}


###
# m4 "mod-4" see "Crop restrictions" in http://avisynth.org/mediawiki/Crop
# m4 is used in may scripts that do resizing.
# m8 "mod-8" is used so that cropping will occur on luma block (8x8) boundaries
# and that resizing will produce an integer number of luma blocks.
# m16 "mod-16" is used so that cropping will occur on YV12 chroma block (16x16) boundaries
# and that resizing will produce an integer number of chroma blocks.
# m16 "mod-m32" is used so that the zw1 mask can be resized to the size of the u and v planes.
function m4(float x) {return(x<16?16:int(round(x/4.0) *4))}
function m8(float x) {return(x<16?16:int(round(x/8.0) *8))}
function m16(float x) {return(x<32?32:int(round(x/16.0)*16))}
function m32(float x) {return(x<64?64:int(round(x/32.0)*32))}


###
#
# This routine builds a zero Weight Mask
# which is used to limit the strength of chroma
# processing in dark areas.
#
# Defaults buildZmask(Cpr=1)
function buildZmask(clip clp, int "Cpr") {
Cpr = default( Cpr, 1 )

clp.SmoothD2(quant=31, num_shift=3, Matrix=29, Qtype=1, ZW=0, ZWce=1, ZWlmDark=0, ZWlmBright=255)
mt_invert()
YlevelsS(90,1.01,240,0,255) # This does a non linear range expansion see http://forum.doom9.org/showthread.php?t=79898
mask = mt_invert()

ZWmask = (Cpr == 0) ? clp
\ : (Cpr == 1) ? mask.ylevels(32, 60.0, 255, 8, 0)
\ : (Cpr == 2) ? mask.ylevels(32, 50.0, 255, 16, 0)
\ : (Cpr == 3) ? mask.ylevels(32, 30.0, 255, 16, 0)
\ : (Cpr == 4) ? mask.ylevels(32, 12.0, 255, 16, 0)
\ : (Cpr == 5) ? mask.ylevels(32, 7.0, 255, 16, 0)
\ : mask.ylevels(32, 1.0, 255, 16, 0)

return(ZWmask)
}




###
#
# This routine moves the color information toward the edges in the luma
#
# Defaults colorWarpToLuma(strength=1)
function colorWarpToLuma(clip clp, int "strength") {
strength = default( strength, 1 )

thresh = (strength == 0) ? 1
\ : (strength == 1) ? 24
\ : (strength == 2) ? 32
\ : (strength == 3) ? 48
\ : (strength == 4) ? 64
\ : (strength == 5) ? 96
\ : (strength == 6) ? 128
\ : (strength == 7) ? 192
\ : 255

blur = strength + 2
warpedColor = clp.aWarpSharp2(thresh=thresh, blur=blur, type=1, depth=14, chroma=4)
mt_merge(clp, clp, warpedColor, y=2, u=5, v=5)

return(last)
}

[TheProfileth]

Very interesting will give this a try in a bit.
I suppose that this will also work for chroma denoising as well as deblocking, or am I mistaken?

[jconklin]

It should work well for chroma denoisning. Qtype=1 will tend to do more deblocking than smoothing, Qtype=4 will tend to do more smoothing, with values 2 and 3 in between. For Qtype=1 Matrix values between 13-29 will do increasingly more smoothing then deblocking. DownSize > 1 greatly increases smothing over deblocking.

WARNING: Last night I discovered a bug in my build system that will cause SmoothD2 to crash (produces an error message from AviSynth) if used with a system that has a msvcr80r.dll that is not compatible with the one used in my build. I have a new build that I am still testing that does not use msvcr80.dll and should fix the problem. I plan to do some more testing tonight and release it tomorrow. I will post as soon as the new release is up.

[jconklin]

A shortened and rather simplified story of what it took to find and squash the bug in SmoothD2
that would, for a few users, cause SmoothD2 to crash with an error message from AviSynth.

First of all. Apologies for having put out a filter that causes problems.
This is what I have been doing to figure out what the problem is and how to fix it.

A question from Bloax. What filters does it depend on?
Given that I have over 150 :eek: filters in my plugin directory. Well you know every time I wanted to try something I would just drop it in and if it didn't break anything I would leave it there. Yea I know, really bad practice.

So

I uninstalled AviSynth.
I checked that the environment variable for the plugin directory had been removed.
I made sure there was no directory in C:\Program Files (x86)\ called AviSynth.
I downloaded the latest version of AviSynth v.2.6.0 ST, 32-bit from SourceForge, file name AviSynth_110525.exe and installed it.
It installed in C:\Program Files (x86)\AviSynth 2.5

In the plugins directory there are 3 files. colors_rgb.avsi, DirectShowSource.dll, and TCPDeliver.dll
On my machine i7-2600K windows 7 64-bit this script, 4 lines of code, runs.
LoadPlugin("C:\SmoothD2-a2\SmoothD2.dll")
AviSource("H:SmoothD2speedTest.avi").ConvertToYV12()
SmoothD2(quant=31, num_shift=4, Matrix=15, Qtype=1, ZW=0, ZWce=1, ZWlmDark=255, ZWlmBright=255, ncpu=1)
return(last)
So

It’s not dependent on some AviSynth filter dll that I am loading.

I uninstalled AviSynth v.2.6.0 ST, 32-bit
I Download and installed Avisynth_258.exe (4.2 MB)
The script still runs
Its not an AviSynth version problem.

Robert Martens mentions Dependency Walker and msvcr80.dll which gets some but not all systems to work.

I get Dependency Walker.

Dependency walker says that SmoothD2.dll needs MSVCR80.DLL file version 8.0.50727.6195
Apparently MSVCR80.DLL is required for any:confused: dll built with Microsoft Visual C++ 2005 Version 8.0.50727.867
Note the version numbers match in the first 3 fields.

OK so it works on my machine and does not care which AviSynth is used.
It works for some people but not others...
Maybe MSVCR80.DLL needs to be in a special place.

When I do a search for MSVCR80.DLL on my C drive the search finds over 50 of them!
Oh that’s right everyone that puts out an application, Photoshop for example, needs to distribute a copy of any dll's that it uses. So most people probably already have it on their system.
When I tracked one down and did a properties on it the description was Microsoft C Runtime Library.
Good, 1 guess confirmed.
When I sorted the list by size there were 5 different sizes. oops. I know I still don't get what the problem is, some part of me has the beginning of a nasty intuition.
When I looked at examples of the different sizes, there were 4 different file versions.:confused:
The same file version came in 778KB and 612KB.
The other sizes were 803KB, 783KB, and 617KB.
Now I am starting to get a bad feeling.

I search further and find
For your convenience, we have provided the following folders for use when redistributing VC++ runtime files.
Subject to the license terms for the software, you may redistribute the folder (unmodified) in the application local folder as a sub-folder with no change to the folder name. You may also redistribute all the files (*.dll and *.manifest)
within a folder, listed below the folder for your convenience, as an entire set.

\VC\redist\x86\Microsoft.VC80.CRT\
msvcm80.dll
msvcp80.dll
msvcr80.dll
Microsoft.VC80.CRT.manifest

Now the bad feeling really kicks in since I realize that the different versions of MSVCR80.DLL are probably not compatible with each other.
OK I will have to distribute the dlls. No big deal. I’ll have to figure out where they should get put so they will be found.
And that won't work either!!!
It can't work because if there are 2 filters that have been built using 2 different versions of MSVCR80.DLL one of them will be a version that was not the version the filter was built with.
Therefore if the versions are different and a called function is incompatible then the filter dies.

At this point, I assume I have my build system set up wrong and probably need to get the MSVCR80.DLL code linked directly into the SmoothD2 dll.
I try a different linker option; the SmoothD2.dll runs on my system and Dependency Walker show no dependencies.

And Bloax tests the new SmoothD2.dll and it works for him.

And I read the AviSynth build a filter and compiler setup docs yet again and this time I realize that of course If Microsoft had given decent hints about the different linker flags in visual studio it would have been obvious :) what option I should have used.

I hope that this has really taken care of the bug.

Thanks for the hints that I have gotten along the way.
Now maybe I can get a good night’s sleep.

[Boulder]

Thanks :) It's refreshing to read someone really explaining how they have traced some weird errors.

By the way, the download link does not show up on the front page of your site, you need to go to the Download page to see it.

[jconklin]

I finally ran a set of speed tests on the latest version of SmoothD2. It took longer then I had planned but it was a lot more3 tests then I had expected to run. There are almost 150 tests. I compared SmoothD2 to 30 filters running on a fast hard disk, a SSD, and a Ram Disk. Also have tests showing the speed improvments available when SmoothD2's multitreading is used. Test results are on https://sites.google.com/site/jconklin754smoothd2/home/speed-tests (I finally ran a set of speed tests on the latest version of SmoothD2. It took longer then I had planned but it was a lot more3 tests then I had expected to run. There are almost 150 tests. I compared SmoothD2 to 30 filters running on a fast hard disk, a SSD, and a Ram Disk. Also have tests showing the speed improvments available when SmoothD2's multitreading is used. Test results are on https://sites.google.com/site/jconklin754smoothd2/home/speed-tests)

[Zaxooz]

I really like your script, thank you for the great job done. There's a question I have about the settings as well. In your example (SmoothD2(quant=31, num_shift=4, Matrix=15, Qtype=1, ZW=0, ZWce=1, ZWlmDark=255, ZWlmBright=255, ncpu=1) you are using Matrix=15, is there any particular reason for that preference? Thanks.

[jconklin]

I have no Idea why I used matrix=15 in the test. I probably meant matrix=5. matrix=15 would probably smooth out all the details of the image.
All matrix numbers larger than 9 should be considered experimental. I put them in as a way to learn how the matrix values modify images.

Thanks for using SmoothD2
Jim

[Zaxooz]

For those interested. Matrices 9 and 15 produce desirable results as strong deblockers when used with low quants (1-3).

[hydra3333]

Thanks.
About to experiment with this badly blocked source.
https://drive.google.com/file/d/0B5RV2aJ2vdhSVk55dm5KMWxaR3M/view?usp=sharing

http://forum.doom9.org/showthread.php?p=1738145#post1738145

[jconklin]

Built on the foundation of smoothD2(), amDCT() provides adaptive strength at the frame and block level based upon an analysis the video frame being processed. It also does local range expansion.

See http://forum.doom9.org/showthread.php?p=1801244#post1801244