Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
29th April 2012, 19:21 | #1 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
SmoothD2 a deblocking and smoothing filter
SmoothD2 is deblocking filter or at stronger settings, a smoothing filter.
It is a major rewrite of SmoothD. See http://forum.doom9.org/showthread.ph...hlight=SmoothD I liked SmoothD both as a deblocker and as a smoother. My problem was when I tried to use it with low quants for deblocking and later in the script with higher quants to produce a prefiltered clip for mvtools it would crash. So I figured I would fix the crash bug. Having restructured and cleaned up some of the code fixing the crash bug, I figured I might as well handle a minor 1/2 bit rounding problem. I did that and I was hooked. Could I make it faster, yes 1.5 times! Are the shift positions optimal, could I tell the difference if I changed them, yes, what is the difference and why, bang head, bang head, ponder, ponder then, 8 shifts on 8x8 squares, the 8 queens problem yes it worked great solution!!! Let's add the other quant-dequant types so we now have the original mpeg_intra, as well as mpeg_inter, h263_intra, and h263_inter processing, nice. How do matrices work? Better redo that code and try a bunch of them and see. Play, play, play ... Well I had fun doing that, sort of got an intuitive sense of what they do, got some you can try, but I don't know how they work, I had too much fun playing. And the last problem, and the universe laughs, can I control the filter strength within a frame, after all it would be nice to decrease the filtering strength as you move from the edge of a macro block toward its center. Think, think, think, try this try that, duh. It's been right there in front of me. The SmoothD zero_weight argument! It controls the filter strength by adding copies of the unshifted original image to the processed images. There are now 5 Zero Weight arguments that provide filter strength control independently at the whole frame level, as a function of the position within a macro block, at a per pixel level via a mask clip, and as a function of the luma value of a pixel. The end result of all of this is a completely reworked SmoothD with a new set of arguments and a new name. SmoothD2(clip clip, int quant, int num_shift, int Matrix, int Qtype, int ZW, int ZWce, clip ZWmask, int ZWlmDark, int ZWlmBright) This is the first release and I expect to add some more features and speed it up some more. Try it and let me know what doesn't work, as well as how it could be better. Main documentation page https://sites.google.com/site/jconklin754smoothd2/home Full source code download page https://sites.google.com/site/jconkl.../home/download Thanks jconklin Read a Tale of a Bug Squash. http://forum.doom9.org/showpost.php?...0&postcount=32 Version A3 fixes a bug that would crash the filter producing a error message from Avisysnth on some users systems. It also adds major speed improvments as well as mult cpu support. Latest version A3 09-08-2012 Last edited by jconklin; 9th September 2012 at 15:05. Reason: Fixed Download Link |
3rd May 2012, 22:52 | #3 | Link |
Leader of Dual-Duality
Join Date: Aug 2010
Location: America
Posts: 134
|
Ooh I liked SmoothD, was a very promising filter. I look forward to this, I also hope for a speed increase but I guess I shall see.
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon |
5th May 2012, 01:43 | #6 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
Thanks! I have been lurking long enough that I figured it was about time I gave something back.
I plan to run a set of speed tests against all the other deblockers I can find. Should have some results in a few days. Will post results when done. I am getting SmoothD2 to deblock color. It works but requires too many additional arguments to get the color deblocking to be reasonable. I want to just give it a color deblock strength argument but haven't yet figured out how to make that work. Early days yet. Bye the way, if "best" referred to my writing style, all of my teachers are going to fall down. |
5th May 2012, 09:24 | #7 | Link |
Registered User
Join Date: Mar 2009
Posts: 3,650
|
Hi! I'm after a filter that smoothes out color variations to achieve flat (clean) colors. ATM I'm using Minblur but I'm looking for a more specialized color cleaner for anime content. Once you perhaps have the color arguments sorted is there a possibility that your filter could achieve this? Or are you aware of something that perhaps already does it? Thanks.
|
5th May 2012, 09:36 | #8 | Link |
Registered User
Join Date: Sep 2009
Posts: 378
|
Would your filter benefit from a 16bit mod for Dither Tools?
http://forum.doom9.org/showpost.php?...59&postcount=3 |
6th May 2012, 07:10 | #9 | Link |
Leader of Dual-Duality
Join Date: Aug 2010
Location: America
Posts: 134
|
A nice 16bit deblocker would be very useful for heavy debanding
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon |
6th May 2012, 08:32 | #10 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
SmoothD2 is an 8 bit luma deblocker/smoother. I am getting it to do color by wrapping 3 SmoothD2 calls in a .avs wrapper. Making it a 16 bit luma deblocker/smoother would require rewriting the xvid routines fdct, dequant, quant, and idct routines which are mmx optimized. It has been decades since I last wrote assembler so it is unlikly that I am going to take on that task. Its late and my eys are starting to close so this is probably nuts. If you can take a 16 bit luma image split it into 2, 8 bit clips with the high 8 bits in one clip and the low 8 bit clips in the other then run each through SmoothD2 and recombine them. If that works at all then there is a chance it can be made to work well.
For anime SmoothD2 with color might be useful. Give me a link to a 5 to 10 second clip of the worst anime you are trying to clean and I will use it in my testing. |
10th May 2012, 23:49 | #11 | Link |
Leader of Dual-Duality
Join Date: Aug 2010
Location: America
Posts: 134
|
Btw regarding this section of the documentation
Code:
Qtype (1, 2, 3, 4): Quant Type. Selects the type of quant-dequant processing that is used in the filtering. They are ordered by their deblocking strength from least strong to strongest. 1. mpeg intra 2. h263 intra 3. mpeg inter 4. h263 inter
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon |
11th May 2012, 01:22 | #12 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
No temporal filtering. It's just different matrices. The basics.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
11th May 2012, 07:38 | #13 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
I knew I was going to need more documentation on Qtype.
In the XVID code that handles the mpeg and h263 processing only mpeg intra and inter allow the specification of a matrix. The h263 routines have the matrix built into their code. I assume they use what the h263 spec calls for. If using h263 intra or h263 inter the matrix argumenht is ignored. I have not supplied "paired matricies" one for mpeg intra and one for mpeg inter. Any matrix, except 4 and 9, can be used with either mpeg intra or mpeg inter. 4 and 9 can only be used with mpeg intra. If there are enough requests I could add a couple of arguments to allow passing in a matrix as a string of 64 numbers. |
25th May 2012, 05:24 | #14 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
Version A2 is out
Fixed: A case where bad arguments were not caught. New: Color deblocking using a script wrapper around SmoothD2 Improved: 1: Slightly better frame edge deblocking. 2: Improved documentation. Didée Thank you for the link to "Understanding a Quantization Matrix". [URL="http://forum.doom9.org/showthread.php?t=54147"] A very good presentation. I have included the link in the documentation. ryrynz Thanks for the question about color. I got me to try color deblocking and smoothing. Version a2 has the color deblock script SmoothD2c(). I expect the script to become more powerful and useful as people add to it. I suspect that if you wrap SmoothD2c() in a temporal smoother such as MDegrain1() it will help your colors. I will download an anime clip and add it to my testing. TheProfileth Thanks for the questions about Qtype. I have upgraded the documentation which I hope will make it a bit clearer. |
25th May 2012, 05:37 | #15 | Link |
Registered User
Join Date: Mar 2006
Posts: 39
|
SmoothD2c script
Here is the SmoothD2c() color deblock code. Some example of it at work https://sites.google.com/site/jconklin754smoothd2/home
Code:
# # REQUIRED ROUTINES # # aWarpSharp2.dll avalable from http://sites.google.com/site/pavelsx2/ # ### # # This file contains the SmoothD2c() script routine. # # SmoothD2c() provides a wrapper around SmoothD2() that provides deblocking of color as well as luma. # The arguments quant, num_shift, Matrix, Qtype, and ZW are the same as those in SmoothD2 and have the same defaults. # The arguments cWarpToLuma and downSize are specific to SmoothD2c(). # cWarpToLuma (0...8) uses aWarpSharp2() to warp the chroma to the luma. Default 0 no warping 8 strong warping. # downSize (1...4) Default 1 no downsizing. This specifies how much the image should be shrunk before deblocking. # This can be useful if you have really bad chroma blocking. see examples at # https://sites.google.com/site/jconklin754smoothd2/home # ### ### # This routine takes a clip that has already had the luma deblocked and uses SmoothD2 to deblock the chroma. # Defaults SmoothD2c(quant=3, num_shift=3, Matrix=3, Qtype=1, ZW=1, cWarpToLuma=0, downSize=1) Function SmoothD2c(clip clp, int "quant", int "num_shift", int "Matrix", int "Qtype", int "ZW", int "cWarpToLuma", int "downSize") { quant = default( quant, 3 ) num_shift = default( num_shift, 3 ) Matrix = default( Matrix, 3 ) Qtype = default( Qtype, 1 ) ZW = default( ZW, 1 ) cWarpToLuma = default( cWarpToLuma, 0 ) downSize = default( downSize, 1 ) Assert( ( quant >= 1 && quant <= 31 ) ? true : false, chr(10) + "'quant' wrong value! [1-31]" + chr(10)) Assert( ( num_shift >= 1 && num_shift <= 4 ) ? true : false, chr(10) + "'num_shift' wrong value! [1-4]" + chr(10)) Assert( ( Matrix >= 1 && Matrix <= 110 ) ? true : false, chr(10) + "'Matrix' wrong value! [1-110]" + chr(10)) Assert( ( Qtype >= 0 && Qtype <= 4 ) ? true : false, chr(10) + "'Qtype' wrong value! [0-4]" + chr(10)) Assert( ( ZW >= 0 && ZW <= 16 ) ? true : false, chr(10) + "'ZW' wrong value! [1-16]" + chr(10)) Assert( ( cWarpToLuma >= 0 && cWarpToLuma <= 8 ) ? true : false, chr(10) + "'cWarpToLuma' wrong value! [0-8]" + chr(10)) Assert( ( downSize >= 1 && downSize <= 4 ) ? true : false, chr(10) + "'downSize' wrong value! [1-4]" + chr(10)) ow = clp.width oh = clp.height sd1 = clp # sd1 = (downSize > 1) ? sd1.bilinearResize(m16(ow/downSize), m16(oh/downSize)) : sd1 sd1 = (downSize > 1) ? sd1.bicubicResize(m16(ow/downSize), m16(oh/downSize)) : sd1 sd1 = (cWarpToLuma > 0) ? sd1.colorWarpToLuma(strength=cWarpToLuma) : sd1 u2 = sd1.UToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1) v2 = sd1.VToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1) u2 = (downSize > 1) ? u2.bicubicResize(m16(ow/2), m16(oh/2)) : u2 v2 = (downSize > 1) ? v2.bicubicResize(m16(ow/2), m16(oh/2)) : v2 sd1 = YToUV(u2, v2).MergeLuma(clp) return(sd1) } ### # m4 "mod-4" see "Crop restrictions" in http://avisynth.org/mediawiki/Crop # m4 is used in may scripts that do resizing. # m8 "mod-8" is used so that cropping will occure on luma block (8x8) boundaries # and that resizing will produce an integer number of luma blocks. # m16 "mod-16" is used so that cropping will occure on YV12 chroma block (16x16) boundaries # and that resizing will produce an integer number of chroma blocks. function m4(float x) {return(x<16?16:int(round(x/4.0) *4))} function m8(float x) {return(x<16?16:int(round(x/8.0) *8))} function m16(float x) {return(x<32?32:int(round(x/16.0)*16))} ### # # This routine moves the color information tward the edges in the luma # # Defaults colorWarpToLuma(strength=1) function colorWarpToLuma(clip clp, int "strength") { strength = default( strength, 1 ) thresh = (strength == 0) ? 1 \ : (strength == 1) ? 24 \ : (strength == 2) ? 32 \ : (strength == 3) ? 48 \ : (strength == 4) ? 64 \ : (strength == 5) ? 96 \ : (strength == 6) ? 128 \ : (strength == 7) ? 192 \ : 255 blur = strength + 2 clp orig=clp aWarpSharp2(thresh=thresh, blur=blur, type=1, depth=14, chroma=4) mt_merge(orig, orig, last, y=2, u=5, v=5) return(last) } |
30th May 2012, 21:03 | #16 | Link |
Leader of Dual-Duality
Join Date: Aug 2010
Location: America
Posts: 134
|
BTW jconklin
What exactly is the point of trying to "warp the chroma to the luma"? Obviously this is easier than rewriting code to allow for chroma denoising but is this nearly as effective as that would be?
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood My New filter is in the works, and will be out soon |
31st May 2012, 00:54 | #17 | Link | |
Registered User
Join Date: Mar 2006
Posts: 39
|
Quote:
The actual chroma deblocking is done by the calls to SmoothD2 that are made in the SmoothD2c routine. I have added an example to https://sites.google.com/site/jconkl...lor_deblocking that shows the same image deblocked with and without using Warp chroma to luma. |
|
7th July 2012, 07:41 | #18 | Link |
Ooo Royalty
Join Date: Aug 2005
Posts: 9
|
Love this filter, especially the warpsharp routine in SmoothD2c.
However, I'm having trouble getting SmoothD2 to load in avs script on an older XP machine. On a newer Windows 7 machine, it works fine. On the old XP box, Avisynth presents a "Plugin not found" error. If I try to force it with LoadPlugin(), it gives error code 0x36b1. My best guess is I'm missing a dependency that comes stock with newer versions of Windows, like a VC runtime. Anyone know what that might be? |
7th July 2012, 08:16 | #19 | Link |
Registered User
Join Date: Feb 2010
Location: New York
Posts: 116
|
Dependency Walker says it needs msvcr80.dll, which if I'm not mistaken corresponds to the Visual C++ 2005 Redistributable.
|
7th July 2012, 12:16 | #20 | Link |
Ooo Royalty
Join Date: Aug 2005
Posts: 9
|
Yes, that got me on the right track. I'd never heard of Dependency Walker, either. Thanks for introducing me to it, and for the help.
It actually depends on this latest side-by-side of VC2005.: https://www.microsoft.com/en-us/down....aspx?id=26347 Older versions won't do. I'm assuming that comes stock with Windows 7, but my XP versions were out of date. |
|
|