Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th April 2012, 19:21   #1  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
SmoothD2 a deblocking and smoothing filter

SmoothD2 is deblocking filter or at stronger settings, a smoothing filter.
It is a major rewrite of SmoothD. See http://forum.doom9.org/showthread.ph...hlight=SmoothD

I liked SmoothD both as a deblocker and as a smoother.
My problem was when I tried to use it with low quants for deblocking and later in the script with higher quants to produce a prefiltered clip for mvtools it would crash.
So I figured I would fix the crash bug.

Having restructured and cleaned up some of the code fixing the crash bug, I figured I might as well handle a minor 1/2 bit rounding problem.
I did that and I was hooked. Could I make it faster, yes 1.5 times!
Are the shift positions optimal, could I tell the difference if I changed them, yes, what is the difference and why, bang head, bang head, ponder, ponder then, 8 shifts on 8x8 squares, the 8 queens problem yes it worked great solution!!!
Let's add the other quant-dequant types so we now have the original mpeg_intra, as well as mpeg_inter, h263_intra, and h263_inter processing, nice.
How do matrices work? Better redo that code and try a bunch of them and see. Play, play, play ... Well I had fun doing that, sort of got an intuitive sense of what they do, got some you can try, but I don't know how they work, I had too much fun playing.

And the last problem, and the universe laughs,
can I control the filter strength within a frame, after all it would be nice to decrease the filtering strength as you move from the edge of a macro block toward its center.
Think, think, think, try this try that, duh. It's been right there in front of me.
The SmoothD zero_weight argument! It controls the filter strength by adding copies of the unshifted original image to the processed images.

There are now 5 Zero Weight arguments that provide filter strength control independently
at the whole frame level,
as a function of the position within a macro block,
at a per pixel level via a mask clip,
and as a function of the luma value of a pixel.

The end result of all of this is a completely reworked SmoothD with a new set of arguments and a new name.

SmoothD2(clip clip, int quant, int num_shift, int Matrix, int Qtype, int ZW, int ZWce, clip ZWmask, int ZWlmDark, int ZWlmBright)

This is the first release and I expect to add some more features and speed it up some more.

Try it and let me know what doesn't work, as well as how it could be better.

Main documentation page https://sites.google.com/site/jconklin754smoothd2/home
Full source code download page https://sites.google.com/site/jconkl.../home/download

Thanks jconklin

Read a Tale of a Bug Squash. http://forum.doom9.org/showpost.php?...0&postcount=32

Version A3 fixes a bug that would crash the filter producing a error message from Avisysnth on some users systems.
It also adds major speed improvments as well as mult cpu support.

Latest version A3 09-08-2012

Last edited by jconklin; 9th September 2012 at 15:05. Reason: Fixed Download Link
jconklin is offline   Reply With Quote
Old 2nd May 2012, 02:59   #2  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
I'll test it and let you know how things went
Zep is offline   Reply With Quote
Old 3rd May 2012, 22:52   #3  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Ooh I liked SmoothD, was a very promising filter. I look forward to this, I also hope for a speed increase but I guess I shall see.
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 4th May 2012, 03:44   #4  |  Link
Mug Funky
interlace this!
 
Mug Funky's Avatar
 
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
Best. First post. Ever.
__________________
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 4th May 2012, 23:08   #5  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Quote:
Originally Posted by Mug Funky View Post
Best. First post. Ever.
Agreed
Edit: Mug Funky don't you mean "Very good"
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 5th May 2012, 01:43   #6  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
Quote:
Originally Posted by Mug Funky View Post
Best. First post. Ever.
Thanks! I have been lurking long enough that I figured it was about time I gave something back.

I plan to run a set of speed tests against all the other deblockers I can find. Should have some results in a few days. Will post results when done.

I am getting SmoothD2 to deblock color. It works but requires too many additional arguments to get the color deblocking to be reasonable. I want to just give it a color deblock strength argument but haven't yet figured out how to make that work. Early days yet.

Bye the way, if "best" referred to my writing style, all of my teachers are going to fall down.
jconklin is offline   Reply With Quote
Old 5th May 2012, 09:24   #7  |  Link
ryrynz
Registered User
 
ryrynz's Avatar
 
Join Date: Mar 2009
Posts: 3,645
Hi! I'm after a filter that smoothes out color variations to achieve flat (clean) colors. ATM I'm using Minblur but I'm looking for a more specialized color cleaner for anime content. Once you perhaps have the color arguments sorted is there a possibility that your filter could achieve this? Or are you aware of something that perhaps already does it? Thanks.
ryrynz is offline   Reply With Quote
Old 5th May 2012, 09:36   #8  |  Link
Yellow_
Registered User
 
Join Date: Sep 2009
Posts: 378
Would your filter benefit from a 16bit mod for Dither Tools?

http://forum.doom9.org/showpost.php?...59&postcount=3
Yellow_ is offline   Reply With Quote
Old 6th May 2012, 07:10   #9  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
A nice 16bit deblocker would be very useful for heavy debanding
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 6th May 2012, 08:32   #10  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
SmoothD2 is an 8 bit luma deblocker/smoother. I am getting it to do color by wrapping 3 SmoothD2 calls in a .avs wrapper. Making it a 16 bit luma deblocker/smoother would require rewriting the xvid routines fdct, dequant, quant, and idct routines which are mmx optimized. It has been decades since I last wrote assembler so it is unlikly that I am going to take on that task. Its late and my eys are starting to close so this is probably nuts. If you can take a 16 bit luma image split it into 2, 8 bit clips with the high 8 bits in one clip and the low 8 bit clips in the other then run each through SmoothD2 and recombine them. If that works at all then there is a chance it can be made to work well.

For anime SmoothD2 with color might be useful. Give me a link to a 5 to 10 second clip of the worst anime you are trying to clean and I will use it in my testing.
jconklin is offline   Reply With Quote
Old 10th May 2012, 23:49   #11  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Btw regarding this section of the documentation
Code:
Qtype (1, 2, 3, 4): Quant Type.  Selects the type of quant-dequant processing that is used in the filtering.  They are ordered by their deblocking strength from least strong to strongest.

1.  mpeg intra 

2.  h263 intra  

3.  mpeg inter 

4.  h263 inter
Do the Qtypes 3 and 4 use other frames? And by that I mean is there temporal filtering going on? It would be really cool if there was but I still would like to know what exactly the difference between the intra and the inter modes as they are never really explained in the documentation.
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 11th May 2012, 01:22   #12  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,389
No temporal filtering. It's just different matrices. The basics.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 11th May 2012, 07:38   #13  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
I knew I was going to need more documentation on Qtype.
In the XVID code that handles the mpeg and h263 processing only mpeg intra and inter allow the specification of a matrix. The h263 routines have the matrix built into their code. I assume they use what the h263 spec calls for. If using h263 intra or h263 inter the matrix argumenht is ignored.
I have not supplied "paired matricies" one for mpeg intra and one for mpeg inter. Any matrix, except 4 and 9, can be used with either mpeg intra or mpeg inter. 4 and 9 can only be used with mpeg intra. If there are enough requests I could add a couple of arguments to allow passing in a matrix as a string of 64 numbers.
jconklin is offline   Reply With Quote
Old 25th May 2012, 05:24   #14  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
Version A2 is out
Fixed: A case where bad arguments were not caught.
New: Color deblocking using a script wrapper around SmoothD2
Improved:
1: Slightly better frame edge deblocking.
2: Improved documentation.


Didée
Thank you for the link to "Understanding a Quantization Matrix". [URL="http://forum.doom9.org/showthread.php?t=54147"]
A very good presentation. I have included the link in the documentation.

ryrynz
Thanks for the question about color. I got me to try color deblocking and smoothing. Version a2 has the color
deblock script SmoothD2c(). I expect the script to become more powerful and useful as people add to it. I suspect that if
you wrap SmoothD2c() in a temporal smoother such as MDegrain1() it will help your colors. I will download an anime clip
and add it to my testing.

TheProfileth
Thanks for the questions about Qtype. I have upgraded the documentation which I hope will make it a bit clearer.
jconklin is offline   Reply With Quote
Old 25th May 2012, 05:37   #15  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
SmoothD2c script

Here is the SmoothD2c() color deblock code. Some example of it at work https://sites.google.com/site/jconklin754smoothd2/home

Code:
#
#  REQUIRED ROUTINES
#  
#  aWarpSharp2.dll  avalable from   http://sites.google.com/site/pavelsx2/
#

###
#
#  This file contains the SmoothD2c() script routine.
#
#  SmoothD2c() provides a wrapper around SmoothD2() that provides deblocking of color as well as luma.
#              The arguments quant, num_shift, Matrix, Qtype, and ZW  are the same as those in SmoothD2 and have the same defaults.
#              The arguments cWarpToLuma and downSize are specific to SmoothD2c().
#              cWarpToLuma (0...8) uses aWarpSharp2() to warp the chroma to the luma. Default 0 no warping 8 strong warping.
#              downSize (1...4) Default 1 no downsizing.  This specifies how much the image should be shrunk before deblocking.
#                               This can be useful if you have really bad chroma blocking. see examples at 
#                               https://sites.google.com/site/jconklin754smoothd2/home
#
###



###
#  This routine takes a clip that has already had the luma deblocked and uses SmoothD2 to deblock the chroma.
# Defaults SmoothD2c(quant=3, num_shift=3, Matrix=3, Qtype=1, ZW=1, cWarpToLuma=0, downSize=1)
Function SmoothD2c(clip clp, int "quant", int "num_shift", int "Matrix", int "Qtype", int "ZW", int "cWarpToLuma", int "downSize") {
quant       = default( quant,          3 )  
num_shift   = default( num_shift,      3 )  
Matrix      = default( Matrix,         3 )  
Qtype       = default( Qtype,          1 )  
ZW          = default( ZW,             1 )  
cWarpToLuma = default( cWarpToLuma,    0 ) 
downSize    = default( downSize,       1 ) 


Assert( ( quant       >= 1  &&  quant       <=  31 ) ? true : false, chr(10) + "'quant' wrong value! [1-31]" + chr(10))
Assert( ( num_shift   >= 1  &&  num_shift   <=   4 ) ? true : false, chr(10) + "'num_shift' wrong value! [1-4]" + chr(10))
Assert( ( Matrix      >= 1  &&  Matrix      <= 110 ) ? true : false, chr(10) + "'Matrix' wrong value! [1-110]" + chr(10))
Assert( ( Qtype       >= 0  &&  Qtype       <=   4 ) ? true : false, chr(10) + "'Qtype' wrong value! [0-4]" + chr(10))
Assert( ( ZW          >= 0  &&  ZW          <=  16 ) ? true : false, chr(10) + "'ZW' wrong value! [1-16]" + chr(10))
Assert( ( cWarpToLuma >= 0  &&  cWarpToLuma <=   8 ) ? true : false, chr(10) + "'cWarpToLuma' wrong value! [0-8]" + chr(10))
Assert( ( downSize    >= 1  &&  downSize    <=   4 ) ? true : false, chr(10) + "'downSize' wrong value! [1-4]" + chr(10))

   ow  = clp.width
   oh  = clp.height
   sd1 = clp 

#   sd1 = (downSize    > 1) ? sd1.bilinearResize(m16(ow/downSize), m16(oh/downSize)) : sd1 
   sd1 = (downSize    > 1) ? sd1.bicubicResize(m16(ow/downSize), m16(oh/downSize)) : sd1 
   sd1 = (cWarpToLuma > 0) ? sd1.colorWarpToLuma(strength=cWarpToLuma)    : sd1

   u2  = sd1.UToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)
   v2  = sd1.VToY().SmoothD2(quant=quant, num_shift=num_shift, Matrix=Matrix, Qtype=Qtype, ZW=ZW) #, ZWlmDark=0, ZWlmBright=255, ZWce=1)

   u2 = (downSize > 1) ? u2.bicubicResize(m16(ow/2), m16(oh/2)) : u2
   v2 = (downSize > 1) ? v2.bicubicResize(m16(ow/2), m16(oh/2)) : v2

   sd1 = YToUV(u2, v2).MergeLuma(clp)

   return(sd1)
}


###
# m4  "mod-4"  see "Crop restrictions" in http://avisynth.org/mediawiki/Crop
#              m4 is used in may scripts that do resizing.
# m8  "mod-8"  is used so that cropping will occure on luma block (8x8) boundaries
#              and that resizing will produce an integer number of luma blocks.
# m16 "mod-16" is used so that cropping will occure on YV12 chroma block (16x16) boundaries
#              and that resizing will produce an integer number of chroma blocks.
function  m4(float x) {return(x<16?16:int(round(x/4.0) *4))}
function  m8(float x) {return(x<16?16:int(round(x/8.0) *8))}
function m16(float x) {return(x<32?32:int(round(x/16.0)*16))}





###
# 
#  This routine moves the color information tward the edges in the luma
#
# Defaults colorWarpToLuma(strength=1)
function colorWarpToLuma(clip clp, int "strength") {
strength = default( strength, 1 )


  thresh = (strength ==  0)  ?    1                   
\        : (strength ==  1)  ?   24
\        : (strength ==  2)  ?   32 
\        : (strength ==  3)  ?   48  
\        : (strength ==  4)  ?   64  
\        : (strength ==  5)  ?   96  
\        : (strength ==  6)  ?  128  
\        : (strength ==  7)  ?  192  
\        :                      255  

blur = strength + 2
clp
orig=clp
aWarpSharp2(thresh=thresh, blur=blur, type=1, depth=14, chroma=4)
mt_merge(orig, orig, last, y=2, u=5, v=5) 

return(last)
}
jconklin is offline   Reply With Quote
Old 30th May 2012, 21:03   #16  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
BTW jconklin
What exactly is the point of trying to "warp the chroma to the luma"? Obviously this is easier than rewriting code to allow for chroma denoising but is this nearly as effective as that would be?
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 31st May 2012, 00:54   #17  |  Link
jconklin
Registered User
 
Join Date: Mar 2006
Posts: 39
Quote:
Originally Posted by TheProfileth View Post
BTW jconklin
What exactly is the point of trying to "warp the chroma to the luma"? Obviously this is easier than rewriting code to allow for chroma denoising but is this nearly as effective as that would be?
Warp chroma to luma is different from deblocking or denoising chroma. What it does, using the warpsharp routine, is if there is a strong luma edge near a chroma edge it will move the chroma edge closer to the luma edge. This is occasionally useful. The downside is it can desaturate colors in dark areas. This is an option that if you have the right kind of damaged video it might be able to help.

The actual chroma deblocking is done by the calls to SmoothD2 that are made in the SmoothD2c routine.

I have added an example to https://sites.google.com/site/jconkl...lor_deblocking that shows the same image deblocked with and without using Warp chroma to luma.
jconklin is offline   Reply With Quote
Old 7th July 2012, 07:41   #18  |  Link
Schwartzvald
Ooo Royalty
 
Join Date: Aug 2005
Posts: 9
Love this filter, especially the warpsharp routine in SmoothD2c.

However, I'm having trouble getting SmoothD2 to load in avs script on an older XP machine.

On a newer Windows 7 machine, it works fine. On the old XP box, Avisynth presents a "Plugin not found" error. If I try to force it with LoadPlugin(), it gives error code 0x36b1.

My best guess is I'm missing a dependency that comes stock with newer versions of Windows, like a VC runtime. Anyone know what that might be?
Schwartzvald is offline   Reply With Quote
Old 7th July 2012, 08:16   #19  |  Link
Robert Martens
Registered User
 
Join Date: Feb 2010
Location: New York
Posts: 116
Dependency Walker says it needs msvcr80.dll, which if I'm not mistaken corresponds to the Visual C++ 2005 Redistributable.
Robert Martens is offline   Reply With Quote
Old 7th July 2012, 12:16   #20  |  Link
Schwartzvald
Ooo Royalty
 
Join Date: Aug 2005
Posts: 9
Yes, that got me on the right track. I'd never heard of Dependency Walker, either. Thanks for introducing me to it, and for the help.

It actually depends on this latest side-by-side of VC2005.:

https://www.microsoft.com/en-us/down....aspx?id=26347

Older versions won't do. I'm assuming that comes stock with Windows 7, but my XP versions were out of date.
Schwartzvald is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:05.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.