Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 2nd February 2012, 09:03   #1  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
My attempt at writing a new shapener - SuperToon

I wanted to try and see if I could make a powerful sharpener that can run in realtime and this is what I came up with.
Code:
#SuperToon is my attempt to optimize/speed up the previous versions of mfToon, vmToon, etc.
#
# requires MaskTools V2.0 (at the time of writing this function the used version was V2.0a35)
# Removegrain(), Unfilter(), and Degrainmedian(), Repair() and Medianblur
# tested only in YV12, although I 'think' it could do other colorspaces it is reccomended to 
# do this in YV12 or YUV2 due to the math involed in this filter.
# thanks to:
# mastrboy for suggestion to use Removegrain() instead of Undot()
# theProfileth for optimizations to use mode=3
# Chikuzen for error checking
# Didée for suggestion for mode =4.


function SuperToon(clip input,   float "power", int  "mode",
\                  int  "Nthr",  int   "Nstr",  int  "Ncap",
\                  int  "Lthr",  int   "Hthr",  int  "Lcap",
\                  int  "cont",  bool  "show")

{

power = default(power,.75)         #values -255.0-255.0 will have impact
                                   # however a range from 0.2-1.5 is usually
                                   # the desired range. Positive values darken lines, 
                                   # negative values brighten them. 0 won't sharpen 
                                   # the video however SuperToon will still consume CPU.
                                   # when setting the Nthr, NCap, Lcap, and pretty much any other
                                   # setting, you can set this to an extreme value like 100, this way 
                                   # you can easily see (as if the edge mask was overlayed on the original)
                                   # what supertoon will be darkening (whether it'd be the lines you want it
                                   # to darken or the noise you didn't want it you detect)
                                   # for mode = 3, Power works differently. it ranges from -100 to 100 and its no
                                   # longer the "strength of darkening" but instead the "strength of smoothing + 
                                   # sharpening"
                                   
mode = default(mode,0)             # ranges from 0-4; tells SuperToon to try different methods
                                   #  no mode is always better than the other, its just that one mode 
                                   #  can fare better for one source and not another. so the user
                                   #  has the option of which mode is best for their source. if you 
                                   #  choose a number outside of the range it will default to 0.
                                   # mode =0: just applies a very simple edge detection mask with
                                   #          no effective noise detection.
                                   # mode =1: creates an edge mask and filters out noise by finding
                                   #          the average of all neighboring pixels and determines 
                                   #          if the current pixel is an "outlier" and not noise, meaning 
                                   #          its an edge to darken.
                                   # mode =2: silmilar to mode 1 but does things differently, it runs a sieve
                                   #          on all the neighboring pixels and then resets the edge mask
                                   #          to be the median of the results from that sieve. 
                                   #          this method is alot slower than the others, but sometimes its
                                   #          the only one that can seem to pull it off that the others can't
                                   # mode =3: does things completely differently in that it performs a "filtered"
                                   #          unfilter on the source. by filtered I mean it sharpens the lines while
                                   #          smoothing everything else. very effective at removing noise and thinning
                                   #          lines, and is very fast. its just not as "strong" at sharpening as I 
                                   #          had hoped as the other modes.
                                   # mode =4: works to ignore non-edge lines like shadow lines so it doesn't darken lines 
                                   #          that weren't meant to be darkened. if you're getting something like black 
                                   #          outlines around stars or snow, this mode is very good at preventing that.
                                   #          this mode doesn't run realtime for me (~20 fps for a 2.01 GHz single core)
                                   
Lthr = default(Lthr,0)             # can range from 0-255, raising this value makes darkened lines thicker and helps 
                                   # with edge detection, but it'll usually give very very thick lines if you set this 
                                   # too high. 9 times out of 10 you will likey not have to change this from the default.
                                   
Hthr = default(Hthr,255)           # can range from 0-255, lowering of this value has a stronger
                                   # impact on line detection than the low threshold.
                                   # most of the time the default is the best setting.
                                   
Lcap = default(Lcap,255)           # brightness cap which tells the function to only darken pixels under a certain luma value
                                   # this is mostly intended to give more control to modes 0-2. mode 3 doesn't use it, and mode 4
                                   # shouldn't ever need it. setting it to 255 basiaclly disables this.

cont = default(cont,180)           # pixels are darkened by an amount relative to their spatial luma change. pixels with very low luma 
                                   # change (like noise) is darkened very very slightly. pixels with large luma change (like a black line)
                                   # are greatly darkened. while moderate change (like a shadowline) is partially darkened.
                                   # this value controls the contrast of these changes. lowering this values makes it so that both low luma
                                   # and high luma changes are subtracted by near the same amount. if you lower this value "power" will have
                                   # to be raised to show a visible difference when used with Ncap, its iw very effective at removing noise.
                                   
Nthr = default(Nthr,4)             # threshold to try to detect noise around the lines to be sharpened set this too high and 
                                   # it'll begin to confuse the lines you want sharpened as noise. this parameter works 
                                   # differently depending on the mode so certain in modes it works better in certain ranges.
                                   # in nearly all modes its used to clip lower values from the generated mask (as noise). for
                                   # modes 1 and 2 it used more to clean the noise from the lines. for 3 its used to filter between
                                   # smoothing or sharpening. for mode 4 its used to overlay mode 0's mask onto mode 4's mask. lower numbers
                                   # add more weight to Mode 0's mask.
                                   
Nstr = default(Nstr,0)             # can range from -100 to 100. 0 will make this do nothing. This is
                                   # usually used to blur the edgemask to try to smooth out residual
                                   # noise. basically its a parameter that uses  Unfliter() on the
                                   # treated mask. negative values work great for reducing noise
                                   # its fast and effective but values near the limits can make demolish lines
                                   # or produce a small halo effect. positive values and help thin some 
                                   # lines which works great if the source doesn't present alot of noise.

Ncap = default(Ncap,30)            # ranges 0-255 increasing it is the best (as far as filter
                                   # speed) at preventing noise from being sharpened, however
                                   # setting it too high and you won't be darkening any lines 
                                   # this setting helps determine how "sensitive" it should be when looking
                                   # at the edge mask. if it looks like the entire picture is darker,
                                   # then its likely this value it too low. 18 is a good starting point
                                   # for most sources I've tried.
                                   
show = default(show,false)         # if you want the edge mask returned instead of the sharpened 
                                   # result
powerF = min(100,max(1,Round(abs(power))))                                 
Spow  = string(power)
SHthr = string(Hthr/4.0)
SNthr = string(Nthr)
SNcap = string(Ncap)
SLcap = string(Lcap)

mask = mt_makediff(input.mt_expand(Hthr),input.mt_inpand(Lthr))

  #mode=1 -> abs(y-x)/x>n/255.0? y: 128
  #mode=2 -> expr= abs(y-x)<n && x>y ? 128: y - x + 128 
mask = (mode==2) ? mask.mt_luts(mask,pixels=mt_square( 4 ), mode="med", expr="y x - abs "+SNthr+" < x y > & 128 y x + 128 - ?")
\                : (mode==1) ? mask.mt_lutf(mask,mode="avg",expr="y x - abs x / "+SNthr+" 255.0 / > y 128 ?")
\                            : mask


  #now this tries a completely different mechanic working more like a "filtered" sharpener AND smoother.
  # very very fast but is still experimental...not effective
hard = mode==3 ?input.degrainmedian().unfilter(powerF,powerF).Removegrain(mode=1):input
soft = mode==3 ?input.degrainmedian().unfilter(-powerF,-powerF).Removegrain(mode=1):input
  #"abs(x-y)>n?y:0"
  #"x==0?y:x"
hard = mode==3 ?input.mt_lutxy(hard,"x y - abs "+SNthr+" > y 0 ?"):hard
hard = mode==3 ?hard.mt_lutxy(soft,"x 0 = y x ?"):hard
hard = mode==3 && Nstr!=0?hard.unfilter(Nstr,Nstr):hard


  #mode 4 works to avoid shadow lines. since the current mask it generates is far too weak, I have it add
  # a portion of the previously generated mask. as a result shadow lines aren't completely removed from  
  # this mask, however their strength is so weak Nthr, Ncap, Nstr, and cont can easily remove it.
  #x - (y-128)/n + 128 
mask = (mode==4) ?input.medianblurcw(2,0,0,0).mt_logic(input,"max").mt_makediff(input).mt_lutxy(mask,"x y 128 - "+SNthr+" / +")
\                :mask

  #these 2 lines workd to clean all/most of the noise caught in the mask
  #as well and reset it's levels so its white to black and not white to grey.
mask = ((Nstr!=0)?mask.unfilter(Nstr,Nstr):mask).Removegrain(mode=1)
mask = mask.levels(128+Nthr,1,255,0,cont).Removegrain(mode=1)

  #residual Noise reduction regardless of mode. the Unfilter  before doesn't 
  # actually remove the noise... but it makes it simpler for the sharp LUT below to see it and 
  # filter it out with NCap
  #"y<n||x>L?x:x-(y-n)*p"
sharp = mode==3 ? hard:input.mt_lutxy(mask,"y "+SNcap+" < x "+SLcap+" > | x x y "+SNcap+" - "+Spow+" * - ?",u=2,v=2)

#if show is true and the mode is not 3 (since it doesn't use an edge mask) return the edge mask
# else return the sharpened video.
return show && mode !=3 ?mask.greyscale():sharp
}
here are some screen caps of before/after of it at work:
http://screenshotcomparison.com/comparison/106060/

and the parameters I used:
Code:
#1 Gurren Lagann     Supertoon(power=.5,mode=0,nthr=8,ncap=40,nstr=10,cont=250)
#2 Soul Eater        Supertoon(power=1,nthr=16,ncap=32,nstr=-70,cont=200)
#3 Full Metal Panic  Supertoon(power=.75,Lthr=0,Nthr=4,Ncap=40,cont=230,nstr=20)
#4 & 5 Fooly Cooly   Supertoon(power=.5,Nthr=24,Ncap=48,cont=230)
#4 and 5 where to show how well it can work with sources that had scenes with strong noise and weak noise

here's some new screen caps showing off the new mode=4
http://screenshotcomparison.com/comparison/107324
and the parameters I used for those screencaps:
Code:
#1 Soul Eater    Supertoon(power=.75,mode=4,Lthr=0,nthr=3,nstr=-15,ncap=32,cont=200)
#2 Gurren Lagann Supertoon(power=.5,mode=4,nthr=4,ncap=40,nstr=-10,cont=200)
all the modes I used would play in realtime when I tested them, except for mode 2 which would run around 10 fps on my PC (2.01 Ghz single-core) and mode 4 (~20fps). Theres still alot of things I want to add and some problems I want to fix. but I wanted to go ahead and post what I've got so far. see if I could get some critiques.

Edit: added more screencaps and updated code.

Last edited by Hadien; 11th February 2012 at 16:30. Reason: code & screencap update
Hadien is offline   Reply With Quote
Old 2nd February 2012, 16:49   #2  |  Link
thebigmunch
Registered User
 
Join Date: Apr 2006
Posts: 6
For your images, you should be able to right-click on the video in the AvsP preview and choose Save Image As to get proper screenshots.
thebigmunch is offline   Reply With Quote
Old 2nd February 2012, 22:23   #3  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
I know how to post a full screenshot, It just didn't occur to me to do so. heres another screencap at full resolution
Hadien is offline   Reply With Quote
Old 2nd February 2012, 23:28   #4  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Since this is not a plugin or a new compiled version of avisynth I think this fits more in the avisynth usage section rather than the avisynth development section.
Just saying
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 2nd February 2012, 23:40   #5  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
well then I was confused. the description for this forum seemed to imply the creation of new filters, functions and such

while the other forum gave me the impression for learning how to code in avisynth, troubleshooting, etc.
Hadien is offline   Reply With Quote
Old 2nd February 2012, 23:45   #6  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Ah well they mean things like dll filters and such but yeah no big deal just make a new thread in the other forum and maybe use real screenshots too lol but yeah I will look forward to testing and giving feedback on this filter in there.
Even better still use this site to compare
http://screenshotcomparison.com/
should be better for everyone and more convenient too
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 3rd February 2012, 15:26   #7  |  Link
Guest
Guest
 
Join Date: Jan 2002
Posts: 21,923
Moved this thread to Avisynth Usage and deleted the crosspost.
Guest is offline   Reply With Quote
Old 3rd February 2012, 17:24   #8  |  Link
mastrboy
Registered User
 
Join Date: Sep 2008
Posts: 288
you should replace undot() with removegrain(mode=1)
mastrboy is offline   Reply With Quote
Old 4th February 2012, 05:08   #9  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
Thanks neuron, I updated my post with suggested changes to the original code and relinked the screencaps I posted in the other thread
Hadien is offline   Reply With Quote
Old 7th February 2012, 10:57   #10  |  Link
Chikuzen
typo lover
 
Chikuzen's Avatar
 
Join Date: May 2009
Posts: 597
I got an error "Script error: expected `:'"
maybe, this will fix.
Code:
--- supertoon.orig.avs	2012-02-07 18:52:06.108082300 +0900
+++ supertoon.fix.avs	2012-02-07 18:51:09.442841300 +0900
@@ -114,7 +114,7 @@
 #"x==0?y:x"
 hard = mode==3 ?input.mt_lutxy(hard,"x y - abs "+SNthr+" > y 0 ?"):hard
 hard = mode==3 ?hard.mt_lutxy(soft,"x 0 = y x ?"):hard
-hard = mode==3 ?(Nstr!=0)?hard.unfilter(Nstr,Nstr):hard
+hard = mode==3 && (Nstr!=0)?hard.unfilter(Nstr,Nstr):hard
 
 #"y<n||x>L?x:x-(y-n)*p"
 sharp = mode==3 ? hard:input.mt_lutxy(mask,"y "+SNcap+" < x "+SLcap+" > | x x y "+SNcap+" - "+Spow+" * - ?",u=2,v=2)
__________________
my repositories
Chikuzen is offline   Reply With Quote
Old 7th February 2012, 13:50   #11  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
yeah I noticed that error, and thats the same fix I did. I haven't been able to post the proper fix lately due to my PC crashing a lot recently (a capacitor blew on my mobo).

I've been trying to wrap my head around a way to subtract the shadow lines from my edgemasks or have it so that the edgemasks don't pick them up in the first place. I don't think shadow lines should be darkened however the way I have my edge detection set up it'll confuse them as lines that should be darkened.

thinking of writing a mask using lut or convolution that detects only shadow lines and then subtract that mask from the edgemask. but not exactly sure on how to write it up.
Hadien is offline   Reply With Quote
Old 10th February 2012, 03:21   #12  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Quote:
Originally Posted by Hadien View Post
I've been trying to wrap my head around a way to subtract the shadow lines from my edgemasks or have it so that the edgemasks don't pick them up in the first place. I don't think shadow lines should be darkened however the way I have my edge detection set up it'll confuse them as lines that should be darkened.

thinking of writing a mask using lut or convolution that detects only shadow lines and then subtract that mask from the edgemask. but not exactly sure on how to write it up.
Well off the top of my head, use 2 masks, 1 for sharpening and the other for darkening. On the darkening mask you could try blurring the image that is to be edge masked. And by not blurring the sharpening mask the sharpening should still be correct and the darkening should be more correct (hopefully) because it will not pick up weaker lines (such as shadows hopefully) or you could do something with thickening/thinning the edgemask to try and cull the shadows that had be included, but I would try the first method before doing that as it seems much simpler
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 10th February 2012, 08:27   #13  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
ugh I hate the fact that if you type a long post and quick reply it suddenly forgets who you are since you are signed out. so you press the back button to find your whole post you wrote up is lost.

the blurring it mostly meant to reduce/prevent noise from being detected and darkened, its never strong enough to effect shadowlines. infact if you check Supertoon's Mask (via show=true) you can see that it can detect shadow lines just as clearly as edge lines. to supertoon, it sees edges and shadowlines as actual lines, and it'll darken everything it confirms as a line (for now, I want to make it more specific).

I think the biggest difference between an edge line and a shadow line is that an edge line is usually one unique, darker color (within a threshold of course) that lies as a boundary between 2 other usually brighter colors. shadow lines are usually 2 different colors pushed together without a darker line splitting the two. so what I would want to do is tell my edge mask to look at its neighboring pixels in the input clip which weren't masked and determine if they were the same color as the pixel that were masked (again within a certain threshold)

what I have tried is overlay the edgemask onto the input (making edges white), then perform an inpand or two. afterwards I'd make a comparision between this overlay and the input stating that if the input pixel is darker than the same pixel in the overlay, its an edge. I'd turn that result into my new mask and tried applying it. While it did mostly demolish the shadow lines, it also broke up much of the actual edges, and somehow made certain other areas darken which weren't edges (notably i motion areas where things got motion blurred). I feel I'm on the right track here, But I think I need to update Masktools (current version is V2.0a35) and see if theres a better function to use here. mt_luts is the only neighboring pixel lut I have and it performs the expression on each pixel before doing the mode (I want it to do that in reverse)

Last edited by Hadien; 10th February 2012 at 08:31. Reason: grammar fixes
Hadien is offline   Reply With Quote
Old 10th February 2012, 17:43   #14  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
A median filter is another possibility to "clamp out" lines, while not reacting to boundary-edges of uniform areas.

> vid.repair(vid.medianblur(2,0,0),1).mt_logic(vid,"max",U=2,V=2)

or maybe FastLineDarken-style?

> vid.repair(vid.mt_expand().mt_expand().mt_inpand(),1,0) # number of ex/inpands acc. to line thickness that shall be catched
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 11th February 2012, 00:48   #15  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Didee would the dithertools box filter be of any use here? As it is certainly much faster than any median filter (as far as I know)
Cause here is a application of it I came up with a while ago after seeing someone ask for a mean filter
Code:
Function vmean (clip v,int "r",int "rc")
	{
	#variable mean
	r = default(r, 4)
	rc= default(rc, r*2)
	v.Dither_convert_8_to_16 ()
	Dither_box_filter16 (radius=r, y=3, u=2, v=2)
	Dither_box_filter16 (radius=rc, y=2, u=3, v=3) # for 4:2:0 chroma subsampling
	DitherPost (mode=-1,stacked=true)
	}
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 11th February 2012, 04:45   #16  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,390
A box filter is a mean filter. For the problem posed, mean filtering is exactly what needs to be avoided.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 11th February 2012, 05:41   #17  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
Quote:
Originally Posted by Didée View Post
A box filter is a mean filter. For the problem posed, mean filtering is exactly what needs to be avoided.
Ok that is what I thought, I just wanted to make sure
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon
TheProfileth is offline   Reply With Quote
Old 11th February 2012, 11:17   #18  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
@Didée:
thanks for that fantastic "clamping" suggestion. I tried it out and while the mask it gave me was still kinda weak (it prevented those shadow lines but would sometimes demolish other lines) I instead played with it some more and I believe I came across a pretty good solution. the mask itself was still weak so what I did was add mode=0's mask back onto this weak mask (mode 0's mask values were greatly reduced to help hide its shadow lines). Together with the noise reductions I already have in place (Nthr, Ncap, Nstr, and cont parameters) the shadow lines in mode 4's mask (mode 4 is the new mode to remove these shadow lines) were made so weak that Supertoon could pass them off as simple noise to filter out. speed-wise mode 4 runs about 20 fps (again on a single core 2.01 Ghz). It feels like I could make some speed improvements to it but I'll worry about that tomorrow

since it's getting late/early here I post a full code update and new comparison screen-caps when I wake up in a few hours.
Hadien is offline   Reply With Quote
Old 11th February 2012, 23:56   #19  |  Link
TheProfileth
Leader of Dual-Duality
 
TheProfileth's Avatar
 
Join Date: Aug 2010
Location: America
Posts: 134
You should be using medianblur(2,0,0) rather than medianblurcw(2,0,0,0) as it is (about) twice as fast and does the same thing in this situation.
That hopefully should allow you to get realtime playback.
__________________
I'm Mr.Fixit and I feel good, fixin all the sources in the neighborhood
My New filter is in the works, and will be out soon

Last edited by TheProfileth; 11th February 2012 at 23:58.
TheProfileth is offline   Reply With Quote
Old 12th February 2012, 00:55   #20  |  Link
Hadien
Infected Mushroom Fan
 
Hadien's Avatar
 
Join Date: Dec 2007
Location: Atlanta,GA
Posts: 17
thanks for the tip and you're right its does greatly speed up that mode to play real-time. I initially thought the center weight version would prove more roboust if I would have it not count the center pixel by setting it to 0. when I changed it back I saw no visual change (though I do remember a change when I first tried it, maybe in the end after adding the difference mask and overlaying the previous mask that got balanced out) but yeah, definite speed boost.

I'm pondering if I should add a hook to the radius variable in median blur cause it might be that a radius of 2 won't always help for every instance.

Besides that, what is everyone's opinion of this sharpener? would you ever see yourself using it? do you like where its going? I'd like to hear some feedback on this.
Hadien is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 08:31.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.