Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.


Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Thread Tools Search this Thread Display Modes
Old 20th October 2004, 23:54   #1  |  Link
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
Here is LimitedSharpen()

Hello friends

Seems this is my millenium post. Since for me this is a rather seldom event, I thought I could at least make a usable and halfway worthy one, instead of doing only small talk in other places.

Time ago, I started a thread in the Development section about "thresholded sharpening". The thread turned into a little idea bin (and is still open for that purpose ), and one of the breed's results is the function


It's been around for a while now, a little wild at times, in several different versions regarding features and so ... but by now, I think it has reached a status where it is working quite nice and sufficiently fast, and has (most) of the features it's supposed to have. Time to make an own, proper thread in Avisynth Usage for this little function.

So, what's the deal?

LimitedSharpen() can be used like a traditional sharpener, but producing much less artefacts. It can be used as a replacement for the common "resize(x4)-XSharpen-resize(x1)" combo, with very similar results (perhaps even better) - but at least 2 times faster, since it requires much less oversampling. And by chaining several instances, it can even be used for something like a "[very] poor man's deconvolution" - but only if one knows how to battle the noise

What's the problem with "normal" sharpening?

Traditional sharpeners like sharpen() or UnsharpMask() compare each pixel against the average of its neighborhood, and emphasize the difference between them. The results (something like "per-pixel contrast enhancement") are good as long as the strength is kept low enough. But artefacts will arise very soon.
XSharpen, being a non-linear sharpener, replaces each pixel with either its darkest or brightest neighbor, depending on which is nearer in range. By the nature of the method, XSharpen produces edges with maximal possible aliasing (jaggyness). So one has either to reduce the percentage, thereby weakening the overall effect, or to work with big supersampling, which makes it both extremely slow and less effective.

Let's have a look at a simple transition from dark to bright, and what our standard sharpeners will do to it:

The blue line represents the original edge, the dark side on the left, the bright side on the right, and in-between the gradient that builds "the edge". Red is (basically) the result of sharpen() or UnsharpMask(), pink is the result of XSharpen().
Obviously, both methods have their pro's, and both have their con's.

Note: Discussion about "ideal sharpening" is lengthy. To make an "optimal" sharpness restauration or enhancement, one would need to know the exact process that did reduce the source's sharpness. Since we never know that, we can only search for compromises that work sufficiently good in most cases.

Now, LimitedSharpen() doesn't re-invent the wheel. It just tries to take the best of both worlds.

Shortly, LimitedSharpen() applies one out of three different sharpeners (two domain sharpeners or a windowed range sharpener) to the source, but will limit the oversharpening (either "hard" or "soft") IF it exceeds a defined "overshoot".

In reference to the graphs above, the script's results look like that (basically) :

(The proportions are not "real" - both graphs were constructed free-handed.)

As you see, LimitedSharpen always takes the enhanced edge steepyness from normal sharpening, but avoids oversharpening in the same way as XSharpen, as long as overshoot is kept at zero, and limiting mode 1 is used.
In many cases, this comes out really nice, and is sufficient.
However, with really strong sharpening there may still occur jaggy edges and/or noticeable loss of gradients in edge neighborhoods. For these cases, limiting mode 2 can be used, as well as supersampling. (The required supersampling factors are only half of what XSharpen usually needs - therefore this script will run much faster in comparison, since only a quarter of image data has to be processed, as opposed to "4*SSXSharpen")

The basic operation is like this:

- make a traditional sharpening operation
- compare each pixel's sharpened result against its brightest and darkest neighbor in the original clip:
-- if the result is inbetween of these, then the pixel is not oversharpened, and the result is used as-is
-- if the result exceeds either the min or max neighbor, then the pixel is oversharpened, and will get limited.

"Neighborhood" of a pixel currently is either its 3x3 or its 5x5 neighborhood, depending on the "wide=true|false" parameter.

Available sharpeners are: UnsharpMask(), Sharpen(), and MinMaxSharpen().

UnsharpMask() and Sharpen() are common - the former sharpens each pixel against a gaussian blurred input, the latter against a 3x3 average. The third mode sharpens each pixel against the average of the brightest & the darkest neighbor. This primitive form of range filtering is less "exact" in theory, but nevertheless turns out useful. The effect is stronger than that of a normal sharpen() operation, and somehow it is less prone to enhance noise and hi-frequency DCT artefacts. The downside is that it does a less optimal job in restoration of blurred corners - but that's mostly neglectable when working with supersampling.

The limiters are: hard limiting, and soft limiting.

With hard limiting, each pixel that becomes either darker than [min_neighbor - overshoot] or brighter than [max_neighbor + overshoot] through sharpening, will simply get clipped to that min|max value.

With soft limiting, no clipping takes place, but a reduction: the effective overshoot from sharpening will get replaced by sqrt(overshoot). This means, a little oversharpening will be present, but it'll be much weaker than it would normally be. As long as the sharpening strength is kept in reasonable range, the oversharpening will still be hardly visible, while still being less prone to loose gradient tone levels in edge neighborhoods.

Lastly, there is the "special" switch. It should be considered as "experimental". Activating this one, the function will perform a simple "smart contrast sharpening": in the range of low levels, pixels will get only brighter through sharpening, not darker. In the high levels range, pixels will get only darker, not brighter. Mid level pixels may get darker/brighter as usual.
This is done by building the comparison frame through a sliding blend of (min_neighbor) for dark pixels and (max_neighbor) for bright pixels, instead of taking the plain average over the whole range. Therefore, in dark areas all values will get sharpened against their darkest neighbor, vice versa in bright areas. Visually, especially in dimmed and dark areas more detail will raise out of the "dark swamp". Simple, not too scientific, but often effective. However, one should use relatively low strengths for this one, or the effect might become strange. But even bigger strengths might come handy for some special tasks of mask creations. Usage of "wide = true" might be a good idea too with lower strengths.
This mode is available only together with Smode=3.

Full function call & parameter description:

LimitedSharpen( float "ss_x",  float "ss_y",     int  "dest_x",  int "dest_y", 
 \              int   "Smode", int   "strength", int  "radius",
 \              int   "Lmode", bool  "wide",     int  "overshoot",
 \              bool  "soft",  int   "edgemode", bool "special",
 \              int   "exborder" )

ss_x, ss_y
As usual, these floats are the factors for supersampled operation. You'll hardly ever need to go higher than 2.0. For simple sharpening tasks, set these to 1.0 (no supersampling). Default is 1.5 each, however.

dest_x, dest_y
These parameters specify an arbitrary output resolution. Comes handy if supersampled operation is used in a processing chain that involves resizing anyways, to avoid an unneeded extra resizing step.
Default is [none], i.e. same resolution as the input clip.

Smode ("Sharpen mode")
1 = UnsharpMask() [from WarpSharp.dll package]
2 = Sharpen()
3 = "MinMaxSharpen()" [private routine of LimitedSharpen]
Default is Smode=3. Change yourself if you prefer another one.

Obviously, the strength of sharpening. For Smode=1, it can be 0~127 (simple sharpening), 128~255 (simple overdrive), 255~4096 (big overdrive).
For Smode=2, values 0~100 are handled over to Sharpen() as 0.0~1.0. Values >100 are mapped to 100.
For Smode=3, 0~100 is common, but 100~inf. can be used if necessary.
Default is strength=160 for Smode=1, and strength=100 for Smode=2|3.

The radius for the unsharp masking of Smode=1. For Smode=2|3, it's simply ignored.
In contrary to former versions, the radius now applies "directly". It's no more scaled along with the ss_x|y values. I like it more this way.
Default is radius=2.

Lmode ("Limiting mode")
1 = hard limiting, together with "overshoot".
2 = soft limiting (use square of real overshoot)
Default is Lmode=1.

false = use min. and max. values of a 3x3 neighborhood for limiting.
true == use min. and max. values of a 5x5 neighborhood for limiting.
Default is wide=false, and this should do the job most times. TRUE might come handy for very blurry sources and/or bigger supersampling factors.

This specifies how much the sharpening result may "shoot over" the min and max limits, before either clipping or reduction will be done. 0=no overshoot allowed at all, 128="make-this-script-useless"
Default is overshoot=1.

A misleading name. If TRUE, then the clip will undergo a blur(1) command only for finding the min and max limits (in order to not acidentially use too high or too low limits, caused by noise, small artefacts, or edge-halos.) This does not blur the processed clip itself.
But attention!! This is mostly useful for soft sources with noise. On sharp sources with little noise, you'll loose some detail nevertheless.
Default is soft=FALSE.

0 = deactivated, process the whole frame
1 = process only edge areas
2 = process only NOT-edge areas
Default is edgemode=0

When TRUE, *and* Smode=3, this will activate the above mentioned smart contrast sharpening. When using other sharpeners, it has simply no effect.
This feature is not fully mature. But be sure to try it out, the effect might be pleasing.
Default is special=FALSE

If the outmost borders of the input clip are not clean, they can be excluded from the sharpening by setting exborder > 0.
values of 1 to 4 will exclude roughly 2 to 8 pixels from each side of the frame, plus an additional soft transition to the processed area.
This feature is thought for cases when you don't want to crop into the image area. The transition to black borders usually contains artefacts. Setting exborder to an approbriate value will prevent these artefacts from getting emphasized.
Default is 0 (no border exclusion). Usage of borderexclusion will cause a small speed loss.

The defaults are equivalent to
LimitedSharpen( ss_x=1.5,   ss_y=1.5,     dest_x=last.width, dest_y=last.height, 
 \              Smode=3,    strength=100, radius=2,
 \              Lmode=1,    wide=false,   overshoot=1,
 \              soft=false, edgemode=0,   special=false,
 \              exborder=0 )


Required are MaskTools >= v1.5.1 and the WarpSharp.dll package.

In case someone's going to use "edgemode": check yourself if the hardcoded values are suited. If not, please adjust them yourself. I myself use it almost never, and don't want to blow up the parameter list unnecessarily.

No RGB input accepted.

Parameter checkings are not water proof, and in no way fool proof.

This is a sharpener. Of course the needed bitrate will grow noticeably. Be careful. Use a denoiser beforehand.


Hah, gotcha! No, I won't kill your fun of toying-around ...

However, the initially mentioned "speedy version" of 4*supersampled XSharpen'ing would look as simple as

LimitedSharpen( ss_x=2.0, ss_y=2.0, Smode=2)

Apart from that, there are so many possibilities to use this script, dependant on the source quality and the effect one wants to achieve ... toy around with the parameters, I'm sure you will get a good grip to it in short time.

And if something is not clear, feel free to ask.

Have fun!

- Didée
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

Last edited by Didée; 24th October 2004 at 14:58.
Didée is offline   Reply With Quote
Old 21st October 2004, 00:04   #2  |  Link
Join Date: Nov 2001
Location: Netherlands
Posts: 6,364
Congrats with your 1000th post!

Btw, you might want to add a link to the plugin/script
Wilbert is offline   Reply With Quote
Old 21st October 2004, 00:13   #3  |  Link
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
Patience. I'm still smoking that cigar.

edit 24 Oct '04: added "exborder" parameter (border exclusion).

edit 26 Nov '04: made the "wide=true" limiting mode working correctly.

# LimitedSharpen()
# A multi-purpose sharpener by Didée

function LimitedSharpen( clip clp, 
 \                       float "ss_x",   float "ss_y", 
 \                       int   "dest_x", int   "dest_y",
 \                       int   "Smode" , int   "strength", int  "radius", 
 \                       int   "Lmode",  bool  "wide",     int  "overshoot", 
 \                       bool  "soft",   int   "edgemode", bool "special",
 \                       int   "exborder" )
ox = clp.width
oy = clp.height
ss_x =     default( ss_x, 1.5 )
ss_y =     default( ss_y, 1.5 )
dest_x =   default( dest_x, ox )
dest_y =   default( dest_y, oy )
Smode =    default( Smode, 3 )
strength = Smode==1 
 \       ? default( strength, 160 )
 \       : default( strength, 100 )
strength = Smode==2&&strength>100 ? 100 : strength
radius =   default( radius, 2 )
Lmode =    default( Lmode, 1 )
wide =     default( wide, false )
overshoot= default( overshoot, 1)
overshoot= overshoot<0 ? 0 : overshoot
soft =     default( soft, false )
edgemode = default( edgemode, 0 )
special =  default( special, false )
exborder = default( exborder, 0)
#radius =   round( radius*(ss_x+ss_y)/2)  #  If it's you, Mug Funky - feel free to activate it again  

clp.isYV12() ? clp : clp.converttoyv12()

ss_x != 1.0 || ss_y != 1.0 ? last.lanczosresize(xxs,yys) : last
tmp = last

edge = logic( tmp.DEdgeMask(0,255,0,255,"5 10 5 0 0 0 -5 -10 -5", divisor=2)
 \           ,tmp.DEdgeMask(0,255,0,255,"5 0 -5 10 0 -10 5 0 -5", divisor=2)
 \           ,"max").levels(0,0.86,128,0,255,false) 

bright_limit = (soft == true) ? tmp.blur(1.0) : tmp
dark_limit1   = bright_limit.inpand()
bright_limit1 = bright_limit.expand()
dark_limit   = (wide==false) ? dark_limit1   : dark_limit1  .inflate.deflate.inpand()
bright_limit = (wide==false) ? bright_limit1 : bright_limit1.deflate.inflate.expand() 
minmaxavg    = special==false
 \           ? yv12lutxy(dark_limit1,bright_limit1,yexpr="x y + 2 /")
 \           : maskedmerge(dark_limit,bright_limit,tmp,Y=3,U=-128,V=-128)

normsharp = Smode==1 ? unsharpmask(strength,radius,0) 
 \        : Smode==2 ? sharpen(float(strength)/100.0) 
 \        :            yv12lutxy(tmp,minmaxavg,yexpr="x x y - "+Str+" * +")

OS = string(overshoot)
Lmode == 1 ? yv12lutxy( bright_limit, normsharp, yexpr="y x "+OS+" + < y x "+OS+" + ?")
 \         : yv12lutxy( bright_limit, normsharp, yexpr="y x "+OS+" + < y x y x - "+OS+" - 1 2 / ^ + "+OS+" + ?")
Lmode == 1 ? yv12lutxy( dark_limit,   last,      yexpr="y x "+OS+" - > y x "+OS+" - ?")
 \         : yv12lutxy( dark_limit,   last,      yexpr="y x "+OS+" - > y x x y - "+OS+" - 1 2 / ^ - "+OS+" - ?")

      edgemode==0  ?  NOP 
 \  : edgemode==1  ?  MaskedMerge(tmp,last,edge.inflate.inflate.blur(1.0),Y=3,U=1,V=1)
 \  :                 MaskedMerge(last,tmp,edge.inflate.inflate.blur(1.0),Y=3,U=1,V=1)

     (ss_x != 1.0 || ss_y != 1.0) 
\ || (dest_x != ox || dest_y != oy) ? lanczosresize(dest_x,dest_y) : last


clp.isYV12() ? ( exborder==0 ? tmp.mergeluma(last)
 \                           : maskedmerge(tmp,last,ex,Y=3,U=1,V=1) )
 \           : ( exborder==0 ? tmp.mergeluma(last.converttoyuy2())
 \                           : tmp.mergeluma( maskedmerge(tmp.converttoyv12(),last,ex,Y=3,U=1,V=1)
 \                                           .converttoyuy2()) )
return last
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

Last edited by Didée; 26th November 2004 at 10:52.
Didée is offline   Reply With Quote
Old 21st October 2004, 01:02   #4  |  Link
Join Date: Jun 2004
Posts: 38
Thanks Didee... looking forward to trying this in place of your previously released and incredibly fast IIP function.

oo_void is offline   Reply With Quote
Old 21st October 2004, 02:37   #5  |  Link
Registered User
joshbm's Avatar
Join Date: Jun 2003
Location: Land of the Noobs & the Home of the Brave
Posts: 349
Wonderful explaination! I've been waiting for the documention for LimitedSharpen(). Now if only we could have an explaination on how to get good input values for IIP .

joshbm is offline   Reply With Quote
Old 21st October 2004, 05:10   #6  |  Link
scharfis_brain's Avatar
Join Date: Mar 2003
Location: Germany
Posts: 3,653
I cannot see the images.
the image-server is terribly slow.

I could host them for you on me arcor-account, which is fast and reliable.
Don't forget the 'c'!

Don't PM me for technical support, please.
scharfis_brain is offline   Reply With Quote
Old 21st October 2004, 15:21   #7  |  Link
Registered User
Piper's Avatar
Join Date: Jul 2002
Location: Canada
Posts: 196
Thank you very much for this Didée! Your explanation/outline of LimitedSharpen is superb! Well done! *humbled*
Piper is offline   Reply With Quote
Old 21st October 2004, 20:51   #8  |  Link
Join Date: Jun 2004
Posts: 38
Well, the results are quite nice save one little problem (at least for me) ... the image borders. Would it be possible to add a function similar to IIP's 'exborder' in a future iteration.

oo_void is offline   Reply With Quote
Old 21st October 2004, 22:24   #9  |  Link
Does it really matter?
ChronoCross's Avatar
Join Date: Jun 2004
Location: Chicago, IL
Posts: 1,542
wow impressive. this might be by far the best documentation I have ever seen. I tested out the filter and all I can say is WOW. The increase in sharpening is amazing. it's scary how effective this is. Keep up the good work.
ChronoCross is offline   Reply With Quote
Old 22nd October 2004, 07:36   #10  |  Link
Moderator, Ex(viD)-Mascot
Teegedeck's Avatar
Join Date: Oct 2001
Posts: 2,564
Congrats 1000! LimitedSharpen gives very good results, but you know that. Easy to use, safe, fine results.
It's a man's life in Doom9's 52nd MPEG division.
"The cat sat on the mat."
ATM I'm thoroughly enjoying the Banshee - a fantastic music player/ripper for Linux. Give it a whirl!
Teegedeck is offline   Reply With Quote
Old 22nd October 2004, 09:59   #11  |  Link
Registered User
Join Date: Jul 2003
Posts: 35
Originally posted by oo_void
Well, the results are quite nice save one little problem (at least for me) ... the image borders. Would it be possible to add a function similar to IIP's 'exborder' in a future iteration.

edgemode=2 didnt help ?
koszopal is offline   Reply With Quote
Old 22nd October 2004, 17:42   #12  |  Link
Join Date: Jun 2004
Posts: 38
Originally posted by koszopal
edgemode=2 didnt help ?
My bad... a bad crop in the source.

oo_void is offline   Reply With Quote
Old 22nd October 2004, 21:19   #13  |  Link
Registered User
Join Date: Jan 2003
Posts: 90
It would appear that this script could be modified to correct existing overshapened sources... worth looking into.
MrTibs is offline   Reply With Quote
Old 22nd October 2004, 23:49   #14  |  Link
Soulhunter's Avatar
Join Date: Apr 2003
Location: Unknown
Posts: 2,812

Visit my IRC channel
Soulhunter is offline   Reply With Quote
Old 23rd October 2004, 13:40   #15  |  Link
Registered User
Join Date: Oct 2004
Location: Québec, Canada
Posts: 107
mod x needed?

Is it better to pass LimitedSharpen() a clip with a resolution which is, e.g., mod32 x mod16? Or the resolution could be mod1 x mod1?
PiXuS is offline   Reply With Quote
Old 24th October 2004, 00:18   #16  |  Link
Registered User
Join Date: Feb 2004
Posts: 156
yv12 needs at least mod 4 width i believe.
malkion is offline   Reply With Quote
Old 24th October 2004, 15:21   #17  |  Link
Registered User
Join Date: Oct 2004
Location: Québec, Canada
Posts: 107
Originally posted by malkion
yv12 needs at least mod 4 width i believe.
I think I didn't explain myself correctly. I wasn't asking for the resolution to pass to XviD (which should be mod16 x mod16 so it can work optimally). I was asking for the filters used by LimitedSharpen.

I know you can't pass whatever resolution to PixieDust. LimitedSharpen doesn't use PixieDust, but it use a library called MaskTools. So the real question was: is there a call to a function of the MaskTools library that require a clip with a resolution which respects a given mod specification (à la PixieDust)?
PiXuS is offline   Reply With Quote
Old 24th October 2004, 15:39   #18  |  Link
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
Thanks everyone for the warming comments. Always a pleasure to serve the community

I have now added the possibility to exclude the borders from processing, as oo_void suggested. It works similar as the same-named parameter in iiP, but allows to control the size of the borders that will be excluded (see updated parameter description).
Tried to implement it in a way that looses as little speed as possible - after all, it means an additional plane copying operation. In case of YUY2 input, also an additional colorspace conversion is needed. That won't affect the output in any way, but makes the operation a little more costly for YUY2 than for YV12 input.
In one respect, the actual implementation is not as smart as it could be: In case that edgemode=1|2 is used, the script will do two separate plane copy operations where the job could be done with only one. It would be not so awful tricky to do it like that ... but I'm currently a little tired of all those "?" and ":"

This function doesn't try to be a replacement for iiP. But I'll implement LimitedSharpen into iiP soon, to make the big boy walk a little faster. In fact, I'm using iiP with this implementation for quite some time now. Only thing is, I'm not sure yet how much LS should be crippled for that purpose, and how much features should be exposed to iiP's function call.

Usage of edgemode=2 is not recommended. At least I don't see much sense in sharpening everything except detail. It's only there for completeness (perhaps someone wants to use that together with UnSmooth(), you never know)

Mug Funky had a similar thought of using this principle for halo reduction. Though I follow the general idea, I still can't see how it actually should work out. Perhaps if you two stick together heads, and fiddle it out ...

Just as malkion said. The script works internally in YV12 colorspace, so MOD4 resolutions are required even for YUY2 input. It were possible to work around that with internal padding - but I'm simply too lazy to do that I think MOD4 for input is fair enough.

I have close to no idea how good or bad LimitedSharpen() works on comic or animee sources, as that's not my world at all. If someone has particular issues with such input, please report them.
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 24th October 2004, 15:58   #19  |  Link
Mug Funky
interlace this!
Mug Funky's Avatar
Join Date: Jun 2003
Location: i'm in ur transfers, addin noise
Posts: 4,555
anime looks fine with limitedsharpen, don't you worry (wasn't one of the test images in the original thread from chobits? hehe... that little robot girl is so cute).

the halo reduction looks like it'll be easier thanks to the new slew of median filters (and hopefully the speed arms-race that competing plugins would create. hmm... competition doesn't work the same for free stuff, though). i've aleady tried using median filtered clips to get cleaner motion-vectors from MVtools, though to be honest i haven't checked to see if they actually work better.
sucking the life out of your videos since 2004
Mug Funky is offline   Reply With Quote
Old 25th October 2004, 03:43   #20  |  Link
Does it really matter?
ChronoCross's Avatar
Join Date: Jun 2004
Location: Chicago, IL
Posts: 1,542
only thing I can say about anime sources is that it makes lines super huge. easily countered with a line thinning script.
ChronoCross is offline   Reply With Quote

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +1. The time now is 08:05.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.